Mission Data Blog
Louisville-based Web Development & Software Engineering

This has been done before with PHP (AJAX upload progress meter for PHP) etc but I needed something a little different because I wanted to upload a file and then have it loaded into a database. I looked around and found that someone had already made something that used the commons file upload package to do the upload part (AJAX Upload progress monitor for Commons-FileUpload Example). It wasn’t exactly what I was looking for but it a good start.

To understand the way this works I think it is easiest to break it down into parts:

1. A file upload extention that counts bytes as they are uploaded
2. An interface that monitors the progress of something running on the server
3. AJAX to pull the monitoring into the current screen

Read the rest of this entry »

One of the downsides of the ruby S3 example code is that it doesn’t support streaming of data (it loads the entire file into memory). It turns out, however, that all that is needed to stream data is a tweak to the ‘request’ method in Net::HTTP.

  require 'net/http'
require 'S3'
require 'pp'

#
# Replace the request method in Net::HTTP to sniff the body type
# and set the stream if appropriate
#
module Net
  class HTTP
    alias __request__ request

    def request(req, body = nil, &block)
      if body != nil && body.respond_to?(:read)
        req.body_stream = body
        return __request__(req, nil, &block)
      else
        return __request__(req, body, &block)
      end
    end
  end
end

#
# Connect to s3 using the ruby API provided by Amazon
#
conn = S3::AWSAuthConnection.new("[PUBLIC]", "[PRIVATE]", false)

#
# Stream a testfile to S3
#
open("testfile") do |stream|
  pp response = conn.put('BUCKET_NAME',
                         "testfile",
                         stream,
                         {
                           "x-amz-acl" => "public-read",
                           "Content-Type" => "text/plain",
                           "Content-Length" =>  FileTest.size("testfile").to_s
                         }
                        )
end

#
# Send a testfile in memory to S3
#
pp response = conn.put('BUCKET_NAME',
                       "testfile",
                       File.read('testfile'),
                       {
                         "x-amz-acl" => "public-read",
                         "Content-Type" => "text/plain"
                       }
                      )

  

A few notes about the code

• When streaming you have to supply the ‘Content-Length’ header
• I had an error about S3.rb calling strip on non-strings, I changed line 49 to ‘interesting_headers[lk] = value.to_s.strip’
• Make sure you replace PUBLIC, PRIVATE, and BUCKET_NAME with appropriate values

We have a couple of fairly high traffic sites that have large images designed to be used for desktop backgrounds. To save a bit of bandwidth, we decided to give Amazon’s S3 webservice a spin.

Signing up was fairly painless. They required a credit card (so they could bill us $.15/G storage and $.20/G transfer). After I signed up I quickly received an email that contained a link to my public and secret keys.

This is a fairly new service and the client tools are just getting started. For my purposes of uploading several images, I decided to use jSh3ll to ‘browse’ my S3 storage and a custom ruby script to upload a large amount of files.

After downloading and installing jSh3ll, I created my first bucket:
Read the rest of this entry »

Setting up ruby to work with Oracle seems to be a pain for a lot of people. Here are the steps I follow to set it up on a linux box from nothing to Active Record or DBI in 7 steps.

1. Gather the installation sources you will need. You have to be registered with oracle to get their instant client packages.
   Download the ruby oci8 drivers
   Download the oracle instant client
   You want the following packages (these examples assume the zip format):
   • Instant Client Package - Basic or Instant Client Package - Basic Lite
   • Instant Client Package - SDK
   • Instant Client Package - SQL*Plus (optional but nice to have)

Read the rest of this entry »

I was looking at implementing a luhn and credit card type check the other day in java and I noticed that there seems to be a lack of code for doing this in ruby. So I figured I would put something together for doing the checks in ruby.

The following function will do a luhn check for a given number (any number not just credit card numbers). The luhn algorithm is fairly simple, if you want to learn more about it check here.

def luhnCheck(ccNumber)
  ccNumber = ccNumber.gsub(/\\D/, '')
  cardLength = ccNumber.length
  parity = cardLength % 2

  sum = 0
  for i in 0...cardLength
    digit = ccNumber[i] - 48

    if i % 2 == parity
      digit = digit * 2
    end

    if digit > 9
      digit = digit - 9
    end

    sum = sum + digit
  end

  return (sum % 10) == 0
end

Before running the luhn check you may want to verify that you have a valid card type or at least one you want to accept. The following function will do that based on the current bin ranges for the differenct companies as of today (for more on this see the following: credit card number information and BIN range information). N.B. Bin ranges change from time to time so this will become dated. It should be easy enough to find the updated ranges.

def ccTypeCheck(ccNumber)
  ccNumber = ccNumber.gsub(/\\D/, '')
  case ccNumber
    when /^3[47]\\d{13}$/ then return "AMEX"
    when /^4\\d{12}(\\d{3})?$/ then return "VISA"
    when /^5\\d{15}|36\\d{14}$/ then return "MC"
    when /^6011\\d{12}|650\\d{13}$/ then return "DISC"
    when /^3(0[0-5]|8[0-1])\\d{11}$/ then return "DINERS"
    when /^(39\\d{12})|(389\\d{11})$/ then return "CB"
    when /^3\\d{15}|1800\\d{11}|2131\\d{11}$/ then return "JCB"
    else return "NA"
  end
end

I’ve been evaluating a couple different mapping software packages recently and the other day I noticed that the same addresses geocoded (for those who don’t know what geocoding is you can find out more about it here) to different locations. They are mostly the same but I figured it was interesting enough to do some more digging and see how different mapping services compared.I looked at the following services. Some of them are commercial services with open apis (ESRI and mapquest) and some of them are non-commercial services with open apis (yahoo and google although google does not have a geocoding api).

• http://www.esri.com/software/awspublicservices/index.html
• http://developer.yahoo.com/maps/index.html
• http://www.mapquest.com/openapi
• http://maps.google.com

For google I viewed the resulting values for latitude and longitude that were generated from a search for the address. For yahoo and ESRI I used their REST geocoding apis and for mapquest I used their java api to their commercial service since their openapi service is only in beta currently.I took 5 addresses located at different points in the US and one in Canada and mapped the returned latitude and longitude from each service. Here are the results:
Read the rest of this entry »

I recently had a need to adapt a script that recrawls a site with nutch. One of my design goals was to use the same command line options as the Fetchtool (one of the steps I had to take to recrawl a site).

It became apparent fairly quickly that bash’s built-in ‘getopts’ didn’t support long command line arguments, so I had to fall back on getopt.

Here is the portion of the script that parses the command line arguments:

set -- `getopt -n$0 -u -a --longoptions="depth: adddays: topN:" "h" "$@"` || usage
[ $# -eq 0 ] && usage

while [ $# -gt 0 ]
do
    case "$1" in
       --depth)   depth=$2;shift;;
       --adddays) adddays=$2;shift;;
       --topN)    topN=$2;shift;;
       -h)        usage;;
       --)        shift;break;;
       -*)        usage;;
       *)         break;;            #better be the crawl directory
    esac
    shift
done

Deconstructing this bit by bit:

Read the rest of this entry »

For alot of people, CVS is history; they’ve switched to SVN and are loving life. For those of us continuing the use this venerable tool, I present the following cobbled together bit of utility.

It’s a bash function. and a nice way to keep an eye on what’s happening on a cvs repository. I work with a team of around 7 guys, and I like to keep an eye on what’s changed and who changed it.

It works like this, summarizing what I consider to be the highlights from the cvs history command, and defaults to showing checkins for the current date:

$ cvshist
M 2006-02-22 10:34 EDT jsmith  1.3  InternalOperation.java         APRJ7A/src/com/ppc/oepica/model/internal
M 2006-02-22 10:53 EDT jsmith  1.37 FooBarCopier.java              APRJ7A/src/com/ppc/oepica/model/external
M 2006-02-22 11:26 EDT darrend 1.23 CommonJobBase.java             APRJ7A/src/com/ppc/oepica/model/common

I use the date command, which gives me some flexibility when specifying a date

$ cvshist yesterday
$ cvshist 2 days ago
$ cvshist january 10
$ cvshist 2006-01-11

Here’s the source. I’ve used this on linux and cygwin; you’ll need bash, gnu date, and cvs.

$ declare -f
cvshist ()
{
  export CVSROOT=:pserver:darrend@shlnxtest1:/home/cvs;
  TARGET_DATE=$(date -d "$*" +%Y-%m-%d);
  cvs history -a -z EDT -c -D $TARGET_DATE | sed "s/\\s*==.*$//" | grep "$TARGET_DATE"
}

At work I received an email from someone that listed libraries I should use when connecting to their AS400 system. I wanted a quick way to convert their whitespace table into a nice comma-separated list. Below is the first 6 lines from about 50 or so.

Sequence
 number   Library
    10    QTEMP            

    20    DDMFLIB2
....

If it’s computable but a one-off, I immediately go to Ruby nowadays. 10 years ago it would have been Perl, 15 or 20 years ago the choice would have been probably a meld of sed|awk.

I always go with __END__ in these situations; Ruby stops processing at the __END__ line, but you are given an enumerable access to the contents that follow with the magical DATA object.

matches = DATA.collect { |line| line.match(/\\\\s+\\\\d+\\\\s+(\\\\w+)/) }
matches.compact!
puts matches.collect { |match| match[1] }.join(', ')

__END__
Initial library list:                            

Sequence
 number   Library
    10    QTEMP            

    20    DDMFLIB2
    30    APPSDEV
    50    FILES2                                                       

   110    QRPG                                                              

Let’s try it out (running Ruby/Windows in a Cygwin terminal):

dday@xpmachine21 ~
$ ruby liblist.rb
QTEMP, DDMFLIB2, APPSDEV, FILES2, QRPG

Success!

We had a problem with how code was being displayed on the blog. IE wasn’t respecting the width of the code block and it was throwing the entire layout off.

Here is how I fixed it:

pre code {
 padding: 5px;
 display: block;
 overflow: auto;
/*I had to add this so ie wouldn't put scrollbars both ways.  Seems to be ignored by firefox*/
 overflow-y: visible;
/*I had to add this for ie to respect the width.  I don't know why it didn't respect its container's width*/
 width: 450px;
}