Mission Data Blog » Systems Integration

rails fixtures with models using set_table_name

darrend — Mon, 23 Apr 2007 03:13:47 +0000

If you have a model that uses set_table_name you may hit a snag when trying to use fixtures and unit tests. The solution is twofold: name the fixture file using the legacy table name, and use the set_fixture_class method in your unit test. I coudn’t find this method mentioned in Agile Web Development with Rails, and there’s only a brief mention on the rubyonrails.com site. Still, some web searches turned up the answers.

Here is a simple example:

We’ll start with a table named libris_book that contains book entries. The name libris_book won’t really fly with the Rails pluralized table name approach. It’s easy, however, to fix that disconnect in the model:

class Book < ActiveRecord::Base
  set_table_name 'libris_book'
end

Let’s take a look at our data:

~/src/libris$ script/console
Loading development environment.
>> Book.count
=> 1230

Good enough. Now, we’ll be good little coders and write some unit tests using fixtures. We’ll create a fixture file and our test class:

books.yml

test_book:
  title: Life in Lawrenceburg
  author: Darren Day

book_test.rb

require File.dirname(__FILE__) + '/../test_helper'

class BookTest < Test::Unit::TestCase
  fixtures :books

  def test_fixtures
    assert_equal(2,Book.count)
    test_book = books(:test_book)
  end
end

Looks good, but it won’t work!

~/src/libris$ rake test
(in /home/darrend/src/libris)
/usr/local/bin/ruby -Ilib:test "/usr/local/lib/ruby/gems/1.8/gems/rake-0.7.2/lib/rake/rake_test_loader.rb" "test/unit/book_test.rb"
Loaded suite /usr/local/lib/ruby/gems/1.8/gems/rake-0.7.2/lib/rake/rake_test_loader
Started
E
Finished in 0.01133 seconds.

  1) Error:
test_fixtures(BookTest):
ActiveRecord::StatementInvalid: Mysql::Error: Table 'libris_test.books' doesn't exist: DELETE FROM books
...

Ugh. That doesn’t look pretty. After some googling around the following comes together

Rename books.yml to libris_book.yml. The fixture yaml file has to share the same name as the database table name.
Change the fixtures declaration to fixtures :libris_book.
Use set_fixture_class to connect the fixture to the model class.

Here is our new and improved book_test.rb file:

require File.dirname(__FILE__) + '/../test_helper'

class BookTest < Test::Unit::TestCase
  set_fixture_class :libris_book => Book
  fixtures :libris_book

  def test_fixtures
    assert_equal(2,Book.count)
    test_book = libris_book(:test_book)
  end
end

How to Create and Overlay KML on a Google Map Using Google’s My Maps

carsonm — Fri, 06 Apr 2007 14:33:48 +0000

A few days ago Google made their “My Maps” announcment and since then there has been nothing but buzz buzz buzz about it. So I figured I would take a minute to show how someone can use this new tool from Google to create their own embedded maps for their site.

The first step is to create your new map. Go to Google Maps and click on the new “My Maps” tab.

Now you create your map. You can create polygons, lines or just points and include whatever you want in the descriptions because it will all be included in the KML.

Once you have completed your map you want to go to click on the KML link at the top right to save the KML for this map to disk.

Just last month Google added support for KML overlays to their map API. It isn’t very well documented on the GGeoXml documentation page but there is only one real issue you have to know about. The main thing to understand is that the KML file must sit on a public webserver somewhere and when you reference it Google needs to be able to access it. For example if you are testing on your local box and you put something like “http://localhost/mykml.kml” in for the URL it will not work. After you get past that the path to using your saved KML file is easy. Here is a full example:



  
    

    Example of KML overlay on a Google Map

Notice that you have to zoom and center to somewhere since that isn’t controled by the KML file but other than that it makes creating embedded maps very simple.

And here are the results from the example:

Enjoy!

S3 Streaming With PHP

carsonm — Sat, 25 Nov 2006 18:45:24 +0000

Steven pointed out that someone was looking for a way to stream to S3 using PHP and said I should figure out how to get it going. Since he made a patch to let you stream data with Ruby using S3 I figured I should do one for PHP. There may be better ways of doing this but since I’ve already done it with C using curl I figured that would be the fastest way.

First you will want to get the S3 PHP library from Amazon. Then you add the following to the file:

/**
 * putObjectStream -- Streams data to a bucket.
 *
 * Takes ($bucket, $key, $streamFunction, $contentType, $contentLength [,$acl, $metadataArray, $md5])
 *
 *
 * - [str] $bucket: the bucket into which file will be written
 * - [str] $key: key of written file
 * - [str] $streamFunction: function to call for data to stream
 * - [str] $contentType: file content type
 * - [str] $contentLength: file content length
 * - [str] $acl: access control policy of file (OPTIONAL: defaults to 'private')
 * - [str] $metadataArray: associative array containing user-defined metadata (name=>value) (OPTIONAL)
 * - [bool] $md5: the MD5 hash of the object (OPTIONAL)
*/
function putObjectStream($bucket, $key, $streamFunction, $contentType, $contentLength, $acl, $metadataArray, $md5){
        sort($metadataArray);
        $resource = "$bucket/$key";
        $resource = urlencode($resource);
        $httpDate = gmdate("D, d M Y G:i:s T");

        $curl_inst = curl_init();

        curl_setopt ($curl_inst, CURLOPT_CONNECTTIMEOUT, 30);
        curl_setopt ($curl_inst, CURLOPT_LOW_SPEED_LIMIT, 1);
        curl_setopt ($curl_inst, CURLOPT_LOW_SPEED_TIME, 180);
        curl_setopt ($curl_inst, CURLOPT_NOSIGNAL, 1);
        curl_setopt ($curl_inst, CURLOPT_READFUNCTION, $streamFunction);
        curl_setopt ($curl_inst, CURLOPT_URL, $this->serviceUrl . $resource);
        curl_setopt ($curl_inst, CURLOPT_UPLOAD, true);
        curl_setopt ($curl_inst, CURLINFO_CONTENT_LENGTH_UPLOAD, $contentLength);

        $header[] = "Date: $httpDate";
        $header[] = "Content-Type: $contentType";
        $header[] = "Content-Length: $contentLength";
        $header[] = "Expect: ";
        $header[] = "Transfer-Encoding: ";
        $header[] = "x-amz-acl: $acl";

        $MD5 = "";
        if($md5){
                $MD5 = $this->hex2b64(md5_file($filePath));
                $header[] = "Content-MD5: $MD5";
        }

        $stringToSign="PUT\n$MD5\n$contentType\n$httpDate\nx-amz-acl:$acl\n";
        foreach($metadataArray as $current){
                if($current!=""){
                        $stringToSign.="x-amz-meta-$current\n";
                        $header = substr($current,0,strpos($current,':'));
                        $meta = substr($current,strpos($current,':')+1,strlen($current));
                        $header[] = "x-amz-meta-$header: $meta";
                }
        }

        $stringToSign.="/$resource";

        $signature = $this->constructSig($stringToSign);

        $header[] = "Authorization: AWS $this->accessKeyId:$signature";

        curl_setopt($curl_inst, CURLOPT_HTTPHEADER, $header);
        curl_setopt($curl_inst, CURLOPT_RETURNTRANSFER, 1);

        $result = curl_exec ($curl_inst);

        $this->responseString = $result;
        $this->responseCode = curl_getinfo($curl_inst, CURLINFO_HTTP_CODE);

        curl_close($curl_inst);
}

Now you have an updated s3.php you are ready to start streaming. Be aware that the content length value needs to be correct. If it is not the correct size your upload will be either truncated (too short) or it will hang (too long). If it hangs the S3 service will timeout after a small amount of time and give you an error.

Here are two examples of how to use it. First with a file:

data, $length);
  }
}

$my_class_inst = new MyClass();

$fsize = filesize("/tmp/largefile.tar.gz");

$my_class_inst->data = fopen("/tmp/largefile.tar.gz", "r");

$s3_inst = new s3("access key", "private key");
$s3_inst->putObjectStream("abucket", "largefile.tar.gz", array($my_class_inst, "stream_function"), "application/x-gzip", $fsize, "public-read", array(), null);

print "Response String: " . $s3_inst->responseString . "\n";
print "Response Code: " . $s3_inst->responseCode . "\n";
print "Parsed XML: " . $s3_inst->parsed_xml . "\n";

fclose($my_class_inst->data);

?>

Another example of streaming some random text:

data;
  }
}

$my_class_inst = new MyClass();

$my_class_inst->data = "A test string";
$size = strlen($my_class_inst->data);

$s3_inst = new s3("access key", "private key");
$s3_inst->putObjectStream("bucketname", "test.txt", array($my_class_inst, "stream_function"), "text/html", $size, "public-read", array(), null);

print "Response String: " . $s3_inst->responseString . "\n";
print "Response Code: " . $s3_inst->responseCode . "\n";
print "Parsed XML: " . $s3_inst->parsed_xml . "\n";

?>

Creating S3 URLs that expire using PHP

carsonm — Thu, 01 Jun 2006 11:50:24 +0000

After reading this post on the S3 forum I realized that other people are thinking about doing some of the same stuff I have. paolonew was looking for a way to for a way to create URLs to S3 objects that expired. I did this a while back when I was thinking about how to host objects that I wanted to protect with some outside scheme. The confusion on the forum seemed to be about the timestamps used to expire the URL. For PHP it is fairly easy.

To clear up the expiration time issue I think these two steps are needed:

1) Keep in mind that the HTTP header dates must be in GMT.
2) The PHP function time() returns the seconds since the epoch January 1 1970 00:00:00 GMT). Notice here this is in GMT as well.
3) The HTTP Date header you see in a response from an S3 server is the time on that server. The machine you use to sign your request should be synced with that time. I think a good guess is that all the Amazon servers are synced with the atomic clock.

There isn’t much to securing a URL after you have that tucked away. Here is an example that will sign a URL so that it is valid for 60 seconds:

hash($stringToSign)));

  return "$S3_URL$bucketName$objectName?AWSAccessKeyId=$keyId&Expires=$expires&Signature=$sig";
}

function hex2b64($str)
{
    $raw = '';
    for ($i=0; $i < strlen($str); $i+=2)
    {
        $raw .= chr(hexdec(substr($str, $i, 2)));
    }
    return base64_encode($raw);
}

?>

The hex2b64 function was pulled from the amazon S3 PHP example library.

Good Techcrunch review of mapping apis

carsonm — Tue, 18 Apr 2006 14:06:21 +0000

Techcrunch has a good review by Frank Gruber of the look and feel of mapping services. I think it is notable that ESRI’s service is not included in the review. I think it is at least as good as the mapquest service. I may have to find time to redo my review of the acuracy of each again and a more technical evaluation of each.

Approximating a circle with a polygon

carsonm — Tue, 18 Apr 2006 02:36:18 +0000

I recently had an opportunity to use ESRI’s ArcSDE again. It is a spatial database interface and in this instance I was using the java api. I wanted to change what used to be a query using a rectangle into a query using a circle. For some reason parts of the java api for ArcSDE require a C library or something. I gave up pretty quickly on trying to make their arc function work since the documentation wasn’t very clear on how it worked. Instead I decided to figure out how to approximate a circle with a polygon and use that instead. Here is the result of that research.

First it helps to have a little background. Most people are familiar with latitude and longitude but you may not be familiar with their exact definitions. I have included the links to the wikipedia pages for both so you can research them more if you want. The main things to know are: latitude runs north/south and each degree of latitude is about 111 km; longitude runs east/west and is 111 km times the cosine of the latitude. The next thing that is helpful to know is the radius of the earth: 6,378.135 km in this case I’ll just use the equatorial radius.

Now that we have the earth taken care of we need to go back to basics. We start by looking at the definition of a circle and the unit circle. The mathematical definition of a circle is the important part here. The following equations will play an important part: x = a + r cos(t) and y = b + r sin(t) where a,b are the center of the circle, x,y are points of the unit circle, r is the radius of the circle, and t is the angle of a line from the origin to x,y in radians. It helps to think of the circle as broken up into triangles.

The last thing to keep in mind is that latitude and longitude are in degrees but when working with cos and sin you need to use radians and not degrees. The conversion between radians and degrees is easy: radians = degrees * (PI / 180) and degrees = radians * (180 / PI).

So now for the main formula in pseudocode:

 earths_radius = 3963 // number of miles in the radius of the earth
 longitude = 80          // longitude of the center of the circle
 latitude = 90            // latitude of the center of the circle
 points = 32              // number of points in the polygon
 circle_radius = 1       // miles of radius

 // find the raidus in lat/lon, units per latitude, units per longitude
 r_latitude = radius_to_degrees(circle_radius / earths_radius)
 r_longitude = r_latitude / cos(degrees_to_radius(latitude))

 // create a point for each edge, we need one extra point to connect the
 // end to the begining
 for point_count = 0 to points + 1
 {
    theta = PI * (point_count / (points / 2))    // find the angle for the current triangle
    circle_x = lng + (r_longitude * cos(theta))  // center a + radius x * cos(theta)
    circle_y = lat + (r_latitude * sin(theta))     // center b + radius y * sin(theta)

    // circle_x, circle_y represents a point on the circle
 }

That is all there is to it. In this example I just picked 32 points for the circle but you may want more or less depending on how large your radius is and how closely you want to approximate the circle. One thing to note about this function is that it produces the points in a counter-clockwise direction.

Now for an example using google maps to display a circle around a point with a radius of one mile:

Missiondata World HQ, Louisville, Ky

And here is the javascript source you need to do it:

var d2r = Math.PI / 180;   // degrees to radians
var r2d = 180 / Math.PI;   // radians to degrees
var earthsradius = 3963; // 3963 is the radius of the earth in miles

var hqpoint = new GPoint(-85.578852, 38.215601);
var map = new GMap(document.getElementById("map"));

map.addControl(new GSmallMapControl());
map.addControl(new GMapTypeControl());
map.centerAndZoom(hqpoint, 4);
map.addOverlay(new GMarker(hqpoint));

drawCircle(-85.578852, 38.215601, map);

function drawCircle(lng, lat, map)
{
   var points = 32;
   var radius = 1;             // radius in miles

   // find the raidus in lat/lon
   var rlat = (radius / earthsradius) * r2d;
   var rlng = rlat / Math.cos(lat * d2r);

   var extp = new Array();
   for (var i=0; i < points+1; i++) // one extra here makes sure we connect the
   {
      var theta = Math.PI * (i / (points/2));
      ex = lng + (rlng * Math.cos(theta)); // center a + radius x * cos(theta)
      ey = lat + (rlat * Math.sin(theta)); // center b + radius y * sin(theta)
      extp.push(new GPoint(ex, ey));
   }

   map.addOverlay(new GPolyline(extp, "#000000", 2));
}

Streaming data to S3 with ruby

steveny — Wed, 29 Mar 2006 18:56:49 +0000

One of the downsides of the ruby S3 example code is that it doesn’t support streaming of data (it loads the entire file into memory). It turns out, however, that all that is needed to stream data is a tweak to the ‘request’ method in Net::HTTP.

  require 'net/http'
require 'S3'
require 'pp'

#
# Replace the request method in Net::HTTP to sniff the body type
# and set the stream if appropriate
#
module Net
  class HTTP
    alias __request__ request

    def request(req, body = nil, &block)
      if body != nil && body.respond_to?(:read)
        req.body_stream = body
        return __request__(req, nil, &block)
      else
        return __request__(req, body, &block)
      end
    end
  end
end

#
# Connect to s3 using the ruby API provided by Amazon
#
conn = S3::AWSAuthConnection.new("[PUBLIC]", "[PRIVATE]", false)

#
# Stream a testfile to S3
#
open("testfile") do |stream|
  pp response = conn.put('BUCKET_NAME',
                         "testfile",
                         stream,
                         {
                           "x-amz-acl" => "public-read",
                           "Content-Type" => "text/plain",
                           "Content-Length" =>  FileTest.size("testfile").to_s
                         }
                        )
end

#
# Send a testfile in memory to S3
#
pp response = conn.put('BUCKET_NAME',
                       "testfile",
                       File.read('testfile'),
                       {
                         "x-amz-acl" => "public-read",
                         "Content-Type" => "text/plain"
                       }
                      )

A few notes about the code

When streaming you have to supply the ‘Content-Length’ header
I had an error about S3.rb calling strip on non-strings, I changed line 49 to ‘interesting_headers[lk] = value.to_s.strip’
Make sure you replace PUBLIC, PRIVATE, and BUCKET_NAME with appropriate values

Offloading web traffic using Amazon’s S3 service

steveny — Tue, 28 Mar 2006 17:43:45 +0000

We have a couple of fairly high traffic sites that have large images designed to be used for desktop backgrounds. To save a bit of bandwidth, we decided to give Amazon’s S3 webservice a spin.

Signing up was fairly painless. They required a credit card (so they could bill us $.15/G storage and $.20/G transfer). After I signed up I quickly received an email that contained a link to my public and secret keys.

This is a fairly new service and the client tools are just getting started. For my purposes of uploading several images, I decided to use jSh3ll to ‘browse’ my S3 storage and a custom ruby script to upload a large amount of files.

After downloading and installing jSh3ll, I created my first bucket:

jSh3ll> bucket www.hpnotiq.com
Bucket set to 'www.hpnotiq.com'
jSh3ll> createbucket
Created bucket 'www.hpnotiq.com'

I then hacked out a quick ruby script using the example ruby S3 library to upload all the image files I wanted to store on S3.

  #!/usr/bin/env ruby

require 'S3'
require 'pp'

conn = S3::AWSAuthConnection.new("", "", false)

Dir["images/*.jpg"].each do |filename|
  basename = filename.split('/')[-1]
  pp response = conn.put('www.hpnotiq.com',
                         "images/wallpapers/#{basename}",
                         File.new(filename).read,
                         {
                           "x-amz-acl" => "public-read",
                           "Content-Type" => "image/jpeg"
                         }
                        )
end

A few notes about the code:

We decided to use the domain name for the bucket and the directory+file for the object id
The S3 ruby libraries don’t stream, so the whole file is loaded into memory and put on the server
The hash at the end sets some headers for the ‘put’. The first tells S3 that this file can be read publicly. The second is the content type for the file(which will be set in the header when the file is downloaded)

Back to JSh3ll to see what’s there:

  jSh3ll> list
Item list for bucket 'www.hpnotiq.com'
key=images/wallpapers/blackdrink_1024.jpg, owner=steveny, size=75625 bytes, last modified=Tue Mar 28 11:54:57 EST 2006
key=images/wallpapers/blackdrink_800.jpg, owner=steveny, size=47251 bytes, last modified=Tue Mar 28 11:54:59 EST 2006
key=images/wallpapers/bottle_1024.jpg, owner=steveny, size=56590 bytes, last modified=Tue Mar 28 11:55:00 EST 2006

Now that all the files are on the service, I just needed to write an apache rewrite rule to redirect people to the images’ new location:

  RewriteEngine on
RewriteRule ^/images/wallpapers/(.*)$ http://s3.amazonaws.com/www.hpnotiq.com/images/wallpapers/$1 [R,L]

Bringing up my browser and looking at the headers, we can see where the request gets redirected to s3.amazonaws.com and the Content-Type is set correctly:

  GET /images/wallpapers/blackdrink_1024.jpg HTTP/1.1

Host: www.hpnotiq.com
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20051010 Firefox/1.0.7 (Ubuntu package 1.0.7)
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache

HTTP/1.x 302 Found
Date: Tue, 28 Mar 2006 17:49:27 GMT
Server: Apache
Location: http://s3.amazonaws.com/www.hpnotiq.com/images/wallpapers/blackdrink_1024.jpg
Content-Length: 326
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1

----------------------------------------------------------

http://s3.amazonaws.com/www.hpnotiq.com/images/wallpapers/blackdrink_1024.jpg

GET /www.hpnotiq.com/images/wallpapers/blackdrink_1024.jpg HTTP/1.1
Host: s3.amazonaws.com
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20051010 Firefox/1.0.7 (Ubuntu package 1.0.7)
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache

HTTP/1.x 200 OK
x-amz-id-2: GxsekKOL2jdljb5K6/RWPlswvBhbNjYrP8klHs2IXGqwZcjzRQ3FIsEhPo/L/Gfe
x-amz-request-id: 6E6912FF565D7B04
Date: Tue, 28 Mar 2006 17:49:29 GMT
Last-Modified: Tue, 28 Mar 2006 16:54:57 GMT
Etag: "a58c016d36905ba89cf17ea99a574cb3"
Content-Type: image/jpeg
Content-Length: 75625
Server: AmazonS3

GIS Geocoding experiments

carsonm — Thu, 09 Mar 2006 03:39:31 +0000

I’ve been evaluating a couple different mapping software packages recently and the other day I noticed that the same addresses geocoded (for those who don’t know what geocoding is you can find out more about it here) to different locations. They are mostly the same but I figured it was interesting enough to do some more digging and see how different mapping services compared.I looked at the following services. Some of them are commercial services with open apis (ESRI and mapquest) and some of them are non-commercial services with open apis (yahoo and google although google does not have a geocoding api).

For google I viewed the resulting values for latitude and longitude that were generated from a search for the address. For yahoo and ESRI I used their REST geocoding apis and for mapquest I used their java api to their commercial service since their openapi service is only in beta currently.I took 5 addresses located at different points in the US and one in Canada and mapped the returned latitude and longitude from each service. Here are the results:

Missiondata, 2300 Hurstbourne Village Drive, Suite 1100, Louisville, KY, 40299
mapquest: 38.215303, -85.578698
google: 38.215601, -85.578852
yahoo: 38.215496, -85.578669
esri: 38.216273, -85.579028

Googleplex, 1600 Amphitheatre Parkway, Mountain View, CA, 94043
mapquest: 37.4238, -122.0901
google: 37.422845, -122.085035
yahoo: 37.42386, -122.090332
esri: 37.42386, -122.090332

IBM New York, 590 Madison Ave, New York, NY, 10022
mapquest: 40.7623, -73.972399
google: 40.762267, -73.972535
yahoo: 40.762245, -73.972644
esri: 40.762245, -73.972644

Lockheed Martin, 6304 Spine Rd, Boulder, CO, 80301
mapquest: 40.0672, -105.206711
google: 40.067084, -105.206555
yahoo: 40.067063, -105.20654
esri: 40.067063, -105.20654

Red Hat Canada, 2323 Yonge Street, Suite #300, Toronto, Ontario M4P 2C9, Canada
yahoo: none
mapquest: 43.708137, -79.3985
google: 37.062500, -95.677068
esri: 79.398592, -43.7081

For the most part all of the services provide very similar results for geocoding in the US. From the limited number of locations I tested it seems that the larger cities have a more reliable set of outputs. In NY for example they almost completely stack on top of each other. I find it interesting too that in 4 out of 6 cases yahoo came back with the exact same results as ESRI provided.

The last thing to notice of course is that the results for Canada are not as good. First off I couldn’t get yahoo’s service to work with a Canadian address for some reason. Also notice that the ESRI latitude and longitude results are swapped. And viewing the results from google for latitude and longitude gave me some wildly incorrect values but the map was actually correct (thus the two maps). The only completely reliable service for Canada was mapquest and google (although google doesn’t do geocoding) with ESRI coming in next just because the results were swapped. All in all I think all 3 did a good job.

I’m not sure why google hasn’t produced something in the way of geocoding yet. I think they have the mapping nailed down but their lack of geocoding will put them at somewhat of a disadvantage at some point since the other 3 services are in the open now and both ESRI and mapquest already have commercial versions of their services that people use widely.