Mission Data Blog
Louisville-based Web Development & Software Engineering

After reading this post on the S3 forum I realized that other people are thinking about doing some of the same stuff I have. paolonew was looking for a way to for a way to create URLs to S3 objects that expired. I did this a while back when I was thinking about how to host objects that I wanted to protect with some outside scheme. The confusion on the forum seemed to be about the timestamps used to expire the URL. For PHP it is fairly easy.

To clear up the expiration time issue I think these two steps are needed:

1) Keep in mind that the HTTP header dates must be in GMT.
2) The PHP function time() returns the seconds since the epoch January 1 1970 00:00:00 GMT). Notice here this is in GMT as well.
3) The HTTP Date header you see in a response from an S3 server is the time on that server. The machine you use to sign your request should be synced with that time. I think a good guess is that all the Amazon servers are synced with the atomic clock.

There isn’t much to securing a URL after you have that tucked away. Here is an example that will sign a URL so that it is valid for 60 seconds:

<?php

require_once('Crypt/HMAC.php');

echo getS3Redirect("/test.jpg") . "\\\\n";

function getS3Redirect($objectName)
{
  $S3_URL = "http://s3.amazonaws.com";
  $keyId = "your key";
  $secretKey = "your secret";
  $expires = time() + 60;
  $bucketName = "/your bucket";

  $stringToSign = "GET\\\\n\\\\n\\\\n$expires\\\\n$bucketName$objectName";
  $hasher =& new Crypt_HMAC($secretKey, "sha1");
  $sig = urlencode(hex2b64($hasher->hash($stringToSign)));

  return "$S3_URL$bucketName$objectName?AWSAccessKeyId=$keyId&Expires=$expires&Signature=$sig";
}

function hex2b64($str)
{
    $raw = '';
    for ($i=0; $i < strlen($str); $i+=2)
    {
        $raw .= chr(hexdec(substr($str, $i, 2)));
    }
    return base64_encode($raw);
}

?>

The hex2b64 function was pulled from the amazon S3 PHP example library.

One of the downsides of the ruby S3 example code is that it doesn’t support streaming of data (it loads the entire file into memory). It turns out, however, that all that is needed to stream data is a tweak to the ‘request’ method in Net::HTTP.

  require 'net/http'
require 'S3'
require 'pp'

#
# Replace the request method in Net::HTTP to sniff the body type
# and set the stream if appropriate
#
module Net
  class HTTP
    alias __request__ request

    def request(req, body = nil, &block)
      if body != nil && body.respond_to?(:read)
        req.body_stream = body
        return __request__(req, nil, &block)
      else
        return __request__(req, body, &block)
      end
    end
  end
end

#
# Connect to s3 using the ruby API provided by Amazon
#
conn = S3::AWSAuthConnection.new("[PUBLIC]", "[PRIVATE]", false)

#
# Stream a testfile to S3
#
open("testfile") do |stream|
  pp response = conn.put('BUCKET_NAME',
                         "testfile",
                         stream,
                         {
                           "x-amz-acl" => "public-read",
                           "Content-Type" => "text/plain",
                           "Content-Length" =>  FileTest.size("testfile").to_s
                         }
                        )
end

#
# Send a testfile in memory to S3
#
pp response = conn.put('BUCKET_NAME',
                       "testfile",
                       File.read('testfile'),
                       {
                         "x-amz-acl" => "public-read",
                         "Content-Type" => "text/plain"
                       }
                      )

  

A few notes about the code

• When streaming you have to supply the ‘Content-Length’ header
• I had an error about S3.rb calling strip on non-strings, I changed line 49 to ‘interesting_headers[lk] = value.to_s.strip’
• Make sure you replace PUBLIC, PRIVATE, and BUCKET_NAME with appropriate values

We have a couple of fairly high traffic sites that have large images designed to be used for desktop backgrounds. To save a bit of bandwidth, we decided to give Amazon’s S3 webservice a spin.

Signing up was fairly painless. They required a credit card (so they could bill us $.15/G storage and $.20/G transfer). After I signed up I quickly received an email that contained a link to my public and secret keys.

This is a fairly new service and the client tools are just getting started. For my purposes of uploading several images, I decided to use jSh3ll to ‘browse’ my S3 storage and a custom ruby script to upload a large amount of files.

After downloading and installing jSh3ll, I created my first bucket:
(more…)