First you will want to get the S3 PHP library from Amazon. Then you add the following to the file:

/**
 * putObjectStream -- Streams data to a bucket.
 *
 * Takes ($bucket, $key, $streamFunction, $contentType, $contentLength [,$acl, $metadataArray, $md5])
 *
 *
 * - [str] $bucket: the bucket into which file will be written
 * - [str] $key: key of written file
 * - [str] $streamFunction: function to call for data to stream
 * - [str] $contentType: file content type
 * - [str] $contentLength: file content length
 * - [str] $acl: access control policy of file (OPTIONAL: defaults to 'private')
 * - [str] $metadataArray: associative array containing user-defined metadata (name=>value) (OPTIONAL)
 * - [bool] $md5: the MD5 hash of the object (OPTIONAL)
*/
function putObjectStream($bucket, $key, $streamFunction, $contentType, $contentLength, $acl, $metadataArray, $md5){
        sort($metadataArray);
        $resource = "$bucket/$key";
        $resource = urlencode($resource);
        $httpDate = gmdate("D, d M Y G:i:s T");

        $curl_inst = curl_init();

        curl_setopt ($curl_inst, CURLOPT_CONNECTTIMEOUT, 30);
        curl_setopt ($curl_inst, CURLOPT_LOW_SPEED_LIMIT, 1);
        curl_setopt ($curl_inst, CURLOPT_LOW_SPEED_TIME, 180);
        curl_setopt ($curl_inst, CURLOPT_NOSIGNAL, 1);
        curl_setopt ($curl_inst, CURLOPT_READFUNCTION, $streamFunction);
        curl_setopt ($curl_inst, CURLOPT_URL, $this->serviceUrl . $resource);
        curl_setopt ($curl_inst, CURLOPT_UPLOAD, true);
        curl_setopt ($curl_inst, CURLINFO_CONTENT_LENGTH_UPLOAD, $contentLength);

        $header[] = "Date: $httpDate";
        $header[] = "Content-Type: $contentType";
        $header[] = "Content-Length: $contentLength";
        $header[] = "Expect: ";
        $header[] = "Transfer-Encoding: ";
        $header[] = "x-amz-acl: $acl";

        $MD5 = "";
        if($md5){
                $MD5 = $this->hex2b64(md5_file($filePath));
                $header[] = "Content-MD5: $MD5";
        }

        $stringToSign="PUT\n$MD5\n$contentType\n$httpDate\nx-amz-acl:$acl\n";
        foreach($metadataArray as $current){
                if($current!=""){
                        $stringToSign.="x-amz-meta-$current\n";
                        $header = substr($current,0,strpos($current,':'));
                        $meta = substr($current,strpos($current,':')+1,strlen($current));
                        $header[] = "x-amz-meta-$header: $meta";
                }
        }

        $stringToSign.="/$resource";

        $signature = $this->constructSig($stringToSign);

        $header[] = "Authorization: AWS $this->accessKeyId:$signature";

        curl_setopt($curl_inst, CURLOPT_HTTPHEADER, $header);
        curl_setopt($curl_inst, CURLOPT_RETURNTRANSFER, 1);

        $result = curl_exec ($curl_inst);

        $this->responseString = $result;
        $this->responseCode = curl_getinfo($curl_inst, CURLINFO_HTTP_CODE);

        curl_close($curl_inst);
}

Now you have an updated s3.php you are ready to start streaming. Be aware that the content length value needs to be correct. If it is not the correct size your upload will be either truncated (too short) or it will hang (too long). If it hangs the S3 service will timeout after a small amount of time and give you an error.

Here are two examples of how to use it. First with a file:

<?php

include 's3.php';

class MyClass
{
  var $data;

  function stream_function($handle, $fd, $length)
  {
    return fread($this->data, $length);
  }
}

$my_class_inst = new MyClass();

$fsize = filesize("/tmp/largefile.tar.gz");

$my_class_inst->data = fopen("/tmp/largefile.tar.gz", "r");

$s3_inst = new s3("access key", "private key");
$s3_inst->putObjectStream("abucket", "largefile.tar.gz", array($my_class_inst, "stream_function"), "application/x-gzip", $fsize, "public-read", array(), null);

print "Response String: " . $s3_inst->responseString . "\n";
print "Response Code: " . $s3_inst->responseCode . "\n";
print "Parsed XML: " . $s3_inst->parsed_xml . "\n";

fclose($my_class_inst->data);

?>

Another example of streaming some random text:

<?php

include 's3.php';

class MyClass
{
  var $data;

  function stream_function($handle, $fd, $length)
  {
    return $this->data;
  }
}

$my_class_inst = new MyClass();

$my_class_inst->data = "A test string";
$size = strlen($my_class_inst->data);

$s3_inst = new s3("access key", "private key");
$s3_inst->putObjectStream("bucketname", "test.txt", array($my_class_inst, "stream_function"), "text/html", $size, "public-read", array(), null);

print "Response String: " . $s3_inst->responseString . "\n";
print "Response Code: " . $s3_inst->responseCode . "\n";
print "Parsed XML: " . $s3_inst->parsed_xml . "\n";

?>

To clear up the expiration time issue I think these two steps are needed:

1) Keep in mind that the HTTP header dates must be in GMT.
2) The PHP function time() returns the seconds since the epoch January 1 1970 00:00:00 GMT). Notice here this is in GMT as well.
3) The HTTP Date header you see in a response from an S3 server is the time on that server. The machine you use to sign your request should be synced with that time. I think a good guess is that all the Amazon servers are synced with the atomic clock.

There isn’t much to securing a URL after you have that tucked away. Here is an example that will sign a URL so that it is valid for 60 seconds:

<?php

require_once('Crypt/HMAC.php');

echo getS3Redirect("/test.jpg") . "\\n";

function getS3Redirect($objectName)
{
  $S3_URL = "http://s3.amazonaws.com";
  $keyId = "your key";
  $secretKey = "your secret";
  $expires = time() + 60;
  $bucketName = "/your bucket";

  $stringToSign = "GET\\n\\n\\n$expires\\n$bucketName$objectName";
  $hasher =& new Crypt_HMAC($secretKey, "sha1");
  $sig = urlencode(hex2b64($hasher->hash($stringToSign)));

  return "$S3_URL$bucketName$objectName?AWSAccessKeyId=$keyId&Expires=$expires&Signature=$sig";
}

function hex2b64($str)
{
    $raw = '';
    for ($i=0; $i < strlen($str); $i+=2)
    {
        $raw .= chr(hexdec(substr($str, $i, 2)));
    }
    return base64_encode($raw);
}

?>

The hex2b64 function was pulled from the amazon S3 PHP example library.

  require 'net/http'
require 'S3'
require 'pp'

#
# Replace the request method in Net::HTTP to sniff the body type
# and set the stream if appropriate
#
module Net
  class HTTP
    alias __request__ request

    def request(req, body = nil, &block)
      if body != nil && body.respond_to?(:read)
        req.body_stream = body
        return __request__(req, nil, &block)
      else
        return __request__(req, body, &block)
      end
    end
  end
end

#
# Connect to s3 using the ruby API provided by Amazon
#
conn = S3::AWSAuthConnection.new("[PUBLIC]", "[PRIVATE]", false)

#
# Stream a testfile to S3
#
open("testfile") do |stream|
  pp response = conn.put('BUCKET_NAME',
                         "testfile",
                         stream,
                         {
                           "x-amz-acl" => "public-read",
                           "Content-Type" => "text/plain",
                           "Content-Length" =>  FileTest.size("testfile").to_s
                         }
                        )
end

#
# Send a testfile in memory to S3
#
pp response = conn.put('BUCKET_NAME',
                       "testfile",
                       File.read('testfile'),
                       {
                         "x-amz-acl" => "public-read",
                         "Content-Type" => "text/plain"
                       }
                      )

  

A few notes about the code

• When streaming you have to supply the ‘Content-Length’ header
• I had an error about S3.rb calling strip on non-strings, I changed line 49 to ‘interesting_headers[lk] = value.to_s.strip’
• Make sure you replace PUBLIC, PRIVATE, and BUCKET_NAME with appropriate values

Signing up was fairly painless. They required a credit card (so they could bill us $.15/G storage and $.20/G transfer). After I signed up I quickly received an email that contained a link to my public and secret keys.

This is a fairly new service and the client tools are just getting started. For my purposes of uploading several images, I decided to use jSh3ll to ‘browse’ my S3 storage and a custom ruby script to upload a large amount of files.

After downloading and installing jSh3ll, I created my first bucket:

jSh3ll> bucket www.hpnotiq.com
Bucket set to 'www.hpnotiq.com'
jSh3ll> createbucket
Created bucket 'www.hpnotiq.com'

I then hacked out a quick ruby script using the example ruby S3 library to upload all the image files I wanted to store on S3.

  #!/usr/bin/env ruby

require 'S3'
require 'pp'

conn = S3::AWSAuthConnection.new("<YOUR PUBLIC KEY>", "<YOUR PRIVATE KEY>", false)

Dir["images/*.jpg"].each do |filename|
  basename = filename.split('/')[-1]
  pp response = conn.put('www.hpnotiq.com',
                         "images/wallpapers/#{basename}",
                         File.new(filename).read,
                         {
                           "x-amz-acl" => "public-read",
                           "Content-Type" => "image/jpeg"
                         }
                        )
end
  

A few notes about the code:

• We decided to use the domain name for the bucket and the directory+file for the object id
• The S3 ruby libraries don’t stream, so the whole file is loaded into memory and put on the server
• The hash at the end sets some headers for the ‘put’. The first tells S3 that this file can be read publicly. The second is the content type for the file(which will be set in the header when the file is downloaded)

Back to JSh3ll to see what’s there:

  jSh3ll> list
Item list for bucket 'www.hpnotiq.com'
key=images/wallpapers/blackdrink_1024.jpg, owner=steveny, size=75625 bytes, last modified=Tue Mar 28 11:54:57 EST 2006
key=images/wallpapers/blackdrink_800.jpg, owner=steveny, size=47251 bytes, last modified=Tue Mar 28 11:54:59 EST 2006
key=images/wallpapers/bottle_1024.jpg, owner=steveny, size=56590 bytes, last modified=Tue Mar 28 11:55:00 EST 2006
  

Now that all the files are on the service, I just needed to write an apache rewrite rule to redirect people to the images’ new location:

  RewriteEngine on
RewriteRule ^/images/wallpapers/(.*)$ http://s3.amazonaws.com/www.hpnotiq.com/images/wallpapers/$1 [R,L]
  

Bringing up my browser and looking at the headers, we can see where the request gets redirected to s3.amazonaws.com and the Content-Type is set correctly:

  GET /images/wallpapers/blackdrink_1024.jpg HTTP/1.1

Host: www.hpnotiq.com
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20051010 Firefox/1.0.7 (Ubuntu package 1.0.7)
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache

HTTP/1.x 302 Found
Date: Tue, 28 Mar 2006 17:49:27 GMT
Server: Apache
Location: http://s3.amazonaws.com/www.hpnotiq.com/images/wallpapers/blackdrink_1024.jpg
Content-Length: 326
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1

----------------------------------------------------------

http://s3.amazonaws.com/www.hpnotiq.com/images/wallpapers/blackdrink_1024.jpg

GET /www.hpnotiq.com/images/wallpapers/blackdrink_1024.jpg HTTP/1.1
Host: s3.amazonaws.com
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20051010 Firefox/1.0.7 (Ubuntu package 1.0.7)
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache

HTTP/1.x 200 OK
x-amz-id-2: GxsekKOL2jdljb5K6/RWPlswvBhbNjYrP8klHs2IXGqwZcjzRQ3FIsEhPo/L/Gfe
x-amz-request-id: 6E6912FF565D7B04
Date: Tue, 28 Mar 2006 17:49:29 GMT
Last-Modified: Tue, 28 Mar 2006 16:54:57 GMT
Etag: "a58c016d36905ba89cf17ea99a574cb3"
Content-Type: image/jpeg
Content-Length: 75625
Server: AmazonS3