Mission Data Blog
Louisville-based Web Development & Software Engineering

Steven pointed out that someone was looking for a way to stream to S3 using PHP and said I should figure out how to get it going. Since he made a patch to let you stream data with Ruby using S3 I figured I should do one for PHP. There may be better ways of doing this but since I’ve already done it with C using curl I figured that would be the fastest way.

First you will want to get the S3 PHP library from Amazon. Then you add the following to the file:

/**
 * putObjectStream -- Streams data to a bucket.
 *
 * Takes ($bucket, $key, $streamFunction, $contentType, $contentLength [,$acl, $metadataArray, $md5])
 *
 *
 * - [str] $bucket: the bucket into which file will be written
 * - [str] $key: key of written file
 * - [str] $streamFunction: function to call for data to stream
 * - [str] $contentType: file content type
 * - [str] $contentLength: file content length
 * - [str] $acl: access control policy of file (OPTIONAL: defaults to 'private')
 * - [str] $metadataArray: associative array containing user-defined metadata (name=>value) (OPTIONAL)
 * - [bool] $md5: the MD5 hash of the object (OPTIONAL)
*/
function putObjectStream($bucket, $key, $streamFunction, $contentType, $contentLength, $acl, $metadataArray, $md5){
        sort($metadataArray);
        $resource = "$bucket/$key";
        $resource = urlencode($resource);
        $httpDate = gmdate("D, d M Y G:i:s T");

        $curl_inst = curl_init();

        curl_setopt ($curl_inst, CURLOPT_CONNECTTIMEOUT, 30);
        curl_setopt ($curl_inst, CURLOPT_LOW_SPEED_LIMIT, 1);
        curl_setopt ($curl_inst, CURLOPT_LOW_SPEED_TIME, 180);
        curl_setopt ($curl_inst, CURLOPT_NOSIGNAL, 1);
        curl_setopt ($curl_inst, CURLOPT_READFUNCTION, $streamFunction);
        curl_setopt ($curl_inst, CURLOPT_URL, $this->serviceUrl . $resource);
        curl_setopt ($curl_inst, CURLOPT_UPLOAD, true);
        curl_setopt ($curl_inst, CURLINFO_CONTENT_LENGTH_UPLOAD, $contentLength);

        $header[] = "Date: $httpDate";
        $header[] = "Content-Type: $contentType";
        $header[] = "Content-Length: $contentLength";
        $header[] = "Expect: ";
        $header[] = "Transfer-Encoding: ";
        $header[] = "x-amz-acl: $acl";

        $MD5 = "";
        if($md5){
                $MD5 = $this->hex2b64(md5_file($filePath));
                $header[] = "Content-MD5: $MD5";
        }

        $stringToSign="PUT\\n$MD5\\n$contentType\\n$httpDate\\nx-amz-acl:$acl\\n";
        foreach($metadataArray as $current){
                if($current!=""){
                        $stringToSign.="x-amz-meta-$current\\n";
                        $header = substr($current,0,strpos($current,':'));
                        $meta = substr($current,strpos($current,':')+1,strlen($current));
                        $header[] = "x-amz-meta-$header: $meta";
                }
        }

        $stringToSign.="/$resource";

        $signature = $this->constructSig($stringToSign);

        $header[] = "Authorization: AWS $this->accessKeyId:$signature";

        curl_setopt($curl_inst, CURLOPT_HTTPHEADER, $header);
        curl_setopt($curl_inst, CURLOPT_RETURNTRANSFER, 1);

        $result = curl_exec ($curl_inst);

        $this->responseString = $result;
        $this->responseCode = curl_getinfo($curl_inst, CURLINFO_HTTP_CODE);

        curl_close($curl_inst);
}

Now you have an updated s3.php you are ready to start streaming. Be aware that the content length value needs to be correct. If it is not the correct size your upload will be either truncated (too short) or it will hang (too long). If it hangs the S3 service will timeout after a small amount of time and give you an error.

Here are two examples of how to use it. First with a file:

<?php

include 's3.php';

class MyClass
{
  var $data;

  function stream_function($handle, $fd, $length)
  {
    return fread($this->data, $length);
  }
}

$my_class_inst = new MyClass();

$fsize = filesize("/tmp/largefile.tar.gz");

$my_class_inst->data = fopen("/tmp/largefile.tar.gz", "r");

$s3_inst = new s3("access key", "private key");
$s3_inst->putObjectStream("abucket", "largefile.tar.gz", array($my_class_inst, "stream_function"), "application/x-gzip", $fsize, "public-read", array(), null);

print "Response String: " . $s3_inst->responseString . "\\n";
print "Response Code: " . $s3_inst->responseCode . "\\n";
print "Parsed XML: " . $s3_inst->parsed_xml . "\\n";

fclose($my_class_inst->data);

?>

Another example of streaming some random text:

<?php

include 's3.php';

class MyClass
{
  var $data;

  function stream_function($handle, $fd, $length)
  {
    return $this->data;
  }
}

$my_class_inst = new MyClass();

$my_class_inst->data = "A test string";
$size = strlen($my_class_inst->data);

$s3_inst = new s3("access key", "private key");
$s3_inst->putObjectStream("bucketname", "test.txt", array($my_class_inst, "stream_function"), "text/html", $size, "public-read", array(), null);

print "Response String: " . $s3_inst->responseString . "\\n";
print "Response Code: " . $s3_inst->responseCode . "\\n";
print "Parsed XML: " . $s3_inst->parsed_xml . "\\n";

?>

               

This entry was posted on Saturday, November 25th, 2006 at 1:45 pm by carsonm and is filed under linux. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.