One of the downsides of the ruby S3 example code is that it doesn’t support streaming of data (it loads the entire file into memory). It turns out, however, that all that is needed to stream data is a tweak to the ‘request’ method in Net::HTTP.
require 'net/http'
require 'S3'
require 'pp'
#
# Replace the request method in Net::HTTP to sniff the body type
# and set the stream if appropriate
#
module Net
class HTTP
alias __request__ request
def request(req, body = nil, &block)
if body != nil && body.respond_to?(:read)
req.body_stream = body
return __request__(req, nil, &block)
else
return __request__(req, body, &block)
end
end
end
end
#
# Connect to s3 using the ruby API provided by Amazon
#
conn = S3::AWSAuthConnection.new("[PUBLIC]", "[PRIVATE]", false)
#
# Stream a testfile to S3
#
open("testfile") do |stream|
pp response = conn.put('BUCKET_NAME',
"testfile",
stream,
{
"x-amz-acl" => "public-read",
"Content-Type" => "text/plain",
"Content-Length" => FileTest.size("testfile").to_s
}
)
end
#
# Send a testfile in memory to S3
#
pp response = conn.put('BUCKET_NAME',
"testfile",
File.read('testfile'),
{
"x-amz-acl" => "public-read",
"Content-Type" => "text/plain"
}
)
A few notes about the code
- When streaming you have to supply the ‘Content-Length’ header
- I had an error about S3.rb calling strip on non-strings, I changed line 49 to ‘interesting_headers[lk] = value.to_s.strip’
- Make sure you replace PUBLIC, PRIVATE, and BUCKET_NAME with appropriate values
1
Probably want to qualify this — you need a pretty recent ruby, 1.8.4, to take advantage of body_stream in Net::HTTPRequest.
2
Hi, I saw your post on the Amazon boards for S3, thanks a ton for blogging this, I thought it was going to be much harder. The only real change I had to make was your code was to give it a binary/octet-stream Content-Type (because I was streaming up media files rather than text). Awesome work.
3
November 9th, 2006 @ 4:53 pm oOcha » Blog Archive » Backup plan Hollered:[...] I also used some information from a great post on the Mission Data Blog. The current code I had opened a file read it into memory and then transferred that to S3. The mentioned blog post explained how to change Net::HTTP to stream a file. [...]
4
Excellent! We use S3 as part of our backup plan and some of the files we transer out can get big. This is just what I was needing.
5
November 25th, 2006 @ 1:45 pm Mission Data Blog » Blog Archive » S3 Streaming With PHP Hollered:[...] Steven pointed out that someone was looking for a way to stream to S3 using PHP and said I should figure out how to get it going. Since he made a patch to let you stream data with Ruby using S3 I figured I should do one for PHP. There may be better ways of doing this but since I’ve already done it with C using curl I figured that would be the fastest way. [...]
6
Thanks a lot for your snippet. It works fine, at least for text files.
But when I try uploading pdf or zip-files, it will only upload half of the file (size is incorrect and I can see only the first “half of the cryptic symbols). I guess the problem is related with the content type. I tried binary/octet-stream and application/octet-stream, but both didnt work.
Do you have any idea why it does not work for me?
Thanks, Markus