One of the downsides of the ruby S3 example code is that it doesn’t support streaming of data (it loads the entire file into memory). It turns out, however, that all that is needed to stream data is a tweak to the ‘request’ method in Net::HTTP.
require 'net/http'
require 'S3'
require 'pp'
#
# Replace the request method in Net::HTTP to sniff the body type
# and set the stream if appropriate
#
module Net
class HTTP
alias __request__ request
def request(req, body = nil, &block)
if body != nil && body.respond_to?(:read)
req.body_stream = body
return __request__(req, nil, &block)
else
return __request__(req, body, &block)
end
end
end
end
#
# Connect to s3 using the ruby API provided by Amazon
#
conn = S3::AWSAuthConnection.new("[PUBLIC]", "[PRIVATE]", false)
#
# Stream a testfile to S3
#
open("testfile") do |stream|
pp response = conn.put('BUCKET_NAME',
"testfile",
stream,
{
"x-amz-acl" => "public-read",
"Content-Type" => "text/plain",
"Content-Length" => FileTest.size("testfile").to_s
}
)
end
#
# Send a testfile in memory to S3
#
pp response = conn.put('BUCKET_NAME',
"testfile",
File.read('testfile'),
{
"x-amz-acl" => "public-read",
"Content-Type" => "text/plain"
}
)
A few notes about the code
- When streaming you have to supply the ‘Content-Length’ header
- I had an error about S3.rb calling strip on non-strings, I changed line 49 to ‘interesting_headers[lk] = value.to_s.strip’
- Make sure you replace PUBLIC, PRIVATE, and BUCKET_NAME with appropriate values