John Carrino

You can now see the large version of images without waiting for my slow, slow cable modem async upload. In fact, all the image loads should be blazing fast as soon as I get them all uploaded. At the rate it has been going, it will take 3 days to upload everything to S3. My script is just about though my china pictures now.

And it is so easy to use. I work at Amazon and have had pretty much constant exposure to S3, but it still surprised me how easy it was to use. I used rSh3ell to get started. I just created a bucket named ‘carrino_albums’, then started uploading things to that bucket with acl ‘public-read’.

Here is the script I used to upload my files. Its pretty hacky. I just reads from a file called input and writes the file with that key to S3 as public-read. Some exception handling is thrown in there. I intend on creating a much better script along the lines of s3sync, which behaves like rsync. S3 makes this easy, becasue it automatically stores the upload date and the md5sum in the metadata, and you can retreive that with a HEAD HTTP call. You can also do a list call on the bucket, which will get you the metadata of all the keys in that bucket.

If I upload a file named china_day2/dsc_2572.sized.jpg to bucket carrino_albums, you can find it here. It’s that easy!

#!/usr/bin/env ruby
require 'S3'

AWS_ACCESS_KEY_ID = 'public key'
AWS_SECRET_ACCESS_KEY = 'private key'
USE_SSL = false
@conn = S3::AWSAuthConnection.new(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, USE_SSL)
@bucket = 'carrino_albums'

def write_file(file_name)
    begin
        bytes = nil
        key = file_name  # NOTE: this is specific to our situation
        File.open(file_name, "rb") {|f| bytes = f.read }
        headers = {
              'x-amz-acl' => "public-read",
              'Content-Length' =>  FileTest.size(file_name).to_s
        }
        response = @conn.put(@bucket, key, bytes, headers).http_response.message
        if response == 'OK'
            puts "#{file_name} SUCCESSFULLY uploaded"
        else
            puts "#{file_name} FAILED to be uploaded"
        end
    rescue Interrupt => e
        puts "Caught Interrupt, bailing out"
        exit 1
    rescue Exception => e
        puts "#{file_name} FAILED to be uploaded and caused an exception. #{e.class.name}: #{e.to_s}"
    end
end

STDOUT.sync = true
File.readlines("input").each { |line|
    line.chomp!
    write_file(line)
}