BarneyBlog

This weekend I ported my big filesystem-based app to S3, and it went like a dream. It's a image-management application, with all the actual images stored on disk. In addition to the standard import/edit/delete, the app provides automatic on-the-fly thumbnail generation, along with primitive editing capabilities (crop, resize, rotate, etc.). With images on local disk, that's all really easy: read them in, do whatever, write them back out. I figured using S3 would make things both more cumbersome and less performant. Both suspicions turned out to be unwarranted.

Building on the 's3Url' UDF that I published last week, I whipped up a little CFC to manage file storage on S3 with a very simple API. It has s3Url, putFileOnS3, getFileFromS3, s3FileExists, and deleteS3File methods, which all do about what you'd expect. You can grab the code here: amazons3.cfc.txt (make sure you remove the ".txt" extension) or visit the project page. It uses the simple HTTP-based interface, so after the authentication is handled, it's all very simple and fast. I haven't looked at the SOAP interface - why bother complicating a simple task?

With that CFC (and an application-specific wrapper to take care of some path-related transforms), porting the whole app took about two hours. I also realized after I was mostly done that the CF image tools accept URLs as well as files, so I switched my image reads to just use URLs instead of pulling the file local and reading it from disk.

As for moving all the actual content, S3Sync was a champ, moving about 4.5GB of data from my Cari server to S3 in a few hours, including gracefully handling a couple errors raised by S3 (which a retry - performed automatically - solved), and a stop/restart in the middle. Total cost: about 65 cents.

Next is porting the blogs, including all the Picasa-based galleries. Unfortunately, that means writing PHP, but with how easy the CF stuff was, I don't think it'll be too much effort.

First task for my Amazon move is getting data assets (non-code-managed files) over to S3. I have a variety of types of data assets that need to move and have references updated, most of which require authentication. To make that easier, I wrote a little UDF to take care of building urls with authentication credentials in there.

<cffunction name="s3Url" output="false" returntype="string">
  <cfargument name="awsKey" type="string" required="true" />
  <cfargument name="awsSecret" type="string" required="true" />
  <cfargument name="bucket" type="string" required="true" />
  <cfargument name="objectKey" type="string" required="true" />
  <cfargument name="requestType" type="string" default="vhost"
    hint="Must be one of 'regular', 'ssl', 'vhost', or 'cname'.  'Vhost' and 'cname' are only valid if your bucket name conforms to the S3 virtual host conventions, and cname requires a CNAME record configured in your DNS." />
  <cfargument name="timeout" type="numeric" default="900"
    hint="The number of seconds the URL is good for.  Defaults to 900 (15 minutes)." />
  <cfscript>
    var expires = "";
    var stringToSign = "";
    var algo = "HmacSHA1";
    var signingKey = "";
    var mac = "";
    var signature = "";
    var destUrl = "";

    expires = int(getTickCount() / 1000) + timeout;
    stringToSign = "GET" & chr(10)
      & chr(10)
      & chr(10)
      & expires & chr(10)
      & "/#bucket#/#objectKey#";
    signingKey = createObject("java", "javax.crypto.spec.SecretKeySpec").init(awsSecret.getBytes(), algo);
    mac = createObject("java", "javax.crypto.Mac").getInstance(algo);
    mac.init(signingKey);
    signature = toBase64(mac.doFinal(stringToSign.getBytes()));
    if (requestType EQ "ssl" OR requestType EQ "regular") {
      destUrl = "http" & iif(requestType EQ "ssl", de("s"), de("")) & "://s3.amazonaws.com/#bucket#/#objectKey#?AWSAccessKeyId=#awsKey#&Signature=#urlEncodedFormat(signature)#&Expires=#expires#";
    } else if (requestType EQ "cname") {
      destUrl = "http://#bucket#/#objectKey#?AWSAccessKeyId=#awsKey#&Signature=#urlEncodedFormat(signature)#&Expires=#expires#";
    } else { // vhost
      destUrl = "http://#bucket#.s3.amazonaws.com/#objectKey#?AWSAccessKeyId=#awsKey#&Signature=#urlEncodedFormat(signature)#&Expires=#expires#";
    }

    return destUrl;
  </cfscript>
</cffunction>

To use it, do something like this:

s3Url(aws_key, aws_secret, "s3.barneyb.com", "test.txt", 'cname');

That will generate a request to the file "test.txt" in the "s3.barneyb.com" bucket, using a CNAME-style URL. Obviously you'll have to know my AWS key and secret for it to work, and I'm not telling, but substitute your own values. You can use regular (bucket name in the request), vhost (bucket name in an S3 subdomain), cname (a vanity CNAME pointing at S3), or ssl (regular over HTTPS) for the 5th type parameter to control the style of URL generated.

Edit: here's a link to the project page.