Amazon S3 URL Builder for ColdFusion

First task for my Amazon move is getting data assets (non-code-managed files) over to S3. I have a variety of types of data assets that need to move and have references updated, most of which require authentication. To make that easier, I wrote a little UDF to take care of building urls with authentication credentials in there.

<cffunction name="s3Url" output="false" returntype="string">
  <cfargument name="awsKey" type="string" required="true" />
  <cfargument name="awsSecret" type="string" required="true" />
  <cfargument name="bucket" type="string" required="true" />
  <cfargument name="objectKey" type="string" required="true" />
  <cfargument name="requestType" type="string" default="vhost"
    hint="Must be one of 'regular', 'ssl', 'vhost', or 'cname'.  'Vhost' and 'cname' are only valid if your bucket name conforms to the S3 virtual host conventions, and cname requires a CNAME record configured in your DNS." />
  <cfargument name="timeout" type="numeric" default="900"
    hint="The number of seconds the URL is good for.  Defaults to 900 (15 minutes)." />
  <cfscript>
    var expires = "";
    var stringToSign = "";
    var algo = "HmacSHA1";
    var signingKey = "";
    var mac = "";
    var signature = "";
    var destUrl = "";

    expires = int(getTickCount() / 1000) + timeout;
    stringToSign = "GET" & chr(10)
      & chr(10)
      & chr(10)
      & expires & chr(10)
      & "/#bucket#/#objectKey#";
    signingKey = createObject("java", "javax.crypto.spec.SecretKeySpec").init(awsSecret.getBytes(), algo);
    mac = createObject("java", "javax.crypto.Mac").getInstance(algo);
    mac.init(signingKey);
    signature = toBase64(mac.doFinal(stringToSign.getBytes()));
    if (requestType EQ "ssl" OR requestType EQ "regular") {
      destUrl = "http" & iif(requestType EQ "ssl", de("s"), de("")) & "://s3.amazonaws.com/#bucket#/#objectKey#?AWSAccessKeyId=#awsKey#&Signature=#urlEncodedFormat(signature)#&Expires=#expires#";
    } else if (requestType EQ "cname") {
      destUrl = "http://#bucket#/#objectKey#?AWSAccessKeyId=#awsKey#&Signature=#urlEncodedFormat(signature)#&Expires=#expires#";
    } else { // vhost
      destUrl = "http://#bucket#.s3.amazonaws.com/#objectKey#?AWSAccessKeyId=#awsKey#&Signature=#urlEncodedFormat(signature)#&Expires=#expires#";
    }

    return destUrl;
  </cfscript>
</cffunction>

To use it, do something like this:

s3Url(aws_key, aws_secret, "s3.barneyb.com", "test.txt", 'cname');

That will generate a request to the file "test.txt" in the "s3.barneyb.com" bucket, using a CNAME-style URL. Obviously you'll have to know my AWS key and secret for it to work, and I'm not telling, but substitute your own values. You can use regular (bucket name in the request), vhost (bucket name in an S3 subdomain), cname (a vanity CNAME pointing at S3), or ssl (regular over HTTPS) for the 5th type parameter to control the style of URL generated.

Edit: here's a link to the project page.

30 responses to “Amazon S3 URL Builder for ColdFusion”

  1. R. Hunter

    OK, that's so sweet. Worked first try.

  2. Johan

    Barney – I am using the same code to create the S3 signature and no problems in Adobe CF. I got round to trying Railo 3 today and the signature returned by the same code is different (and of course incorrect). Any ideas why?

  3. Andrew

    Johan, am having the same problem, did you find a solution?

    Andrew

  4. Andrew

    answered my own question……

    had a problem that any "GET" request would fail in Railo (railo-3.0.1.000-railo-express-6.1.0-with-jre-windows) (works perfectly in CF7)
    a complete guess/intuition here: I added

    and added text/html to var cs

    now all is ok!

    Andrew.

  5. John Matlock

    This whole signature thing has left me feeling particularly stupid. I've looked over your code and don't understand it at all. It's a function. But how do you call it, what parameters do you send it, what do you get back.

    What I really want is a simple Cold Fusion function that gives me the signature. Is that what this does?

    John Matlock

  6. John Matlock

    I tried putting your code in a test page and browsed to it. It didn't work, giving the following:

    Invalid parser construct found on line 21 at position 5.
    ColdFusion was looking at the following text:var
    Invalid expression format. The usual cause is an error in the expression structure.

    I suspect that this came because we are using an older version of Cold Fusion. Any further thoughts.

    Second question. Could you provide a short sample of how to call this function. I got the first two arguments OK. But what's the "bucket" and the "objectKey"? And what do you get when you call the function?

    Third is really a comment:

    Thanks for taking the time to post this code and for your prompt response to my first question. I'm feeling particularly stupid with this whole thing. I'm also feeling a bit annoyed as my site has been running for a lot of years (I started with CF Version 2.) and I have other things to do with my time.

    One last question. Do you have any comment on the service that http://www.amazonapisigning.com is doing?

    John Matlock

  7. John Matlock

    Unfortunately that was about what I was expecting.

    I'm using CF Version 5. When Version 6 came out they changed the code base from C to Java. I tested V6 and found enough differences (mostly minor to be sure) that I would have had to go through and test several hundred templates. As Version 6 didn't seem to offer enough new features to justify changing I put it off.

    Meanwhile time has gone on and versions have come about that if anything would make it even more important to test everything on even more hundreds of templates. And Version 5 is stable, doing everything I need, up to this Amazon problem.

    It's also expensive to buy new versions for all the servers and development machines every time they decide to issue a new version. As I remember their licensing agreement you have (or are supposed) to purchase a separate copy for each machine.

    Although I haven't used it much, I'd want to seriously look at becoming a PHP shop rather than CF. Any comments?

    My current thinking is that I'll give http://www.amazonapisigning.com a try and if it works fine, otherwise I'll just stop working with Amazon. In looking at the threads on the Amazon API forum it seems that a lot of people are thinking this way.

    I don't quite understand just what problem Amazon was trying to solve with this new security system. Did they get badly hacked or something? It may well be that they will lose enough revenue to re-think their approach.

    Oh well, thanks again.

    John

  8. Jo C

    I've been trying to use this on CF 9 without much success – The code appears to work beautifully but its producing the wrong signature and I can't work out why – I've even tried replacing the signature generation with cf_hmac calls and it's still doing it incorrectly

    It's a great piece of code saving me hours of work – if I could just get the signature working – I would be very very happy – This is somewhat a proof of concept and I really want to get it going to prove to the boss that it works – Any clues you could give me would really be appreciated

  9. Jo C

    I grabbed the latest code from the link – the only reason I've not posted it there is that you can't and this appears to be the only post you have on it

    Thanks for responding

  10. Jo C

    Got to be honest this is driving me nuts – I've seen it working using a java testing suite – works fine so why isn't the hmac_sha1 encode working ? I've also now tried it on CF8 and get the same error – So its not Coldfusion Server per se that is the problem – its something in the code I'm doing wrong – it must be

  11. Andrew

    could it be the version of java you are now using? that would explain why it used to work on CF8?

  12. Jo C

    I don't think so – I'm using the same JDK for both Coldfusion and Java – if that was the case – the Java testsuite wouldn't run either

  13. Jo C

    I've actually found a work around for this problem – I took the Amazon S3 Library for Rest in Java http://developer.amazonwebservices.com/connect/entry.jspa?externalID=132&categoryID=47 and compiled the files below com into a jar – Which was dropped into the coldfusion server/lib

    I then used the examples to build the following coldfusion function

    s3conn = CreateObject("java", "com.amazon.s3.AWSAuthConnection");
    s3conn.init("#variables.awsKey#","#variables.awsSecret#");
    s3object = CreateObject("java", "com.amazon.s3.S3Object");
    s3object.init(content.GetBytes());
    headers = createObject("java","java.util.TreeMap");
    contentTypeArray[1] = "#contentType#";
    headers.put("Content-Type",contentTypeArray);
    response = s3conn.put(bucket, objectKey, s3object, headers).connection.getResponseMessage();
    return response;

    Which works perfectly – At last – I've just got to build the rest of the suite now so that we can upload, delete, create URLs etc.

    Very relieved I found a work around and part of me is very pleased that good old Java provided the solution having spent 9 years as a Java programmer before starting work in Coldfusion

  14. Ted

    I'm getting this error a little more than half the time using your function:

    SignatureDoesNotMatch
    The request signature we calculated does not match the signature you provided. Check your key and signing method.

    It works on some files, but not others. Any ideas?

    Here's an example of one that does NOT work (gives error above):

    http://media.pilatesanytime.com.s3.amazonaws.com/2009/12/Amy_Level%201_1205_hd.mp4?AWSAccessKeyId=AKIAI2VTKXRAZRK5HBIQ&Signature=rq2gfu34bGVigE28ciyouE79EI0%3D&Expires=1260572977

    On the other hand, this one DOES work:

    http://media.pilatesanytime.com.s3.amazonaws.com/2009/12/merideth_color_correct_hd.mp4?AWSAccessKeyId=AKIAI2VTKXRAZRK5HBIQ&Signature=qNlu4xSwL4oxXtS6Ykn%2FirrcLhs%3D&Expires=1260573100

    It's got me banging my head against the desk.

  15. Ted

    Hmm… Here is another that is failing and I don't think it has a space:

    http://media.pilatesanytime.com.s3.amazonaws.com/2009/12/class3_full_med.mp4?AWSAccessKeyId=AKIAI2VTKXRAZRK5HBIQ&Signature=R1kkMOStaBSRWWpGK3JyI0yo%2FFA%3D&Expires=1260573829

    I didn't do any URL encoding… I think that might have happened from your blog software perhaps? The link I copy and pasted actually did have a space in that example.

  16. Ted

    OK… I added a URLEncodedFormat() function onto the filename I passed your function. This seems to have it working. So the function call looks like this now:

    Seem like it should have made a difference? I guess it has…

  17. Gregory Matthews

    Do you have an Coldfusion CFC example for creating a Cloudfront Signed URL? I've looked everywhere for the last four days and cannot find anything at all.

    Thanks in advance for your help.

    Gregory

  18. Ted

    As far as I know, there is no such thing as signed Cloudfront URLs. This is the biggest problem with cloudfront– the only way to use it is to have your files wide open to being read by anybody (by changing the permissions).

  19. Gregory Matthews

    Ted,

    Actually, there is: http://docs.amazonwebservices.com/AmazonCloudFront/latest/DeveloperGuide/index.html?PrivateContent.html#CreateRSAKey

    It falls under "Serving Private Content". The problem is nobody has created this CFC, "yet". I just spoke with Barney about getting this done. Be on the lookout … it's coming!

    Gregory

  20. Ted

    That would be fantastic. I have a web site that streams Pilates videos to subscribers (http://www.pilatesanytime.com) using S3 signed URLS, which is why I am using this CFC. I realized that the speed of S3 is not good enough to deliver HD quality videos to people overseas, so I want to switch to Cloudfront, but we really need some protection for our URLs to avoid bandwidth stealing. This would be very useful to me.

  21. Gregory Matthews

    Ted,

    You're describing the exact same situation I've been dealing with for quite some time (only my topic is different!). Glad to hear that you'll be able to put it into immediate use as well.

    Best,

    Gregory

  22. Gregory Matthews

    Ted,

    By the way, I took a quick look at your site and think you did a great job with your implementation of CFWINDOW to tease potential subscribers into signing up for your free trial.

    Clean, fast, and good user experience.

    Shoot me an email to goxmedia@gmail.com and let me know how the site is doing for you. I'm doing something very similar, different topic, heavy JQuery and Coldfusion (perfect fit), and a fairly advanced CMS. I'd like to share experiences and tips if you're up for it.

    Either way, take it easy – and keep stretchin'!

    Gregory

  23. Larry Larson

    Hello –

    This code looks valuable, but it is generating a java error for me in both CF9 and CF8:

    Element AWSSECRET is undefined in a Java object of type class.

    Any ideas?

Leave a Reply

You must escape tags within your comment. Use &lt; for < and &gt; for >.