First task for my Amazon move is getting data assets (non-code-managed files) over to S3. I have a variety of types of data assets that need to move and have references updated, most of which require authentication. To make that easier, I wrote a little UDF to take care of building urls with authentication credentials in there.
<cffunction name="s3Url" output="false" returntype="string">
<cfargument name="awsKey" type="string" required="true" />
<cfargument name="awsSecret" type="string" required="true" />
<cfargument name="bucket" type="string" required="true" />
<cfargument name="objectKey" type="string" required="true" />
<cfargument name="requestType" type="string" default="vhost"
hint="Must be one of 'regular', 'ssl', 'vhost', or 'cname'. 'Vhost' and 'cname' are only valid if your bucket name conforms to the S3 virtual host conventions, and cname requires a CNAME record configured in your DNS." />
<cfargument name="timeout" type="numeric" default="900"
hint="The number of seconds the URL is good for. Defaults to 900 (15 minutes)." />
<cfscript>
var expires = "";
var stringToSign = "";
var algo = "HmacSHA1";
var signingKey = "";
var mac = "";
var signature = "";
var destUrl = "";
expires = int(getTickCount() / 1000) + timeout;
stringToSign = "GET" & chr(10)
& chr(10)
& chr(10)
& expires & chr(10)
& "/#bucket#/#objectKey#";
signingKey = createObject("java", "javax.crypto.spec.SecretKeySpec").init(awsSecret.getBytes(), algo);
mac = createObject("java", "javax.crypto.Mac").getInstance(algo);
mac.init(signingKey);
signature = toBase64(mac.doFinal(stringToSign.getBytes()));
if (requestType EQ "ssl" OR requestType EQ "regular") {
destUrl = "http" & iif(requestType EQ "ssl", de("s"), de("")) & "://s3.amazonaws.com/#bucket#/#objectKey#?AWSAccessKeyId=#awsKey#&Signature=#urlEncodedFormat(signature)#&Expires=#expires#";
} else if (requestType EQ "cname") {
destUrl = "http://#bucket#/#objectKey#?AWSAccessKeyId=#awsKey#&Signature=#urlEncodedFormat(signature)#&Expires=#expires#";
} else { // vhost
destUrl = "http://#bucket#.s3.amazonaws.com/#objectKey#?AWSAccessKeyId=#awsKey#&Signature=#urlEncodedFormat(signature)#&Expires=#expires#";
}
return destUrl;
</cfscript>
</cffunction>
To use it, do something like this:
s3Url(aws_key, aws_secret, "s3.barneyb.com", "test.txt", 'cname');
That will generate a request to the file "test.txt" in the "s3.barneyb.com" bucket, using a CNAME-style URL. Obviously you'll have to know my AWS key and secret for it to work, and I'm not telling, but substitute your own values. You can use regular (bucket name in the request), vhost (bucket name in an S3 subdomain), cname (a vanity CNAME pointing at S3), or ssl (regular over HTTPS) for the 5th type parameter to control the style of URL generated.
Edit: here's a link to the project page.
OK, that's so sweet. Worked first try.
R.,
Glad it worked for you.
Barney – I am using the same code to create the S3 signature and no problems in Adobe CF. I got round to trying Railo 3 today and the signature returned by the same code is different (and of course incorrect). Any ideas why?
Johan,
Don't know why it wouldn't work on Railo 3, though if you're using Railo, can't you just use the built-in S3 support? I haven't done any S3 stuff on Railo to this point, so I know nothing about it (like whether you can even do secured stuff).
Can you isolate when the value changes between CF and Railo? I don't have a dev environment where I am right now, nor the time to get one set up (particularly with CF's 300MB download), so I can't check.
Johan, am having the same problem, did you find a solution?
Andrew
answered my own question……
had a problem that any "GET" request would fail in Railo (railo-3.0.1.000-railo-express-6.1.0-with-jre-windows) (works perfectly in CF7)
a complete guess/intuition here: I added
and added text/html to var cs
now all is ok!
Andrew.
This whole signature thing has left me feeling particularly stupid. I've looked over your code and don't understand it at all. It's a function. But how do you call it, what parameters do you send it, what do you get back.
What I really want is a simple Cold Fusion function that gives me the signature. Is that what this does?
John Matlock
John,
Look at the CFARGUMENT tags. The first four are all that is required, and they're named for what they are. The first two are your account credentials and the second two identify the resource you want a URL for.
I tried putting your code in a test page and browsed to it. It didn't work, giving the following:
Invalid parser construct found on line 21 at position 5.
ColdFusion was looking at the following text:var
Invalid expression format. The usual cause is an error in the expression structure.
I suspect that this came because we are using an older version of Cold Fusion. Any further thoughts.
Second question. Could you provide a short sample of how to call this function. I got the first two arguments OK. But what's the "bucket" and the "objectKey"? And what do you get when you call the function?
Third is really a comment:
Thanks for taking the time to post this code and for your prompt response to my first question. I'm feeling particularly stupid with this whole thing. I'm also feeling a bit annoyed as my site has been running for a lot of years (I started with CF Version 2.) and I have other things to do with my time.
One last question. Do you have any comment on the service that http://www.amazonapisigning.com is doing?
John Matlock
John,
bucket and objectKey are the standard S3 concepts. For example, I have a bucket named 'potd-s3.barneyb.com' with an object at key 'test.txt'. That corresponds to the url 'http://potd-s3.barneyb.com.s3.amazonaws.com/test.txt', along with a couple other variations on a theme.
Assuming these two invocations of s3Url (made at 15:08 US Pacific time on 7/28/09):
#s3Url(myKey, mySecret, "potd-s3.barneyb.com", "test.txt")#
#s3Url(myKey, mySecret, "potd-s3.barneyb.com", "test.txt", "cname", 86400)#
I'll be rewarded with these two URLs (the first expiring after 15 minutes, the second after 24 hours):
http://potd-s3.barneyb.com.s3.amazonaws.com/test.txt?AWSAccessKeyId=0YVN1G49J71QKD4Y0982&Signature=AISFgtnwAzx0cnC4dzdyP4WFyKA%3D&Expires=1248819782
http://potd-s3.barneyb.com/test.txt?AWSAccessKeyId=0YVN1G49J71QKD4Y0982&Signature=7eU%2FEVrFcZ8olFZRAPXt2J5wwXo%3D&Expires=1248905282
What CFML engine and version are you running? I would expect this to work on ColdFusion 7 and newer, Railo 3.0 and newer, and Open BlueDragon 1.0 and newer, though I've only tested on ColdFusion 8 and Railo 3.1. That error message makes me think you're on either ColdFusion 6.0 or 6.1, which I wouldn't necessarily expected to fail, but wouldn't have been confident saying would work.
The UDF in the post has since morphed into a CFC that provides other functionality as well. Don't know if you care, but might be worth a look. http://www.barneyb.com/barneyblog/projects/amazon-s3-cfc/
I don't know anything about amazonapisigning.com.
Unfortunately that was about what I was expecting.
I'm using CF Version 5. When Version 6 came out they changed the code base from C to Java. I tested V6 and found enough differences (mostly minor to be sure) that I would have had to go through and test several hundred templates. As Version 6 didn't seem to offer enough new features to justify changing I put it off.
Meanwhile time has gone on and versions have come about that if anything would make it even more important to test everything on even more hundreds of templates. And Version 5 is stable, doing everything I need, up to this Amazon problem.
It's also expensive to buy new versions for all the servers and development machines every time they decide to issue a new version. As I remember their licensing agreement you have (or are supposed) to purchase a separate copy for each machine.
Although I haven't used it much, I'd want to seriously look at becoming a PHP shop rather than CF. Any comments?
My current thinking is that I'll give http://www.amazonapisigning.com a try and if it works fine, otherwise I'll just stop working with Amazon. In looking at the threads on the Amazon API forum it seems that a lot of people are thinking this way.
I don't quite understand just what problem Amazon was trying to solve with this new security system. Did they get badly hacked or something? It may well be that they will lose enough revenue to re-think their approach.
Oh well, thanks again.
John
I've been trying to use this on CF 9 without much success – The code appears to work beautifully but its producing the wrong signature and I can't work out why – I've even tried replacing the signature generation with cf_hmac calls and it's still doing it incorrectly
It's a great piece of code saving me hours of work – if I could just get the signature working – I would be very very happy – This is somewhat a proof of concept and I really want to get it going to prove to the boss that it works – Any clues you could give me would really be appreciated
Jo,
Are you using the code in this post, or did you grab the CFC from http://www.barneyb.com/barneyblog/projects/amazon-s3-cfc/ ? The latter is a better starting point, though I've not done any CF9 testing.
I grabbed the latest code from the link – the only reason I've not posted it there is that you can't and this appears to be the only post you have on it
Thanks for responding
Got to be honest this is driving me nuts – I've seen it working using a java testing suite – works fine so why isn't the hmac_sha1 encode working ? I've also now tried it on CF8 and get the same error – So its not Coldfusion Server per se that is the problem – its something in the code I'm doing wrong – it must be
could it be the version of java you are now using? that would explain why it used to work on CF8?
I don't think so – I'm using the same JDK for both Coldfusion and Java – if that was the case – the Java testsuite wouldn't run either
I've actually found a work around for this problem – I took the Amazon S3 Library for Rest in Java http://developer.amazonwebservices.com/connect/entry.jspa?externalID=132&categoryID=47 and compiled the files below com into a jar – Which was dropped into the coldfusion server/lib
I then used the examples to build the following coldfusion function
s3conn = CreateObject("java", "com.amazon.s3.AWSAuthConnection");
s3conn.init("#variables.awsKey#","#variables.awsSecret#");
s3object = CreateObject("java", "com.amazon.s3.S3Object");
s3object.init(content.GetBytes());
headers = createObject("java","java.util.TreeMap");
contentTypeArray[1] = "#contentType#";
headers.put("Content-Type",contentTypeArray);
response = s3conn.put(bucket, objectKey, s3object, headers).connection.getResponseMessage();
return response;
Which works perfectly – At last – I've just got to build the rest of the suite now so that we can upload, delete, create URLs etc.
Very relieved I found a work around and part of me is very pleased that good old Java provided the solution having spent 9 years as a Java programmer before starting work in Coldfusion
I'm getting this error a little more than half the time using your function:
SignatureDoesNotMatch
The request signature we calculated does not match the signature you provided. Check your key and signing method.
It works on some files, but not others. Any ideas?
Here's an example of one that does NOT work (gives error above):
http://media.pilatesanytime.com.s3.amazonaws.com/2009/12/Amy_Level%201_1205_hd.mp4?AWSAccessKeyId=AKIAI2VTKXRAZRK5HBIQ&Signature=rq2gfu34bGVigE28ciyouE79EI0%3D&Expires=1260572977
On the other hand, this one DOES work:
http://media.pilatesanytime.com.s3.amazonaws.com/2009/12/merideth_color_correct_hd.mp4?AWSAccessKeyId=AKIAI2VTKXRAZRK5HBIQ&Signature=qNlu4xSwL4oxXtS6Ykn%2FirrcLhs%3D&Expires=1260573100
It's got me banging my head against the desk.
Hey Ted. It looks like you have a space in the failing URL's filename, and it's being URL encoded when you generate the S3 URL. Are you ensuring that the objectKey you're passing to the CFC is also URL encoded (i.e. you're doing the URL encoding before assembling the URL, rather than after)?
If you pass the key "my neat file.txt" to generate the URL, but then the URL you send people to is "my%20neat%20file.txt", S3 is going to complain that the signature doesn't match, because it'll do the comparison against the latter.
Hmm… Here is another that is failing and I don't think it has a space:
http://media.pilatesanytime.com.s3.amazonaws.com/2009/12/class3_full_med.mp4?AWSAccessKeyId=AKIAI2VTKXRAZRK5HBIQ&Signature=R1kkMOStaBSRWWpGK3JyI0yo%2FFA%3D&Expires=1260573829
I didn't do any URL encoding… I think that might have happened from your blog software perhaps? The link I copy and pasted actually did have a space in that example.
OK… I added a URLEncodedFormat() function onto the filename I passed your function. This seems to have it working. So the function call looks like this now:
Seem like it should have made a difference? I guess it has…
Do you have an Coldfusion CFC example for creating a Cloudfront Signed URL? I've looked everywhere for the last four days and cannot find anything at all.
Thanks in advance for your help.
Gregory
As far as I know, there is no such thing as signed Cloudfront URLs. This is the biggest problem with cloudfront– the only way to use it is to have your files wide open to being read by anybody (by changing the permissions).
Ted,
Actually, there is: http://docs.amazonwebservices.com/AmazonCloudFront/latest/DeveloperGuide/index.html?PrivateContent.html#CreateRSAKey
It falls under "Serving Private Content". The problem is nobody has created this CFC, "yet". I just spoke with Barney about getting this done. Be on the lookout … it's coming!
Gregory
That would be fantastic. I have a web site that streams Pilates videos to subscribers (http://www.pilatesanytime.com) using S3 signed URLS, which is why I am using this CFC. I realized that the speed of S3 is not good enough to deliver HD quality videos to people overseas, so I want to switch to Cloudfront, but we really need some protection for our URLs to avoid bandwidth stealing. This would be very useful to me.
Ted,
You're describing the exact same situation I've been dealing with for quite some time (only my topic is different!). Glad to hear that you'll be able to put it into immediate use as well.
Best,
Gregory
Ted,
By the way, I took a quick look at your site and think you did a great job with your implementation of CFWINDOW to tease potential subscribers into signing up for your free trial.
Clean, fast, and good user experience.
Shoot me an email to goxmedia@gmail.com and let me know how the site is doing for you. I'm doing something very similar, different topic, heavy JQuery and Coldfusion (perfect fit), and a fairly advanced CMS. I'd like to share experiences and tips if you're up for it.
Either way, take it easy – and keep stretchin'!
Gregory
Hello –
This code looks valuable, but it is generating a java error for me in both CF9 and CF8:
Element AWSSECRET is undefined in a Java object of type class.
Any ideas?
Larry,
Without seeing your code I can't say for sure, but I'd wager you're omitting the 'awsSecret' argument (perhaps because the name is entered wrong – there's a double 's'). In any case, I'd recommend using the S3 CFC (http://www.barneyb.com/barneyblog/projects/amazon-s3-cfc/) instead of just the UDF. It has the same functionality, plus a lot more, including initializing with your principal and credential so you don't need to pass them to every invocation.