I'm early in the move to Amazon, of course, but already some specific tools are indispensable. I'm sure the list will grow, but here's where I'm at right now:
- S3Sync – A simple rsync-like command line tool (called 's3sync') for syncing stuff from a computer to S3 or the reverse. Also includes the 's3cmd' tool that roughly implements the web service API (list your buckets, put a file, etc.). This is the cornerstone of the plan for moving all my data files from my current server and backups to S3. Once the migration is complete, s3cmd will probably be the tool of choice for manipulating S3 programatically. Written in Ruby, and requires 1.8.4+; my CentOS 4 box couldn't find a new enough RPM, so I had to compile from source (which was totally painless).
- S3 Firefox Organizer (S3Fox)- a client for S3 following the standard FTP client paradigms. It has it's own proprietary definition of folders, but they're unobstrusive. Since I'm getting stuff into S3 mostly with s3sync, I'm mostly using this for read-only oversight.
- EC2 UI – a client for managing your EC2 "stuff" from Firefox. While not FTP-like at all, it shares a lot of the same UI as S3Fox for setting up accounts and the like.
I found JetS3t works great. I use the synchronize command line tool between my Windows and Linux computers. http://jets3t.s3.amazonaws.com/applications/applications.html
S3Sync didn't work for me well. The backup time was very long, It alway re upload the modify files all over again :(
I move to http://www.s3rsync.com and now using rsync to s3. Modified file are partly upload the my backup is much faster.
Naama G
Naama,
s3rsync is still doing a full-file load into S3, it's just going through an intermediate EC2 server. If the source of your backups are already on EC2, then it doesn't matter. Further, all your data must pass through S3rsync's servers, which is far from desirable with sensitive stuff.
Hi barneyb,
I don't have EC2 so it save me the trouble (and cost) to deal with EC2.
> s3rsync is still doing a full-file load into S3
Why should I care, as long as it fast and easy to use?
Naama,
Fast and easy to use are great, but sending your data to a third party is less than ideal, and the full load into S3 means you're paying for the full bandwidth every time.
Neither of those are necessarily deal breakers, but definitely worth considering if you're backing up a lot of data and/or it contains sensitive information.
Bameyb
> full load into S3 means you're paying for the full bandwidth every time.
No, I'm paying only for the bandwidth from my Server to S3rsync servers (which very efficient base on rsync) I'm not pay for the bandwidth from S3rsync servers to S3 storage.