I am running latest [easyazon-link asin=”1783283114″ locale=”us”] GNU Debian Linux 7.x [/easyazon-link] and I would like to rsync my important files from local drive to [easyazon-link asin=”B004OR1JF0″ locale=”us”] Amazon S3 [/easyazon-link] storage with rsync tool.
You will need:
- Amazon AWS account
- Amazon Access Key and Secret Key
- Amazon S3 bucket name
- s3fs utility installed and configured on the [easyazon-link asin=”111821854X” locale=”us”] Debian [/easyazon-link]server
First we need to install few requirements on the [easyazon-link asin=”1783283114″ locale=”us”] Debian [/easyazon-link] server:
su apt-get install build-essential libfuse-dev fuse-utils libcurl4-openssl-dev libxml2-dev mime-support apt-get update
Second, now you can download the latest build of s3fs from the project link https://code.google.com/p/s3fs/ , untar and compile.
su cd Downloads wget https://s3fs.googlecode.com/files/s3fs-1.74.tar.gz tar xzvf s3fs-1.74.tar.gz cd s3fs* ./configure make make install
By now you should have installed the s3fs utility on the system. You can try to run it by typing s3fs you should see message that it is not configured yet.
If installed and not configured – you will see this message
s3fs: missing BUCKET argument Usage: s3fs BUCKET:[PATH] MOUNTPOINT [OPTION]...
Now we will create a local directory that will be mounted to [easyazon-link asin=”0596515812″ locale=”us”] Amazon S3 [/easyazon-link] bucket
su mkdir /mnt/s3 chown yourusername:yourusername /mnt/s3
Next step is create a password file with the Amazon keys:
su echo "ACCESS_KEY_ID:SECRET_ACCESS_KEY" > /etc/passwd-s3fs chmod 0600 /etc/passwd-s3fs
This is covering the basic setup, installation and configuration s3fs on the [easyazon-link asin=”1593270690″ locale=”us”] Debian [/easyazon-link] server. Lets try it now.
su s3fs bucket-name /mnt/s3 cd /mnt/s3 ls -l
If you have all setup correctly, you should be able to connect to S3 storage bucket, mount it as the /mnt/s3 and be able to navigate to the directory and list files in the S3 bucket. If all works, you can now write small script and create a cron job so all is done automatically and you will receive email upon rsync completion.
############# SAMPLE SCRIPT ################## #!/bin/bash # Script : Mount S3 bucket as a local drive /mnt/s3 and rsync files bucket="mys3-bucket" destination="/mnt/s3" backup="/data/MYFILES" # mounting S3 bucket for backup as a local drive s3fs $bucket $destination # rsync local files to S3 bucket rsync -avz $backup $destination # Once rsync finished - send email and unmount the S3 bucket from local system ls -lh $destination | mail -s "Rsync completed" sysadmin@system.com umount $destination ######### END OF SAMPLE SCRIPT #######
Modify the variables according your needs, save it as a s3fs.sh file, change permission and you can add it to crontab so it runs on daily basis.
su chmod 700 s3fs.sh ---------------------- su crontab -e 0 0 * * * /path/to/s3fs.sh # this runs rsync to s3 every day at midnight
By the way there was released a new book on Amazon covering best Administration practices for Debian 7 🙂
[easyazon-image align=”none” asin=”1783283114″ locale=”us” height=”160″ src=”http://ecx.images-amazon.com/images/I/51KgrYCMzoL._SL160_.jpg” width=”130″]