Mount S3 bucket as a local drive and use rsync to sync files on Debian 7

I am running latest [easyazon-link asin=”1783283114″ locale=”us”] GNU Debian Linux 7.x [/easyazon-link] and I would like to rsync my important files from local drive to [easyazon-link asin=”B004OR1JF0″ locale=”us”] Amazon S3 [/easyazon-link] storage with rsync tool.

You will need:

  • Amazon AWS account
  • Amazon Access Key and Secret Key
  • Amazon S3 bucket name
  • s3fs utility installed and configured on the [easyazon-link asin=”111821854X” locale=”us”] Debian [/easyazon-link]server

First we need to install few requirements on the [easyazon-link asin=”1783283114″ locale=”us”] Debian [/easyazon-link] server:

su
apt-get install build-essential libfuse-dev fuse-utils libcurl4-openssl-dev libxml2-dev mime-support
apt-get update

Second, now you can download the latest build of s3fs from the project link  https://code.google.com/p/s3fs/ , untar and compile.

su
cd Downloads
wget https://s3fs.googlecode.com/files/s3fs-1.74.tar.gz
tar xzvf s3fs-1.74.tar.gz
cd s3fs*
./configure
make
make install

By now you should have installed the s3fs utility on the system. You can try to run it by typing s3fs you should see message that it is not configured yet.

If installed and not configured – you will see this message

s3fs: missing BUCKET argument
Usage: s3fs BUCKET:[PATH] MOUNTPOINT [OPTION]...

Now we will create a local directory that will be mounted to [easyazon-link asin=”0596515812″ locale=”us”] Amazon S3 [/easyazon-link] bucket

su
mkdir /mnt/s3
chown yourusername:yourusername /mnt/s3

Next step is create a password file with the Amazon keys:

su
echo "ACCESS_KEY_ID:SECRET_ACCESS_KEY" > /etc/passwd-s3fs
chmod 0600 /etc/passwd-s3fs

This is covering the basic setup, installation and configuration s3fs on the [easyazon-link asin=”1593270690″ locale=”us”] Debian [/easyazon-link] server. Lets try it now.

su
s3fs bucket-name /mnt/s3
cd /mnt/s3
ls -l

If you have all setup correctly, you should be able to connect to S3 storage bucket, mount it as the /mnt/s3 and  be able to navigate to the directory and list files in the S3 bucket. If all works, you can now write small script  and create a cron job so all is done automatically and you will receive email upon rsync completion.

############# SAMPLE SCRIPT ##################
#!/bin/bash
# Script : Mount S3 bucket as a local drive /mnt/s3 and rsync files

bucket="mys3-bucket"
destination="/mnt/s3"
backup="/data/MYFILES"

# mounting S3 bucket for backup as a local drive
s3fs $bucket $destination

# rsync local files to S3 bucket
rsync -avz $backup $destination

# Once rsync finished - send email and unmount the S3 bucket from local system
ls -lh $destination | mail -s "Rsync completed" sysadmin@system.com
umount $destination
######### END OF SAMPLE SCRIPT #######

Modify the variables according your needs, save it as a s3fs.sh file, change permission and you can add it to crontab so it runs on daily basis.

su
chmod 700 s3fs.sh
----------------------
su
crontab -e
0 0 * * * /path/to/s3fs.sh # this runs rsync to s3 every day at midnight

By the way there was released a new book on Amazon covering best Administration practices for Debian 7 🙂

[easyazon-image align=”none” asin=”1783283114″ locale=”us” height=”160″ src=”http://ecx.images-amazon.com/images/I/51KgrYCMzoL._SL160_.jpg” width=”130″]

Leave a Reply