Creating your own backups with Tarsnap

in SecurityStabilityTools

Managing backups yourself

If you choose to create your own backups, we recommend using Tarsnap as an offsite backup provider.
This very robust third party backup service is well known for it’s excellent security and their low prices.

Tarsnap encrypts your data and then stores it on Amazon S3 storage platform. S3 is extremely reliable, as the data is duplicated to at least three different data centers. Nobody but you has access to the data, as it is encrypted using the best open standards (not even the NSA can get to it).

Tarsnap charges $0.25 per GB per month (for storage and bandwidth), but only transfers and stores changed data (incremental updates).

Backup your database

This requires some free space on your Hypernode. For a typical 1GB database, the compressed dump file takes 20MB (the index data is not copied). You can use df -h ~ to verify the available amount of available disk space.

If you want to store your database backup offsite (IE: with Tarsnap), we recommend to create the database backup before copying the file content to an offsite backup location. This way you are ensured your database backup is saved with the file backup on an offsite backup location.

Create a database backup

Add the following lines your crontab. It will not lock your database, so it can be safely run at busy times. This example runs nightly.

MAILTO=yourmailadress@here
PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin"

0 2 * * * mkdir -p ~/backup; flock -n ~/.mysqldump chronic n98-magerun db:dump --root-dir=~/public --compression=gz --add-time=suffix --no-interaction --strip @stripped ~/backup

0 5 * * * chronic find /data/web/backup/ -type f -mtime +7 -delete

NB: If you are using Magento 2, replace /data/web/public with /data/web/magento2 and use the n98-magerun2 utility for Magento 2 instead of the the n98-magerun utility.

Configuring Tarsnap

Step One – Create a Tarsnap account

Register for an account at Tarsnap and upload some prepaid funds. For an average Magento shop, €100,- will pay for a year of backup service.
You will receive a notification from Tarsnap when you are almost running out of credits.

Step Two – Generate a private key file

You will need to create a private key to encrypt your data. This is necessary to create and decrypt your backups and is needed when creating or restoring backups from Tarsnap.

On your Hypernode, run the following command to generate a key file:

tarsnap-keygen --keyfile ~/tarsnap.key --user <email> --machine <mysite>

You should copy this tarsnap.key file to a safe location, as your backup is inaccessible without it!
A local USB stick in a safe at home or in the office is a good idea.

Step Three – Configure Tarsnap

Create a Tarsnap configuration file by running the following script:

cat > ~/.tarsnaprc <<EOM
cachedir ~/.tarsnapcache
keyfile ~/tarsnap.key
print-stats
humanize-numbers

exclude .git
exclude *.log
exclude *.zip
exclude *.lock
exclude *.gz
exclude .tarsnapcache
exclude data/web/*/var/*/*
exclude data/web/*/media/catalog/product/cache/*
exclude data/web/*/media/js/*
exclude data/web/*/media/css/*.css
EOM

And second, create a directory where Tarsnap can temporary store and cache files, before sending them to a Tarsnap server:

mkdir -p ~/.tarsnapcache

Step Four – Test the setup

Ensure yourself the backup works as expected. Create a backup and if possible, restore the backup as well to double check the created backup is useful when you need it.

The initial upload could take up to several hours. To create a first backup, run the following command:

# Create a backup archive called "backup-WEEKDAY"
tarsnap -c -f backup-`date "+%A"` -v  ~

Step 5 – Configure a cronjob

Implement this as a daily cron. To make optimal use of the Tarsnap efficiency algorithm, you should NOT use compression on your local database dump and you should dump always to the same file.

Lets put this in a tiny script first. Save the following script as /data/web/backup/makebackup.sh:

cat > ~/backup/makebackup.sh <<EOM
#!/bin/sh
TODAY=\$(date "+%A");
flock -n ~/.mysqldump chronic n98-magerun db:dump --root-dir=~/public --no-interaction --strip @stripped ~/backup/mysql-latest.sql;
flock -n ~/.tarsnap.lock tarsnap -d -f backup-\$TODAY 2>/dev/null;
flock -n ~/.tarsnap.lock chronic tarsnap -c -f backup-\$TODAY ~
EOM

NB: If you are using Magento 2, use the n98-magerun2 utility for Magento 2 instead of the the n98-magerun utility.

Next, make the script executable:

chmod 755 ~/backup/*.sh

If you want to save some space on your Hypernode, you can delete the created backup by adding rm ~/backup/mysql-latest.sql to your backup script. This implicates there is no database backup present for recovery, so if you need your backup you’ve got to retrieve it from Tarsnap first in case of emergency.

Next, add a cronjob to run the backup script nightly:

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=yourmailadress@here
0 2 * * * flock -n ~/.makebackup.lock ~/backup/makebackup.sh

This will create an archive for every day of the week. It will overwrite old archives. You will only pay for the unique, compressed amount of data. For an average shop, this is 1/3 of the size.

Restore your webshop from Tarsnap

Note: restoring will use the same directory structure.

For example, when you restore from the backup with path “/data/web/public” (in case of Magento 1), the “data/web/public” structure will be created in the current working directory. So do a cd / first, to overwrite the old files in the right location.

Restoring your content from a Tarsnap backup

  • List the available backup archives from Tarsnap:
    tarsnap --list-archives --verbose
    
  • List the contents of a specific archive:
    tarsnap -t -f <archive>
    
  • Extract a file from Tarsnap:
    tarsnap -x -f <archive> <file or files>
    tarsnap -x -f backup-Monday data/web/site.com/Nieuwsbrief.png
    

Restoring your database from a Tarsnap backup

  • Retrieve the backup from Tarsnap (if you remove the dump file after pushing it to Tarsnap)
    cd / 
    tarsnap -x -f backup-Monday data/web/backup/mysql-latest.sql
    
  • Then, restore a database dump, use n98-magerun (For Magento 2 n98-magerun2), to import the database dump:
    n98-magerun db:import --root-dir=~/public ~/backup/mysql-latest.sql
    

0