Complete backup solution implementation

The following is the complete backup solution implemented using combination of amanda and rsync scripts. This is the documentation for the backup process.

Backups are being handled with primary backup server running amanda server package version 3.1.2-1.rhel5.i386 and clients running amanda client packages. Rsync backup script is used for synchronizing amanda configuration files and amanda vtapes (files containing amanda backups) to the remote backup server.

The two backup servers are redbck01 and redbck02. Both are hp proliant DL360 running centos 5.5 with 1.8 Tb of storage and 6 GB memory. redbck01 is primary backup server and redbck02 is the secondary offisite backup server.

Amanda server configuration:
redbck01 is configured as amanda server which runs the configuration 'DailySet1' everynight for backing up the Disk List entries or clients as specified in the disk list file. Amanda configuration files for 'DailySet1' are located in /etc/amanda/DailySet1/ . amanda uses amanda.conf and disklist files located in config directory while running the backup configuration.
The following are some of the lines in /etc/amanda/DailySet1/amanda.conf which define details of backup configuration:
mailto "root" # emails the amanda report after the amanda backup runs_
tpchanger "chg-disk" # tape changer for virtual tapes
tapedev "file://space/vtapes/DailySet1/slots" # the tape device (where the virtual tapes are defined and data is stored)_
tapetype "HARDDISK" # use hard disk instead of tapes (vtape config)
dumpcycle "4 weeks" # the number of days in dump cycle before starting to reuse the tapes
runspercycle 20 # the number of amdump runs in dumpcycle days
tapecycle 25 tapes # the number of tapes in rotation
logdir "/etc/amanda/DailySet1/log" # log directory
define tapetype HARDDISK {
lenght 50000 mbytes # the maximum capacity of each slot is 50 GB * 25 slots = 1.25 TB max
}
auth "bsdtcp" # use bsdtcp for authentication between amanda server and clients
define dumptype comp-user-tar {
user-tar # we're using comp-user-tar in our disk list entries as the dump type as it uses fast compression
compress client fast } Amanda determines the backup levels by its own algorithm which guarantees full backups of all disklist entries in a dumpcycle and incremental backups according to the necessity.
The following are the disklist entries from the /etc/amanda/DailySet1/disklist file which defines the clients and directories being backup up along with dumptype:
xenora02 / comp-user-tar
clxbld01 /etc comp-user-tar
clxbld01 /root comp-user-tar
clxbld01 /home comp-user-tar
clxbld01 /usr/svn comp-user-tar
clxbld01/usr/local comp-user-tar

Amanda server uses /var/lib/amanda/.amandahosts file to allow the backup clients to connect back to the server when doing restores. Specify fully qualified domain names of all clients as follows:

xenora02 root amindexd amidxtaped

To check and verify the amanda configuration is working without any errors: On redbkc01, run the amcheck tool as amandabackup user:
-sh-3.2$ amcheck DailySet1
To run the backup manually on redbck01, as amandabackup, run amdump to start the DailySet1 backup.
-sh-3.2$ amdump DailySet1
after amdump finishes it will email the report to the specified addresses in the config file.
To find out what has been backed up we can use amadmin tool with find argument for a quich summary of entire backups.
-sh-3.2$ amadmin DailySet1 find
amdump will run automatically every night. Crontab entries for the amandabackup user are:
0 16 * * 1-5 /usr/sbin/amcheck -m DailySet1 # every week day at 4 pm run amcheck to verify the configuration is working
5 21 * * 1-6 /usr/sbin/amdump DailySet1 # every mon-sat at 5 past mid night run the amdump on DailySet1 config
amdump maximum runtime at the current configuration is approximately: 4 hours
Synchronizing redbck01 and redbck02:
Our main approach for backups is that in case of client failure we must be able to recover the data as early as possible without any single point of failure. So for example a client is dead and data has to be restored to the newly created client we can get the data from either redbck01 or redbck02. But as the redbck01 is the primary backup server and is located inside the network it is quicker to recover from it. If in case of a disaster, redbck01 is lost then we can get the data from offsite backup server redbck02 by configuring amanda client accordingly.
Therefore synchronising the data between redbck01 and redbck02 is done by using rsync. Amanda server package is installed in redbck02, the data needed from redbck01 is '/etc/amanda/' which holds amanda configuration files and '/space/vtapes' which holds the backups from clients (Disk list entries). A rsync backup script 'redbck02rsync.sh' is being used for syncing the data from redbck01 to redbck02 every night.
Crontab entries for user root on redbck01 which runs the rsync script every night are:
#0 0 * * 1-6 /root/redbck02rsync.sh -s /etc/amanda -d redbck02:/etc#0 1 * * 1-6 /root/redbck02rsync.sh -s /space/vtapes -d redbck02:/space
Amanda Client Configuration:
Amanda clients use /var/lib/amanda/.amandahosts file for allowing backup server to connect to the amanda client. So the file must have the following entry of backup server:
redbck01.
Amanda client configuration file is /etc/amanda/amanda-client.conf which is used for connecting to the server for restoring data. It contains details of configuration name, index server (either redbck01 or redbck02 depending on situation), tape_server and authentication protocol information. The following are some of the lines of /etc/amanda/amanda-client.conf which defines the client configuration file.
conf "DailySet1" # your config name
index_server "redbck01." # your amindexd server
tape_server "redbck01." # your amidxtaped serverauth "bsdtcp"
Recovery
To recover data to the client in the event of data loss or system failure, we can either use primary backup server redbck01 or offsite backup server redbck02. we must specify the configuration, index_server and tape_server details in /etc/amanda/amanda-client.conf file.
after the configuration, as root run amrecover to initiate the data recovery process.
[root@dlxmkt01 ~]# amrecover
AMRECOVER Version 3.1.2. Contacting server on redbck01.
...220 redbck01 AMANDA index server (3.1.2) ready.Setting restore date to today (2010-09-16)200
Working date set to 2010-09-16.200 Config set to DailySet1.501
Host dlxmkt01 is not in your disklist.Trying host dlxmkt01 ...
501 Host dlxmkt01 is not in your disklist.Trying host dlxmkt01 ...501 Host dlxmkt01 is not in your disklist.
Use the sethost command to choose a host to recover
amrecover>
The list of commands below will demonstrate a recovery of a set of different files and directories to the "/tmp/amanda/phooper/" directory. amrecover may not ask for sethost command if the system hostname is fully qualified domain name.
amrecover> listhost
200- List hosts for config DailySet1
201- xenora02.
201- clxbld01.
201- isobck01.
200 List hosts for config DailySet1
amrecover> sethost dlxmkt01.
200 Dump host set to dlxmkt01.
amrecover> listdisk
200- List of disk for host dlxmkt01.
201- /
200 List of disk for host dlxmkt01.
amrecover> setdisk /
200 Disk set to /.
amrecover> cd /
amrecover> ls
2010-09-16-00-05-03 var/
2010-09-16-00-05-03 usr/
2010-09-16-00-05-03 tmp/
2010-09-16-00-05-03 tdev15/
2010-09-16-00-05-03 sys/
2010-09-16-00-05-03 srv/
2010-09-16-00-05-03 selinux/
2010-09-16-00-05-03 sbin/
2010-09-16-00-05-03 root/
2010-09-16-00-05-03 proc/
2010-09-16-00-05-03 opt/
2010-09-16-00-05-03 mnt/
2010-09-16-00-05-03 media/
2010-09-16-00-05-03 lib64/
2010-09-16-00-05-03 lib/
2010-09-16-00-05-03 initrd/
2010-09-16-00-05-03 home/
2010-09-16-00-05-03 etc/
2010-09-16-00-05-03 dev/
2010-09-16-00-05-03 data/
2010-09-16-00-05-03 boot/
2010-09-16-00-05-03 bin/
2010-09-16-00-05-03 .
2010-09-14-00-05-03 init
2010-09-14-00-05-03 .autofsck
amrecover> lcd /
amrecover> add /tmp/amanda/phooper
Added dir /tmp/amanda/phooper/ at date 2010-09-16-00-05-03
amrecover> extract
Extracting files using tape drive changer on host redbck01.
The following tapes are needed: DailySet1-15
Extracting files using tape drive changer on host redbck01.
Load tape DailySet1-15 nowContinue [?/Y/n/s/d]? y
amrecover> quit
200 Good bye.
If redbck01 fails we can recover the data from redbck02 by running the rsync script 'redbck02rsync.sh' which is located in /root/ directory from redbck02