Linux Archiving: rsync, Tape, CD & DVD Media

How to archive data using rsync, DDS-3/4, CD-R/RW & DVD-R/RW drives under Linux

Introduction

This note is intended to help the X-Ray Facility (XRF) users archive (backup) their data both at home and at synchrotron using DDS-3/4, CD-R/RW, or DVD-R/RW drives under Linux operating system (o/s). Similar archiving under Win2000/WinXP platforms is fairly straight forward and therefore will not be covered. A copy of this Note will be posted in XRF Resources page shortly after receiving suggestions and corrections from the users. This note was written in May 10, 2003 and last updated August 19, 2004.

UNIX command: rsync

rsync is an incremental archiving UNIX/Linux utility. It has several options (secure copy, file compression, directory tree, and file comparison) allowing a user to copy files from a source to a destination incrementally over the network. The command is fairly easy to use and therefore will be shown with an example. It is best to write a script and run the script, say, once an hour; alternatively one can even include a loop so that the script is invoked automatically after several minutes of ‘sleep’ time.

The following script is executed in the destination (local) computer and when invoked the user logs-in to the target (remote) computer and copies the file to the local machine:

#!/bin/sh

#This script will copy data from a source (remote) to my destination (local)

rsync –avz –e ssh --exclude “*.imx_0” /data/d1/xtal1 /mnt/fire1/my_dir/xtal1

Explanation for the script is given below:

#!/bin/sh: invokes the appropriate shell

# This script : A line starting with a ‘#’ is a comment and not interpreted by the shell

rsync: the command itself

-a: archive mode (back-up)

-v: verbose mode

-z: compress the file while transferring

-e ssh: the remote shell will be a ‘secure shell’

--exclude: any files that need to be excluded; wild card permitted

/data/d1/xtal1: source directory for copying (remote; detector’s computer)

/mnt/fire1/my_dir/xtal1: destination directory for storing (local; your computer)

#!/bin/sh

# This script will copy data from a source to my laptop; sleep for 1 minute

# and do it again

while true

do

rsync –avz –e ssh --exclude “*.imx_0” /data/d1/xtal1 /mnt/fire1/my_dir/xtal1

sleep 3600

done

rsync session

A typical session is given below:

#!/bin/sh

# This script is called getdata.sh when executed in tampa.sb.fsu.edu will copy # data from spruce.sb.fsu.edu (marCCD machine) to my laptop tampa.sb.fsu.edu # (local machine)

rsync –avz –e ssh /home/marccd/d2/ccd-data /home/soma/data

# Target directory is spruce:/home/marccd/d2/ccd-data

# Destination directory is tampa:/home/soma/data

soma@tampa[2:28pm]~>./getdata.sh

soma@spruce's password:***********

receiving file list ... done

./

ccd_xtal_001.imx

wrote 32 bytes read 467 bytes 76.77 bytes/sec

total size is 15183 speedup is 30.43

wait for 1 minute

soma@spruce's password:***********

receiving file list ... done

./

soma_ccd.dat

wrote 32 bytes read 4939 bytes 301.27 bytes/sec

total size is 27171 speedup is 5.47

SCSI Tape Drive

Archiving data sets both for transport and long-term storage is very important to all the XRF users, since it is the only way to have access to older data. In the past DDS (Digital Data Storage), DDS-3 and DDS-4 magnetic tapes have been the media of choice for archiving. The reasons are the fact that large amounts of data could be archived in modestly priced tapes and can compactly be stored for several years without degradation. However, tape media are still susceptible to strong magnetic and electrical fields, deteriorate over time (7-10 years) and finally don’t provide random-access to the stored data. CD-Rs and recently DVD-Rs have alleviated the problem by providing a stable medium, universal availability, random-access to data and a compact form. One drawback with DVD-R is the cost/MB is more than that of DDS-4 media and cost adds up if the storage needs are in hundreds of GB. Here we will cover the procedure for archiving data using the tape media since it is still widely used. Later we will cover the archiving using CD-R & DVD-R.

Figure below shows an external DDS-3 SCSI Tape Drive from Seagate Technology (Certance).

External SCSI DDS-3/4 drive from Certance (formerly Seagate Technology)

raccoon.sb.fsu.edu (128.186.103.108; RH Linux 7.1)

Raccoon.sb.fsu.edu (located in KLB 410A) is a 933 MHz Linux machine dedicated both for data processing and archiving. It features an external SCSI DDS-3 tape drive, one internal IDE DVD-ROM drive and one internal SCSI CD-R/-RW drive. DDS-3 tape drives are backward compatible in reading and writing data to DDS (sometimes referred to as DAT), DDS-2 in addition to DDS-3 media. Depending upon the level of compression feasible for the data, DDS-3 media can accommodate uncompressed capacity of 12 GB and compressed capacity of 24 GB.

neptune.sb.fsu.edu (128.186.103.106; RH Linux 7.3)

Neptune.sb.fsu.edu (located in KLB 410A) is a 933 MHz Linux machine dedicated both for archiving and data processing. It features an internal SCSI DDS-4 tape drive, one internal SCSI CD-ROM drive and one external FireWire hard drive. DDS-4 tape drives are backward compatible in reading and writing data to DDS (sometimes referred to as DAT), DDS-2 and DDS-3 in addition to DDS-4 media. Depending upon the level of compression feasible for a data DDS-3 media can accommodate uncompressed capacity of 20 GB and compressed capacity of 40 GB.

The picture below shows the various DDS media logos:

DDS Media Logos

All XRF users will be able to read from and write to the tape drive. Since it is the only SCSI tape drive it has the device allocation of /dev/st0 and its status can be checked using the mt command and data archived using tar command. Typical commands that are used during archiving are listed below with a brief explanation:

mt –f /dev/st0 status |Check status of the first SCSI tape drive /dev/st0

tar cvf /dev/st0 my_dir |Write contents of my_dir to tape

tar tvf /dev/st0 |Look at or list contents of previously archived tape

tar -xvf /dev/st0 parent_dir/my_dir |Extract contents of my_dir

mt= magnetic tape; tar= tape archive; -c= create; -v= verbose;-x= extract; st= scsi tape; -t= list; -f= file(system).

Reading DDS-3/4 media

[root@raccoon /]# mt -f /dev/st0 status |Check status of tape drive

SCSI 2 tape drive: File number=0, block number=0, partition=0.

Tape block size 512 bytes. Density code 0x25 (DDS-3).

General status bits on (45010000):

BOT WR_PROT ONLINE IM_REP_EN |Beginning of tape; write-protected tape

[root@raccoon DDS]# tar tvf /dev/st0 |List contents of tape

drwxr-xr-x soma/users 0 2001-09-19 15:04:14 TapeDDS3-024/

drwxr-xr-x soma/users 0 2001-09-18 13:44:17 TapeDDS3-024/4mmtp116/

drwxr-xr-x soma/users 0 2001-09-18 12:39:05 TapeDDS3-024/4mmtp116/AdamekF1

[root@raccoon DDS]# tar-xvf /dev/st0 |Extract contents of the tape media

fire2/soma/

fire2/soma/all.list

fire2/soma/Blaber_Fgf/

fire2/soma/Blaber_Fgf/fgf_2/

fire2/soma/Blaber_Fgf/fgf_2/snap.runlist

fire2/soma/Blaber_Fgf/fgf_2/snap_0_001.img

[root@raccoon DDS]# tar -xvf /dev/st0 fire2/soma/Blaber_Fgf |Extract contents of only Blaber_Fgf directory and NOT all

fire2/soma/Blaber_Fgf/

fire2/soma/Blaber_Fgf/fgf_2/

fire2/soma/Blaber_Fgf/fgf_2/snap.runlist

fire2/soma/Blaber_Fgf/fgf_2/snap_0_001.img

[root@raccoon DDS]# mt -f /dev/st0 offline |Eject the tape

Writing DDS-3/4 media

[root@raccoon /]# mt -f /dev/st0 status |Check status of tape drive

SCSI 2 tape drive: File number=0, block number=0, partition=0.

Tape block size 512 bytes. Density code 0x25 (DDS-3).

General status bits on (45010000):

BOT ONLINE IM_REP_EN |Beginning of tape; ready to write

[soma@spruce ~]$ tar -cvvf /dev/st0 tmp/ |Write contents of tmp directory

drwxrwxr-x soma/marccd 0 2003-04-30 16:41:21 tmp/

-rwxr-xr-x soma/marccd 103756 2000-12-22 23:53:35 tmp/font.exe

-rw------- soma/marccd 7223800 2001-05-16 11:36:03 tmp/abc10003.osc

-rw-rw-r-- soma/marccd 1737 2001-08-07 15:10:20 tmp/uniqueify.log

[soma@spruce ~]$ tar -cvvf /dev/st0 erase |Erase contents of tape. Careful!!!

[soma@spruce ~]$ tar -cvf /dev/st0 my_dir | & tee dds4-listing & |Simultaneously (