Linux Archiving: rsync, Tape, CD & DVD Media

How to archive data using rsync, DDS-3/4, CD-R/RW & DVD-R/RW drives under Linux

Introduction

This note is intended to help the X-Ray Facility (XRF) users archive (backup) their data both at home and at synchrotron using DDS-3/4, CD-R/RW, or DVD-R/RW drives under Linux operating system (o/s). Similar archiving under Win2000/WinXP platforms is fairly straight forward and therefore will not be covered. A copy of this Note will be posted in XRF Resources page shortly after receiving suggestions and corrections from the users. This note was written in May 10, 2003 and last updated August 19, 2004.

UNIX command: rsync

rsync is an incremental archiving UNIX/Linux utility. It has several options (secure copy, file compression, directory tree, and file comparison) allowing a user to copy files from a source to a destination incrementally over the network. The command is fairly easy to use and therefore will be shown with an example. It is best to write a script and run the script, say, once an hour; alternatively one can even include a loop so that the script is invoked automatically after several minutes of ‘sleep’ time.

The following script is executed in the destination (local) computer and when invoked the user logs-in to the target (remote) computer and copies the file to the local machine:

#!/bin/sh

#This script will copy data from a source (remote) to my destination (local)

rsync –avz –e ssh --exclude “*.imx_0” /data/d1/xtal1 /mnt/fire1/my_dir/xtal1

Explanation for the script is given below:

#!/bin/sh: invokes the appropriate shell

# This script : A line starting with a ‘#’ is a comment and not interpreted by the shell

rsync: the command itself

-a: archive mode (back-up)

-v: verbose mode

-z: compress the file while transferring

-e ssh: the remote shell will be a ‘secure shell’

--exclude: any files that need to be excluded; wild card permitted

/data/d1/xtal1: source directory for copying (remote; detector’s computer)

/mnt/fire1/my_dir/xtal1: destination directory for storing (local; your computer)

#!/bin/sh

# This script will copy data from a source to my laptop; sleep for 1 minute

# and do it again

while true

do

rsync –avz –e ssh --exclude “*.imx_0” /data/d1/xtal1 /mnt/fire1/my_dir/xtal1

sleep 3600

done

rsync session

A typical session is given below:

#!/bin/sh

# This script is called getdata.sh when executed in tampa.sb.fsu.edu will copy # data from spruce.sb.fsu.edu (marCCD machine) to my laptop tampa.sb.fsu.edu # (local machine)

rsync –avz –e ssh /home/marccd/d2/ccd-data /home/soma/data

# Target directory is spruce:/home/marccd/d2/ccd-data

# Destination directory is tampa:/home/soma/data

soma@tampa[2:28pm]~>./getdata.sh

soma@spruce's password:***********

receiving file list ... done

./

ccd_xtal_001.imx

wrote 32 bytes read 467 bytes 76.77 bytes/sec

total size is 15183 speedup is 30.43

wait for 1 minute

soma@spruce's password:***********

receiving file list ... done

./

soma_ccd.dat

wrote 32 bytes read 4939 bytes 301.27 bytes/sec

total size is 27171 speedup is 5.47

SCSI Tape Drive

Archiving data sets both for transport and long-term storage is very important to all the XRF users, since it is the only way to have access to older data. In the past DDS (Digital Data Storage), DDS-3 and DDS-4 magnetic tapes have been the media of choice for archiving. The reasons are the fact that large amounts of data could be archived in modestly priced tapes and can compactly be stored for several years without degradation. However, tape media are still susceptible to strong magnetic and electrical fields, deteriorate over time (7-10 years) and finally don’t provide random-access to the stored data. CD-Rs and recently DVD-Rs have alleviated the problem by providing a stable medium, universal availability, random-access to data and a compact form. One drawback with DVD-R is the cost/MB is more than that of DDS-4 media and cost adds up if the storage needs are in hundreds of GB. Here we will cover the procedure for archiving data using the tape media since it is still widely used. Later we will cover the archiving using CD-R & DVD-R.

Figure below shows an external DDS-3 SCSI Tape Drive from Seagate Technology (Certance).

External SCSI DDS-3/4 drive from Certance (formerly Seagate Technology)

raccoon.sb.fsu.edu (128.186.103.108; RH Linux 7.1)

Raccoon.sb.fsu.edu (located in KLB 410A) is a 933 MHz Linux machine dedicated both for data processing and archiving. It features an external SCSI DDS-3 tape drive, one internal IDE DVD-ROM drive and one internal SCSI CD-R/-RW drive. DDS-3 tape drives are backward compatible in reading and writing data to DDS (sometimes referred to as DAT), DDS-2 in addition to DDS-3 media. Depending upon the level of compression feasible for the data, DDS-3 media can accommodate uncompressed capacity of 12 GB and compressed capacity of 24 GB.

neptune.sb.fsu.edu (128.186.103.106; RH Linux 7.3)

Neptune.sb.fsu.edu (located in KLB 410A) is a 933 MHz Linux machine dedicated both for archiving and data processing. It features an internal SCSI DDS-4 tape drive, one internal SCSI CD-ROM drive and one external FireWire hard drive. DDS-4 tape drives are backward compatible in reading and writing data to DDS (sometimes referred to as DAT), DDS-2 and DDS-3 in addition to DDS-4 media. Depending upon the level of compression feasible for a data DDS-3 media can accommodate uncompressed capacity of 20 GB and compressed capacity of 40 GB.

The picture below shows the various DDS media logos:

DDS Media Logos

All XRF users will be able to read from and write to the tape drive. Since it is the only SCSI tape drive it has the device allocation of /dev/st0 and its status can be checked using the mt command and data archived using tar command. Typical commands that are used during archiving are listed below with a brief explanation:

mt –f /dev/st0 status |Check status of the first SCSI tape drive /dev/st0

tar cvf /dev/st0 my_dir |Write contents of my_dir to tape

tar tvf /dev/st0 |Look at or list contents of previously archived tape

tar -xvf /dev/st0 parent_dir/my_dir |Extract contents of my_dir

mt= magnetic tape; tar= tape archive; -c= create; -v= verbose;-x= extract; st= scsi tape; -t= list; -f= file(system).

Reading DDS-3/4 media

[root@raccoon /]# mt -f /dev/st0 status |Check status of tape drive

SCSI 2 tape drive: File number=0, block number=0, partition=0.

Tape block size 512 bytes. Density code 0x25 (DDS-3).

General status bits on (45010000):

BOT WR_PROT ONLINE IM_REP_EN |Beginning of tape; write-protected tape

[root@raccoon DDS]# tar tvf /dev/st0 |List contents of tape

drwxr-xr-x soma/users 0 2001-09-19 15:04:14 TapeDDS3-024/

drwxr-xr-x soma/users 0 2001-09-18 13:44:17 TapeDDS3-024/4mmtp116/

drwxr-xr-x soma/users 0 2001-09-18 12:39:05 TapeDDS3-024/4mmtp116/AdamekF1

[root@raccoon DDS]# tar-xvf /dev/st0 |Extract contents of the tape media

fire2/soma/

fire2/soma/all.list

fire2/soma/Blaber_Fgf/

fire2/soma/Blaber_Fgf/fgf_2/

fire2/soma/Blaber_Fgf/fgf_2/snap.runlist

fire2/soma/Blaber_Fgf/fgf_2/snap_0_001.img

[root@raccoon DDS]# tar -xvf /dev/st0 fire2/soma/Blaber_Fgf |Extract contents of only Blaber_Fgf directory and NOT all

fire2/soma/Blaber_Fgf/

fire2/soma/Blaber_Fgf/fgf_2/

fire2/soma/Blaber_Fgf/fgf_2/snap.runlist

fire2/soma/Blaber_Fgf/fgf_2/snap_0_001.img

[root@raccoon DDS]# mt -f /dev/st0 offline |Eject the tape

Writing DDS-3/4 media

[root@raccoon /]# mt -f /dev/st0 status |Check status of tape drive

SCSI 2 tape drive: File number=0, block number=0, partition=0.

Tape block size 512 bytes. Density code 0x25 (DDS-3).

General status bits on (45010000):

BOT ONLINE IM_REP_EN |Beginning of tape; ready to write

[soma@spruce ~]$ tar -cvvf /dev/st0 tmp/ |Write contents of tmp directory

drwxrwxr-x soma/marccd 0 2003-04-30 16:41:21 tmp/

-rwxr-xr-x soma/marccd 103756 2000-12-22 23:53:35 tmp/font.exe

-rw------- soma/marccd 7223800 2001-05-16 11:36:03 tmp/abc10003.osc

-rw-rw-r-- soma/marccd 1737 2001-08-07 15:10:20 tmp/uniqueify.log

[soma@spruce ~]$ tar -cvvf /dev/st0 erase |Erase contents of tape. Careful!!!

[soma@spruce ~]$ tar -cvf /dev/st0 my_dir | & tee dds4-listing & |Simultaneously (tee) archive data from my_dir to a tape and write the filenames into another file (dds4-listing).

spruce.sb.fsu.edu (128.186.103.109; RedHat Linux 8.0)

Spruce.sb.fsu.edu the 650 MHz Linux machine located in KLB 410A that controls marCCD165 data collection has an internal DDS-4 tape drive. This SCSI DDS-4 tape drive can read and write to DDS2, DDS3 and DDS4 media. However, it is best to use DDS4 media, since it has a native/compressed capacity of 20/40 GB. IP/CCD data will compress anywhere between 10 to 60% depending upon the level of detail in the original image. The same tar and mt commands shown above can be used to archive data using this tape drive. Please read the mt and tar man pages or check with Soma, or Mike for further help.

General Information & Trouble Shooting of Tape Drive/Media

Data Archiving @ APS

Archive Creation Time: Sun Apr 13 21:36:47 CDT 2003 @ 14-BM-C Beamline

bm14cbu.cars.aps.anl.gov | 164.54.161.114

I inserted a DDS-3 tape into HP Model C1537A DDS-3 Tape drive attached to a Linux workstation running RedHat Linux Version 7.3 Valhalla 2.4.8-27.7.x and archived the data using regular tar command:

[userbmc@bm14cbu]$ tar cvvf /dev/st0 blaber_0304/

Data Retrieval @ FSU

Archive Extraction Time: Thu Apr 17 13:31:14 EDT 2003 @ XRF (spruce.chem.fsu.edu)

Later at home when the same DDS3 tape was read using the DDS4 drive in spruce.sb.fsu.edu I received an error. More details about the error and how the problem was solved are explained below.

Error Management and Trouble-shooting @ FSU

Note Creation Time: Thu Apr 17 13:31:14 EDT 2003

Linux spruce.chem.fsu.edu 2.2.16-22-marccd #1 Wed Oct 25 15:23:06 CDT 2000 i686 /home/soma/tapeerror.txt

[soma@spruce ~]$ tar tvf /dev/st0 |DDS-3 tape created @ APS read @ FSU

tar: /dev/st0: Cannot read: Input/output error

tar: At beginning of tape, quitting now

tar: Error is not recoverable: exiting now

I then checked the status of the tape

[soma@spruce ~]$ mt -f /dev/st0 status |Set at default block size of 512; recognizes DDS-3

SCSI 2 tape drive:

File number=0, block number=0, partition=0.

Tape block size 512 bytes. Density code 0x25 (DDS-3).

Soft error count since last status=0

General status bits on (45010000):

BOT WR_PROT ONLINE IM_REP_EN

I then got a suggestion from the website: http://www.ls.eso.org/lasilla/sciops/2p2/E2p2M/WFI/tapes/TapeProb.html

to set the block size to be variable (set to zero) while reading under Linux

(I used 'reading HP Linux DDS3 tape' under Google search)

[soma@spruce ~]$ mt -f /dev/st0 setblk 0 |Set to variable block size

[soma@spruce ~]$ mt -f /dev/st0 status |Check the status again to confirm block size

SCSI 2 tape drive:

File number=0, block number=0, partition=0.

Tape block size 0 bytes. Density code 0x25 (DDS-3).

Soft error count since last status=0

General status bits on (45010000):

BOT WR_PROT ONLINE IM_REP_EN

Now I was able to read the tape (here, only listing is shown, but I could do tar xvf as well)

[soma@spruce ~]$ tar tvf /dev/st0 |Get a listing of files in archive

drwxrwxrwx userbmc/users 0 2003-04-13 15:36:14 fgf_1/

-rw-r--r-- userbmc/users 57 2003-04-13 11:57:02 fgf_1/snap.runlist

-rw-r--r-- userbmc/users 10617344 2003-04-13 11:26:18 fgf_1/snap_0_009.img

-rw-r--r-- userbmc/users 10617344 2003-04-13 11:29:56 fgf_1/snap_0_010.img

--------------------------------------------------

General Tape Media Trouble-shooting

Now if the blk size is set to zero (= set to variable size)...

[soma@spruce ~]$ mt -f /dev/st0 status

SCSI 2 tape drive:

File number=0, block number=0, partition=0.

Tape block size 0 bytes. Density code 0x26 (unknown to this mt) |a DDS-4 tape

Soft error count since last status=0

General status bits on (45010000):

BOT WR_PROT ONLINE IM_REP_EN

... and if you try to read a tape written with default blk size of 512 you may encounter the following problem shown below.

[soma@spruce ~]$ tar tvf /dev/st0

tar: Record size = 1 blocks

drwxrwxr-x marccd/marccd 0 2001-09-28 11:14:49 d2/

drwxr-xr-x marccd/marccd 0 2000-10-27 01:18:30 d2/lost+found/

drwxrwxr-x soma/marccd 0 2001-09-28 10:55:57 d2/APS2001/

drwxr-xr-x soma/marccd 0 2001-09-27 02:38:38 d2/APS2001/MSP1/

-rw------- soma/marccd 8964 2001-09-27 05:54:58 d2/APS2001/MSP1/.listing

-rw-r--r-- soma/marccd 57 2001-09-27 07:37:00 d2/APS2001/MSP1/snap.runlist

-rw-r--r-- soma/marccd 10617344 2001-09-27 04:18:00 d2/APS2001/MSP1/snap_0_203.img

...the tape reading virtually stops at this stage.

In that event, reset the blk size back to 512 and then you can read it.

[soma@spruce ~]$ mt -f /dev/st0 setblk 512 | Set block size back to the default value of 512

[soma@spruce ~]$ mt -f /dev/st0 status |Check the status again

SCSI 2 tape drive:

File number=0, block number=0, partition=0

Tape block size 512 bytes. Density code 0x26 (unknown to this mt)

Soft error count since last status=0

General status bits on (45010000):

BOT WR_PROT ONLINE IM_REP_EN

[soma@spruce ~]$ tar tvf /dev/st0 |Now you can read the tape properly

drwxrwxr-x marccd/marccd 0 2001-09-28 11:14:49 d2/

drwxrwxr-x soma/marccd 0 2001-09-28 10:55:57 d2/APS2001/

drwxr-xr-x soma/marccd 0 2001-09-27 02:38:38 d2/APS2001/MSP1/

-rw------- soma/marccd 8964 2001-09-27 05:54:58 d2/APS2001/MSP1/.listing

-rw-r--r-- soma/marccd 57 2001-09-27 07:37:00 d2/APS2001/MSP1/snap.runlist

Reading Media Created in Silicon Graphics (SGI) in Linux o/s

Mon Apr 21 14:51:25 EDT 2003

Linux spruce.sb.fsu.edu 2.2.16-22-marccd #1 Wed Oct 25 15:23:06 CDT 2000 i686

I tried to read an old DDS tape created in thr.sb.fsu.edu (Silicon Graphics Irix 5.3; IP20 MIPS) using the DDS4 internal tape-drive in spruce.sb.fsu.edu. Earlier, I had had some problems reading the tapes written using SGI. So first I got some information about the tape drive from SGI (the machine that wrote the tape).

thr.sb.fsu.edu----------------------------SGI Irix 5.3---------------

I logged into thr.sb.fsu.edu, with no tape inside tape drive,

root@thr.sb.fsu.edu> mt -f /dev/rmt/tps0d3 status

Controller: SCSI

Device: ARCHIVE: Python 25601-XXX2.63

Status: 0x202

Drive Type: DAT

Media: Not Ready

root@thr.sb.fsu.edu> mt -f /dev/rmt/tps0d3 blksize

Recommended tape I/O size 262144 bytes (512 512-byte blocks)

Minimum block size: 1 byte(s)

Maximum block size: 16777215 bytes

Current block size: Variable.

Then with DDS tape number 43 (DDS MRS 90M Sony) created under SGI o/s inside the tape drive,

root@thr.sb.fsu.edu> mt -f /dev/rmt/tps0d3 status

Controller: SCSI

Device: ARCHIVE: Python 25601-XXX2.63

Status: 0x20266

Drive Type: DAT

Media: Ready, write protected, at BOT

root@thr.sb.fsu.edu> mt -f /dev/rmt/tps0d3 blksize

Recommended tape I/O size 262144 bytes (512 512-byte blocks)

Minimum block size: 1 byte(s)

Maximum block size: 16777215 bytes

Current block size: 262144

Then I took the same tape and inserted inside DDS4 internal tape drive in spruce.sb.fsu.edu with Linux o/s

spruce.sb.fsu.edu-----------------------RH Linux 7.1---------------

Then I logged into spruce.sb.fsu.edu. I inserted the tape number 43 in DDS4 drive (/dev/st0). And first I tried with default block size of 512 (It should have worked since the default value of the media was 512? but didn't.) and I got an error message:

[soma@spruce ~]$ tar tvf /dev/st0

tar: /dev/st0: Cannot read: Input/output error

tar: At beginning of tape, quitting now

tar: Error is not recoverable: exiting now

I checked the default block size was 512

[soma@spruce ~]$ mt -f /dev/st0 status

SCSI 2 tape drive:

File number=0, block number=0, partition=0.

Tape block size 512 bytes. Density code 0x13 (DDS (61000 bpi)).

Soft error count since last status=0

General status bits on (45010000):

BOT WR_PROT ONLINE IM_REP_EN

Then I tried to change the blksize to 256

[soma@spruce ~]$ mt -f /dev/st0 setblk 256

[soma@spruce ~]$ mt -f /dev/st0 status

SCSI 2 tape drive:

File number=0, block number=0, partition=0.

Tape block size 256 bytes. Density code 0x13 (DDS (61000 bpi)).

Soft error count since last status=0

General status bits on (45010000):

BOT WR_PROT ONLINE IM_REP_EN

I still got the same error message

[soma@spruce ~]$ tar tvf /dev/st0

tar: /dev/st0: Cannot read: Input/output error

tar: At beginning of tape, quitting now

tar: Error is not recoverable: exiting now

Then I tried to change the blksize 0 (variable block size)

[soma@spruce ~]$ mt -f /dev/st0 setblk 0

[soma@spruce ~]$ mt -f /dev/st0 status

SCSI 2 tape drive:

File number=0, block number=0, partition=0.

Tape block size 0 bytes. Density code 0x13 (DDS (61000 bpi)).

Soft error count since last status=0

General status bits on (45010000):

BOT WR_PROT ONLINE IM_REP_EN

[soma@spruce ~]$ tar tvf /dev/st0

drwxr-xr-x raxis/user 0 1997-10-27 11:50:33 ./

drwxr-xr-x raxis/user 0 1997-09-26 16:06:16 Aav/

drwxr-xr-x raxis/user 0 1997-09-26 16:06:42 Aav/Figures/

-rw-r--r-- raxis/user 2809446 1997-08-26 10:53:39 Aav/Figures/gel1.tif

tar: Skipping to next header

-rw-r--r-- raxis/user 7786496 1997-10-09 22:49:21 Ak_Hi1/Data/akhi1026.osc

tar: Skipping to next header

-rw-r--r-- raxis/user 4995053 1997-10-14 00:36:36 Diao_rt10/insph5020.osc.gz

tar: Skipping to next header

It only worked partly and stopped.

-----

Finally I got a suggestion from the website about ‘dd’ (UNIX command)

(Google search: 'sgi dds block size')

http://wks.uts.ohio-state.edu/sun/faqs/HP-4mmDAT-info.txt

"dd if=/dev/st0 ibs=20k obs=512 bs=512k | tar tvf -"

And from the website related HP/Unix

http://hpux.tn.tudelft.nl/hppd/FAQ/9-13.html

9-13 How do I read an SGI-written tar format DDS tape?

dd if=/dev/rmt/0m ibs=512k obs=10k| tar -xvf -

-----

I tried with the default value mentioned in the website. It worked partly,

[soma@spruce ~]$ dd if=/dev/st0 ibs=20k obs=512 bs=512k | tar tvf -

drwxrwxr-x raxis/user 0 2000-03-25 13:41:05 HChenMar-005/

drwxrwxr-x raxis/user 0 2000-03-28 14:29:07 HChenMar-005/Tamo/

-rw-rw-r-- raxis/user 5190385 2000-03-24 18:43:38 HChenMar-005/Tamo/tamo001.osc.Z

tar: Skipping to next header

-rw-rw-r-- raxis/user 4998991 2000-03-24 19:56:24 HChenMar-005/Tamo/tamo009.osc.Z

tar: Skipping to next header

-rw-rw-r-- raxis/user 5182763 2000-03-25 14:17:19 HChenMar-005/Tamo/tamo2003.osc.Z

tar: Skipping to next header

-rw-rw-r-- raxis/user 5180609 2000-03-25 15:02:49 HChenMar-005/Tamo/tamo2008.osc.Z

tar: Skipping to next header

0+152 records in

76+0 records out

Then I tried with modified command (cf bs=256k to 512k). Then it worked nicely.

[soma@spruce ~]$ dd if=/dev/st0 ibs=20k obs=512 bs=256k | tar tvf -

drwxrwxr-x raxis/user 0 2000-03-25 13:41:05 HChenMar-005/

drwxrwxr-x raxis/user 0 2000-03-28 14:29:07 HChenMar-005/Tamo/

-rw-rw-r-- raxis/user 5190385 2000-03-24 18:43:38 HChenMar-005/Tamo/tamo001.osc.Z

-rw-rw-r-- raxis/user 5193270 2000-03-24 18:52:44 HChenMar-005/Tamo/tamo002.osc.Z

-rw-rw-r-- raxis/user 5173483 2000-03-24 19:01:50 HChenMar-005/Tamo/tamo003.osc.Z

-rw-rw-r-- raxis/user 5159591 2000-03-24 19:10:56 HChenMar-005/Tamo/tamo004.osc.Z

-rw-rw-r-- raxis/user 5128517 2000-03-24 19:20:01 HChenMar-005/Tamo/tamo005.osc.Z

113+0 records in

113+0 records out

Tapes with DDS (or DDS1) and 60M (60 meters) symbols were not readable (ejected out) by DDS4 drive.

SCSI CD-R/-RW Drive

Reading from CD-ROM drive/CD-RW Drive

CD-ROM drive and CD-RW/-R drives can be used both for direct data processing and copying the files from the drives to the disks (CD-R/-RW drives). The drives are first mounted as a file systems (remember, in Linux even the devices are filesystems, f/s) to allow the operating system (o/s) access to the existing data. Once the processing is completed the drives can unmounted. Figure below shows the internal HP CD-Writer 9500.

Internal SCSI CD-RW/R drive HP-CD Writer 9500 from HP

First, place a data CD in a CD-ROM or CD-RW drive and mount the device if not mounted. Process the data and unmount the device. A typical session might look like the following:

xray@raccoon[9:28am]~> df -k –l |Check the status local disks (-l option)

/dev/hda10 8815372 20 8367544 1% /d5

/dev/sda1 17639220 7234608 9508592 44% /d3

/dev/sda2 17639248 7208136 9535088 44% /d4

xray@raccoon[9:28am]~> mount /mnt/cdrom |Now mount CD-ROM device (f/s)

xray@raccoon[9:28am]~> df -k –l |Check local disks again; confirm mounting

/dev/hda10 8815372 20 8367544 1% /d5

/dev/sda1 17639220 7234608 9508592 44% /d3

/dev/sda2 17639248 7208136 9535088 44% /d4

/dev/hdc 633432 633432 0 100% /mnt/cdrom

xray@raccoon[9:28am]~> ls -lt /mnt/cdrom |Check contents of CD-ROM

-r-xr-xr-x 1 root root 18006000 Oct 26 1999 lysodata002.osc

-r-xr-xr-x 1 root root 18006000 Oct 26 1999 lysodata001.osc

-r-xr-xr-x 1 root root 9006000 Oct 26 1999 lysodata_screen002.osc

-r-xr-xr-x 1 root root 9006000 Oct 26 1999 lysodata_screen001.osc

xray@raccoon[9:28am]~> umount /mnt/cdrom |Unmount (umount) CD-ROM

Repeat the same procedure for CD-RW drive replacing /mnt/cdrom by /mnt/cdrom1. Remember, however, for successful mounting CD-R and CD-RW should have pre-existing data in them (blank CD-R and CD-RW will give an error: no medium found!).

Writing to CD-RW Drive:

Raccoon.sb.fsu.edu has a CD-RW drive that will allow archiving of your data into a CD-R or CD-RW medium. However, before archiving the data the user needs first to create an ISO 9660 image of their data and then write that image to the CD-R or CD-RW. While creating the ISO 9660 image of the data remember not to exceed ~ 640 MB of total space. If however, the user data is large, the data can be split into manageable segments, see § Splitting Large Directory section below to see how this can be accomplished. Once the ISO9660 image of appropriate size has been created, the command cdrecord can be used to archive (write) data to the CD-R or CD-RW. A typical session might look like the following:

xray@raccoon[4:49pm]~> cd /d3 |Go to parent directory of user data

xray@raccoon[4:49pm]/d3> du -sk Images/|Check the size of archiving directory

509656 Images |~500 Mbytes

xray@raccoon[4:49pm]/d3> mkisofs -l -R -v -o LysImages.raw Images/ |Make an ISO 9660 filesystem image with RockRidge protocol

[-l= long file names; -R= Rock Ridge protocol, -v=verbose, -o=output image name]

mkisofs 1.13 (i686-pc-linux-gnu) |Program starts

Scanning Images

1.97% done, estimate finish Thu May 31 16:49:36 2001

..98.26% done, estimate finish Thu May 31 16:50:20 2001

Total rockridge attributes bytes: 6529

254421 extents written (496 Mb) |Image creation is completed

xray@raccoon[4:55pm]/d3> ls -lt LysImages.raw |Confirm the size

-rw-r--r-- 1 xray users 521054208 May 31 16:50 LysImages.raw

xray@raccoon[4:55pm]/d3> cdrecord -v -dummy speed=4 dev=1,0,0 LysImages.img |Check the process with a dummy write; skip this step after few successful write sessions

xray@raccoon[4:55pm]/d3> cdrecord -v speed=4 dev=1,0,0 LysImages.img |Make an archive in CD-R. The real writing with commence with this command

[-v=verbose; speed=4=4x write speed; dev=1,0,0=SCSI number, id number, lun number]

Now try to check whether the archived data is accessible.

xray@raccoon[9:28am]~> mount /mnt/cdrom1

xray@raccoon[9:28am]~> df -k –l |Confirm the mounting

/dev/sda1 17639220 7234608 9508592 44% /d3

/dev/sda2 17639248 7208136 9535088 44% /d4

/dev/scd0 581650 581650 0 100% /mnt/cdrom1

xray@raccoon[9:28am]~> ls -lt /mnt/cdrom1 |Confirm the contents

-r-xr-xr-x 1 root root 18006000 Oct 26 1999 mydata002.osc

-r-xr-xr-x 1 root root 18006000 Oct 26 1999 mydata001.osc

xray@raccoon[9:28am]~> umount /mnt/cdrom1 |Unmount the drive

Take the CD-RW or CD-R media from the drive carefully label it and store.

Splitting Large Directory

Sometime users may encounter large data directories that occupy more than 600 MB of space. One-way to deal with large data set is to split the directory into several ~600 MB chunks and write separate CDs. The another way is write the data into DVD-R\-RW media (see § Writing to DVD-R). The following section shows how to split large directory and create several chunks of 600 MB (for CD-R\-RW) or 4.5 GB (for DVD-R\-RW).

soma@raccoon[11:20am]/d3/Ip/> du -ks DKGRA4 |Find the directory size

1192408 DKGRA4 |Approximately 1.2GB; larger than 1 CD-R capacity

This is obviously more than the capacity of one CD-R media (~650 MB). So first get the listing of all files in the directory and redirect the output to a new file, here it is called, ‘contents.list’

soma@raccoon[11:20am]/d3/Ip/DKGRA4> ls -lt >contents.list

Upon examination of the 'contents.list', one realizes the actual names of the files start at column 57 and higher (use File Info under Edit menu, if using gedit). So using the 'cut' command we make ourselves a name-only list of all the files and we call this ‘file.list’. This is done by first applying 'cut' to 'contents.list' and then re-directing the out put to ‘file.list’.

soma@raccoon[11:20am]/d3/Ip> gedit contents.list&

soma@raccoon[11:23am]/d3/Ip> cut -c57- contents.list>file.list

Now split this file.list into two halves, assuming all files occupy equal amount of space, renaming top half as ‘filetop.list’ and bottom half as ‘filebot.list’. Save both the files.

soma@raccoon[11:23am]/d3/Ip> ls -lt *.list

-rw-r--r-- 1 soma users 5004 Aug 10 10:39 file.list

-rw-r--r-- 1 soma users 22921 Aug 10 10:38 contents.list

-rw-r--r-- 1 soma users 2502 Aug 10 10:37 filebot.list

-rw-r--r-- 1 soma users 2502 Aug 10 10:37 filetop.list

Now use the ‘mkisofs’ command option ‘–exclude-list’ and specify the files to be excluded, here say, filetop.list and this will create an image of only bottom half of the directory DKGRA4, satisfying our objective of keeping the image less than 650 MB.

soma@raccoon[11:24am]/d3/Ip> mkisofs -l -R -v -exclude-list filetop.list -o Dkgra4top.raw DKGRA4/ | Exclude some files

Now make another image, however, this time replace the filebot.list as the excluded file. With the two images, you can create two separate CDs one after another.

More Detailed Information

For more detailed information contact Soma at 850-644-6648 or by e-mail: soma@sb.fsu.edu.

FireWire DVD-R\-RW Drive

Writing to DVD-\-RW Drive

Neptune.sb.fsu.edu has a FireWire DVD-R\-RW drive that will allow archiving of user data into a DVD-R or DVD-RW medium. However, before archiving the data the user needs first to create an ISO 9660 image of their data, similar to the procedure we saw in the section § Writing to CD-R\-RW, and then write that image to the DVD-R or DVD-RW. While creating the ISO 9660 image of the data remember not to exceed ~ 4.6 GB. Important Note: Unlike CD-R & CD-RW there are several DVD media available and not all of them are compatible with our drive. So ensure that you get a DVD-R (some times referred to as “DVD minus R” or “DVD dash R”) or its re-writeable counter-part DVD-RW. DVD+R, DVD+RW, and DVD-RAM are NOT compatible with our drive.

FireWire DVD-R/-RW drive from Lacie

Once the ISO9660 image of appropriate size has been created, the command dvdrecord can be used to archive (write) data to the DVD-R or DVD-RW. A typical session might look like the following:

[root@neptune /]# cd /home/Dump | Move to correct directory

[root@neptune Dump]# mkisofs -v -l -R -o HongLi-ALS.raw /home/Extract/ | Make an image of the required directory

1.0% done, estimate finish Thu Dec 19 09:49:13 2002

. . .

99.82% done, estimate finish Thu Dec 19 09:49:14 2002

1507696 extents written (2944 Mb) | Image has been created

[root@neptune Dump]# mount HongLi-ALS.raw -r -t iso9660 -o loop /mnt | Mount the image as a file-system to check

[root@neptune Dump]# df –k –l | Test the mounted f/s before writing to DVD

Filesystem 1k-blocks Used Available Use% Mounted on

/dev/hda1 1019864 140640 827416 15% /

/dev/hda6 11867516 6087092 5177588 55% /home

none 127828 0 127828 0% /dev/shm

/dev/hda3 3035952 1840412 1041316 64% /usr

/dev/hda5 3035920 57792 2823908 3% /usr/local

/home/Dump/HongLi-ALS.raw

3015392 3015392 0 100% /mnt

[root@neptune Dump]# umount /mnt | Unmount f/s before writing to DVD

[root@neptune Dump]# dvdrecord –scanbus | Find DVD-R Drive Allocation

scsibus2:

2,0,0 201) 'PIONEER ' 'DVD-RW DVR-104 ' '1.20' Removable CD-ROM

2,1,0 200) 'Maxtor ' '1394 storage ' 'v1.2' Disk

2,2,0 202) *

2,6,0 206) *

2,7,0 207) *

[root@neptune Dump]# dvdrecord -v -dao speed=2 dev=2,0,0 HongLi-ALS.raw | Write DVD with verbose (-v); disk-at-once (-dao); on SCSI device=2,0,0

dvdrtools - based on:

TOC Type: 1 = CD-ROM

scsidev: '2,0,0'

scsibus: 2 target: 0 lun: 0

Linux sg driver version: 3.1.22

Using libscg version 'schily-0.5'

atapi: 1

Vendor_info : 'PIONEER'

Identifikation: 'DVD-RW DVR-104 '

Revision : '1.20'

Device seems to be: Generic mmc2 DVD.

Using generic SCSI-3/mmc DVD-R(W) driver (mmc_mdvd).

Track 01: 2944 of 2944 MB written (fifo 100%).

Track 01: Total bytes read/written: 3087761408/3087761408 (1507696 sectors).

Writing time: 2168.070s

Fixating...

Fixating time: 115.125s

dvdrecord: fifo had 48636 puts and 48636 gets.

dvdrecord: fifo was 0 times empty and 26393 times full, min fill was 92%.

Once the data has been written the user can insert the DVD-R\-RW into another DVD-ROM drive; mount that drive and process the data directly from the drive.

Conclusion

With several options available for archiving, I hope the users of the Facility find it easier to back-up their data both for short- and long-term storage. I also would like to remind that the backing-up the data is the responsibility of the user and not that of the Facility Director. Remember that one lost data set is worth several months of YOUR work and may be NOT easily be replaced.

As noted in the section on DDS tapes, problems tend to rise while trying to read an old archive either due to hardware or software incompatibilities. So it is imperative that the users periodically move their archives to the latest and stable media of choice. Please send your suggestions and comments to Soma.