414 Kasha
Laboratory Institute of
Molecular Biophysics Phone: |
Soma’s Computer Notes
Linux
Archiving: rsync, Tape, CD & DVD
Media
How to archive
data using rsync, DDS-3/4, CD-R/RW and DVD-R/RW drives under Linux
Table of contents
raccoon.sb.fsu.edu
(128.186.103.108; RH Linux 7.1)
neptune.sb.fsu.edu
(128.186.103.106; RH Linux 7.3)
spruce.sb.fsu.edu
(128.186.103.109; RedHat Linux 8.0)
General
Information & Trouble Shooting of Tape Drive/Media
Error
Management and Trouble-shooting @ FSU..
General
Tape Media Trouble-shooting
Reading
Media Created in Silicon Graphics (SGI) in Linux o/s
Reading
from CD-ROM drive/CD-RW Drive
ã
2000-2004 Thayumanasamy Somasundaram
414 Kasha
Laboratory
Institute
of Molecular Biophysics,
E-mail: soma@sb.fsu.edu • URL: http://www.sb.fsu.edu/~soma
Phone
August 19,
2004
Logos,
Figures & Photos ã
of the respective Instrument Manufacturers
Linux Archiving: rsync, Tape, CD & DVD Media
How to archive data using rsync, DDS-3/4, CD-R/RW & DVD-R/RW drives under Linux
This note is intended to help the X-Ray Facility (XRF) users archive
(backup) their data both at home and at synchrotron using DDS-3/4, CD-R/RW, or
DVD-R/RW drives under Linux operating system (o/s). Similar archiving under Win2000/WinXP
platforms is fairly straight forward and therefore will not be covered. A copy of this Note will be posted in XRF Resources page
shortly after receiving suggestions and corrections from the users. This note was written in
rsync is an incremental archiving UNIX/Linux utility. It has several options (secure copy, file compression, directory tree, and file comparison) allowing a user to copy files from a source to a destination incrementally over the network. The command is fairly easy to use and therefore will be shown with an example. It is best to write a script and run the script, say, once an hour; alternatively one can even include a loop so that the script is invoked automatically after several minutes of ‘sleep’ time.
The following script is executed in the destination (local) computer and when invoked the user logs-in to the target (remote) computer and copies the file to the local machine:
#!/bin/sh
#This script will copy data from a source (remote) to
my destination (local)
rsync –avz –e ssh --exclude “*.imx_0”
/data/d1/xtal1 /mnt/fire1/my_dir/xtal1
Explanation for the script is given below:
#!/bin/sh: invokes the appropriate shell
# This script : A line starting with a ‘#’ is a
comment and not interpreted by the shell
rsync:
the command itself
-a:
archive mode (back-up)
-v:
verbose mode
-z:
compress the file while transferring
-e
ssh: the remote shell will be a ‘secure shell’
--exclude:
any files that need to be excluded; wild card permitted
/data/d1/xtal1:
source directory for copying (remote; detector’s computer)
/mnt/fire1/my_dir/xtal1:
destination directory for storing (local; your computer)
#!/bin/sh
# This script will copy data from a source to my laptop;
sleep for 1 minute
# and do it again
while true
do
rsync –avz –e ssh --exclude “*.imx_0”
/data/d1/xtal1 /mnt/fire1/my_dir/xtal1
sleep 3600
done
A typical session is given below:
#!/bin/sh
# This script is called getdata.sh when executed in tampa.sb.fsu.edu
will copy
# data from spruce.sb.fsu.edu (marCCD
machine) to my laptop tampa.sb.fsu.edu
# (local machine)
rsync –avz –e ssh
/home/marccd/d2/ccd-data /home/soma/data
# Target directory is spruce:/home/marccd/d2/ccd-data
# Destination directory is
soma@tampa[2:28pm]~>./getdata.sh
soma@spruce's password:***********
receiving file list ... done
./
ccd_xtal_001.imx
wrote 32 bytes read 467
bytes 76.77 bytes/sec
total size is 15183 speedup
is 30.43
wait for 1 minute
soma@spruce's password:***********
receiving file list ... done
./
soma_ccd.dat
wrote 32 bytes read 4939
bytes 301.27 bytes/sec
total size is 27171 speedup
is 5.47
Figure below shows an external DDS-3 SCSI Tape Drive from Seagate Technology (Certance).
External SCSI DDS-3/4 drive from Certance (formerly Seagate Technology)
Raccoon.sb.fsu.edu (located in KLB 410A) is a 933 MHz Linux machine dedicated both for data processing and archiving. It features an external SCSI DDS-3 tape drive, one internal IDE DVD-ROM drive and one internal SCSI CD-R/-RW drive. DDS-3 tape drives are backward compatible in reading and writing data to DDS (sometimes referred to as DAT), DDS-2 in addition to DDS-3 media. Depending upon the level of compression feasible for the data, DDS-3 media can accommodate uncompressed capacity of 12 GB and compressed capacity of 24 GB.
Neptune.sb.fsu.edu (located in KLB 410A) is a 933 MHz Linux machine dedicated both for archiving and data processing. It features an internal SCSI DDS-4 tape drive, one internal SCSI CD-ROM drive and one external FireWire hard drive. DDS-4 tape drives are backward compatible in reading and writing data to DDS (sometimes referred to as DAT), DDS-2 and DDS-3 in addition to DDS-4 media. Depending upon the level of compression feasible for a data DDS-3 media can accommodate uncompressed capacity of 20 GB and compressed capacity of 40 GB.
The picture below shows the various DDS media logos:
DDS Media Logos
All XRF users will be able to read from and write to the tape drive. Since it is the only SCSI tape drive it has the device allocation of /dev/st0 and its status can be checked using the mt command and data archived using tar command. Typical commands that are used during archiving are listed below with a brief explanation:
mt
–f /dev/st0 status |Check
status of the first SCSI tape drive /dev/st0
tar
cvf /dev/st0 my_dir |Write contents of my_dir to tape
tar
tvf /dev/st0 |Look at
or list contents of previously archived tape
tar
-xvf /dev/st0 parent_dir/my_dir |Extract contents of my_dir
mt=
magnetic tape; tar= tape archive; -c= create; -v= verbose;-x= extract; st= scsi
tape; -t= list; -f= file(system).
[root@raccoon
/]# mt -f /dev/st0 status |Check
status of tape drive
SCSI 2 tape drive: File
number=0, block number=0, partition=0.
Tape block size
512 bytes. Density
code 0x25 (DDS-3).
General status bits on
(45010000):
BOT WR_PROT ONLINE IM_REP_EN |Beginning of tape; write-protected
tape
[root@raccoon
DDS]# tar tvf /dev/st0 |List contents of tape
drwxr-xr-x soma/users 0 2001-09-19 15:04:14 TapeDDS3-024/
drwxr-xr-x soma/users 0 2001-09-18 13:44:17
TapeDDS3-024/4mmtp116/
drwxr-xr-x soma/users 0 2001-09-18 12:39:05
TapeDDS3-024/4mmtp116/AdamekF1
[root@raccoon
DDS]# tar-xvf /dev/st0 |Extract
contents of the tape media
fire2/soma/
fire2/soma/all.list
fire2/soma/Blaber_Fgf/
fire2/soma/Blaber_Fgf/fgf_2/
fire2/soma/Blaber_Fgf/fgf_2/snap.runlist
fire2/soma/Blaber_Fgf/fgf_2/snap_0_001.img
[root@raccoon
DDS]# tar -xvf /dev/st0 fire2/soma/Blaber_Fgf |Extract contents of only
Blaber_Fgf directory and NOT all
fire2/soma/Blaber_Fgf/
fire2/soma/Blaber_Fgf/fgf_2/
fire2/soma/Blaber_Fgf/fgf_2/snap.runlist
fire2/soma/Blaber_Fgf/fgf_2/snap_0_001.img
[root@raccoon
DDS]# mt -f /dev/st0 offline |Eject the tape
[root@raccoon
/]# mt -f /dev/st0 status |Check
status of tape drive
SCSI 2 tape drive: File
number=0, block number=0, partition=0.
Tape block size
512 bytes. Density
code 0x25 (DDS-3).
General status bits on
(45010000):
BOT ONLINE IM_REP_EN |Beginning of tape;
ready to write
[soma@spruce
~]$ tar -cvvf /dev/st0 tmp/ |Write contents of tmp directory
drwxrwxr-x soma/marccd 0 2003-04-30 16:41:21 tmp/
-rwxr-xr-x soma/marccd 103756 2000-12-22 23:53:35 tmp/font.exe
-rw------- soma/marccd
7223800 2001-05-16 11:36:03 tmp/abc10003.osc
-rw-rw-r-- soma/marccd 1737 2001-08-07 15:10:20 tmp/uniqueify.log
[soma@spruce
~]$ tar -cvvf /dev/st0 erase |Erase contents of tape. Careful!!!
[soma@spruce
~]$ tar -cvf /dev/st0 my_dir | & tee dds4-listing & |Simultaneously
(tee) archive
data from my_dir to a
tape and write the filenames into another file (dds4-listing).
Spruce.sb.fsu.edu the 650 MHz Linux machine located in KLB 410A that controls marCCD165 data collection has an internal DDS-4 tape drive. This SCSI DDS-4 tape drive can read and write to DDS2, DDS3 and DDS4 media. However, it is best to use DDS4 media, since it has a native/compressed capacity of 20/40 GB. IP/CCD data will compress anywhere between 10 to 60% depending upon the level of detail in the original image. The same tar and mt commands shown above can be used to archive data using this tape drive. Please read the mt and tar man pages or check with Soma, or Mike for further help.
Archive
Creation Time: Sun
bm14cbu.cars.aps.anl.gov | 164.54.161.114
I inserted a DDS-3 tape into HP Model C1537A DDS-3 Tape drive
attached to a Linux workstation running RedHat Linux Version 7.3 Valhalla
2.4.8-27.7.x and archived the data using regular tar command:
[userbmc@bm14cbu]$ tar cvvf /dev/st0
blaber_0304/
Archive
Extraction Time: Thu
Later at home when the same DDS3 tape was read using the DDS4
drive in spruce.sb.fsu.edu I received an error.
More details about the error and how the problem was solved are
explained below.
Note Creation Time: Thu
Linux spruce.chem.fsu.edu
2.2.16-22-marccd #1 Wed
[soma@spruce ~]$ tar tvf /dev/st0 |DDS-3 tape created @ APS read @ FSU
tar: /dev/st0: Cannot read: Input/output
error
tar: At beginning of tape, quitting now
tar: Error is not recoverable: exiting now
I then checked the status of the tape
[soma@spruce ~]$ mt -f /dev/st0 status |Set at default block size of 512; recognizes DDS-3
SCSI 2 tape drive:
File number=0, block number=0,
partition=0.
Tape block size 512
bytes. Density code 0x25 (DDS-3).
Soft error count since last status=0
General status bits on (45010000):
BOT WR_PROT ONLINE IM_REP_EN
I then got a suggestion from the website: http://www.ls.eso.org/lasilla/sciops/2p2/E2p2M/WFI/tapes/TapeProb.html
to set the block size to be variable (set
to zero) while reading under Linux
(I used 'reading
[soma@spruce ~]$ mt -f /dev/st0 setblk
0 |Set to
variable block size
[soma@spruce ~]$ mt -f /dev/st0 status |Check the status again to confirm block size
SCSI 2 tape drive:
File number=0, block number=0,
partition=0.
Tape block size 0 bytes.
Density code 0x25 (DDS-3).
Soft error count since last status=0
General status bits on (45010000):
BOT
WR_PROT ONLINE IM_REP_EN
Now I was able to read the tape (here,
only listing is shown, but I could do tar xvf as well)
[soma@spruce ~]$ tar tvf /dev/st0 |Get a listing
of files in archive
drwxrwxrwx userbmc/users 0 2003-04-13 15:36:14 fgf_1/
-rw-r--r-- userbmc/users 57 2003-04-13 11:57:02 fgf_1/snap.runlist
-rw-r--r-- userbmc/users 10617344
2003-04-13 11:26:18 fgf_1/snap_0_009.img
-rw-r--r-- userbmc/users 10617344
2003-04-13 11:29:56 fgf_1/snap_0_010.img
--------------------------------------------------
Now if the blk size is set to zero (= set
to variable size)...
[soma@spruce ~]$
mt -f /dev/st0 status
SCSI 2 tape drive:
File number=0, block number=0,
partition=0.
Tape block size 0 bytes.
Density code 0x26 (unknown to this mt) |a DDS-4 tape
Soft error count since last status=0
General status bits on (45010000):
BOT
WR_PROT ONLINE IM_REP_EN
... and if you try to read a tape written
with default blk size of 512 you may encounter the following problem shown
below.
[soma@spruce ~]$
tar tvf /dev/st0
tar: Record size = 1 blocks
drwxrwxr-x
marccd/marccd 0 2001-09-28 11:14:49
d2/
drwxr-xr-x marccd/marccd 0 2000-10-27 01:18:30 d2/lost+found/
drwxrwxr-x soma/marccd 0 2001-09-28 10:55:57 d2/APS2001/
drwxr-xr-x soma/marccd 0 2001-09-27 02:38:38 d2/APS2001/MSP1/
-rw------- soma/marccd 8964 2001-09-27 05:54:58
d2/APS2001/MSP1/.listing
-rw-r--r-- soma/marccd 57 2001-09-27 07:37:00
d2/APS2001/MSP1/snap.runlist
-rw-r--r-- soma/marccd 10617344 2001-09-27
04:18:00 d2/APS2001/MSP1/snap_0_203.img
...the tape reading virtually stops at
this stage.
In that event, reset the blk size back to
512 and then you can read it.
[soma@spruce ~]$
mt -f /dev/st0 setblk 512 | Set block size
back to the default value of 512
[soma@spruce ~]$ mt -f /dev/st0 status |Check the
status again
SCSI 2 tape drive:
File number=0, block number=0, partition=0
Tape block size 512 bytes. Density code
0x26 (unknown to this mt)
Soft error count since last status=0
General status bits on (45010000):
BOT
WR_PROT ONLINE IM_REP_EN
[soma@spruce ~]$
tar tvf /dev/st0 |Now you can
read the tape properly
drwxrwxr-x marccd/marccd 0 2001-09-28 11:14:49 d2/
drwxrwxr-x soma/marccd 0 2001-09-28 10:55:57 d2/APS2001/
drwxr-xr-x soma/marccd 0 2001-09-27 02:38:38 d2/APS2001/MSP1/
-rw------- soma/marccd 8964 2001-09-27 05:54:58
d2/APS2001/MSP1/.listing
-rw-r--r-- soma/marccd 57 2001-09-27 07:37:00 d2/APS2001/MSP1/snap.runlist
Mon
Linux spruce.sb.fsu.edu 2.2.16-22-marccd
#1 Wed
I tried to read an old DDS tape created in thr.sb.fsu.edu (Silicon
Graphics Irix 5.3; IP20 MIPS) using the DDS4 internal tape-drive in spruce.sb.fsu.edu. Earlier, I had had some problems reading the
tapes written using SGI. So first I got
some information about the tape drive from SGI (the machine that wrote the
tape).
thr.sb.fsu.edu----------------------------SGI
Irix 5.3---------------
I logged into thr.sb.fsu.edu, with no tape inside tape drive,
root@thr.sb.fsu.edu> mt -f
/dev/rmt/tps0d3 status
Controller:
SCSI
Device:
ARCHIVE: Python 25601-XXX2.63
Status:
0x202
Drive
Type: DAT
Media:
Not Ready
root@thr.sb.fsu.edu> mt -f
/dev/rmt/tps0d3 blksize
Recommended
tape I/O size 262144 bytes (512 512-byte blocks)
Minimum
block size: 1 byte(s)
Maximum
block size: 16777215 bytes
Current
block size: Variable.
Then with DDS tape number 43 (DDS MRS 90M Sony) created under SGI
o/s inside the tape drive,
root@thr.sb.fsu.edu> mt -f
/dev/rmt/tps0d3 status
Controller:
SCSI
Device:
ARCHIVE: Python 25601-XXX2.63
Status:
0x20266
Drive
Type: DAT
Media:
Ready, write protected, at BOT
root@thr.sb.fsu.edu> mt -f
/dev/rmt/tps0d3 blksize
Recommended
tape I/O size 262144 bytes (512 512-byte blocks)
Minimum
block size: 1 byte(s)
Maximum
block size: 16777215 bytes
Current
block size: 262144
Then I took the same tape and inserted inside DDS4 internal tape
drive in spruce.sb.fsu.edu with Linux o/s
spruce.sb.fsu.edu-----------------------RH
Linux 7.1---------------
Then I logged into spruce.sb.fsu.edu. I inserted the tape number 43 in DDS4 drive
(/dev/st0). And first I tried with default
block size of 512 (It should have worked since the default value of the media
was 512? but didn't.) and I got an error message:
[soma@spruce ~]$ tar tvf /dev/st0
tar: /dev/st0: Cannot read: Input/output
error
tar: At beginning of tape, quitting now
tar: Error is not recoverable: exiting now
I checked the default block size was 512
[soma@spruce ~]$ mt -f /dev/st0 status
SCSI 2 tape drive:
File number=0, block number=0,
partition=0.
Tape block size 512 bytes. Density code
0x13 (DDS (61000 bpi)).
Soft error count since last status=0
General status bits on (45010000):
BOT
WR_PROT ONLINE IM_REP_EN
Then I tried to change the blksize to 256
[soma@spruce ~]$ mt -f /dev/st0 setblk
256
[soma@spruce ~]$ mt -f /dev/st0 status
SCSI 2 tape drive:
File number=0, block number=0,
partition=0.
Tape block size 256 bytes. Density code
0x13 (DDS (61000 bpi)).
Soft error count since last status=0
General status bits on (45010000):
BOT
WR_PROT ONLINE IM_REP_EN
I still got the same error message
[soma@spruce ~]$ tar tvf /dev/st0
tar: /dev/st0: Cannot read: Input/output
error
tar: At beginning of tape, quitting now
tar: Error is not recoverable: exiting now
Then I tried to change the blksize 0 (variable block size)
[soma@spruce ~]$ mt -f /dev/st0 setblk
0
[soma@spruce ~]$ mt -f /dev/st0 status
SCSI 2 tape drive:
File number=0, block number=0,
partition=0.
Tape block size 0 bytes. Density code 0x13
(DDS (61000 bpi)).
Soft error count since last status=0
General status bits on (45010000):
BOT
WR_PROT ONLINE IM_REP_EN
[soma@spruce ~]$ tar tvf /dev/st0
drwxr-xr-x raxis/user 0 1997-10-27 11:50:33 ./
drwxr-xr-x raxis/user 0 1997-09-26 16:06:16 Aav/
drwxr-xr-x raxis/user 0 1997-09-26 16:06:42 Aav/Figures/
-rw-r--r-- raxis/user 2809446 1997-08-26 10:53:39 Aav/Figures/gel1.tif
tar: Skipping to next header
-rw-r--r-- raxis/user 7786496 1997-10-09 22:49:21
Ak_Hi1/Data/akhi1026.osc
tar: Skipping to next header
-rw-r--r-- raxis/user 4995053 1997-10-14 00:36:36
Diao_rt10/insph5020.osc.gz
tar: Skipping to next header
It only worked partly and stopped.
-----
Finally I got a suggestion from the website about ‘dd’ (UNIX
command)
(Google search: 'sgi dds block size')
http://wks.uts.ohio-state.edu/sun/faqs/HP-4mmDAT-info.txt
"dd if=/dev/st0 ibs=20k obs=512
bs=512k | tar tvf -"
And from the website related HP/Unix
http://hpux.tn.tudelft.nl/hppd/FAQ/9-13.html
9-13 How do I read an SGI-written tar format DDS tape?
dd if=/dev/rmt/0m ibs=512k obs=10k| tar
-xvf -
-----
I tried with the default value mentioned in the website. It worked partly,
[soma@spruce ~]$ dd if=/dev/st0 ibs=20k
obs=512 bs=512k | tar tvf -
drwxrwxr-x raxis/user 0
2000-03-25 13:41:05 HChenMar-005/
drwxrwxr-x raxis/user 0 2000-03-28 14:29:07
HChenMar-005/Tamo/
-rw-rw-r-- raxis/user 5190385 2000-03-24 18:43:38
HChenMar-005/Tamo/tamo001.osc.Z
tar: Skipping to next header
-rw-rw-r-- raxis/user 4998991 2000-03-24 19:56:24
HChenMar-005/Tamo/tamo009.osc.Z
tar: Skipping to next header
-rw-rw-r-- raxis/user 5182763 2000-03-25 14:17:19
HChenMar-005/Tamo/tamo2003.osc.Z
tar: Skipping to next header
-rw-rw-r-- raxis/user 5180609 2000-03-25 15:02:49 HChenMar-005/Tamo/tamo2008.osc.Z
tar: Skipping to next header
0+152 records in
76+0 records out
Then I tried with modified command (cf bs=256k to 512k). Then it worked nicely.
[soma@spruce ~]$ dd if=/dev/st0 ibs=20k
obs=512 bs=256k | tar tvf -
drwxrwxr-x raxis/user 0
2000-03-25 13:41:05 HChenMar-005/
drwxrwxr-x raxis/user 0 2000-03-28 14:29:07
HChenMar-005/Tamo/
-rw-rw-r-- raxis/user 5190385 2000-03-24 18:43:38
HChenMar-005/Tamo/tamo001.osc.Z
-rw-rw-r-- raxis/user 5193270 2000-03-24 18:52:44 HChenMar-005/Tamo/tamo002.osc.Z
-rw-rw-r-- raxis/user 5173483 2000-03-24 19:01:50
HChenMar-005/Tamo/tamo003.osc.Z
-rw-rw-r-- raxis/user 5159591 2000-03-24 19:10:56
HChenMar-005/Tamo/tamo004.osc.Z
-rw-rw-r-- raxis/user 5128517 2000-03-24 19:20:01
HChenMar-005/Tamo/tamo005.osc.Z
113+0 records in
113+0 records out
Tapes with DDS (or DDS1) and 60M (60 meters) symbols were not
readable (ejected out) by DDS4 drive.
CD-ROM drive and CD-RW/-R drives can be used both for direct data processing and copying the files from the drives to the disks (CD-R/-RW drives). The drives are first mounted as a file systems (remember, in Linux even the devices are filesystems, f/s) to allow the operating system (o/s) access to the existing data. Once the processing is completed the drives can unmounted. Figure below shows the internal HP CD-Writer 9500.
Internal SCSI CD-RW/R drive HP-CD Writer 9500 from HP
First, place a data CD in a CD-ROM or CD-RW drive and mount the device if not mounted. Process the data and unmount the device. A typical session might look like the following:
xray@raccoon[9:28am]~> df -k –l |Check the status local disks (-l
option)
/dev/hda10 8815372 20
8367544 1% /d5
/dev/sda1
17639220 7234608
9508592 44% /d3
/dev/sda2
17639248 7208136 9535088
44% /d4
xray@raccoon[9:28am]~> mount /mnt/cdrom |Now
xray@raccoon[9:28am]~> df -k –l |Check local disks again; confirm mounting
/dev/hda10 8815372 20
8367544 1% /d5
/dev/sda1
17639220 7234608 9508592
44% /d3
/dev/sda2
17639248 7208136 9535088
44% /d4
/dev/hdc 633432 633432 0 100% /mnt/cdrom
xray@raccoon[9:28am]~> ls -lt /mnt/cdrom |Check contents of CD-ROM
-r-xr-xr-x
1 root root 18006000
-r-xr-xr-x
1 root root 18006000
-r-xr-xr-x
1 root root 9006000
-r-xr-xr-x
1 root root 9006000
xray@raccoon[9:28am]~> umount /mnt/cdrom |Unmount
(umount) CD-ROM
Repeat the same procedure for CD-RW drive replacing /mnt/cdrom by /mnt/cdrom1. Remember, however, for successful mounting CD-R and CD-RW should have pre-existing data in them (blank CD-R and CD-RW will give an error: no medium found!).
Raccoon.sb.fsu.edu has a CD-RW drive that will allow archiving of your data into a CD-R or CD-RW medium. However, before archiving the data the user needs first to create an ISO 9660 image of their data and then write that image to the CD-R or CD-RW. While creating the ISO 9660 image of the data remember not to exceed ~ 640 MB of total space. If however, the user data is large, the data can be split into manageable segments, see § Splitting Large Directory section below to see how this can be accomplished. Once the ISO9660 image of appropriate size has been created, the command cdrecord can be used to archive (write) data to the CD-R or CD-RW. A typical session might look like the following:
xray@raccoon[4:49pm]~> cd /d3 |Go
to parent directory of user data
xray@raccoon[4:49pm]/d3> du -sk Images/|Check the size of archiving directory
509656 Images
|~500 Mbytes
xray@raccoon[4:49pm]/d3> mkisofs -l -R -v -o LysImages.raw
Images/ |Make an ISO 9660 filesystem image with RockRidge
protocol
[-l= long file names; -R= Rock
Ridge protocol, -v=verbose, -o=output image name]
mkisofs 1.13 (i686-pc-linux-gnu) |Program starts
Scanning Images
1.97% done,
estimate finish Thu
..98.26% done, estimate finish Thu
Total rockridge attributes bytes: 6529
254421 extents written (496 Mb) |Image creation is completed
xray@raccoon[4:55pm]/d3> ls -lt LysImages.raw |Confirm the size
-rw-r--r--
1 xray users 521054208
xray@raccoon[4:55pm]/d3> cdrecord -v -dummy speed=4 dev=1,0,0 LysImages.img |Check the process with a dummy write; skip this step after few successful write sessions
xray@raccoon[4:55pm]/d3> cdrecord -v speed=4 dev=1,0,0 LysImages.img |Make an archive in CD-R. The real writing with commence with
this command
[-v=verbose; speed=4=4x write speed; dev=1,0,0=SCSI
number, id number, lun number]
Now try to check whether the archived data is accessible.
xray@raccoon[9:28am]~> mount /mnt/cdrom1
xray@raccoon[9:28am]~> df -k –l |Confirm the
mounting
/dev/sda1
17639220 7234608 9508592
44% /d3
/dev/sda2
17639248 7208136 9535088
44% /d4
/dev/scd0 581650 581650 0 100% /mnt/cdrom1
xray@raccoon[9:28am]~> ls -lt /mnt/cdrom1 |Confirm the
contents
-r-xr-xr-x
1 root root 18006000
-r-xr-xr-x
1 root root 18006000
xray@raccoon[9:28am]~> umount /mnt/cdrom1 |Unmount the
drive
Take the CD-RW or CD-R media from the drive carefully label it and store.
Sometime users may encounter large data directories that occupy more than 600 MB of space. One-way to deal with large data set is to split the directory into several ~600 MB chunks and write separate CDs. The another way is write the data into DVD-R\-RW media (see § Writing to DVD-R). The following section shows how to split large directory and create several chunks of 600 MB (for CD-R\-RW) or 4.5 GB (for DVD-R\-RW).
soma@raccoon[11:20am]/d3/Ip/> du -ks DKGRA4 |Find the directory size
1192408 DKGRA4 |Approximately 1.2GB; larger than 1 CD-R capacity
This is obviously more than the capacity of one CD-R
media (~650 MB). So first get the
listing of all files in the directory and redirect the output to a new file,
here it is called, ‘contents.list’
soma@raccoon[11:20am]/d3/Ip/DKGRA4> ls -lt >contents.list
Upon examination of the 'contents.list', one realizes the actual
names of the files start at column 57 and higher (use File Info under Edit
menu, if using gedit). So using the 'cut' command we make ourselves
a name-only list of all the files and we call this ‘file.list’. This is done by first applying 'cut' to 'contents.list' and then re-directing the
out put to ‘file.list’.
soma@raccoon[11:20am]/d3/Ip>
gedit contents.list&
soma@raccoon[11:23am]/d3/Ip>
cut -c57- contents.list>file.list
Now split this file.list into two halves, assuming
all files occupy equal amount of space, renaming top half as ‘filetop.list’ and bottom half as ‘filebot.list’. Save both the files.
soma@raccoon[11:23am]/d3/Ip>
ls -lt *.list
-rw-r--r-- 1 soma
users 5004
-rw-r--r-- 1 soma
users 22921
-rw-r--r-- 1 soma
users 2502
-rw-r--r-- 1 soma
users 2502
Now use the ‘mkisofs’ command option ‘–exclude-list’ and specify the files to
be excluded, here say, filetop.list and this will create an image of only bottom
half of the directory DKGRA4, satisfying our objective of keeping the image
less than 650 MB.
soma@raccoon[11:24am]/d3/Ip> mkisofs -l -R -v -exclude-list filetop.list -o
Dkgra4top.raw DKGRA4/ |
Exclude some files
Now make another image, however, this time replace
the filebot.list as the excluded file. With the
two images, you can create two separate CDs one after another.
For more detailed information contact Soma at
Neptune.sb.fsu.edu has a FireWire
Once the ISO9660 image of appropriate size has been created, the command dvdrecord can be used to archive (write) data to the DVD-R or DVD-RW. A typical session might look like the following:
[root@neptune /]# cd /home/Dump | Move to correct directory
[root@neptune Dump]# mkisofs -v -l -R -o
HongLi-ALS.raw /home/Extract/ | Make an image of the required directory
1.0% done, estimate finish Thu
.
. .
99.82% done, estimate finish Thu
1507696 extents written (2944 Mb) | Image has been
created
[root@neptune
Dump]#
[root@neptune
Dump]# df –k –l | Test the mounted f/s before writing to DVD
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/hda1 1019864 140640
827416 15% /
/dev/hda6 11867516 6087092
5177588 55% /home
none 127828 0
127828 0% /dev/shm
/dev/hda3 3035952 1840412
1041316 64% /usr
/dev/hda5 3035920 57792
2823908 3% /usr/local
/home/Dump/HongLi-ALS.raw
3015392 3015392 0 100% /mnt
[root@neptune
Dump]# umount
/mnt |
Unmount f/s before writing to DVD
[root@neptune
Dump]# dvdrecord –scanbus | Find DVD-R Drive Allocation
Cdrecord
1.11a15 (i686-pc-linux-gnu)(C) 1995-2001 Jörg Schilling
scsibus2:
2,0,0 201) 'PIONEER ' 'DVD-RW DVR-104 ' '1.20' Removable CD-ROM
2,1,0
200) 'Maxtor ' '1394 storage ' 'v1.2' Disk
2,2,0
202) *
2,6,0
206) *
2,7,0
207) *
[root@neptune
Dump]# dvdrecord -v -dao speed=2 dev=2,0,0 HongLi-ALS.raw |
Write DVD with verbose (-v); disk-at-once (-dao); on SCSI device=2,0,0
dvdrtools
- based on:
Cdrecord
1.11a15 (i686-pc-linux-gnu) (C) 1995-2001 Jörg Schilling
TOC
Type: 1 = CD-ROM
scsidev:
'2,0,0'
scsibus:
2 target: 0 lun: 0
Linux
sg driver version: 3.1.22
Using
libscg version 'schily-0.5'
atapi:
1
Vendor_info : 'PIONEER'
Identifikation:
'DVD-RW DVR-104 '
Revision : '1.20'
Device
seems to be: Generic mmc2 DVD.
Using
generic SCSI-3/mmc DVD-R(W) driver (mmc_mdvd).
Track
01: 2944 of 2944 MB written (fifo 100%).
Track
01: Total bytes read/written: 3087761408/3087761408 (1507696 sectors).
Writing
time: 2168.070s
Fixating...
Fixating
time: 115.125s
dvdrecord:
fifo had 48636 puts and 48636 gets.
dvdrecord:
fifo was 0 times empty and 26393 times full, min fill was 92%.
Once the data has been written the user can insert the DVD-R\-RW into another DVD-ROM drive; mount that drive and process the data directly from the drive.
With several options available for archiving, I hope the users of the Facility find it easier to back-up their data both for short- and long-term storage. I also would like to remind that the backing-up the data is the responsibility of the user and not that of the Facility Director. Remember that one lost data set is worth several months of YOUR work and may be NOT easily be replaced.
As noted in the section on DDS tapes, problems tend to rise while trying to read an old archive either due to hardware or software incompatibilities. So it is imperative that the users periodically move their archives to the latest and stable media of choice. Please send your suggestions and comments to Soma.