Thayumanasamy Somasundaram, Ph D

414 Kasha Laboratory

Institute of Molecular Biophysics

Florida State University, Tallahassee, FL 32306-4380

Phone: (850) 644-6448 | Fax: (850) 644-7244

Utsomasundaram@fsu.eduU | Uwww.sb.fsu.edu/~somaU


Soma’s Computer Notes

LTO 1 and 2 (Ultrium 1 and 2) tape archiving

 

Troubleshooting for LTO-1 tape in LTO-2 drive


 

 

Key words: LTO-1, LTO-2, Ultrium 1, Ultrium 2, tape, tape drive, archiving, back-up, timeout error, tape read error, mt, tar, Linux, SGI, UNIX, HP, Certance, Quantum, DAT, DDS, DAT2, DDS3, DDS4, DDS5, DAT72, Linear Tape Open 1, Digital Data Storage, Digital Audio Tape, Exabyte, Digital8, 8mm tape


Table of contents

Introduction. 3

Tape archiving (back-up) procedures at XRF. 3

Primer 3

DDS and DAT tapes. 3

SGI to Linux archive migration. 3

DDS to LTO migration. 4

DDS3 and DAT72 Drives. 4

Problem with LTO 1 tape in LTO 2 drive. 5

UNIX commands used: mt and tar. 5

Solution to LTO 1 tape in LTO 2 drive read error. 7

Conclusion. 7

ã 2000-2010 Thayumanasamy Somasundaram

414, Kasha Laboratory

Institute of Molecular Biophysics, Florida State University,

Tallahassee, FL 32306-4380

E-mail: Utsomasundaram@fsu.eduU • URL: Uhttp://www.sb.fsu.edu/~somaU

Phone 850.644.6448 • Fax 850.644.7244

May 20, 2010 | Version 1.0

Key words: LTO-1, LTO-2, Ultrium 1, Ultrium 2, tape, tape drive, archiving, back-up, timeout error, tape read error, mt, tar, Linux, SGI, UNIX, HP, Certance, Quantum, DAT, DDS, DAT2, DDS3, DDS4, DDS5, DAT72, Linear Tape Open 1, Digital Data Storage, Digital Audio Tape, Exabyte, Digital8, 8mm tape

 

Logos, Figures & Photos ã of the respective Instrument Manufacturers

 


LTO 1 and 2 (Ultrium 1 and 2) tape archiving

Troubleshooting for LTO-1 tapes and LTO-2 drives

Version: 1.0 May 20, 2010

Introduction

This note is intended to help the IMB UX-Ray FacilityU (XRF) users troubleshoot problems with reading and writing ULTOU 1[1] and LTO 2 (aka, UUltriumU 1 and Ultrium-2) tapes using the tape drives at the Facility. A copy of this Note will be posted in UXRF Resources pageU shortly after receiving suggestions and corrections from the users. This note was first written on May 20, 2010 by the X-Ray Facility Director Thayumanasamy Somasundaram.

Tape archiving (back-up) procedures at XRF

Primer

The IMB X-Ray Facility archives large amounts of macromolecular x-ray diffraction data collected both at home-source and synchrotron-sources. Typical data collection from home-source per data set is around 3 GB and typical data collection from the synchrotron-source per trip is close to 350 GB. So even with the current 1 TB storage server capacity, we need to archive (back-up) the data into tapes for long-term storage. The two common tape formats we use at the Facility are Digital Data Storage (DDS) tape for home data and Linear Tape Open (LTO) tape for synchrotron data. The X-Ray Facility has been archiving data using DDS format since 1993 and using LTO format since 2005.

DDS and DAT tapes

X-Ray Facility’s UDDSU tape archiving dates back to the time when the Facility’s computing, processing, and modeling platforms (early to mid 1990s) were dominated by Silicon Graphics Inc (SGI) computers. SGI’s native/internal tape drive had a native capacity of 2 GB and was DDS format (aka, DDS1 or UDATU; DAT, Digital Audio Tape developed for audio industry found a use in computer data archiving as DDS tapes similar to 8mm and Digital8 (D8) tapes developed for video industry found usage in computer archiving as Exabyte tapes). Over the years the Facility has used DDS, DDS2, DDS3, DDS4, and DDS5 (aka, DAT72) tapes to archive the data sets.

SGI to Linux archive migration

In early 2001 we had to retire all SGI machines since the x-ray generator vendor upgraded the R-Axis IIc data collection platform from SGI Indigo running Irix 5.3 to Microsoft Windows NT 4.0 and the newly acquired marCCD 165 data collection platform was running RH Linux 7.0. Meanwhile, the support for SGI operating system upgrades were getting pricier and infrequent and the crystallographic community was moving toward Linux platform. This forced us to migrate all the data archived using SGI tape-drive to Linux platform. This turned out be more painful than we originally thought. Due to either the SGI specific software or hardware or a combination of both, the SGI written-tapes were unreadable under Linux software/hardware. The only way we could migrate the data was to extract the data from the DDS tapes using the same SGI machine (that wrote the tape) to an external hard disk. This hard disk in turn was later mounted to a Linux machine attached to a DDS3 tape drive. Eventually all DDS tapes contents were written to DDS3 tapes using Linux operating system (the average migration ratio was 5 DDS tapes = 1 DDS3 tape).

DDS to LTO migration

Now we are contemplating two options. Option1: migration of DDS3 and DDS4 tape archives to LTO 2 or LTO 3 tapes. The reason for this migration is to keep all the archives in one single medium to avoid readability problem in the future and to reduce upgrading several format hardware. DDS compatibility chart is shown below:

DDS-1 through DAT-72 compatibility chart. Note that DDS3 drives can read DDS3, DDS2 and DDS1 tapes. DAT72 drives can read DAT72, DDS4, and DDS3 but not DDS1 or DDS2 (Two generation backward compatibility). Source: www.dat-mgm.com

Option2: is to continue to support this format. Since majority of the current archives are in DDS3, DDS4, and DAT 72 tapes we need to retain at least two working DAT 72 drives and acquire a DAT 160 drive.

DDS3 and DAT72 Drives

To keep the Option 2 mentioned above viable we are keeping one each of DDS3, DDS4, and DAT 72 tape drives and plan to acquire a DAT 160 drive.

Seagate DDS3 external tape drive

Quantum DAT72 internal tape drive

 

Problem with LTO 1 tape in LTO 2 drive

The synchrotron-source data are usually archived in LTO-1 (Ultrium-1; native/compressed capacity 100/200 GB) tapes due to their large capacity and diffraction data set’s ability to be compressible. Occasionally, we use LTO-2 tapes (Ultrium-2; native/compressed capacity 200/400 GB) to accommodate larger data sets. The LTO 1[2] tapes (LTO 1, LTO-1, Ultrium 1, and Ultrium-1 all refer to the same tape) were written using rad.sb.fsu.edu and LTO 2 tapes were written using sie.sb.fsu.edu.

Users sometime find it impossible to read the LTO 1 tapes in LTO 2 drive even though it is allowed (Note that the LTO 2 drives are compatible with both ULTO 1 and LTO 2 tapesU. However, LTO 1 drivers can read only ULTO 1 tapesU and NO ULTO 2 tapes. UBackwardU read compatibility is assured for current and two former generations and backward write compatibility is assured for current and one former generation).

UNIX commands used: mt and tar

HP StorageWorks Ultrium 215e tape drive

Quantum external Ultrium 2 tape drive

 

While writing the LTO 1 tapes in rad.sb.fsu.edu Linux mt command was used in combination with Linux tar command. The computer was running Ubuntu 7.04 with Linux kernel ver. 2.6.20-16-generic and mt-st v. 0.8. The tape drive was HP StorageWorks Ultrium 215e.An example is shown below for archiving Linux data:

tsomasun@rad:/imb/xrf/backup$ mt -f /dev/st0 status

SCSI 2 tape drive:

File number=0, block number=0, partition=0.

Tape block size 0 bytes. Density code 0x40 (DLT1 40 GB, or Ultrium).

Soft error count since last status=0

General status bits on (41010000):

 BOT ONLINE IM_REP_EN

 

tsomasun@rad:/imb/xrf/backup$ ls -lt

total 24

drwxr-xr-x  4 soma users   31 2009-10-27 10:35 dds3-001

drwxr-xr-x 19 soma users 4096 2009-10-26 10:37 dds3-002

 

tsomasun@rad:/imb/xrf/backup$ tar cvf /dev/st0 dds3-001 dds3-002 >&1 | tee dds-bup-001.list &

[1] 27901

tsomasun@rad:/imb/xrf/backup$ dds3-001/

dds3-001/raxis/

dds3-001/raxis/GattisMar00-1/

dds3-001/raxis/GattisMar00-1/ipdata.img

dds3-001/raxis/GattisMar00-1/soln000.stl

dds3-001/raxis/GattisMar00-1/solnsBS000.stl

 

The same tape was now taken over to gau.sb.fsu.edu for reading. This computer was running Ubuntu 8.04.2 ane Linux kernel ver. 2.6.24-24-generic and mt-st v. 0.09b . The tape drive was Quantum LTO 2 drive. Then when attempted to extract the data using tar gives the following error:

tsomasun@gau:~$ tar tvf /dev/st0

tar: /dev/st0: Cannot read: Input/output error

tar: At beginning of tape, quitting now

tar: Error is not recoverable: exiting now

 

tsomasun@gau:~$ mt -f /dev/st0 status

SCSI 2 tape drive:

File number=0, block number=0, partition=0.

Tape block size 512 bytes. Density code 0x40 (DLT1 40 GB, or Ultrium).

Soft error count since last status=0

General status bits on (45010000):

 BOT WR_PROT ONLINE IM_REP_EN

 

However, when LTO 2 tape written in gau.sb.fsu.edu and read in gau.sb.fsu.edu doesn’t give any problem, as shown below:

tsomasun@gau:~$ mt -f /dev/st0 status

SCSI 2 tape drive:

File number=0, block number=0, partition=0.

Tape block size 512 bytes. Density code 0x42 (LTO-2).

Soft error count since last status=0

General status bits on (45010000):

 BOT WR_PROT ONLINE IM_REP_EN

tsomasun@gau:~$ tar tvf /dev/st0

dr-xr-xr-x root/root         0 2008-03-20 11:42 xray2/

drwxr-xr-x soma/users        0 2008-02-13 21:51 xray2/Bb/

drwxr-xr-x soma/users        0 2008-02-13 19:30 xray2/Bb/pf/

-rw-r--r-- soma/users 18878464 2008-02-13 19:29 xray2/Bb/pf/11311_s.0001

-rw-r--r-- soma/users    15633 2008-02-13 21:55 xray2/Bb/pf/sergui_log.html

drwxr-xr-x soma/users        0 2008-02-13 20:47 xray2/Bb/pf0001a3/

-rw-r--r-- soma/users 18878464 2008-02-13 20:42 xray2/Bb/pf1a3/pf11311_1.0111

-rw-r--r-- soma/users      142 2008-02-13 20:33 xray2/Bb/pf1a3/autosave.run

-rw-r--r-- soma/users 18878464 2008-02-13 20:37 xray2/Bb/pf1a3/pf11311_1.0051

So, where does the problem lie in our inability to read in gau using a Quantum LTO-2 drive the LTO 1 tape written in rad with HP StorageWorks Ultrium 215e drive? If one notices the boxed up portion of the output from mt -f /dev/st0 status command when the tape was written LTO 1 drive had the setting of Tape block size 0 bytes. However, in the LTO 2 drive default setting is Tape block size 512 bytes. So the tape written in LTO 2 can be read in LTO 2 without problem, but the tape written LTO 1 can’t be read in LTO 2 due to Tape Block size problem.

Solution to LTO 1 tape in LTO 2 drive read error

So, how does one solve this problem? It turns out that there is a simple solution. Simply set the Tape Block size to variable in the LTO 2 drive. This is done with setblk 0 Linux command. Then the status is checked and tar command is issued. Now one can read the tape as shown below:

First the Tape Block size is set to variable (setblk 0)

tsomasun@gau:~$ mt -f /dev/st0 setblk 0

 

Then the status is checked with (mt –f /dev/st0 status)

 

tsomasun@gau:~$ mt -f /dev/st0 status

SCSI 2 tape drive:

File number=0, block number=0, partition=0.

Tape block size 0 bytes. Density code 0x40 (DLT1 40 GB, or Ultrium).

Soft error count since last status=0

General status bits on (41010000):

 BOT ONLINE IM_REP_EN

Note the change to variable block size (Tape block size 0 bytes). Now we can read the tape without a problem.

tsomasun@gau:~$ tar tvf /dev/st0

drwxr-xr-x soma/users        0 2007-01-31 15:35 dds3-001/

drwxrwxr-x soma/users        0 2000-04-04 17:01 dds3-001/raxis/

drwxr-xr-x soma/users        0 2000-03-22 17:06 dds3-001/raxis/GattisMar00-1/

-rw-r--r-- soma/users   413696 2000-03-22 16:53 dds3-001/raxis/GattisMar00-1/ipdata.img

-rw-r--r-- soma/users  1947648 2000-03-22 16:21 dds3-001/raxis/GattisMar00-1/soln000.stl

Note also that with the variable block size set, we could even read the LTO 2 tape written with Tape Block size 512 bytes, i.e., it is always better to have the Tape Block size set to variable (aka 0) since it can read ANY block size. But the contrary is NOT true, .i.e., one can’t read a tape written with variable block size using a tape drive set to a particular block size.

Conclusion

So when one encounters problems with reading a tape one possible solution is to modify the Tape Block Size to variable block size. But there is one caveat to this story. Since mt or tar generally don’t give clues why they are failing it is important that we need to note down the original parameters during archiving of the data to avoid future problems. So follow the rules when you archive:

*       Try to use generic tar command and note down the o/s and hardware

*       Write down output of mt -f /dev/st0 status of tape drive used for archiving

*       Match the output of mt -f /dev/st0 status of extracting machine to that of archived machine

*       Try to archive with variable block size with mt -f /dev/st0 setblk 0

Please send your suggestions and comments to USomaU.



[1] LTO 1 refers to Linear Tape Open 1

[2] The names LTO 1, LTO-1, Ultrium 1, Ultrium-1 are used interchangeably and refer to the same tape.