DVD-Vault/Opti-Vault Software Overview

 

 

Software is based on Fedora Core 2 distribution.  Addon software is extensions to standard automounter software.  There are open source programs that perform some parts of this function currently, but no open source software is installable and maintainable for a SCSI library (Medium Changer) solution due to various issues.  For example, one program resides entirely in the operating system kernel and requires a system reboot to import or export media.

Software will work on all SCSI medium changer hardware, including both DVD-RAM libraries and the new Blu-Ray libraries from Sony and Kubota.  There are three versions currently on the roadmap:

  • Version 1 has basic ability to read and write single DVD platters, on a single side.  All data is written in UDF format.  This part is functionally complete as of BETA 1 (to be posted) and is just getting additional enhancements added.
  • Version 2 supports multiple platters as a single file system.  Data is concatenated on the multiple platters.
  • Version 3 supports both concatenation and striping.  Striping is added for performance purposes.

 

The software consists of several programs:

  1. Inventory Program

 Looks for a DVD library attached to the system.  When found, will open the library and mount each piece of media.  Each media label is used to create a mount point in a /archive directory but updating the /etc/auto.archive and /etc/migration.conf files.  In the future, a  complete directory is retrieved from each DVD and the following  information is stored in a MySQL database:

  • Media ID
  • File Name
  • MD5 checksum for file
  • Create/Access/Modification time/date
  • File size in bytes

Program has the following options

  • Read file entries on DVDs
  • (future)Generate MD5 Checksum
  • (future)Store information in MySQL database
  • Raw device name of Medium Changer (Library)
  • Raw device names of DVD Drives

If raw device names are not specified, program will attempt to locate a library and associated drives.  If more than one library is found, error is returned.  If DVD drives are at duplicate addresses or on different busses, and error is returned that the configuration cannot be determined.

  1. Automounter (in BETA release, multiple drives supported)

Automounter is run at boot time by /etc/init.d/autofs.  Automounter is called by the autofs file system when references to directories in /archive are made by application programs.  Automounter will then take the least recently used/longest mounted disk drive and will issue the system calls to dismount it.  If the drive is busy (still is use by an application), another drive will be selected.  When a drive is dismounted, the pending media will be mounted in that drive and a success status will be returned to the kernel indicating that the pending application program can proceed.  The /etc/migration.conf file is updated with the time and data of last access, along with the free space remaining on the media.

  1. Media Scan (not released yet)

Utility which will scan each piece of media and verify that the UDF file system was correctly closed on the media.  This program will run on all pieces of media found in the library at boot time and will place the media back into slots in the library.  An option will allow the program to retrieve all pieces of media in the library and check them in a DVD drive.  Incorrect UDF file systems will be repaired.  Failure to repair the media will cause the program to mark the media as unmountable in the automounter table.

  1. Override media scan (not released yet)

Utility to reverse a media marked as unmountable and make it mountable again.

  1. Web management Interface  (not released yet)

A web management interface allows for shows the current automounter list, provides the ability to start and stop the automounter, and to export and import media.  This is a plugin for the webmin software.

  1. Media importer (not released yet)

Opens the import slot, retrieves media from the slot if inserted, inventories the media in a drive and stores the media in an unused slot.  The automounter table is updated with the new media id.

  1. File Duplication Reporter (not released yet)

Program that retrieves MySQL records and indicates which files are exact duplicates of other files that are stored in the library.  The file names do not have to match, just the contents.  Program runs by using the MD5 checksum, not a read from the media.

8. Data Migrator (in BETA release)
A daemon run by cron that relocates files from hard disk to DVD platters.  During relocation the MD5 checksum is calculated and stored in a MySQL database along with the original location of the file, the new location (platter, directory, filename) and metadata.  After relocation, a symbolic link is created from the original file location to the autofs directory for the platter.

9. MCLIB (In BETA release)
A library and setup of utility programs that gives basic commands for manipulating a SCSI-based DVD Library.  SCSI also could be iSCSI or Fibre Channel.  I tried using MTX but there were some problems with response times, so I ported over the Solaris/Windows MCLIB to LINUX for this program.