Ranting Datastore

bak2disc-tpb

2009-07-29T10:17:00.001-04:00

#!/bin/bash
################################################################################
#
# bak2disc - backup to disc volumes while maintaining original data structure
# Copyright (C) 2005-2006 dorphell <dorphell [AT] gmail [DOT] com>
#
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software
#   Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
#
################################################################################

#--------------------------------------------------------------------
# PUBLIC CONSTANTS [Default Parameters]
#--------------------------------------------------------------------

## Dir used for meta-data (a few MBs of ASCII files) ##
TMP_DIR="${TEMP:-/tmp/bak2disc}"
mkdir -p $TMP_DIR

## Disc type (valid types: CD74, CD80, DVD5, DVD9, DLT7, LTO2, LTO3) ##
MEDIA_TYPE="DVD5"

## Space allocated for index metadata (MiB)##
INDEX_SIZE=4

## mkisofs options used by growisofs ##
MKISOFS_OPTS=" -r -J "

## Limit for the volume-filling search in the element array ##
SEARCH_RANGE=150

Label="Volume"

#--------------------------------------------------------------------
# PRIVATE CONSTANTS [Do not Modify]
#--------------------------------------------------------------------

VERSION="0.7-tpb"
PROG_NAME="bak2disc"
GB=1073741824         # 1024^3
MB=1048576            # 1024^2

## ANSI Color modifiers ##
C_BOLD="\e[1m"        # Bold
C_NORM="\e[m\e[0;39m" # Normal
C_BLUE="\e[1;34m"     # Blue
C_RED="\e[1;31m"      # Red

## Only use newlines for field separation ##
IFS=$'\n'

## Error Messages ##
ErrorMsgs=(
"00: No error"
"01: Unknown error"
"02: Program \'\$2\' not in \\\$PATH"
"03: Working directory \'\$2\' already exists"
"04: Permission denied or file/directory does not exist: \$2"
"05: File[s] too big, choose a higher-capacity disc or suppress oversized file[s]"
"06: Temporary directory \'\$2\' exists"
"07: Could not create temporary directory \'\$2\'"
"08: User reported failed burn for disc volume \$2, --resume to try again"
"09: Option \'\$2\' requires an argument"
"10: Invalid option: \$2"
"11: Previous session metadata not found. Incorrect --temp parameter?"
"12: No options given, what do you want to do?"
"13: No \'--include\' files/directories defined"
"14: Disc \$2 index larger than allocated space, increase --index size"
"15: Please provide a single valid operating mode"
"16: Unknown disc type \'\$2\'. Valid types: cD74, CD80, DVD5, DVD9, DLT7, LTO2, LTO3"
"17: SIGINT caught"
"18: Invalid --speed argument, value must be an integer"
"19: mkisofs is really genisoimage which will not work")

#--------------------------------------------------------------------
# FUNCTIONS
#--------------------------------------------------------------------

Help() {
cat << EOF
$PROG_NAME v$VERSION (c) 2006 dorphell
Usage: $PROG_NAME <mode> [options]
Backup to disc volumes while maintaining original data structure

    Informative flags:
    -h, --help      Print help screen
    -v, --version   Print version information

    Operating mode flags: (one required)
    -f, --fresh     Start a new backup session
    -n, --noburn    Generate backup volume information but don't burn to disc
    -r, --resume    Resume the burn phase of a previous backup session (relies
                    on original metadata)
    -w, --wipe      Clean up temporary directory and exit

    Accessibility flags:
    -j, --eject     Open drive tray for disc loading/unloading and skip the
                    verification prompt.
    -l, --exlarge   Automatically exclude files too large for backup media
    -c, --nocolor   Turn off all text color/highlighting

    Configuration options: (every option requires an argument)
    -i, --include   Colon-separated file/directory paths to include (multiple
                    definitions and bash globbing wildcards are also vallid)
    -e, --exclude   File/directory paths to exclude (same syntax as --include)
    -t, --temp      Temporary directory [default: $TMP_DIR]
    -o, --mkisofs   Burning options passed onto mkisofs/growisofs
                    [default: $MKISOFS_OPTS]
    -m, --mtype     Disc type (CD74, CD80, DVD5, DVD9, DLT7, LTO2, LTO3) [default: $MEDIA_TYPE]
    -x, --index     Space allocated for index metadata out of disc capacity
                    [default: $INDEX_SIZE]
    -g, --range     Number of files/directories to poll when filling slack
                    space on volumes [default: $SEARCH_RANGE]
    -s, --speed     Speed factor of the writing process (integer value)
                    [default: Maximum supported by the drive]
    --label         Label the .iso and volume name

    Examples:
    $PROG_NAME --fresh --include "/etc:/mnt/HD*:/opt" -d /dev/hdc --mtype DVD5
    $PROG_NAME -n -i /usr -i /var -e /var/log --temp /tmp --range 10 -m DVD5
    $PROG_NAME --resume --temp /var/tmp

    GNUtar:
    gtar cf <tape> -T $TMP_DIR/Volume_1

EOF
}

Version() {
   echo "$PROG_NAME $VERSION"
}

## ErrorExit "int error_code" "string reference_name" ##
ErrorExit() {
   echo -ne "$C_BLUE" >&2
   echo -ne ">> "
   echo -ne "$C_RED" >&2
   eval echo -n "ERROR ${ErrorMsgs[$1]}"
   echo -ne "$C_NORM" >&2
   echo
   exit $1
}

## PrintBold "string message" ##
PrintBold() {
   echo -ne "$C_BOLD" >&2
   echo -ne "$1"
   echo -ne "$C_NORM" >&2
}

## PrintStatus "string message" ##
PrintStatus() {
   echo -ne "$C_BLUE" >&2
   echo -ne ">>"
   echo -ne "$C_NORM$C_BOLD " >&2
   echo -ne "$1"
   echo -ne "$C_NORM" >&2
   echo
}

## YesNo "string question" "boolean default" [y/n] ##
YesNo() {
   OPTIONS="[y/n]"
   [ "$2" == "y" ] && OPTIONS="[Y/n] "
   [ "$2" == "n" ] && OPTIONS="[y/N] "

   PrintBold "$1 $OPTIONS"
   IFS=$' \t\n' read ANSWER
   [ ! "$ANSWER" ] && [ "$2" ] && ANSWER="$2"
   while [ "$ANSWER" != 'y' ] && [ "$ANSWER" != 'n' ] && [ "$ANSWER" != 'Y' ] && [ "$ANSWER" != 'N' ]; do
      PrintBold "Invalid response, please answer 'y' or 'n': "
      IFS=$' \t\n' read ANSWER
   done
   [[ "$ANSWER" == 'y' || "$ANSWER" == 'Y' ]] && return 0 || return 1
}

## CheckPath "string program_name" ##
CheckPath() {
   while [ "$1" ]; do
      ! type "$1" &>/dev/null && ErrorExit 2 "$1"
      shift
   done
}

## ParseArgs "string args" (e.g. "$@") ##
ParseArgs() {
   while [ "$1" ]; do
   ## Informative flags ##
      case "$1" in
         -h|--help)
            Help;                  exit ;;
         -v|--version)
            Version;               exit ;;
   ## Operating modes flags ##
         -f|--fresh)
            FRESH=1;              shift ;;
         -r|--resume)
            RESUME=1;             shift ;;
         -n|--noburn)
            NOBURN=1;             shift ;;
         -w|--wipe)
            WIPE=1;               shift ;;
   ## Accessibility flags ##
         -j|--eject)
            CheckPath 'eject'
            EJECT=1;              shift ;;
         -l|--exlarge)
            AUTO_EXCLUDE=1;       shift ;;
         -c|--nocolor)
            NOCOLOR=1;            shift ;;
   ## Configuration options ##
         -i|--include)
            [ ! "$2" ] && ErrorExit 9 "$1"
          INCLUDE="$INCLUDE:$2"; shift 2;;
         -e|--exclude)
            [ ! "$2" ] && ErrorExit 9 "$1"
          EXCLUDE="$EXCLUDE:$2"; shift 2;;
         -o|--mkisofs)
            [ ! "$2" ] && ErrorExit 9 "$1"
            MKISOFS_OPTS="$2";   shift 2;;
         -m|--mtype)
            [ ! "$2" ] && ErrorExit 9 "$1"
            MEDIA_TYPE="$2";     shift 2;;
         -x|--index)
            [ ! "$2" ] && ErrorExit 9 "$1"
            INDEX_SIZE="$2";     shift 2;;
         -t|--temp)
            [ ! "$2" ] && ErrorExit 9 "$1"
            TMP_DIR="$2";        shift 2;;
         -g|--range)
            [ ! "$2" ] && ErrorExit 9 "$1"
            SEARCH_RANGE="$2";   shift 2;;
         -s|--speed)
            [ ! "$2" ] && ErrorExit 9 "$1"
      [ "${2##*[^0-9]*}" ] || ErrorExit 18
            SPEED="-speed=$2";   shift 2;;
         --label)
             [ ! "$2" ] && ErrorExit 9 "$1"
             Label="$2"; shift 2;;
         *)
            Help;      ErrorExit 10 "$1";;
      esac
   done
}

## SetMediaSize "string preset" (e.g. SetMediaSize "dvd9") ##
SetMediaSize() {
   MEDIA_TYPE="$(echo $1| tr a-z A-Z)"
   case "$MEDIA_TYPE" in
      CD74) MEDIA_SIZE=650 ;;
      CD80) MEDIA_SIZE=703 ;;
      DVD5) MEDIA_SIZE=4482 ;;
      DVD9) MEDIA_SIZE=8144 ;;
      DLT7) MEDIA_SIZE=30000 ;;
      LTO2) MEDIA_SIZE=190000 ;;
      LTO2c) MEDIA_SIZE=323000 ;;
      LTO3) MEDIA_SIZE=390000;;
      LTO3c) MEDIA_SIZE=663000;;
         *) Help;
            ErrorExit 16 "$1" ;;
   esac
   let "MEDIA_SIZE-=INDEX_SIZE"
}

## ValidatePath "string path" "string output_variable" ##
## If path is readable, return its proper pathname; else error + exit ##
ValidatePath() {
   [ ! -r "$1" ] && CleanUp && ErrorExit 4 "$1"
   [ "$1" == '/' ] && return
   eval "$2=\"$(echo "$1"| sed 's|/\+|/|g;s|/$||')\""
}

## GetList "string du_output_syntax" (e.g. "1234 <TAB> /path/to/dir") ##
## Returns a list of files/dirs that are less than media size ##
GetList() {
   ## Set Size and Name ##
   VARS=(${1//$'\t'/$'\n'})
   Size=${VARS[0]}
   Name=${VARS[1]}
   if [ $Size -gt $((MEDIA_SIZE*MB)) ] && [ -d "$Name" ]; then
      for x in $($DU -ba --max-depth=1 "$Name"| gawk -F '\t' '$2 != "'$Name'" {print}'); do
         GetList "$x"
      done
   else
      printf "%s\t%s\n" "$Name" "$Size"
   fi
}

## Burn disc volumes defined in TMP_DIR ##
BurnVolumes() {
   NumOfDiscs=$(head -n 1 "$TMP_DIR/summary" 2>/dev/null| gawk '{print $8}')
   MEDIA_TYPE=$(head -n 1 "$TMP_DIR/summary" 2>/dev/null| gawk '{print $9}')

   ## Check if metadata exists ##
   [ ! $NumOfDiscs ] && ErrorExit 11

   ## If --resume, reprint distribution summary ##
   if [ $RESUME ]; then
      PrintStatus "Resuming burn session..."
      while read line; do
         vol=$(echo $line| gawk '$1 == "Disc" {printf "%i\n", $2}')
         [ ${line:0:1} == " " ] && echo "$line"        && continue
         [ ${line:0:1} == "I" ] && PrintStatus "$line" && continue
         [ -z $vol ] && PrintBold "$line\n" && continue
         [ -e "$TMP_DIR/Volume_$vol" ] && PrintBold "$line\n" || echo "$line [Done]"
      done < "$TMP_DIR/summary"
      echo
   fi

## Make ISO files
#   for (( DISC_NUM=1; DISC_NUM<=$NumOfDiscs; DISC_NUM++ )); do
   DISC_NUM=1
   while [ $DISC_NUM -le $NumOfDiscs ]; do
      ## Volume already burned ##
      [ ! -e "$TMP_DIR/Volume_$DISC_NUM" ] && continue

      ## Burn the volume ##
      ALL_MKISOFS_OPTS="$MKISOFS_OPTS -quiet -V ${Label}-${DISC_NUM}_of_$NumOfDiscs -graft-points -exclude-list $EXC_LIST -path-list $TMP_DIR/Volume_$DISC_NUM -o ${Label}_$DISC_NUM.iso"

      IFS=$' \n';
      echo "mkisofs $ALL_MKISOFS_OPTS"
      mkisofs $ALL_MKISOFS_OPTS
      IFS=$'\n'

      ## Volume successfully burned, delete volume metadata ##
      rm "$TMP_DIR/Volume_$DISC_NUM" "$TMP_DIR/volume_${DISC_NUM}_of_${NumOfDiscs}"
      DISC_NUM=$(expr $DISC_NUM + 1)
   done

   ## Backup complete, wipe out metadata ##
   PrintStatus "All $NumOfDiscs backup volumes burned."
   CleanUp
}

## CleanUp (removes "$TMP_DIR" entirely) ##
CleanUp() {
   PrintStatus "Cleaning up temporary data"
   rm -rf "$TMP_DIR"
}

Terminate() {
   echo
   ErrorExit 17
}

#--------------------------------------------------------------------
# SCRIPT START
#--------------------------------------------------------------------

## Capture ^C exit code ##
trap Terminate SIGINT

## Check if utilities are in $PATH ##
# Solaris /bin/du won't work. Look for GNU du named gdu
[ -f $(which gdu) ] && DU=$(which gdu) || DU=$(which du)
[ $(uname -s) == "SunOS" -a $(basename $DU) != "gdu" ] && ErrorExit 2 "gdu"

CheckPath 'gawk' 'sed' 'printf' 'sort' "$DU" 'stat' 'comm' 'mkisofs'

mkisofs --version | grep -s genisoimage > /dev/null
[ $? -eq 0 ] && echo "mkisofs is really genisoimage which will not work"

## Parse command-line arguments ##
[ -z "$1" ] && Help && ErrorExit 12
ParseArgs "$@"

ValidatePath "$TMP_DIR" "TMP_DIR"

TMP_DIR="$TMP_DIR/$PROG_NAME-$USER"
FILE_LIST="$TMP_DIR/include"
EXC_LIST="$TMP_DIR/exclude"

SetMediaSize "$MEDIA_TYPE"

[ $NOCOLOR ] && unset C_BOLD C_NORM C_BLUE C_RED

## No operating modes defined, error ##
[ $((FRESH+NOBURN+RESUME+WIPE)) -ne 1 ] && Help && ErrorExit 15

## If --resume specified, just burn using preexisting metadata ##
[ $RESUME ] && BurnVolumes && exit

## Delete any garbage/leftovers and start fresh ##
[ $WIPE ] && CleanUp && exit

## Check for leftovers, if not clean, ask what to do ##
if [ -e "$TMP_DIR" ]; then
   ! YesNo "Leftover metadata detected; DELETE and start fresh?" "n" && PrintBold "Nothing dnoe... exiting\n" && exit
   CleanUp
fi

## Check/Ask for required variables ##
[ ! "$INCLUDE" ] && Help && ErrorExit 13

PrintBold "$C_RED-------------------------------------------------------------------$C_NORM\n"
PrintBold " Do not modify included files until they have been written to disc\n"
PrintBold " Excessive growth may cause volume to overflow disc capacity\n"
PrintBold "$C_RED-------------------------------------------------------------------$C_NORM\n"

## Start a fresh backup now ##
mkdir -p "$TMP_DIR" || ErrorExit 7 "$TMP_DIR"

## Convert path lists to arrays ##
INCLUDE=(${INCLUDE//:/$'\n'})
EXCLUDE=(${EXCLUDE//:/$'\n'})

## Validate all dir/file paths ##
x=0
while [ $x -lt ${#INCLUDE[@]} ]; do ValidatePath "${INCLUDE[$x]}" "INCLUDE[$x]"; x=$(expr $x + 1); done
x=0
while [ $x -lt ${#EXCLUDE[@]} ]; do ValidatePath "${EXCLUDE[$x]}" "EXCLUDE[$x]"; x=$(expr $x + 1); done

## Remove all dupes ##
for x in "${INCLUDE[@]}"; do INCLUDE=($(echo "${INCLUDE[*]}"| sed "\|^$x/|d"| sort -u)); done
for x in "${EXCLUDE[@]}"; do EXCLUDE=($(echo "${EXCLUDE[*]}"| sed "\|^$x/|d"| sort -u)); done

## Generate element filelist from INCLUDE paths ##
for x in "${INCLUDE[@]}"; do
   GetList "$($DU -bs "$x")"
done| sort > "$FILE_LIST"

## Remove excluded items from filelist ##
if [ "${EXCLUDE[*]}" ]; then
   for item in $($DU -bs "${EXCLUDE[@]}"); do
      item=(${item//$'\t'/$'\n'})
      Size=${item[0]}
      Name=${item[1]}

      ## *Exact* match ##
      Exists=$(gawk -F '\t' '$1 == "'$Name'" || $1 ~ "'$Name'/" {print $1}' "$FILE_LIST")

      if [ "$Exists" ]; then
         for x in $Exists; do sed "\|^$x\t.*|d" -i "$FILE_LIST"; done
      else
         Parent=""
         while [ ! "$Parent" ] && [ "$Name" ]; do
            Name="${Name%/*}"
            Parent=$(gawk -F '\t' '$1 == "'$Name'" {print}' "$FILE_LIST")
         done
         Parent=(${Parent//$'\t'/$'\n'})
         PName=${Parent[0]}
         PSize=${Parent[1]}
         NewSize=$((PSize-Size))
         sed "s|$^$PName$\t$.*$$|\1\t$NewSize|" -i "$FILE_LIST"
      fi
   done
fi

## Initialize element array ##
ELEMENTS=($(cat "$FILE_LIST"))

## Fill content elements into disc volumes ##
Current=0; Count=0; CountRange=0; declare -i Size; LastElement=${#ELEMENTS[@]}

while [ "${ELEMENTS[*]}" ]; do
   line=(${ELEMENTS[$Count]//$'\t'/$'\n'})
   Name="${line[0]}" # File name
   Size="${line[1]}" # File size

   if [ $Size -gt $((MEDIA_SIZE*MB)) ]; then                      # file bigger than disc capacity
      AEXCLUDE[${#AEXCLUDE[@]}]="$Name"                           # add file to auto-exclude list
#      unset ELEMENTS[$Count] Name
      unset ELEMENTS[$Count]
      unset Name
   fi

   if [ ! $Name ] && [ $Count -ne $CountRange ]; then             # skip null elements
      :
   elif [ $CountRange -ne 0 ] && [ $Count -eq $CountRange ]; then # end of search range, volume filled
      CountRange=0
      let "Count=KeepCount-1"                                     # go to 1st element that didn't fit
      let "Current++"
#      echo "$Current"
#      echo "$((VolSizes[$Current]+Size)) $((MEDIA_SIZE*MB))"

   elif [ $((VolSizes[$Current]+Size)) -lt $((MEDIA_SIZE*MB)) ]; then
      VolSizes[$Current]=$((${VolSizes[$Current]}+$Size))
      VolNames[$Current]="${VolNames[$Current]}$Name\n"
      unset ELEMENTS[$Count]
   else                                                           # element too big for current vol
      if [ $CountRange -eq 0 ]; then                              # remember element position
         KeepCount=$Count
         let "CountRange=Count+SEARCH_RANGE"
         if [ $CountRange -gt $LastElement ]; then                # don't set range outside the array!
            CountRange=$LastElement
         fi
      fi
   fi
   let "Count++"
done

## Ask if big files should be auto-excluded ##
if [ "$AEXCLUDE" ]; then
   if [ ! $AUTO_EXCLUDE ]; then
      for x in "${AEXCLUDE[@]}"; do echo "| $x"; done
      YesNo "+ The preceding file[s] will not fit on a $MEDIA_TYPE disc, exclude and continue?" "y" || ErrorExit 5
      echo
   fi
   EXCLUDE=("${EXCLUDE[@]}" "${AEXCLUDE[@]}")
fi

## Account for excluded files in distribution summary ##
find "${EXCLUDE[@]}" ! -type d 2>/dev/null| sort > "$EXC_LIST"

## Generate index and filelists ##
x=0;
while [ $x -lt ${#VolNames[@]} ]; do
   BurnName="Volume_$((x+1))"                      # e.g. Volume_1
   IndexName="volume_$((x+1))_of_${#VolNames[@]}" # e.g. volume_1_of_16

   ## Generate mkisofs graft-points formated filelist ##
   echo -e "${VolNames[$x]}"| gawk -F '\n' '$1 {printf "data%s=%s\n", $1, $1}' > "$TMP_DIR/$BurnName"

   ## Generate volume reference index ##
   VolFiles[$x]=$(comm -23 <(find $(echo -e "${VolNames[$x]}") ! -type d 2>/dev/null| sort) "$EXC_LIST")
   echo "${VolFiles[$x]}" > "$TMP_DIR/$IndexName"

   ## Generate master catalog ##
   echo "### $IndexName ###" >> "$TMP_DIR/catalog.idx"
   echo -e "${VolFiles[$x]}\n"| sed 's|^/|'$((x+1))': /|' >> "$TMP_DIR/catalog.idx"

   ## Append reference index and master catalog to mkisofs input filelist ##
   #echo "$IndexName=$TMP_DIR/$IndexName" >> "$TMP_DIR/$BurnName"
   echo "catalog.idx=$TMP_DIR/catalog.idx" >> "$TMP_DIR/$BurnName"
   x=$(expr $x + 1 )
done

## Generate mkisofs exclude list file ##
echo "${EXCLUDE[*]}" > "$EXC_LIST"

TotalSize=$(echo "${VolSizes[*]}"| gawk '{sum+=$1} END {printf "%.2f", (sum/'$GB') }')
Index_Size=$(stat -c %s "$TMP_DIR/catalog.idx" 2>/dev/null)
TotalFiles=$(echo "${VolFiles[*]}"| grep -c "[^\n]")
NumOfDiscs=${#VolNames[@]}

## Overall backup summary ##
SUM_TOTALS=$(printf "Summary: %s files (%s GiB) divided into %s %s disc volumes" "$TotalFiles" "$TotalSize" "$NumOfDiscs" "$MEDIA_TYPE")
SUM_INC=$(for x in "${INCLUDE[@]}"; do echo "   $x"; done)
SUM_EXC=$(for x in "${EXCLUDE[@]}"; do echo "   $x"; done)

PrintBold "$SUM_TOTALS\n"; echo -e "$SUM_TOTALS" > "$TMP_DIR/summary"
                PrintStatus "Items Included:" && echo "$SUM_INC" && echo -e "Items Included:\n$SUM_INC" >> "$TMP_DIR/summary"
[ $EXCLUDE ] && PrintStatus "Items Excluded:" && echo "$SUM_EXC" && echo -e "Items Excluded:\n$SUM_EXC" >> "$TMP_DIR/summary"

## Disc volume distribution summary ##
x=0
while [ $x -lt $NumOfDiscs ]; do
   DISC_NUM=$((x+1))
   DISC_FILES=$(echo "${VolFiles[$x]}"| grep -c "[^\n]")
   VIDX_SIZE=$(stat -c %s "$TMP_DIR/volume_${DISC_NUM}_of_${NumOfDiscs}")
   DISC_SIZE=$(echo "${VolSizes[$x]}" $Index_Size $VIDX_SIZE $MB| gawk '{printf "%f", ($1+$2+$3)/$4 }')
   DISC_FREE=$(echo $MEDIA_SIZE $INDEX_SIZE $DISC_SIZE| gawk '{printf "%i", $1+$2-$3}')
   PRINT_DIST="$PRINT_DIST$(printf "Disc %3d: %5d files -> %8.2f MiB, %4i MiB free" "$DISC_NUM" "$DISC_FILES" "$DISC_SIZE" "$DISC_FREE")\n"
   [ $DISC_FREE -lt 1 ] && ErrorExit 14 $DISC_NUM
   x=$(expr $x + 1 )
done; PrintBold $PRINT_DIST

## Save distribution summary ##
echo -ne $PRINT_DIST >> "$TMP_DIR/summary"

## If --noburn specified, exit now ##
[ $NOBURN ] && PrintStatus "Backup metadata generated." && exit

## Ask whether to burn now or leave metadata for later ##
YesNo "\nBegin burn sequence now?" "y" && BurnVolumes

#--------------------------------------------------------------------
# SCRIPT END
#--------------------------------------------------------------------

Archiving a tree to DVD with bak2disc

2009-07-29T09:16:00.005-04:00

bak2disc by dorphell is a script to create DVDs of a directory tree, preserving the structure. It will create an index file on the DVD so you can find which DVD the files are on.

I modified the script to create .iso files. You can also feed the file lists created to tar. So I modified the script to create lists contining X GB of data. I feed that to tar and can tar files to a DLT, SDLT, LTO, etc tape.

I sent an email and didn't get a reply. I'll try to get a version up somewhere.

Micro Irrigation

2008-06-11T12:58:00.001-04:00

Article showing typical layout

HOWTO install a system
From Colorado State

RainBird automation
Tutorials
Irrigo systems - nice picture
Drip Works - end user vendor
Toro Micro Irrigation

Gardening

2008-06-04T16:03:00.001-04:00

Just a few quick sites

Getting rid of weeds from http://www.thisgardenisillegal.com

Composting

http://myfolia.com/ is a garden tracking journal

According to http://www.garden.org, We are Zone 6a. And they have a nice plant zone finder.

Oh My!! GardenMon

Unsolicited phone calls

2008-02-08T16:29:00.001-05:00

From Slashdot:

Here is an effective (though laborious) way to deal with that.

1. Register on National Do-Not-Call list.

2. Wait 3 month beginning period.

3. Get caller ID.

4. Wait for another call.

5. Be pleasant to the person, if you can order something cheap, say $10, do it.

6. Get their address and phone number as you place the order.

7. Photograph the Caller ID display as evidence.

8. Take good notes including date, time, person talked to, company name, as more evidence.

9. Copy the bill you receive for $10 as conclusive evidence of marketing intent.

10. Go to your county courthouse, lodge a small claim for $500 for a telemarketing violation.

11. Send them proper notice they are being sued.

12. Since they are often out of state, they won't show and you get default judgment.

13. If they do show, you have proof of listing, notice, call, and call purpose.

14. For bonus dollars, ($500 per item) look into whether they have,
train to, practice and publish upon demand the required company calling
policies.

15. Profit!!!

Powered by ScribeFire.

Podcasting with an M-Audio Microtrack

2008-02-03T19:51:00.000-05:00
Think of the MT (Microtrack) as you would a cassette recorder. Don't worry about listening. That's for later on the computer. The MT is to capture the sound. Much you'd use a camera to take lots of pictures of an event. You'll select relevant photos for your presentation, possibly editing and cleaning them up. You might change the order of the photos to better fit your presentation.

Make sure the battery is charged up.

Use the electret microphone on the wire, not the T mic. This lets you put the microphone near the sound.

On one side of the MT (microtrack) is a L/M/H switch. Put it on M for Microphone.
Put the microphone in its jack.

Press record to start recording. Do a test recording to check sound levels.

On the screen you'll see 2 bars, labeled L and R, for Left and Right.
The levels are the sensitivity of the sound. You want to adjust the level or sensitivity of the microphone so you get all the sound but not so sensitive that the sound is distorted.

Too low a level & the mic will not pickup (hear) any sound. Too high a level and there will be too much static and distortion. You might be able to hear someone's heartbeat ;-)

You want to adjust the levels so they are near the right side, but not hitting the end. Read your script while doing this until the levels are adjusted.

Hit record again to stop the recording. The MT will display "Writing" as it saves the recording as an MP3 file.

I'd suggest keeping a list of what you record:
Sound check
Script take #1
Script take #2
recording of speech to students
The 1st MP3 file on the MT is 1), etc...

Now that you've recorded, bring the MT over to the computer. Plug the USB cable into the MT and the computer.

The MT will appear as a disk called "MICRODRIVE" with all your recordings named file001.mp3, file002.mp3, etc. There is one for each time you pressed rec to start & then rec to end. Your log will help you figure out which is which.

They might not start number 001, but they will be numbered sequentially.

Create a folder in your My Documents folder and copy the files from the MICRODRIVE into it. Do not erase them from the MICRODRIVE yet. These are your originals. Some people make a CD of them so they have a read only copy.

Now you have your sound files to edit. MT supplies Audacity which is a good choice. You can copy it from the CD or download it off the internet.

I've never used Audacity, but basically it lets you edit your sound files. You can filter it to change the sound if your original recording wasn't optimal. Mostly, you'll be cutting out dead air and splicing snippets into place. You might find splicing the 1st half of take #2 with the 2nd half of take #3 to be the best choice. The better your original, the less to do latter.

Linux Video Editing

2008-01-15T13:51:00.001-05:00

The Grumpy Editor's video journey, part 1 [LWN.net]
The Grumpy Editor's video journey part 3: DVD authoring [LWN.net]

Powered by ScribeFire.

Smart Network Switches

2008-01-15T13:40:00.001-05:00

A Switch Even a Penguin Can Love Part 1

A Switch Even a Penguin Can Love: QoS, User Management and More Part 2

Free Media sources

2007-12-28T16:46:00.001-05:00

20+ Places for Public Domain E-Books
Internet Archive: Moving Image Archive

TiVo

2007-12-27T15:56:00.001-05:00

Tivo Desktop
Running on Windows. Grabs the shows to a local VM disk. It won't save to a mapped drive. Lame.

VideoReDo
Converts the .TiVo to an mpg that the TiVo can view. Also strips commercials. Costs about $30. Windows.

TVAP
Watches the local VM disk for new shows. It then runs VideoReDo on the shows and saves them to the file server. It creates a text file with the show metadata in it. Windows.

pyTiVo
Publishes the shows on the file server for the TiVo to view again.

Runs on the file server (Solaris) but could be anything that can run python and ffmpeg (mplayer?). Unix, MacOSX, Windows. Uses the show metadata from TVAP.

Apparently pyTivo doesn't like "-" in the name of a share. Anything I tried to transfer from such a share would pause after 10 seconds and eventually kill the transfer. Removing the "-" from the sharename fixes it.

pyTivo has a version that will create a share for every subfolder you have. It works well.

Encoding and Ripping

Lifehacker Top 10: Top 10 Free Video Rippers, Encoders, and Converters

DIY tools

2007-12-27T15:47:00.001-05:00

Build a Microwave Transformer Homemade Welder

Powered by ScribeFire.

Organizing blogs

2007-12-27T15:13:00.001-05:00

I've been reading a number of blogs about organizing, GTD (Getting Things Done) and dealing with clutter. Here's a list of links:

Unclutterer - and dealing with paper clutter
Declutter an area from Good housekeeping

43 Folders deals with GTD and is a source of the hipster PDA (paper based)
Lifehacker has tips on GTD and changing your life habits
Zen Habit's simplification of GTD Nice graphic of flow

Resources for Decluttering your home from http://zenhabits.net

WWLD? No. 2: Keeping Connected | 43 Folders
WWLD? No. 3: Organizing your environment | 43 Folders
WWLD? No. 4: Living Your Life | 43 Folders

Powered by ScribeFire.

Sysadmin

2007-12-05T10:14:00.000-05:00

Brendan Gregg has a ton of Solaris tools and stuff that works elsewhere. Portping can substitute where pings are not returned but another service is.

Clonezilla is a ghost clone

How to Make a Network Cable

Macintosh software

2007-10-26T10:25:00.000-04:00
XStrimmer - trim universal binaries & other languages from apps
Macports - Free software for Mac. was darwinports

More SCSI issues

2007-10-26T09:11:00.000-04:00
I got another card with an LSI 53c895. It should be supported by the glm driver in 64 bit solaris. *sigh* Nope.

So now, I've got the Adaptec 2940au in the Linux box. It's seen but not the tape drive. I have a 2940U2 card to try also.

When did SCSI get difficult?

Solaris 10 x86 SCSI cards

2007-10-19T09:30:00.000-04:00
I just got a cast off DLT 7000 drive to put in my Solaris x86 box. I've had SCSI on most of my systems dating back to a 286 with an Adaptec 1542b card. I ran Minix and DOS at the time with a Syquest 88MB drive. I often used that drive on Macintosh at a nearby college to download from simtel20.

Later I got a 486 and a 2GB 4mm DAT drive. I used the same card. The syquest and DAT worked with versions of tar for MacOS 7, SunOS 4.x, OS/2, DOS and Linux. Back when my drives were < 500MB, 2GB was quite large.

Along the way, I picked up other SCSI adapters; they were new or salvaged. I have a Symbios 53c825, an Adaptec 2940u, a Compaq Ultra 2. I thought they'd work great in my AMD dual core Solaris box.

*sigh*. The Symbios uses the ncrs driver which is 32 bit only. The 2940 uses the adp which is only 32bit. The Compaq uses the cadp which is also only 32 bit. They will work in Linux and probably MacOSX. I've found some stuff on the net about the adp and cadp drivers having issues and the glm driver for LSI chips is in house. The glm driver is the only one of the lot to be 64 bits.

So, for Solaris, I need a new card. The best bet I think is an LSI (ne symbios) card that uses the glm driver. The 53c825 isn't supported by glm, but many older/newer ones are. Sun used LSI chips. Some of the LSI cards are much cheaper then the Adaptec (the adp320 driver will work) I would need to get. I'm getting one on eBay at 1/3 the price of the Adaptecs that would work. More information later.

I could run backups on my Linux box, but I want to have the drive local to the disks, etc. Gigabit is about 60 MB/s and the drive is 8 (5?) MB/s so there's no lack of bandwidth, but all the data is on the Solaris file server.

Today's links

2007-09-11T08:56:00.000-04:00
DVD Flick - DVD Authoring w/ chapterising
Libra - Personal media inventory w/ lending. Barcode reading via webcam, cuecat
ZFS quick guide.
Automatic ZFS snapshotting - Not quite Netapp, but better then anything else out there IMO

Podcasting links

2007-09-07T09:15:00.000-04:00
A friend of my wife is interested in starting podcasting so I googled some links.
She has a Macintosh so I'm focusing on Mac tools

O'Reilly Digital Media Article - a good site for all digital media
Audacity is a good audio editor. It's cross platform and open source too. Free.
Engadget has a HOWTO that mentions free tools for receiving and creating
A Linux setup for Recording Video - many of the tools are available for MacOSX
From about.com - more info
iLounge is a cool iPod site with a Beginner's Guide

Most of these articles mention Free/Open Source software. You don't need to buy software to podcast. You can get by with a microphone, a computer, free software and maybe a webcam. Better microphones and a digital camcorder are obvious upgrades.

There might be Legal issues. This is the Creative Commons take on it. It's based on EFF's Guide. The EFF is all about protecting your legal rights in digital media. Since you're publishing, you care about Intellectual Property (IP) and Copyright. Lawrence Lessig is worth paying attention to.

I'd focus on creating the content, the audio MP3. I'd leave distribution to one of the podcasting sites. The about.com link has some sites.

Fly Fusion Pen

2007-09-07T09:06:00.000-04:00
My wife & I got a Fly Fusion pen to try. You write on special paper and the pen records your writing.

You can hook it up to a USB cable and import the drawing & writing. Output is an image or OCR will transform the text into a document.

In addition, the pen had a speaker and headphone jack to play music and give feedback for other functions. There is a music playing app: You draw a keyboard and can then play it with the pen. It will translate spanish writing to english speaking or vice versa. A calculator.

Your lettering needs to be block style like elementry school. I have pretty bad handwriting and decent block lettering. The OCR works well for me.

It's targeted at kids for homework and the applications are geared that way. For note taking, I think it's good for business use too. It's less obtrusive in a meeting then a laptop or especially a table computer. It's quicker then using a Palm. It captures drawings better then both. At $70 plus $8 per notebook, it's a nobrainer.

pyTivo issues

2007-09-05T09:10:00.000-04:00
Apparently pyTivo doesn't like "-" in the name of a share. Anything I tried to transfer from such a share would pause after 10 seconds and eventually kill the transfer. Removing the "-" from the sharename fixes it.

TiVo links

2007-08-31T09:22:00.000-04:00
pyTivo is discussed here
TVAP is discussed here and here

More post TiVo processing

2007-08-31T09:00:00.000-04:00
I found an app called TiVo-VideoReDo-AutoProcessor or TVAP. It's a wrapper for VideoReDo that:
watches a folder for transfers
Processes the files after transfer
with Ad-detective
or QSP
Saves the result to an mpg file
The VideoReDo forums discuss it & you can get it from there.

I'd prefer a .tivo save as, but an mpg with commercials semi automatically stripped gets me 80% of the way.

Actually, if I can get pyTiVo to transfer the .mpg a bett better &/or keep the .TiVo metadata in the .mpg, I'd be ok with .mpg.

Cool DIY stuff

2007-08-17T08:54:00.000-04:00
Solar water heater

Safer Windows

2007-08-16T10:53:00.000-04:00
Safe Surfing Drop admin privileges before running a browser
Disable disk writes Sandboxie intercepts writes from an app and saves them in a cache. That cache goes away when you quit the app. Reads come from the disk though
Uninstaller Complete unistaller

VideoReDo

2007-08-14T09:19:00.001-04:00
It works directly on .TiVo files if you have windows media player installed. Great tool!

Tivo videos

2007-08-13T08:29:00.000-04:00
The Mac transfer is taking too long. Ok, I'm punting. I running TiVo Desktop in VMware. I can't save to a network drive so I'll have to manually move files from the "local" disk to the server.

On another note, VideoReDo works very well. I have to tivodecode to mpg and rename the saved .tivo to .mpg. But it's great for stripping commercials out. I think I'm going to buy it.

TiVo Video and togo

2007-08-08T14:57:00.000-04:00
I have two TiVos and a file server running Solaris at home. I also have a Linux server running VMware with XPpro in it. Finally I have a PPC Mac running OSX 10.4 Tiger. That should cover the gaumet 'eh? Oh, I have Gigabit ethernet (it's not that expensive anymore!). If only the TiVo wasn't limited to USB 2.0 10/100 adapters. I get 60MB/s between the Linux and Solaris box.

Transfer
To get the .TiVos to the file server I got TiVo desktop for Windows. It really only goes to local disk; mapped drives not allowed. I tried the Macintosh version but it doesn't transfer! TiVo says buy Toast to get what's free on Windows.

I have Galleon installed on Solaris. It was much easier (really) then the Linux install I used to use. The gui to select transfers is *slowwww*. It works well enough for auto transfers though. It's a Java app and works on all OSen. Development has finally been picked up so maybe it will get faster.

I also tried using a web browser directly. That worked as long as I did one at a time & didn't switch away. Definitely single threaded.

I bought Toast. It's slower (hours, days) then the PC version, but it works on a mapped drive. And it's faster then Galleon.

Burning
At first, I got Sonic MyDVD as recommended by TiVo. Ugh. Crashes consistantly. So I bought Nero.

Nero had issued burning directly to a DVD, but going to an .nrg file worked well enough. An update from Nero fixed that. Nero would always rencode so it was slow. Finally it didn't handle .TiVo. I needed to transcode to mpg. See below.

I then bought Toast on the Mac. That works well! Transcoding, etc. Ideally I could go to an ISO file so I could burn on any system, but they have a .toast format. Maybe some digging through the docs.

Ideally, I want to create ISO files that can be burned on any of my OSen. Toast will work though.

Transcoding
I mentioned that Nero wouldn't do .TiVo right? Luckily there are various ways of transcoding to .mpg.

TiVo Desktop has a .dll that works with DirectShow on windows so Windows Media Player can view the videos. There is a kludge that will let you use the DirectShow stuff to transcode & demux, etc. It's a tedious GUI that should be automated.

DirectShowDump is a nice GUI tool that will let you point at multiple .TiVo files and turn them into .mpg

Someone created tivodecode that does the job on the command line. It's GPL code and has been ported to Windows, MacOSX, Linux and any POSIX system. An easy compile runs it on Solaris. Now it can be batched. On the Mac, there are a few GUI wrappers that will download and transcode.

GoBack
Ok, I've got all these .TiVo and .mpg files on my file server. How can I pull them back to the TiVo like I do the multi room viewing?

Tivo Desktop will serve .TiVo files up. Did I mention that it doesn't do mapped drives?

The Mac version will work.

Galleon will do it with .TiVo and .mpg files in the proper format. mpg files don't have the meta data with the description; you only get a filename to describe an mpg.

Recent versions of the TiVo OS will not deal with subfolders. Everything in 1 directory. That doesn't scale well and I divvy'd up my shows so I can't use anything above because they show up as 1 TiVo.

pyTivo to the rescue. It lets you setup multiple shares. I serve up each subfolder as a share. I see a TiVo as Good_Eats and that's my Good Eats subfolder. It was easy to get going on the Solaris box, but it should run everywhere Python and ffmpeg can run. Oh, pyTivo will also convert files to .mpg the tivo can understand, on the fly. As long as ffmpeg can transcode it. Fantastic. I have run into issues with mp4 and avi files but that might be because the Solaris box doesn't have the codecs. I might need to go to Linux.

Commercials
There are a few apps out there that will let you cut up .avi or mpg files. avidemux will search for blank frames that broadcasters are required to put between commercials and shows. ffmpeg will transcode from mpg to avi:
ffmpeg -i file -target ntsc-dvd -b 2500 -s 352x480 -hq -ab 160 file.mpg
There's still a large, tedious process for cutting.

cbreak is a windows .NET app that is supposed to do it automatically. There's even a batch file that'll convert .mpg with ffmpeg.exe 1st. It's drag & drop. Perfect. Except the resulting file isn't .TiVo compatible. My Solaris ffmpeg can't convert the avi either. Displaying it via pyTivo locks up the TiVo too!

There are commercial apps out there like VideoReDo that automate much of it. MythTV has skipping built in, but I don't know if it can be used as a filter.

Virtualization

2007-08-08T14:42:00.000-04:00
I have Fedora 7 on an AM2 dual core system. I've been doing alot with VMware Server and have owned VMware Workstation since 2.0. One of the issues with VMware is reinstalling every time the kernel is updated. Add to that the fact that VMware has only precompiled for certain kernels on certain distribution releases. *sigh*

So I decided to try out the qemu-kvm stuff in Fedora. Fedora 7 has a GUI manager for Xen and (I think qemu-kvm).

To make a long story short, qemu is easy except for the networking. I keep everything on the file server and nothing on my clients so I need networking. The GUI doesn't reveal anything about the networking and doesn't quite work with qemu yet. All the documentation I've Googled say 'qemu is in Fedora 7' but all of the networking stuff is written up for earlier versions.

No problems, except the virt manager sets up some networking stuff. And isn't documented. And uses different names from the older examples.

*sigh* So I punted and installed VMware server. There's a patch for the install that will let you compile the module for any kernel. Joy! And it works. It'd be nice if VMware included it or linked to it on the web.

In any event, I've got XPpro running in a VM. I can RDP into it from my laptop and run things like cbreak to cut commercials out of .TiVo streams after I convert 'em to mpg.

Today's Links

2007-08-01T09:47:00.000-04:00
Systems
iSync adapting
Linux Audio Troubleshooting Nice guide to figuring out the hardware in linux
Solaris keeps running. The post is in german, but they stayed running. No one noticed until they checked the room.
OpenKVM
Windows
Disable services
AD tools
Open Source tools

Google
Cheatsheet

Video
Transcoding

Today's Links

2007-07-31T08:14:00.000-04:00
Linux
QEMU bridging

Lockpicking for kids
ZFS snapshots

Today's links

2007-07-30T11:20:00.000-04:00
GTD
Liferemix has a whole bunch of productivity blogs
GTD primer

Scan all incoming paper
From unclutterer.com
Ken Silver

iPod fixes

Linux
Dolby 5.1 sound
Mounting ISOs

IPTV info

Back of the napkin Network speed testing

2007-07-25T14:07:00.000-04:00
I've been trying to compare iSCSI, NFS, and general network speed lately.

Awhile ago, my main Linux system was a dual PIII 500MHz. I was getting 20MB/s over the gigabit ethernet which seemed slow to me. At work I've seen 40MB/s to 60MB/s. I upgraded the server to an AMD dual core system and get 60MB/s.

My back of the napkin testing was pushing a gigabyte with dd:
time dd if=/dev/zero bs=1048576 count=1024 of=

GNU dd can also display MB/s.

I figure /dev/zero is the fastest source of bits. I can output to /dev/null to get the fastest data sink. Or local disk or NFS or SMB. But that still didn't measure just the network.

I found ttcp which sets up a client/server. On the sink I do ttcp -r > /dev/null. On the source, I pipe dd to ttcp -t . Netcat by Hobbit could do the same, but now there's a GNU nc that doesn't let you do -lp! I suppose I should just compile my own.

Something like:
On sink: nc -lvnp 5150 > /dev/null
On source: dd blah | nc -v -w 2 serverip

Bill McGonigle has this note about iperf. It looks interesting too.
Iperf's home
TCP tuning

The Iperf tarball has a test directory (read the README in it):
one side: perl server
other: perl client tests remote local | tee iperf.log
Tune some stuff
Run it again to iperf2.log
grep Mbits.s /tmp/iperf*.log | awk '{print $1, $(NF-1)}' | sort -n +1 | less
If iperf2.log is at the bottom, you got more megabits.

Is it accurate? I don't know. Does it help measure change? Yep. And that's what tuning is about.

Links

2007-07-25T08:26:00.001-04:00
Online Media God
Sun BigAdmin Not just for Solaris
Fix RPM db
Google Video (FLV) conversion with ffmpeg

Lifehacking/Productivity
Productivity/Life Hack site
GTD Good Easy on a mac
Bit Literacy Review
Text based todo

Why run VMs? part 1

2007-07-24T09:15:00.000-04:00
I'm a Unix guy, but often work in a windows environ. Usually there's something that only runs in windows that I need to run.

I've used DOSemu, Executor (Mac OS 7), Win4Lin, VMware Workstation (2.x -> 6.x), Bochs, QEMU, Basillisk and wine. All have various capabilities and impacts.

If you just want to run windows apps, having a 2nd computer running terminal server over a gigabit net is probably the best for features. You get full hardware access, reasonable display speed (gigabit switches and KVMs are inexpensive).

For a Macintosh, VNC over an SSH connection works well on Mac OSX. I've used VNC to control System 7 systems. I find VNC on a mac works better then on windows. Windows doesn't let VNC at the login screen while OSX does.

VMware and Win4Lin depended on the kernel so any update to that meant a reconfigure. *sigh* And hopefully that kernel was supported. Otherwise you had a choice to make. I get tired of all the reconfiguring. Of course Windows doesn't update its kernel often so the host is more stable.

I have hopes for QEMU lately. And KVM now that I have a chip with virtualization. I've gotten QEMU running with Solaris as a host, but the networking was a bit tricky. I'd love to see VMware hosted on Solaris but I doubt the port will be done. Hardware is inexpensive enough that having a Linux box is doable.

Today's Links

2007-07-24T09:14:00.000-04:00
Linux KVM Wiki
Google 411
ZFS updates

Today's links

2007-07-23T16:36:00.000-04:00
enjoy-life-now-and-save-for-later

Laserwriter Serial Cable

2007-07-23T08:52:00.000-04:00

25 pin ports
printer pc
2 3
3 2
4 5
5 4
7 7 (8?)
8 20
20 6 (7?) (8?)

Cheap storage

2007-07-22T22:58:00.001-04:00
Right now it's SATA. If you're running RAID (and you should), SCSI isn't going to buy you much more reliability. It's not going to give you much more speed on the low end either. Anyways, this is about cheap.

You have a PC with slots in it. It's going to be your server. I suggest having the OS on a RAID1 mirror. Don't have data on this so the OS can go as fast as possible. The RAID buys some reliability. If it's IDE, no slaves. Just masters. Mixing them slows things down quite a bit. Ok, you can put the CD/DVD on it.

Get a 4 port SATA card. You do not care about the RAID in it. Most of them are really software RAID anyways and each card does it differently. By doing software RAID, you can use any adapter. Hardware RAID means your data is at the mercy of a brand or even model of that card if it fails. Besides, you're going to be running an iSCSI SAN or an NFS/Samba file server here. Your CPU is dedicated to storage. Your bottleneck will be gigabit ethernet to your clients.

Get 4 SATA drives of the same size. I like to have a standard size that will be available in the future, like 500GB.

Now the hard part: how to power and cool them? If you have a case like the Antec PB180, there's a fan and a 4 drive chamber. Just get some SATA cables and maybe some 4 pin to SATA power adapters.

If not, you need to get an external drive. There are some nice ones under $150 with power and cooling for 4 SATA drives. You can also build your own with a PC power supply, a fan, power adapters and something to bolt the drives to that the fan pulls air through.

Now get some long SATA cables. I've used 42" ones. They do not have to be eSATA. I've used internal SATA from the drives through a card slot to the internal SATA card and they work fine. Using short SATA extenders lets you have a nice disconnection outside the PC or drive case.

Hook it all up, install your OS and make it a software RAID5 setup. Cheap.

Which OS? Do you want a NAS or SAN device? I really like ZFS on my servers and that means Solaris right now. Solaris 10u3 doesn't have an iSCSI target so that means NAS only. Newer versions of OpenSolaris have iSCSI targer. Linux has an iSCSI target also and I don't think it's as close to bleeding edge as OpenSolaris, but there are many that feel comfortable with it. There's a linux distribution called OpenFiler that does iSCSI target, Samba and NFS with a web admin interface for you non unix types.

Speeds? On a Dual PIII 500MHz I got 20-25MB/s. On a Xeon 2.4 or on an AMD 4000+ I get about 120MB/s. The PIII can't keep up with Gigabit. The other 2 can do 90MB/s over gigabit. That's faster then local disk on older systems.

4 * 500GB @$120/ea -> $480 for disk. Put a $500 PC with gigabit ethernet under that and you have 1.5TB or RAID5 for under $1000.

Storage rant #1

2007-07-22T22:18:00.000-04:00
I've been doing sysadmin for awhile now. I've seen disk space go from > $4 per megabyte(!) to less then $240 for a terabyte. Space requirements have gone up too. You used to be able to have your compiler, editor and source on a floppy.

My general rule has been local disk for databases and other things that need locking, NAS for everything else. The local store can be a SAN of course. Centralize storage as much as possible (but no more so) to keep backups from going over the network. Because it's centralized, you can do RAID to increase reliability.

Backups are not archives! Backups are so you can recover your setup as close as possible to the latest good state if the hardware fails completly. If you want to go back to a point in time, that's an archive.

Backups have changed dramatically over the years. I don't think there's such a thing as an inexpensive tape anymore. At least something that's dramatically cheaper then disk. I once bought a 2GB 4mm DAT to backup my home systems. I probably had 1GB at the time. Now you're going to need multiple tapes to span your disks. Because of that, you probably also want an automatic tape system as well.

After you figure out your backup cycle (how far back a backup goes (archives are forever)) and how much data you have, you arrive at the total data storage. If you go tape, figure out the automated drive cost with a full compliment of tapes. You might find that is cheaper to buy a disk farm of some sort for your backup store. Remember, backups are not archives.

You'll still probably want to put data offsite. But the disks do have the advantage of matching the speed of incoming data. Your backups will always be much quicker to disk then tape so your window will be larger. When you create media for offsite, it'll be done on the disk farm so it will go quicker.

LVM notes

2007-07-22T21:55:00.000-04:00
Commands

ls /sbin/vg* /sbin/lv*/sbin/pv*
Xscan, Xdisplay, Xcreate, Xremove, Xextend

Initialize for lvm
pvcreate -v /dev/md3
pvdisplay /dev/md3
Scan & build /etc/lvmtab stuff
vgscan
create the volume group /dev/vg from /dev/md3
vgcreate vg /dev/md3
vgdisplay
show Allocated and Free space
sudo vgdisplay | egrep '^[AF].*Size'
Create a partition
lvcreate --size 2048m vg
ls -l /dev/vg
mke2fs -j /dev/vg/lvol
mkdir /test
mount -t ext3 /dev/vg/lvol /test
lvdisplay /dev/vg/lvol[n]
expand it!
umount /dev/vg/lvol?
lvextend -L + /dev/vg/lvol?
e2fsck -f /dev/vg/lvol?
resize2fs -p /dev/vg/lvol?
mount it
remove it!
umount /dev/vg/lvol?
lvremove /dev/vg/lvol?
rmdir mountpoint
vi /etc/fstab