Friday, December 28, 2007
Thursday, December 27, 2007
TiVo
Tivo Desktop
Running on Windows. Grabs the shows to a local VM disk. It won't save to a mapped drive. Lame.
VideoReDo
Converts the .TiVo to an mpg that the TiVo can view. Also strips commercials. Costs about $30. Windows.
TVAP
Watches the local VM disk for new shows. It then runs VideoReDo on the shows and saves them to the file server. It creates a text file with the show metadata in it. Windows.
pyTiVo
Publishes the shows on the file server for the TiVo to view again.
Runs on the file server (Solaris) but could be anything that can run python and ffmpeg (mplayer?). Unix, MacOSX, Windows. Uses the show metadata from TVAP.
Apparently pyTivo doesn't like "-" in the name of a share. Anything I tried to transfer from such a share would pause after 10 seconds and eventually kill the transfer. Removing the "-" from the sharename fixes it.
pyTivo has a version that will create a share for every subfolder you have. It works well.
Encoding and Ripping
Lifehacker Top 10: Top 10 Free Video Rippers, Encoders, and Converters
Running on Windows. Grabs the shows to a local VM disk. It won't save to a mapped drive. Lame.
VideoReDo
Converts the .TiVo to an mpg that the TiVo can view. Also strips commercials. Costs about $30. Windows.
TVAP
Watches the local VM disk for new shows. It then runs VideoReDo on the shows and saves them to the file server. It creates a text file with the show metadata in it. Windows.
pyTiVo
Publishes the shows on the file server for the TiVo to view again.
Runs on the file server (Solaris) but could be anything that can run python and ffmpeg (mplayer?). Unix, MacOSX, Windows. Uses the show metadata from TVAP.
Apparently pyTivo doesn't like "-" in the name of a share. Anything I tried to transfer from such a share would pause after 10 seconds and eventually kill the transfer. Removing the "-" from the sharename fixes it.
pyTivo has a version that will create a share for every subfolder you have. It works well.
Encoding and Ripping
Lifehacker Top 10: Top 10 Free Video Rippers, Encoders, and Converters
Organizing blogs
I've been reading a number of blogs about organizing, GTD (Getting Things Done) and dealing with clutter. Here's a list of links:
Unclutterer - and dealing with paper clutter
Declutter an area from Good housekeeping
43 Folders deals with GTD and is a source of the hipster PDA (paper based)
Lifehacker has tips on GTD and changing your life habits
Zen Habit's simplification of GTD Nice graphic of flow
Resources for Decluttering your home from http://zenhabits.net
WWLD? No. 2: Keeping Connected | 43 Folders
WWLD? No. 3: Organizing your environment | 43 Folders
WWLD? No. 4: Living Your Life | 43 Folders
Unclutterer - and dealing with paper clutter
Declutter an area from Good housekeeping
43 Folders deals with GTD and is a source of the hipster PDA (paper based)
Lifehacker has tips on GTD and changing your life habits
Zen Habit's simplification of GTD Nice graphic of flow
Resources for Decluttering your home from http://zenhabits.net
WWLD? No. 2: Keeping Connected | 43 Folders
WWLD? No. 3: Organizing your environment | 43 Folders
WWLD? No. 4: Living Your Life | 43 Folders
Powered by ScribeFire.
Wednesday, December 5, 2007
Sysadmin
Brendan Gregg has a ton of Solaris tools and stuff that works elsewhere. Portping can substitute where pings are not returned but another service is.
Clonezilla is a ghost clone
How to Make a Network Cable
Clonezilla is a ghost clone
How to Make a Network Cable
Friday, October 26, 2007
Macintosh software
More SCSI issues
I got another card with an LSI 53c895. It should be supported by the glm driver in 64 bit solaris. *sigh* Nope.
So now, I've got the Adaptec 2940au in the Linux box. It's seen but not the tape drive. I have a 2940U2 card to try also.
When did SCSI get difficult?
So now, I've got the Adaptec 2940au in the Linux box. It's seen but not the tape drive. I have a 2940U2 card to try also.
When did SCSI get difficult?
Friday, October 19, 2007
Solaris 10 x86 SCSI cards
I just got a cast off DLT 7000 drive to put in my Solaris x86 box. I've had SCSI on most of my systems dating back to a 286 with an Adaptec 1542b card. I ran Minix and DOS at the time with a Syquest 88MB drive. I often used that drive on Macintosh at a nearby college to download from simtel20.
Later I got a 486 and a 2GB 4mm DAT drive. I used the same card. The syquest and DAT worked with versions of tar for MacOS 7, SunOS 4.x, OS/2, DOS and Linux. Back when my drives were < 500MB, 2GB was quite large.
Along the way, I picked up other SCSI adapters; they were new or salvaged. I have a Symbios 53c825, an Adaptec 2940u, a Compaq Ultra 2. I thought they'd work great in my AMD dual core Solaris box.
*sigh*. The Symbios uses the ncrs driver which is 32 bit only. The 2940 uses the adp which is only 32bit. The Compaq uses the cadp which is also only 32 bit. They will work in Linux and probably MacOSX. I've found some stuff on the net about the adp and cadp drivers having issues and the glm driver for LSI chips is in house. The glm driver is the only one of the lot to be 64 bits.
So, for Solaris, I need a new card. The best bet I think is an LSI (ne symbios) card that uses the glm driver. The 53c825 isn't supported by glm, but many older/newer ones are. Sun used LSI chips. Some of the LSI cards are much cheaper then the Adaptec (the adp320 driver will work) I would need to get. I'm getting one on eBay at 1/3 the price of the Adaptecs that would work. More information later.
I could run backups on my Linux box, but I want to have the drive local to the disks, etc. Gigabit is about 60 MB/s and the drive is 8 (5?) MB/s so there's no lack of bandwidth, but all the data is on the Solaris file server.
Later I got a 486 and a 2GB 4mm DAT drive. I used the same card. The syquest and DAT worked with versions of tar for MacOS 7, SunOS 4.x, OS/2, DOS and Linux. Back when my drives were < 500MB, 2GB was quite large.
Along the way, I picked up other SCSI adapters; they were new or salvaged. I have a Symbios 53c825, an Adaptec 2940u, a Compaq Ultra 2. I thought they'd work great in my AMD dual core Solaris box.
*sigh*. The Symbios uses the ncrs driver which is 32 bit only. The 2940 uses the adp which is only 32bit. The Compaq uses the cadp which is also only 32 bit. They will work in Linux and probably MacOSX. I've found some stuff on the net about the adp and cadp drivers having issues and the glm driver for LSI chips is in house. The glm driver is the only one of the lot to be 64 bits.
So, for Solaris, I need a new card. The best bet I think is an LSI (ne symbios) card that uses the glm driver. The 53c825 isn't supported by glm, but many older/newer ones are. Sun used LSI chips. Some of the LSI cards are much cheaper then the Adaptec (the adp320 driver will work) I would need to get. I'm getting one on eBay at 1/3 the price of the Adaptecs that would work. More information later.
I could run backups on my Linux box, but I want to have the drive local to the disks, etc. Gigabit is about 60 MB/s and the drive is 8 (5?) MB/s so there's no lack of bandwidth, but all the data is on the Solaris file server.
Tuesday, September 11, 2007
Today's links
DVD Flick - DVD Authoring w/ chapterising
Libra - Personal media inventory w/ lending. Barcode reading via webcam, cuecat
ZFS quick guide.
Automatic ZFS snapshotting - Not quite Netapp, but better then anything else out there IMO
Libra - Personal media inventory w/ lending. Barcode reading via webcam, cuecat
ZFS quick guide.
Automatic ZFS snapshotting - Not quite Netapp, but better then anything else out there IMO
Friday, September 7, 2007
Podcasting links
A friend of my wife is interested in starting podcasting so I googled some links.
She has a Macintosh so I'm focusing on Mac tools
O'Reilly Digital Media Article - a good site for all digital media
Audacity is a good audio editor. It's cross platform and open source too. Free.
Engadget has a HOWTO that mentions free tools for receiving and creating
A Linux setup for Recording Video - many of the tools are available for MacOSX
From about.com - more info
iLounge is a cool iPod site with a Beginner's Guide
Most of these articles mention Free/Open Source software. You don't need to buy software to podcast. You can get by with a microphone, a computer, free software and maybe a webcam. Better microphones and a digital camcorder are obvious upgrades.
There might be Legal issues. This is the Creative Commons take on it. It's based on EFF's Guide. The EFF is all about protecting your legal rights in digital media. Since you're publishing, you care about Intellectual Property (IP) and Copyright. Lawrence Lessig is worth paying attention to.
I'd focus on creating the content, the audio MP3. I'd leave distribution to one of the podcasting sites. The about.com link has some sites.
She has a Macintosh so I'm focusing on Mac tools
O'Reilly Digital Media Article - a good site for all digital media
Audacity is a good audio editor. It's cross platform and open source too. Free.
Engadget has a HOWTO that mentions free tools for receiving and creating
A Linux setup for Recording Video - many of the tools are available for MacOSX
From about.com - more info
iLounge is a cool iPod site with a Beginner's Guide
Most of these articles mention Free/Open Source software. You don't need to buy software to podcast. You can get by with a microphone, a computer, free software and maybe a webcam. Better microphones and a digital camcorder are obvious upgrades.
There might be Legal issues. This is the Creative Commons take on it. It's based on EFF's Guide. The EFF is all about protecting your legal rights in digital media. Since you're publishing, you care about Intellectual Property (IP) and Copyright. Lawrence Lessig is worth paying attention to.
I'd focus on creating the content, the audio MP3. I'd leave distribution to one of the podcasting sites. The about.com link has some sites.
Fly Fusion Pen
My wife & I got a Fly Fusion pen to try. You write on special paper and the pen records your writing.
You can hook it up to a USB cable and import the drawing & writing. Output is an image or OCR will transform the text into a document.
In addition, the pen had a speaker and headphone jack to play music and give feedback for other functions. There is a music playing app: You draw a keyboard and can then play it with the pen. It will translate spanish writing to english speaking or vice versa. A calculator.
Your lettering needs to be block style like elementry school. I have pretty bad handwriting and decent block lettering. The OCR works well for me.
It's targeted at kids for homework and the applications are geared that way. For note taking, I think it's good for business use too. It's less obtrusive in a meeting then a laptop or especially a table computer. It's quicker then using a Palm. It captures drawings better then both. At $70 plus $8 per notebook, it's a nobrainer.
You can hook it up to a USB cable and import the drawing & writing. Output is an image or OCR will transform the text into a document.
In addition, the pen had a speaker and headphone jack to play music and give feedback for other functions. There is a music playing app: You draw a keyboard and can then play it with the pen. It will translate spanish writing to english speaking or vice versa. A calculator.
Your lettering needs to be block style like elementry school. I have pretty bad handwriting and decent block lettering. The OCR works well for me.
It's targeted at kids for homework and the applications are geared that way. For note taking, I think it's good for business use too. It's less obtrusive in a meeting then a laptop or especially a table computer. It's quicker then using a Palm. It captures drawings better then both. At $70 plus $8 per notebook, it's a nobrainer.
Wednesday, September 5, 2007
pyTivo issues
Apparently pyTivo doesn't like "-" in the name of a share. Anything I tried to transfer from such a share would pause after 10 seconds and eventually kill the transfer. Removing the "-" from the sharename fixes it.
Friday, August 31, 2007
More post TiVo processing
I found an app called TiVo-VideoReDo-AutoProcessor or TVAP. It's a wrapper for VideoReDo that:
I'd prefer a .tivo save as, but an mpg with commercials semi automatically stripped gets me 80% of the way.
Actually, if I can get pyTiVo to transfer the .mpg a bett better &/or keep the .TiVo metadata in the .mpg, I'd be ok with .mpg.
- watches a folder for transfers
- Processes the files after transfer
- with Ad-detective
- or QSP
- Saves the result to an mpg file
I'd prefer a .tivo save as, but an mpg with commercials semi automatically stripped gets me 80% of the way.
Actually, if I can get pyTiVo to transfer the .mpg a bett better &/or keep the .TiVo metadata in the .mpg, I'd be ok with .mpg.
Friday, August 17, 2007
Thursday, August 16, 2007
Safer Windows
Safe Surfing Drop admin privileges before running a browser
Disable disk writes Sandboxie intercepts writes from an app and saves them in a cache. That cache goes away when you quit the app. Reads come from the disk though
Uninstaller Complete unistaller
Disable disk writes Sandboxie intercepts writes from an app and saves them in a cache. That cache goes away when you quit the app. Reads come from the disk though
Uninstaller Complete unistaller
Tuesday, August 14, 2007
Monday, August 13, 2007
Tivo videos
The Mac transfer is taking too long. Ok, I'm punting. I running TiVo Desktop in VMware. I can't save to a network drive so I'll have to manually move files from the "local" disk to the server.
On another note, VideoReDo works very well. I have to tivodecode to mpg and rename the saved .tivo to .mpg. But it's great for stripping commercials out. I think I'm going to buy it.
On another note, VideoReDo works very well. I have to tivodecode to mpg and rename the saved .tivo to .mpg. But it's great for stripping commercials out. I think I'm going to buy it.
Wednesday, August 8, 2007
TiVo Video and togo
I have two TiVos and a file server running Solaris at home. I also have a Linux server running VMware with XPpro in it. Finally I have a PPC Mac running OSX 10.4 Tiger. That should cover the gaumet 'eh? Oh, I have Gigabit ethernet (it's not that expensive anymore!). If only the TiVo wasn't limited to USB 2.0 10/100 adapters. I get 60MB/s between the Linux and Solaris box.
Transfer
To get the .TiVos to the file server I got TiVo desktop for Windows. It really only goes to local disk; mapped drives not allowed. I tried the Macintosh version but it doesn't transfer! TiVo says buy Toast to get what's free on Windows.
I have Galleon installed on Solaris. It was much easier (really) then the Linux install I used to use. The gui to select transfers is *slowwww*. It works well enough for auto transfers though. It's a Java app and works on all OSen. Development has finally been picked up so maybe it will get faster.
I also tried using a web browser directly. That worked as long as I did one at a time & didn't switch away. Definitely single threaded.
I bought Toast. It's slower (hours, days) then the PC version, but it works on a mapped drive. And it's faster then Galleon.
Burning
At first, I got Sonic MyDVD as recommended by TiVo. Ugh. Crashes consistantly. So I bought Nero.
Nero had issued burning directly to a DVD, but going to an .nrg file worked well enough. An update from Nero fixed that. Nero would always rencode so it was slow. Finally it didn't handle .TiVo. I needed to transcode to mpg. See below.
I then bought Toast on the Mac. That works well! Transcoding, etc. Ideally I could go to an ISO file so I could burn on any system, but they have a .toast format. Maybe some digging through the docs.
Ideally, I want to create ISO files that can be burned on any of my OSen. Toast will work though.
Transcoding
I mentioned that Nero wouldn't do .TiVo right? Luckily there are various ways of transcoding to .mpg.
TiVo Desktop has a .dll that works with DirectShow on windows so Windows Media Player can view the videos. There is a kludge that will let you use the DirectShow stuff to transcode & demux, etc. It's a tedious GUI that should be automated.
DirectShowDump is a nice GUI tool that will let you point at multiple .TiVo files and turn them into .mpg
Someone created tivodecode that does the job on the command line. It's GPL code and has been ported to Windows, MacOSX, Linux and any POSIX system. An easy compile runs it on Solaris. Now it can be batched. On the Mac, there are a few GUI wrappers that will download and transcode.
GoBack
Ok, I've got all these .TiVo and .mpg files on my file server. How can I pull them back to the TiVo like I do the multi room viewing?
Tivo Desktop will serve .TiVo files up. Did I mention that it doesn't do mapped drives?
The Mac version will work.
Galleon will do it with .TiVo and .mpg files in the proper format. mpg files don't have the meta data with the description; you only get a filename to describe an mpg.
Recent versions of the TiVo OS will not deal with subfolders. Everything in 1 directory. That doesn't scale well and I divvy'd up my shows so I can't use anything above because they show up as 1 TiVo.
pyTivo to the rescue. It lets you setup multiple shares. I serve up each subfolder as a share. I see a TiVo as Good_Eats and that's my Good Eats subfolder. It was easy to get going on the Solaris box, but it should run everywhere Python and ffmpeg can run. Oh, pyTivo will also convert files to .mpg the tivo can understand, on the fly. As long as ffmpeg can transcode it. Fantastic. I have run into issues with mp4 and avi files but that might be because the Solaris box doesn't have the codecs. I might need to go to Linux.
Commercials
There are a few apps out there that will let you cut up .avi or mpg files. avidemux will search for blank frames that broadcasters are required to put between commercials and shows. ffmpeg will transcode from mpg to avi:
cbreak is a windows .NET app that is supposed to do it automatically. There's even a batch file that'll convert .mpg with ffmpeg.exe 1st. It's drag & drop. Perfect. Except the resulting file isn't .TiVo compatible. My Solaris ffmpeg can't convert the avi either. Displaying it via pyTivo locks up the TiVo too!
There are commercial apps out there like VideoReDo that automate much of it. MythTV has skipping built in, but I don't know if it can be used as a filter.
Transfer
To get the .TiVos to the file server I got TiVo desktop for Windows. It really only goes to local disk; mapped drives not allowed. I tried the Macintosh version but it doesn't transfer! TiVo says buy Toast to get what's free on Windows.
I have Galleon installed on Solaris. It was much easier (really) then the Linux install I used to use. The gui to select transfers is *slowwww*. It works well enough for auto transfers though. It's a Java app and works on all OSen. Development has finally been picked up so maybe it will get faster.
I also tried using a web browser directly. That worked as long as I did one at a time & didn't switch away. Definitely single threaded.
I bought Toast. It's slower (hours, days) then the PC version, but it works on a mapped drive. And it's faster then Galleon.
Burning
At first, I got Sonic MyDVD as recommended by TiVo. Ugh. Crashes consistantly. So I bought Nero.
Nero had issued burning directly to a DVD, but going to an .nrg file worked well enough. An update from Nero fixed that. Nero would always rencode so it was slow. Finally it didn't handle .TiVo. I needed to transcode to mpg. See below.
I then bought Toast on the Mac. That works well! Transcoding, etc. Ideally I could go to an ISO file so I could burn on any system, but they have a .toast format. Maybe some digging through the docs.
Ideally, I want to create ISO files that can be burned on any of my OSen. Toast will work though.
Transcoding
I mentioned that Nero wouldn't do .TiVo right? Luckily there are various ways of transcoding to .mpg.
TiVo Desktop has a .dll that works with DirectShow on windows so Windows Media Player can view the videos. There is a kludge that will let you use the DirectShow stuff to transcode & demux, etc. It's a tedious GUI that should be automated.
DirectShowDump is a nice GUI tool that will let you point at multiple .TiVo files and turn them into .mpg
Someone created tivodecode that does the job on the command line. It's GPL code and has been ported to Windows, MacOSX, Linux and any POSIX system. An easy compile runs it on Solaris. Now it can be batched. On the Mac, there are a few GUI wrappers that will download and transcode.
GoBack
Ok, I've got all these .TiVo and .mpg files on my file server. How can I pull them back to the TiVo like I do the multi room viewing?
Tivo Desktop will serve .TiVo files up. Did I mention that it doesn't do mapped drives?
The Mac version will work.
Galleon will do it with .TiVo and .mpg files in the proper format. mpg files don't have the meta data with the description; you only get a filename to describe an mpg.
Recent versions of the TiVo OS will not deal with subfolders. Everything in 1 directory. That doesn't scale well and I divvy'd up my shows so I can't use anything above because they show up as 1 TiVo.
pyTivo to the rescue. It lets you setup multiple shares. I serve up each subfolder as a share. I see a TiVo as Good_Eats and that's my Good Eats subfolder. It was easy to get going on the Solaris box, but it should run everywhere Python and ffmpeg can run. Oh, pyTivo will also convert files to .mpg the tivo can understand, on the fly. As long as ffmpeg can transcode it. Fantastic. I have run into issues with mp4 and avi files but that might be because the Solaris box doesn't have the codecs. I might need to go to Linux.
Commercials
There are a few apps out there that will let you cut up .avi or mpg files. avidemux will search for blank frames that broadcasters are required to put between commercials and shows. ffmpeg will transcode from mpg to avi:
ffmpeg -i file -target ntsc-dvd -b 2500 -s 352x480 -hq -ab 160 file.mpgThere's still a large, tedious process for cutting.
cbreak is a windows .NET app that is supposed to do it automatically. There's even a batch file that'll convert .mpg with ffmpeg.exe 1st. It's drag & drop. Perfect. Except the resulting file isn't .TiVo compatible. My Solaris ffmpeg can't convert the avi either. Displaying it via pyTivo locks up the TiVo too!
There are commercial apps out there like VideoReDo that automate much of it. MythTV has skipping built in, but I don't know if it can be used as a filter.
Virtualization
I have Fedora 7 on an AM2 dual core system. I've been doing alot with VMware Server and have owned VMware Workstation since 2.0. One of the issues with VMware is reinstalling every time the kernel is updated. Add to that the fact that VMware has only precompiled for certain kernels on certain distribution releases. *sigh*
So I decided to try out the qemu-kvm stuff in Fedora. Fedora 7 has a GUI manager for Xen and (I think qemu-kvm).
To make a long story short, qemu is easy except for the networking. I keep everything on the file server and nothing on my clients so I need networking. The GUI doesn't reveal anything about the networking and doesn't quite work with qemu yet. All the documentation I've Googled say 'qemu is in Fedora 7' but all of the networking stuff is written up for earlier versions.
No problems, except the virt manager sets up some networking stuff. And isn't documented. And uses different names from the older examples.
*sigh* So I punted and installed VMware server. There's a patch for the install that will let you compile the module for any kernel. Joy! And it works. It'd be nice if VMware included it or linked to it on the web.
In any event, I've got XPpro running in a VM. I can RDP into it from my laptop and run things like cbreak to cut commercials out of .TiVo streams after I convert 'em to mpg.
So I decided to try out the qemu-kvm stuff in Fedora. Fedora 7 has a GUI manager for Xen and (I think qemu-kvm).
To make a long story short, qemu is easy except for the networking. I keep everything on the file server and nothing on my clients so I need networking. The GUI doesn't reveal anything about the networking and doesn't quite work with qemu yet. All the documentation I've Googled say 'qemu is in Fedora 7' but all of the networking stuff is written up for earlier versions.
No problems, except the virt manager sets up some networking stuff. And isn't documented. And uses different names from the older examples.
*sigh* So I punted and installed VMware server. There's a patch for the install that will let you compile the module for any kernel. Joy! And it works. It'd be nice if VMware included it or linked to it on the web.
In any event, I've got XPpro running in a VM. I can RDP into it from my laptop and run things like cbreak to cut commercials out of .TiVo streams after I convert 'em to mpg.
Wednesday, August 1, 2007
Today's Links
Systems
iSync adapting
Linux Audio Troubleshooting Nice guide to figuring out the hardware in linux
Solaris keeps running. The post is in german, but they stayed running. No one noticed until they checked the room.
OpenKVM
Windows
Disable services
AD tools
Open Source tools
Google
Cheatsheet
Video
Transcoding
iSync adapting
Linux Audio Troubleshooting Nice guide to figuring out the hardware in linux
Solaris keeps running. The post is in german, but they stayed running. No one noticed until they checked the room.
OpenKVM
Windows
Disable services
AD tools
Open Source tools
Cheatsheet
Video
Transcoding
Tuesday, July 31, 2007
Monday, July 30, 2007
Today's links
GTD
Liferemix has a whole bunch of productivity blogs
GTD primer
Scan all incoming paper
From unclutterer.com
Ken Silver
iPod fixes
Linux
Dolby 5.1 sound
Mounting ISOs
IPTV info
Liferemix has a whole bunch of productivity blogs
GTD primer
Scan all incoming paper
From unclutterer.com
Ken Silver
iPod fixes
Linux
Dolby 5.1 sound
Mounting ISOs
IPTV info
Wednesday, July 25, 2007
Back of the napkin Network speed testing
I've been trying to compare iSCSI, NFS, and general network speed lately.
Awhile ago, my main Linux system was a dual PIII 500MHz. I was getting 20MB/s over the gigabit ethernet which seemed slow to me. At work I've seen 40MB/s to 60MB/s. I upgraded the server to an AMD dual core system and get 60MB/s.
My back of the napkin testing was pushing a gigabyte with dd:
time dd if=/dev/zero bs=1048576 count=1024 of=
GNU dd can also display MB/s.
I figure /dev/zero is the fastest source of bits. I can output to /dev/null to get the fastest data sink. Or local disk or NFS or SMB. But that still didn't measure just the network.
I found ttcp which sets up a client/server. On the sink I do ttcp -r > /dev/null. On the source, I pipe dd to ttcp -t. Netcat by Hobbit could do the same, but now there's a GNU nc that doesn't let you do -lp! I suppose I should just compile my own.
Something like:
On sink: nc -lvnp 5150 > /dev/null
On source: dd blah | nc -v -w 2 serverip
Bill McGonigle has this note about iperf. It looks interesting too.
Iperf's home
TCP tuning
The Iperf tarball has a test directory (read the README in it):
one side: perl server
other: perl client tests remote local | tee iperf.log
Tune some stuff
Run it again to iperf2.log
grep Mbits.s /tmp/iperf*.log | awk '{print $1, $(NF-1)}' | sort -n +1 | less
If iperf2.log is at the bottom, you got more megabits.
Is it accurate? I don't know. Does it help measure change? Yep. And that's what tuning is about.
Awhile ago, my main Linux system was a dual PIII 500MHz. I was getting 20MB/s over the gigabit ethernet which seemed slow to me. At work I've seen 40MB/s to 60MB/s. I upgraded the server to an AMD dual core system and get 60MB/s.
My back of the napkin testing was pushing a gigabyte with dd:
time dd if=/dev/zero bs=1048576 count=1024 of=
GNU dd can also display MB/s.
I figure /dev/zero is the fastest source of bits. I can output to /dev/null to get the fastest data sink. Or local disk or NFS or SMB. But that still didn't measure just the network.
I found ttcp which sets up a client/server. On the sink I do ttcp -r > /dev/null. On the source, I pipe dd to ttcp -t
Something like:
On sink: nc -lvnp 5150 > /dev/null
On source: dd blah | nc -v -w 2 serverip
Bill McGonigle has this note about iperf. It looks interesting too.
Iperf's home
TCP tuning
The Iperf tarball has a test directory (read the README in it):
one side: perl server
other: perl client tests remote local | tee iperf.log
Tune some stuff
Run it again to iperf2.log
grep Mbits.s /tmp/iperf*.log | awk '{print $1, $(NF-1)}' | sort -n +1 | less
If iperf2.log is at the bottom, you got more megabits.
Is it accurate? I don't know. Does it help measure change? Yep. And that's what tuning is about.
Links
Online Media God
Sun BigAdmin Not just for Solaris
Fix RPM db
Google Video (FLV) conversion with ffmpeg
Lifehacking/Productivity
Productivity/Life Hack site
GTD Good Easy on a mac
Bit Literacy Review
Text based todo
Sun BigAdmin Not just for Solaris
Fix RPM db
Google Video (FLV) conversion with ffmpeg
Lifehacking/Productivity
Productivity/Life Hack site
GTD Good Easy on a mac
Bit Literacy Review
Text based todo
Tuesday, July 24, 2007
Why run VMs? part 1
I'm a Unix guy, but often work in a windows environ. Usually there's something that only runs in windows that I need to run.
I've used DOSemu, Executor (Mac OS 7), Win4Lin, VMware Workstation (2.x -> 6.x), Bochs, QEMU, Basillisk and wine. All have various capabilities and impacts.
If you just want to run windows apps, having a 2nd computer running terminal server over a gigabit net is probably the best for features. You get full hardware access, reasonable display speed (gigabit switches and KVMs are inexpensive).
For a Macintosh, VNC over an SSH connection works well on Mac OSX. I've used VNC to control System 7 systems. I find VNC on a mac works better then on windows. Windows doesn't let VNC at the login screen while OSX does.
VMware and Win4Lin depended on the kernel so any update to that meant a reconfigure. *sigh* And hopefully that kernel was supported. Otherwise you had a choice to make. I get tired of all the reconfiguring. Of course Windows doesn't update its kernel often so the host is more stable.
I have hopes for QEMU lately. And KVM now that I have a chip with virtualization. I've gotten QEMU running with Solaris as a host, but the networking was a bit tricky. I'd love to see VMware hosted on Solaris but I doubt the port will be done. Hardware is inexpensive enough that having a Linux box is doable.
I've used DOSemu, Executor (Mac OS 7), Win4Lin, VMware Workstation (2.x -> 6.x), Bochs, QEMU, Basillisk and wine. All have various capabilities and impacts.
If you just want to run windows apps, having a 2nd computer running terminal server over a gigabit net is probably the best for features. You get full hardware access, reasonable display speed (gigabit switches and KVMs are inexpensive).
For a Macintosh, VNC over an SSH connection works well on Mac OSX. I've used VNC to control System 7 systems. I find VNC on a mac works better then on windows. Windows doesn't let VNC at the login screen while OSX does.
VMware and Win4Lin depended on the kernel so any update to that meant a reconfigure. *sigh* And hopefully that kernel was supported. Otherwise you had a choice to make. I get tired of all the reconfiguring. Of course Windows doesn't update its kernel often so the host is more stable.
I have hopes for QEMU lately. And KVM now that I have a chip with virtualization. I've gotten QEMU running with Solaris as a host, but the networking was a bit tricky. I'd love to see VMware hosted on Solaris but I doubt the port will be done. Hardware is inexpensive enough that having a Linux box is doable.
Monday, July 23, 2007
Sunday, July 22, 2007
Cheap storage
Right now it's SATA. If you're running RAID (and you should), SCSI isn't going to buy you much more reliability. It's not going to give you much more speed on the low end either. Anyways, this is about cheap.
You have a PC with slots in it. It's going to be your server. I suggest having the OS on a RAID1 mirror. Don't have data on this so the OS can go as fast as possible. The RAID buys some reliability. If it's IDE, no slaves. Just masters. Mixing them slows things down quite a bit. Ok, you can put the CD/DVD on it.
Get a 4 port SATA card. You do not care about the RAID in it. Most of them are really software RAID anyways and each card does it differently. By doing software RAID, you can use any adapter. Hardware RAID means your data is at the mercy of a brand or even model of that card if it fails. Besides, you're going to be running an iSCSI SAN or an NFS/Samba file server here. Your CPU is dedicated to storage. Your bottleneck will be gigabit ethernet to your clients.
Get 4 SATA drives of the same size. I like to have a standard size that will be available in the future, like 500GB.
Now the hard part: how to power and cool them? If you have a case like the Antec PB180, there's a fan and a 4 drive chamber. Just get some SATA cables and maybe some 4 pin to SATA power adapters.
If not, you need to get an external drive. There are some nice ones under $150 with power and cooling for 4 SATA drives. You can also build your own with a PC power supply, a fan, power adapters and something to bolt the drives to that the fan pulls air through.
Now get some long SATA cables. I've used 42" ones. They do not have to be eSATA. I've used internal SATA from the drives through a card slot to the internal SATA card and they work fine. Using short SATA extenders lets you have a nice disconnection outside the PC or drive case.
Hook it all up, install your OS and make it a software RAID5 setup. Cheap.
Which OS? Do you want a NAS or SAN device? I really like ZFS on my servers and that means Solaris right now. Solaris 10u3 doesn't have an iSCSI target so that means NAS only. Newer versions of OpenSolaris have iSCSI targer. Linux has an iSCSI target also and I don't think it's as close to bleeding edge as OpenSolaris, but there are many that feel comfortable with it. There's a linux distribution called OpenFiler that does iSCSI target, Samba and NFS with a web admin interface for you non unix types.
Speeds? On a Dual PIII 500MHz I got 20-25MB/s. On a Xeon 2.4 or on an AMD 4000+ I get about 120MB/s. The PIII can't keep up with Gigabit. The other 2 can do 90MB/s over gigabit. That's faster then local disk on older systems.
4 * 500GB @$120/ea -> $480 for disk. Put a $500 PC with gigabit ethernet under that and you have 1.5TB or RAID5 for under $1000.
You have a PC with slots in it. It's going to be your server. I suggest having the OS on a RAID1 mirror. Don't have data on this so the OS can go as fast as possible. The RAID buys some reliability. If it's IDE, no slaves. Just masters. Mixing them slows things down quite a bit. Ok, you can put the CD/DVD on it.
Get a 4 port SATA card. You do not care about the RAID in it. Most of them are really software RAID anyways and each card does it differently. By doing software RAID, you can use any adapter. Hardware RAID means your data is at the mercy of a brand or even model of that card if it fails. Besides, you're going to be running an iSCSI SAN or an NFS/Samba file server here. Your CPU is dedicated to storage. Your bottleneck will be gigabit ethernet to your clients.
Get 4 SATA drives of the same size. I like to have a standard size that will be available in the future, like 500GB.
Now the hard part: how to power and cool them? If you have a case like the Antec PB180, there's a fan and a 4 drive chamber. Just get some SATA cables and maybe some 4 pin to SATA power adapters.
If not, you need to get an external drive. There are some nice ones under $150 with power and cooling for 4 SATA drives. You can also build your own with a PC power supply, a fan, power adapters and something to bolt the drives to that the fan pulls air through.
Now get some long SATA cables. I've used 42" ones. They do not have to be eSATA. I've used internal SATA from the drives through a card slot to the internal SATA card and they work fine. Using short SATA extenders lets you have a nice disconnection outside the PC or drive case.
Hook it all up, install your OS and make it a software RAID5 setup. Cheap.
Which OS? Do you want a NAS or SAN device? I really like ZFS on my servers and that means Solaris right now. Solaris 10u3 doesn't have an iSCSI target so that means NAS only. Newer versions of OpenSolaris have iSCSI targer. Linux has an iSCSI target also and I don't think it's as close to bleeding edge as OpenSolaris, but there are many that feel comfortable with it. There's a linux distribution called OpenFiler that does iSCSI target, Samba and NFS with a web admin interface for you non unix types.
Speeds? On a Dual PIII 500MHz I got 20-25MB/s. On a Xeon 2.4 or on an AMD 4000+ I get about 120MB/s. The PIII can't keep up with Gigabit. The other 2 can do 90MB/s over gigabit. That's faster then local disk on older systems.
4 * 500GB @$120/ea -> $480 for disk. Put a $500 PC with gigabit ethernet under that and you have 1.5TB or RAID5 for under $1000.
Storage rant #1
I've been doing sysadmin for awhile now. I've seen disk space go from > $4 per megabyte(!) to less then $240 for a terabyte. Space requirements have gone up too. You used to be able to have your compiler, editor and source on a floppy.
My general rule has been local disk for databases and other things that need locking, NAS for everything else. The local store can be a SAN of course. Centralize storage as much as possible (but no more so) to keep backups from going over the network. Because it's centralized, you can do RAID to increase reliability.
Backups are not archives! Backups are so you can recover your setup as close as possible to the latest good state if the hardware fails completly. If you want to go back to a point in time, that's an archive.
Backups have changed dramatically over the years. I don't think there's such a thing as an inexpensive tape anymore. At least something that's dramatically cheaper then disk. I once bought a 2GB 4mm DAT to backup my home systems. I probably had 1GB at the time. Now you're going to need multiple tapes to span your disks. Because of that, you probably also want an automatic tape system as well.
After you figure out your backup cycle (how far back a backup goes (archives are forever)) and how much data you have, you arrive at the total data storage. If you go tape, figure out the automated drive cost with a full compliment of tapes. You might find that is cheaper to buy a disk farm of some sort for your backup store. Remember, backups are not archives.
You'll still probably want to put data offsite. But the disks do have the advantage of matching the speed of incoming data. Your backups will always be much quicker to disk then tape so your window will be larger. When you create media for offsite, it'll be done on the disk farm so it will go quicker.
My general rule has been local disk for databases and other things that need locking, NAS for everything else. The local store can be a SAN of course. Centralize storage as much as possible (but no more so) to keep backups from going over the network. Because it's centralized, you can do RAID to increase reliability.
Backups are not archives! Backups are so you can recover your setup as close as possible to the latest good state if the hardware fails completly. If you want to go back to a point in time, that's an archive.
Backups have changed dramatically over the years. I don't think there's such a thing as an inexpensive tape anymore. At least something that's dramatically cheaper then disk. I once bought a 2GB 4mm DAT to backup my home systems. I probably had 1GB at the time. Now you're going to need multiple tapes to span your disks. Because of that, you probably also want an automatic tape system as well.
After you figure out your backup cycle (how far back a backup goes (archives are forever)) and how much data you have, you arrive at the total data storage. If you go tape, figure out the automated drive cost with a full compliment of tapes. You might find that is cheaper to buy a disk farm of some sort for your backup store. Remember, backups are not archives.
You'll still probably want to put data offsite. But the disks do have the advantage of matching the speed of incoming data. Your backups will always be much quicker to disk then tape so your window will be larger. When you create media for offsite, it'll be done on the disk farm so it will go quicker.
LVM notes
Commands
ls /sbin/vg* /sbin/lv*/sbin/pv*
Xscan, Xdisplay, Xcreate, Xremove, Xextend
Initialize for lvm
remove it!
ls /sbin/vg* /sbin/lv*/sbin/pv*
Xscan, Xdisplay, Xcreate, Xremove, Xextend
Initialize for lvm
- pvcreate -v /dev/md3
- pvdisplay /dev/md3
- vgscan
- create the volume group /dev/vg from /dev/md3
- vgcreate vg /dev/md3
- vgdisplay
- show Allocated and Free space
- sudo vgdisplay | egrep '^[AF].*Size'
- lvcreate --size 2048m vg
- ls -l /dev/vg
- mke2fs -j /dev/vg/lvol
- mkdir /test
- mount -t ext3 /dev/vg/lvol /test
- lvdisplay /dev/vg/lvol[n]
- umount /dev/vg/lvol?
- lvextend -L + /dev/vg/lvol?
- e2fsck -f /dev/vg/lvol?
- resize2fs -p /dev/vg/lvol?
- mount it
umount /dev/vg/lvol? lvremove /dev/vg/lvol? - rmdir mountpoint
- vi /etc/fstab
Subscribe to:
Posts (Atom)
