*scottstuff*

Soo Valley, near Whistler, B.C.

I spent Thursday and Friday in Whistler along with a few hundred co-workers, enjoying the Seattle version of Google’s annual ski trip. I took the opportunity to go snowmobiling for the first time, and took my camera along.

I need to find another excuse to go play in the snow with my camera; that was too much fun.

One of my favorite places to take pictures is on Fir Island, in Skagit County, Washington. It’s a farming community in the Skagit River delta, and it’s home to around 1 million migratory birds every winter, including Snow Geese and Swans. Ever see a cloud of geese turn the sky white?

It’d been weeks since I’ve been able to spend time hiking around with my camera, so I drove up Monday morning before dawn to see what I could find.

Dawn on Fir Island

There were geese and swans flying by all morning, but I never got a really great shot of any of them from up close. There were thousands of them visible in the distance, though.

Morning flight

It’s been really cold lately, and there was ice everywhere, including the sea shore. There was a weird layer of office over everything; I assume that it was left behind by the falling tide. The plants looked like they’d been wrapped in cellophane.

Plants in Ice

Finally, on the way out, I spotted this Heron hiding in a drainage ditch a few feet from the road. He was my third or fourth heron of the day.

Cold Heron

I can’t help thinking that a bit of fill-flash would have helped there, but he was close enough that it probably would have startled him.

As any experienced digital photographer can tell you, the trick to getting repeatable color out of a lab is to get a good ICC color profile for the lab’s printer and using it for every print that you make. I’ve been a big fan of Dry Creek Photo’s printer profiling service for years; they work with labs to build quality profiles and then publish them for free on their website, along with some documentation on how to use them.

Unfortunately, actually using the profiles is a pain for most users. The process looks roughly like this:

1. Using a color-profiled monitor, get things looking the way that you want on the screen. Save.
2. Resize the image to the correct resolution for the print size and resolution that are required for your lab. Different labs want 300, 320, or 400 DPI.
3. Convert the image to the color profile for that lab that you’re using. Possibly re-adjust colors slightly.
4. Pad the image out to a specific number of pixels to keep the lab’s printer from trying to re-scale your image. The exact settings depend on the printer and paper size. There’s a big chart on Dry Creek Photo’s website.

That’s not a big deal if you’re printing one or two pictures for framing, but it’s a huge pain when you have dozens or hundreds of pictures to process. Photoshop actions can help, but it seems like something is always going wrong.

When Adobe released their Lightroom SDK, I had high hopes that it’d make it easier to automate most of this. Unfortunately, Adobe didn’t expose their profile conversion engine to the SDK in the first SDK release, so it’s not really possible to build pre-profiled images directly out of Lightroom without using some external tool to do the profile conversion. A few weeks ago, Timothy Armes released LR/Mogrify, which uses ImageMagick’s mogrify command-line tool for profile conversion. It’s a cool tool, but it replaces a 5-6 step process in Photoshop with a dozen text boxes that you need to fill in to get good results. It shows promise, but it’s not quite the tool that I’m looking for.

What I really want is a Lightroom export plugin that asks you three simple questions:

1. Where are you going to print this?
2. Which paper are you going to use?
3. What size do you want?

Then it does all of the hard work on its own. It’d fetch the correct profiles from Dry Creek, install them, figure out which resolution to use, handle image rotation, do some amount of pre-print sharpening, and then spit out a JPEG for you. Yeah, you could do a bit better with Photoshop and spending some time dealing with soft-proofing and sharpening, but I’m not willing to do that for 50 4x6 prints.

I’ve spent a bit of time this week building the first part of the tool–a machine-readable profile database. I’ve extracted a list of 765 current ICC profiles from Dry Creek Photo’s website, discarded the profiles that haven’t been updated in years, and produced an XML file that looks sort of like this:

<lab>
  <name>Costco #747</name>
  <address>24008 Snohomish-Woodinville Rd. SE, Woodinville, WA 98072</address>
  <phone>425-806-7708</phone>
  <printer>Noritsu 3411</printer>
  <paper>Fuji Crystal Archive</paper>
  <resolution>320</resolution>
  <notes>Note: This lab has multiple printers. Request your profiled prints be run on Noritsu 34?Pro-B.</notes>
  <profile>
    <name>Glossy paper profile</name>
    <date>January 17, 2008</date>
    <url>http://www.drycreekphoto.com/icc/Profiles/IccFiles/Washington/Costco-WA-Woodinville-Gls.icc</url>
  </profile>
  <profile>
    <name>Lustre paper profile</name>
    <date>January 17, 2008</date>
    <url>http://www.drycreekphoto.com/icc/Profiles/IccFiles/Washington/Costco-WA-Woodinville-Lus.icc</url>
  </profile>
  <size>4x6in</size>
  <size>5x7in</size>
  <size>8x10in</size>
  <size>8x12in</size>
  <size>12x12in</size>
  <size>12x18in</size>
</lab>
<lab>
  ...

The <resolution> and <size> blocks are semi-manual additions; the import script sets resolution automatically when it sees a printer type that only has one resolution setting, but Noritsu 3XXX printers can run at either 320 or 300. I’ve filled in the one or two labs that I use most frequently, and I’ll to add others as time permits. The <size> bit is fully manual, and I’m not really sure that it’s worth the effort to populate.

Right now, the XML file lists 380 labs. Entertainingly, that’s 379 Costcos plus Adorama. It looks like Dry Creek has dropped almost everyone else. If there’s a second source of non-Costco profiles, I’d love to know about it.

So, anyway, I have this XML available at http://profiles.sigkill.org/profiles.xml. It’s mostly automatically generated, and I can rebuild it in under 5 minutes. I’ll keep it up to date if people are interested in the data, if not it’ll die off eventually. If you have a tool that wants to use it, then send me mail and let me know that it’s useful. If you have any changes that you’d like to see, or anything that I should add, let me know and I’ll see what I can do.

I’m really not looking forward to going home from my vacation.

I bought my wife an Olympus 790SW point-and-shoot camera before we left for Hawaii on vacation, and I’m growing increasingly fond of the little thing. It doesn’t really compare to my Canon 5D’s image quality, but it’s so small and handy that it’s easier to carry. Even better, it seems to be indestructible–it’s submersible and can be dropped up to 4.5 feet without breaking anything.

In other words, it’s the perfect beach camera for families with small kids. Plus, you can take it snorkeling, just in case one of these pops up:

or some of these:

There are more pictures on Flickr if you’re interested.

It also takes semi-decent video. I wouldn’t confuse it with a HD camcorder, but I wouldn’t take the camcorder in the water, either. Here’s my son’s first time snorkeling:

It’s around $260 on Amazon, if you’re interested.

So, I’m in Hawaii this week for vacation with the family. It’s been a while since I’ve had a real vacation that didn’t involve cross-country drives or tight deadlines. I’ll write more eventually, but for now, have a few pictures:

It looks like Lightroom 1.3 is out. More excitingly, the Lightroom Export SDK is now available. I guess it’s time to learn Lua.

I might wait until after next week’s vacation to pick up Lua, though. I have a beach calling my name.

Interesting trivia: staggered drive spin-up is an optional part of the SATA II spec. Unlike most SCSI drives, though, it’s not controlled through a jumper. Instead, it uses one pin on the SATA power connector. If pin 11 is floating, then the drive is supposed to wait to start spinning.

Apparently the 650W power supply in my new server only provides enough current to spin up 10 drives at once, because adding an 11th drive makes it turn off on its own immediately. Sigh. I wonder if anyone makes delayed-spin SATA power dongles?

I mentioned that I bought a Gigabyte GC-RAMDISK (a.k.a. i-RAM) to go in my new home file server, largely to see if using it as a solid-state log device would improve ZFS performance.

Unfortunately, I’ve been completely and totally unable to get the card to do anything at all. I’m not sure if I have a defective card or if Gigabyte’s SATA implementation is just really buggy. When I plugged it into the motherboard’s ICH9R SATA ports, the BIOS didn’t even show it on the boot-up scan and Solaris reported it as failing to initialize correctly. When I plugged it into the Supermicro AOC-SAT2-MV8 8-port SATA card, the BIOS could see it but Solaris gave a similar error. Connecting it to the motherboard’s Marvell eSATA ports made the Marvell hang at bootup and made Solaris really unhappy, spewing drive failure messages all over the console.

I can’t find a single review that suggests that anyone has got this to work with a recent motherboard. Digging through the Linux kernel mailing list suggests that it has a really spotty SATA implementation. Apparently they developed it using a couple Windows drivers as a comparison, instead of actually paying attention to the SATA spec.

So, it’s going back to Amazon today. It was a nice idea, but it just doesn’t appear to work. I don’t know if it’s broken or just poorly designed, but either way it’s not useful to me.

I now have Solaris up and running and reasonably stable-looking, after only 12 hours of work. A number of things turned out to be bigger issues than I’d anticipated, largely because it’s been years since I last used Solaris and, frankly, Solaris’s disk partitioning and formatting tools suck.

• My first problem is still unresolved: my BIOS refuses to boot from the IDE DVD drive that I installed. Once the system boots, it works just fine, so I’m not sure what’s up. Maybe a BIOS bug. Fortunately, the system’s perfectly happy booting off a USB DVD drive, and (amazingly) Solaris is happy installing from it.
• The GC-RAMDISK card that I was looking forward to testing is a complete failure so far. I don’t know if I have a bad card or if it’s simply incompatible with both SATA chips in the system, but the BIOS completely ignores it if its plugged into the motherboard, and Solaris fails to talk to it on either bus. If it’s plugged into the MB, then I get a device failed initialization error; if it’s plugged into the PCI-X SATA card then I get device on port 5 still busy after reset. I’ve swapped cables and RAM. I’d really like to get it to work, so I’m going to try it with an older system before RMAing it.
• Actually getting a working Solaris install took me 3 tries. The first time I installed it to the wrong drive (the first disk on the PCI-X card, not the first disk on the motherboard), and it was unable to mount the root partition after rebooting. Next, I managed to install it onto an EFI partition, and that wouldn’t boot either. Finally, I installed it onto the second drive on the right bus, and that worked.
• Since Solaris’s installer doesn’t support ZFS yet, I had to manually copy the root filesystem onto a newly created ZFS filesystem mirrored across a pair of drives. The directions are helpful, but I kept screwing things up. First, I accidentally created the ZFS pool using the entire disk, which made ZFS re-label the drives with EFI, which makes them unbootable. Then, I missed the line in the directions that says to run format -e instead of using format; that left me with a pair of nicely partitioned drives that still used EFI. The third try worked, and the system is now booting off of ZFS via GRUB without problems. Er…
• Well, one problem–I can’t change the GRUB menu.lst file for some reason. I don’t know where GRUB is looking for it, but it’s not in /boot/grub/menu.lst on my boot array. My changes are being completely ignored. I can live with this for the weekend.
• OpenSolaris doesn’t ship with drivers for the ASUS P5K WS’s onboard Ethernet chips. I had to grab them from Marvell, but it was easy enough to install.
• Creating an 8-drive ZFS filesystem is trivial. One command takes care of RAID, logical volume management, creates the filesystem, and mounts it: zpool create -f space raidz2 c0t1d0 c0t2d0 c0t3d0 c0t4d0 c3t2d0 c3t3d0 c3t4d0 c3t5d0.
• ZFS performance is decent. Here’s a bonnie with and without ZFS compression, using 10 GB of data on a box with 2 GB of RAM:

              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    GB M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU  /sec %CPU
zfs        10 105.0 55.3 163.3 27.3 121.0 30.4 119.2 88.4 287.1 36.2   169  1.8
zfs+c      10 112.9 59.7 181.5 30.3 127.8 29.1 118.1 86.0 424.9 52.2   198  2.1

• 163 MB/sec writing and 287 MB/sec reading is good enough for me. I was expecting slightly higher numbers, but there’s nothing here to complain about. Adding compression improves writing a bit and makes a big difference reading. It’s quite a bit faster then GigE, which was my goal.

A few days ago, I mentioned that my home NAS box had failed, and that I was considering replacing it with a PC server running OpenSolaris and ZFS. I’ve read a pile of ZFS docs, and it looks like the best option available to me today, so I decided to order some suitable hardware.

At that point, pretty much everything broke down. I have a hard enough time keeping track of which hardware works with Linux this week, and OpenSolaris is completely new to me. Sun’s list of officially-supported hardware is pretty sparse, and digging through their mailing list archives gets frustrating quickly. From what I can tell, it boils down to:

• Current Intel and AMD CPUs are all fine.
• Most of Intel’s chipsets are fine.
• Most of nVidia’s AMD chipsets are fine.
• nVidia and Intel video chips are good.
• Most common Ethernet chipsets are either supported natively or have drivers available.
• The only SATA controllers that work are Intel’s ICH southbridges, Silicon Image’s PCI and PCI-E chips, Marvell’s PCI chips, and nVidia’s southbridges. It’s not clear that Marvell’s PCI-E chips work. Most motherboards with additional, non-southbridge SATA ports probably won’t work.
• Venturing too far outside of this list will probably result in problems.

I was looking for a motherboard with 8 SATA ports, and was hoping that the Intel D975XBX2 (“Bad Axe 2”) would work, but 4 of its 8 SATA ports belong to a Marvell PCI-E SATA chip that doesn’t appear to be supported. I went through every single 8-port motherboard in Newegg’s database and ended up flunking all of them out for some reason–unsupported chipset, unsupported SATA chip, not enough slots, too expensive, and so on. In the end, I ended up ordering an Asus P5K WS (the ‘WS’ is important–the P5K is a different board). It only has 6 on-board SATA ports, but it includes a PCI-X slot. That’ll let me use the Supermicro AOC-SAT2-MV8, which is far and away the cheapest 8-port SATA card on the market. That’ll give me a total of 14 SATA ports, which should be enough for a whatever I want to throw at it. The Marvell PCI-X chip at the heart of the Supermicro card is the same one used in Sun’s Sun Fire x4500 48-drive server, so it’s safe to assume that Sun has put a lot of effort into the driver.

Most of the test of the system is fairly generic–a cheap nVidia 7200GS video card (the cheapest PCI-E card that NewEgg carries), a nice case and power supply, RAM, and a boatload of drives.

The one odd component that I’ve added is a Gigabyte GC-RAMDISK with 1 GB of RAM. The GC-RAMDISK is a battery-backed SATA ramdisk; it looks like a hard drive to the system and can survive up to 18 hours without power. I’ve had my eye on this thing for years, and it looks like it’ll be a perfect external log device for GFS. I had to ask to see how ZFS will behave if the device fails, and it looks like manual intervention may be required after an 18+ hour power outage, but it should be pretty minimal. I’m planning on posting some benchmarks here once I’ve had a chance to try it out.

Assuming that I’m able to get this whole mess to work at all, I should have lots to write about here over the next week or so. I’m going to start by explaining why I want to use Solaris instead of Linux or *BSD, and why I’m building something instead of buying a pre-build NAS box.

So, as part of my new home server series, I want to explain why I’m using OpenSolaris instead of Linux.

I’ve used Linux since 0.97.1, in August of 1992. I’ve had at least one Linux box at home continuously since 1993 or so. I’ve had a few small chunks of my code added to the kernel over the years. I’ve built several install disks and one embedded appliance distro from scratch, starting with a kernel and busybox and going on up from there. I’ve written X drivers, camera drivers, and drivers for embedded devices on the motherboard. I’ve managed Great Heaping Big Gobs of Hardware at various jobs. Basically, I know Linux well, and I’ve used it for almost half of my life.

That in itself might mean that it’s time for a change–professionally, I’ve been very tightly focused on Linux, and diversity is a good thing. But that’s not why I’m using Solaris this week. I’m using it because I’m fed up with losing data to weird RAID issues with Linux, and I believe that OpenSolaris with ZFS will be substantially more reliable long-term. Things I’m specifically fed up with:

• md (the Linux RAID driver)’s response to any sort of drive error, even a transient timeout, is to kick the drive from the array, no matter what. Most of the IDE drives that I’ve had over the years have been prone to random timeouts every few months, at least once you bundle more then 2 or 3 of them in a single box and then try snaking massive ribbon cable through the case. My SATA experiences haven’t been substantially better. Linux will happily bump an otherwise working 4-drive RAID 5 array to a 3-drive degraded RAID 5 array on the first failure, and then on to a 2-drive failed array on the second failure. Even when a simple retry would have cleared both errors. This has cost me data repeatedly, because I’ve been forced to manually intervene and re-add “failed” disks to RAID arrays. If I was too slow, then a second drive failure risked total data loss. Even worse, these random transient failures blind you to real drive failures, like the one that ate my NAS box last weekend.
• Actual drive failures can hang the kernel. I’ve had at least 3 cases at home where broken drives either caused system lockups or completely kept the system from booting. That sucks. Odds are some drivers are good while others are broken; apparently I’ve just had bad luck.
• None of Linux’s filesystems are particularly resilient in the face of on-disk data corruption. Compare with ZFS, which checksums everything that it reads or writes.

In short: everything works great when things are perfect, but building a reliable multi-drive storage system requires careful component and kernel compatibility work, and then you have to stay right on top of things if you want everything to keep working. When things stop working, they usually fail badly. That’s almost the complete antithesis of what I want for home: plug it in, and it just keeps working. I don’t want small failures to cascade through the system. Little failures should isolated, identified, and automatically repaired whenever possible. OpenSolaris and ZFS seems to provide that, while Linux with md and ext3 does not.

That’s why I’m planning on using ZFS. My logic for building a server vs. buying another little NAS box is simple: none of the little NAS boxes on the market use ZFS right now, and none of the cheap ones have room for more then 5 drives. I’m planning on using a double-parity system (RAID 6 or ZFS’s raidz2, where the system can cope with a 2-drive failure) plus a spare drive, and that’d only leave me with 2 data disks. The only way that I can get enough data with only 2 disks would be to use 1TB drives, and they’re too pricy right now.

So, I’m willing to spend the time to build a somewhat complex server because I believe (hope?) that it’ll save me time in the future, and it’ll let me avoid ever having to do the reconstruct-from-the-source dance again. I don’t think I lost anything critical last weekend, and I’m reasonably confident that I’ll be able to get things limping along well enough to recover data anyway, but I’ve now done this 3 times in the past 4 years, and I’ve had it.

Coming up soon: backups, OpenSolaris hardware compatibility, and GC-RAMDISK performance benchamarks. Stay tuned :-).

So, the comments on yesterday’s post about my nasty RAID failure encouraged me to spend some time looking at ZFS on OpenSolaris, and I really like what I see. I’ve ordered some new hardware, so I should have lots to write about by next weekend.

Reading the ZFS docs reminded me of my Holy Grail of Storage: a storage system that could actually do reasonably smart things with 3–5 drives. Imagine a system where you could start with 3 drives and simply plug new drives in as you need more space, without worrying about RAID or data layout. When you run out of slots, then just unplug the oldest, smallest drive and plug in a new, larger one, and the data will resync, giving you more disk space without needing any special work on your part. For bonus points, you’d be able to designate specific bits of your data as more or less important, so Bittorrent files might not be replicated at all, while your Word documents might be replicated onto every available drive.

I’ve wanted that for years, but I’ve largely dismissed it as a pipe dream, because it doesn’t fit cleanly into the drive/RAID/LVM/filesystem model that everything uses. The only thing that I’ve seen that even comes close is Drobo, and it’s supposedly fairly slow and really just too “magic” for me to trust.

I realized this morning that it’d be easy to build a storage system like this using ZFS. Just create a zpool with 3 drives to start, and then create zfs filesystems with copies=2 on top of it. When you add new drives, just add them to the pool. Blindly removing a single old drive will only leave you with a single copy of some of your files, but that shouldn’t be fatal, and ZFS can copy everything off of it if you give it a chance. There are some corner cases that will give you less redundancy–if you manage to fill the system 98% full before adding a new drive, then all of the replicas of new data will probably end up on the same disk. There are a couple obvious workarounds, and Sun will probably add replication rebalancing at some point, if it isn’t there already.

What’s worse then the sound of one hard drive going “click, click?” Why, two drives going “click, click, click” in the same RAID 5 array, of course.

I’m not very happy with my little Infrant NAS box right now. I think I’ve had it with RAID 5–if I’m going to pile my life onto a disk array, then I really want something that can survive a 2-drive failure without croaking, and that’s basically impossible in a 4-drive enclosure.

I’m seriously considering replacing the Infrant with an OpenSolaris box running ZFS over RAID-Z2 with 6–10 drives; that should live through 2-drive failures, right? Anyone feel the need to talk me out of it?

Today’s the big release day for Charlie Stross’s new book, Halting State. I had the good fortune to pick up a review copy through work last week, since we’re one of the official stops on his tour schedule (w00t!).

Short review: go buy it. You can thank me later.

Slightly longer review:

Stross is my favorite science fiction (-ish) author right now, and has been for a few years. He’s written a number of amazing short stories, but his novels have been a bit hit-or-miss. The Atrocity Archives is deeply awesome, as is Accelerando (which is basically a collection of 9 of his short stories), but Singularity Sky didn’t really work for me.

Halting State, on the other hand, is the best science fiction novel that I’ve read in years; perhaps since The Atrocity Archives. Like most of his work, Halting State is less scifi then “geekfi”–it’s about people and technology extrapolated slightly into the future.

In this case, it’s set in 2014-ish Scotland, newly independent from England. It all started with a police call. The caller, panicked, reported a theft. Something about a bank robbery, which made the police operator sit up and take notice, until the caller started blathering about orcs and a dragon. They didn’t send anyone out to investigate until the second call, which still had orcs and the dragon, but sounded more insistent. The detective follows the address provided and ends up in the middle of nowhere, at a former nuclear command bunker surrounded by expensive cars, to discover that someone did rob a bank. Not a brick-and-mortar bank, though–this bank’s is inside of a MMORPG. The total take? About 120 million Euros, once you count the ebay-able value of the missing goods and the hit to the newly-IPOed “in-game economic stabilization” company. The dragon blew the doors off of the bank and the orcs grabbed the loot. They escaped into a portal to another MMORPG where the bank’s managers don’t have root access. The cop’s stumped–how do you investigate this? You can’t dust for dragon prints. So she logs back into CopSpace to see what she can find…

By the time Stross is done, he’s wrapped up gaming, gamers, virtual reality, cryptographic security, in-game economics, the nanny state run amok, mobile phones, and something much more sinister then any of the characters expected to find.

Just go read it, and thank me afterwards. Or thank Stross, that’s probably more appropriate.