Recovering From a Failed ZFS Root Mirror Disk

Hypothetical. Say, for example, you’ve got a ZFS mirror on disk0 and disk1, from which your Nexenta system boots. A year or two later, disk0 crashes, and starts spewing messages such as the following:

 ahci: [ID 860969 kern.warning] WARNING: ahci0: ahci_port_reset port 0 the device hardware has been initialized and the power-up diagnostics failed

Shortly thereafter there’s a storm and the power goes out for several hours and drains your entire humongous UPS, and when the power finally comes back on, the system won’t boot — it stops at the grub prompt; issuing a standard boot command fails.

A Tale of Two Chairs

It was the best of products and it was the worst of products. It was a tale of one man trying to find an affordable chair.

Several years back I shopped at WalMart and bought a card table with four folding chairs for around $50. I was really disappointed because I didn’t think a card table would be worth more than $30 or so. This story however is not about the card table, but about chairs.

JFS on Linux = Surprisingly Lossy

I set up a RAID media server a couple years ago, and decided to give JFS2 a try, since it’s touted as being fast and reliable across the spectrum of usecases. My setup is primarily write-once-read-many, for storing the terabytes of audio and video recordings I’ve made over the last decade for project ruori and the like.

Several weeks ago, the power went out for an extended period of time while I was away, and, while it was on UPS backup, it failed to shutdown cleanly and the power was suddenly cut when the UPS ran out.

When I brought the machine back up, the volume wouldn’t mount, so I ran jfs_fsck on it. jfs_fsck said that the journal was corrupt, and started block-scanning. It came up with a pretty big list of files and directories that were irrevocably corrupt. Parts of a few of them got linked into /lost+found, but the majority simply vanished.

Funny thing is, I hadn’t made any changes to these files in several years. I could understand if maybe some very recent FS updates were lost due to write-caching, but why did it lose track of these ancient files?

This reminds me of the rampant table corruption of MS-DOS’s FAT16 filesystem, which couldn’t keep track of a needle dancing on the point of a needle.

So, plus one for backups of backups, and minus one for JFS on Linux. I think I’ll be rebuilding the machine with ZFS-fuse. Or OpenSolaris, for that matter.


Steve Mokris is a developer at Kosada, Inc.

Drupalcon tidbits

Cool things I learned about at Drupalcon DC include (but are not limited to) the following:

  • There is a Drupal 6 theme starter app whose maintainer is constantly improving it.
  • Usability studies have proven that I am not the only one who couldn’t figure out WTF to do with the welcome screen on my first Drupal install.
  • A lot of the modules that seem like they should be in core but probably won’t be in Drupal 7, are left out because Token is unlikely to be included and many of them rely on Token.

As usual, more coming…

DrupalCon 2009: First Impressions

I can positively affirm that the free tee-shirts given out with DrupalCon 09 registration are quality. The medium size fits me perfectly and the cotton is nice and soft and thick. Overall very wearable. Caveat: they’re not pre-shrunk so they will shrink.

I have a lot of actually relevant info to post but that will have to wait. Topics will include:

  • Theming tips and best practices
  • Awesome stuff (eg. fields!) that will be built-in to Drupal 7
  • Awesome stuff that won’t be in core in Drupal 7, and why it won’t (Hint: if it depends on something that won’t make core, it can’t go in core either)
  • Some other stuff.

Overall it’s been enjoyable and packed with information so far. I only wish I could go to more sessions.

memset() vs. bzero() — Ultimate Showdown

There are a few functions used to zero out memory on most unix variants. memset(), bzero(), and calloc() are all a few such functions. calloc() isn’t very useful for clearing already allocated memory, so it won’t be appearing much more in this article. However, the other two are somewhat more interesting than meets the eye.

Kineme is now (c)ubercoolische

In case you’re not super minimal like we are and don’t already know, Richie Hawtin is the Berlincoolest musician ever. His label, M-NUS, is celebrating its 10th anniversary with, among other things, a blue lighted cube (pictured left). Ali Demirel, visualist for Richie Hawtin, includes Quartz Composer in his arsenal of interactive and realtime media tools, and has been using a couple of Kineme tools to control and interact with his visuals.

He used Particle Tools to get some interesting effects in the visuals displayed behind the musicians during their set. He modified the “Fire” sample to fit with the other visuals and the music.

He also used Kineme File Tools String with URL to access the information acquired by the Cube. Members had RFID chipcards, and it would get their name whenever anyone held up their RFID chipcard to it. Demirel then chose the right moment to display their name in the visuals.

For an interview with Mr. Demirel, including links to videos, check out the full story on Kosada.com.

90% Easy, 10% Impossible

90% Easy, 10% Impossible Pie ChartFrom time to time, I get these insatiable urges to read what other Cocoa developers blog about. Sometimes they’re informative, sometimes they’re funny, sometimes they read like college textbooks, and sometimes they’re just downright terrible, but I read them anyway. It comes and goes in waves, every 2 or 3 months. A couple weeks ago, one such binge happened, and I started reading Aaron Hillegass’ critique of NSController (since I was hating it at the time, and wanted to feel justified in hating it). In the critique, the following statement was made: