But this morning he called me with a rather odd problem:

“I hooked up my Treo and started syncing it, then walked away for a few minutes. When I returned, the screen was covered with a bunch of squares, and I can’t do anything.”

He took a few photos with his cellphone camera and emailed them to me (click for full-size):

(You can see bits of the menu-bar scattered around the screen, and it looks like Missing Sync had launched iPhoto. Possibly related?)

I talked him through a few tests. He was able to see the mouse cursor and move the mouse, but clicking didn’t cause any apparent response. He could activate Dashboard — Dashboard itself looked fine, and the checkerboard showed through in the background. He then tapped the power button, and the “Are you sure you want to shut down your computer now?” dialog appeared, but it was scrambled and the buttons didn’t work.

I instructed him to power it off by holding the power button down. Since rebooting, it’s been working fine.

*Confused.*

I googled it a bit but wasn’t able to find any clear reports of this happening to other people.

So, how do we get this elusive MAC address anyway? For many devices, it’s printed right on the device. This is by far the easiest and fastest way to get it. But some devices hide this information, and have no interface for figuring it out. For cases like this, we can whip out a handy utility called arp and some network-jutsu, and purloin these delicious octets with reckless abandon.

arp is a program that manipulates the ARP cache on a system. The ARP cache translates an IP address into a hardware, or MAC, address. So, by pulling data out of this cache, we can find out which MAC address our system uses when we speak to a specific IP address.

Using arp is very simple for what we’re looking to do. arp [ip address] will tell us what a given IP or hostname’s MAC address is. This usage looks like this on linux:

cwright@leikata:~>arp 10.1.1.7
Address HWtype HWaddress Flags Mask Iface
vaeltaja.softpixel.com ether 00:11:24:E9:92:AC C eth0

or this on OS X:

cwright@phendrana:~>arp 192.168.255.5
server.example.com (192.168.255.5) at 0:30:6e:29:b2:55 on en1 [ethernet]


So far, so good. We can pull data out of the cache, and the MAC address is right there. Is there a catch? Of course there is.

Not having an address in the cache will, of course, make a cache query useless. This is what such a condition looks like on linux:

cwright@leikata:~>arp google.com
google.com (64.233.187.99) -- no entry

or this on OS X:

cwright@phendrana:~>arp 35.42.42.42
35.42.42.42 (35.42.42.42) -- no entry

No entry!? How can this be? In this example, we’re trying to get the MAC address from an address that isn’t on our local network. Since we only use MAC address for communication on the same LAN, we don’t cache, and thus cannot acquire, MAC addresses from non-local addresses.

Sometimes, however, we’ll receive that error for a machine that is on the network. This happens because we haven’t communicated with the destination machine. To solve this, we simply need to initiate some communication so that we’ll cache the MAC address. I like to use the ping utility, like this:

cwright@phendrana:~>arp 192.168.255.11
192.168.255.11 (192.168.255.11) -- no entry

cwright@phendrana:~>ping 192.168.255.11
PING 192.168.255.11 (192.168.255.11): 56 data bytes
64 bytes from 192.168.255.11: icmp_seq=0 ttl=30 time=2.612 ms
^C
--- 192.168.255.11 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 2.612/2.612/2.612/0.000 ms

cwright@phendrana:~>arp 192.168.255.11
? (192.168.255.11) at 0:d0:d9:2:f7:6 on en1 [ethernet]

In the first arp, we have no data in the cache. So, we ping the destination, causing some communication, and thus some cached data. After that, we can query again, and we have our MAC address ready and waiting. That wasn’t so hard, now was it?

Excitingly, these new machines broke the mold (they’re slightly older, considerably cheaper machines), so we had to tweak the image a bit. This was thankfully quite painless. In a burst of excitement, we also decided that on the morrow we would take our working image, and use that as the source for the other machines, rather than using an archive from the e-mail/intranet server. By doing this, we could completely uncap transfer speeds on a tiny network (which we have up until now been heavily capping because saturating the network makes all the other network services shut down).

The reason why this is at all interesting or important? Folding@Home.

The expected deployment date for these new machines is not in the immediate future. Up until now we’ve usually just put undeployed machines under a desk to bide their time until they go live. With this new batch of machines though, we’d be crazy to not utilize this block of unused computing power for something altruistic.

The only tool kit we used is the SystemRescueCD.

First, we configured the network settings. This is easily accomplished using the well known tool ifconfig. Nothing fancy there.

Next, we configured the root password. Since this was a completely private network, we chose a weak password. Not a big deal. passwd is your friend.

Then, we started actually trying sftp, ftp, and ssh, and found that they’re all not running out of the box. A quick /usr/sbin/sshd reveals an interesting error: no key in /etc/ssh/ssh_host_rsa_key or /etc/ssh/ssh_host_dsa_key. To generate the keys, we did this: ssh-keygen -b 768 -f /etc/ssh/ssh_host_rsa_key -N '' and then ssh-keygen -b 768 -f /etc/ssh/ssh_host_dsa_key -N ''. Then a quick /usr/sbin/sshd and the image master was in business.

This revealed yet another problem, this time a conservative bug in sftp: you can’t get /dev/hda because it’s “not a regular file.” This posed quite a problem. However, we found that by repeating the steps above on each machine, we could have the image master “push” the image onto the others in this fashion: cat /dev/hda | ssh root@[ip address] cat \> /dev/hda. This caused the image master to dump the harddrive, and then tunnel it to the others over ssh, where they would take the tunneled data and place it on /dev/hda, which is the harddrive.

In the process of imaging each of these machines, we were also able to test the efficacy of ssh’s -C compression switch. We discovered that while uncompressed machines ran at about 10-11 Megabytes per second (100Mbps limit), compressed streams ran at only 100 Kilobytes per second. We couldn’t really measure how much the data was compressed, but we were pretty sure compressed streams were being soundly outpaced by the uncompressed ones. We eventually threw our hands up in disgust and restarted the compressed stream test machine so that it would use uncompressed streams and finish before it was time to go retire.

This worked pretty well. It made our Linksys switches want to retire. No time profiling was done. We learned that this isn’t too useful out of the box without some tweaks, since intervention was required on every machine. But for a small cluster like this, it wasn’t a problem at all.

Being open source, their projects and products are often enhanced by the contributions of others. These contributions often come in the form of a “patch” — a file that tells the computer what to change in the source code to add the contribution. Each organization has their own policy for submitting patches, and their own rules for who gets to apply patches, what to do if one of them breaks something, etc.

On the Mozilla site, there is a huge corpus of documentation describing all the rules, regulations, and policies regarding path submission, application, and review. They also document how to create patches. The source of information is completely staggering. This is a mixed bag of blessing (all the information you need is available …) and curse (… if you’re patient enough to find it)… I’m more of a visual-step-by-step kind of learner, so I’m going to cover some of their basic patch creation and submission policies here. Note that just about all of this can be gleaned by reading all their documentation. But I found that this method was more intuitive to me, and perhaps a bit snappier (even if it did break their protocol a few times).

This is the process and timeline that I have experienced with my first patch — a simple modification to Mozilla Thunderbird to add a new “Add to Bcc:” button to the message composition window.

2007.04.02 — I complete the initial (alpha) version of the patch, and do some personal testing. Testing is very important.

2007.04.03 11:17:52 PDT — I revise my patch. This includes some layout changes, and some public testing. At this point, I create a patch for submission to Mozilla. The command I used: # diff -Naur mozilla.orig mozilla. When submitting a patch, they have several drop-down checkboxes with archaic options. This meant little to me, but I figured + was the best-looking option I had. I check that, and submitted the patch.

2007.04.03 14:18:39 PDT — My first submission feedback arrives: I didn’t diff against the correct original source branch, and I missed a few preferred diff parameters. Suggested command: # diff -pu8 . Also, the + option was apparently special, and I wasn’t supposed to check that (+ means the patch has been reviewed by a module owner, and has been found acceptable for superreview.) I do some testing, and find that -pu8 seems inappropriate; -p is for C function names (my patch was to some XML), and -u8 is deprecated. Correct usage is # diff -U 8 for the desired result.

2007.04.03 20:11:08 PDT — At this point, I contact the feedback guy for more information, asking what I should do to create a more acceptable patch. I create a patch against the current CVS, using similar commands to the above. Submitted a new patch to obsolete the old one.

2007.04.03 09:25:14 +0200 — He responds with some very helpful feedback regarding things a patch must include. It must be versioned against the current CVS, and it must also include index lines for revision information. I quickly form an appropriate patch with the correct format. The correct command isn’t diff at all, but this: # cvs diff -p -u -8. This makes a patch that can be applied to the source in-place. It also indexes which files are modified, and marks their revision number in CVS. This makes backing-out the patch — if necessary — easier.

2007.04.04 02:03:03 PDT — I generate the correct form of patch, and submit again (3rd patch, obsoleting the previous two). I also request review from bienvenu. It’s a long wait, so I go do other things, like work on other patches.

2007.04.05 08:19:43 PDT — The patch is granted review (it gets that + that I mistakenly set on the first one), and automatically gets promoted to superreview from mscott. However, there is a question attached that indicates a lack of actual testing. The reviewer asks about the aesthetics of the new widget.

Then, the standard Open Source Controversy takes control…
2007.04.05 13:37:10 PDT — concerned-user

> looks reasonable to me - one question - does the contacts siderbar initially
> come up wide enough to see all three buttons?

No, definitely not. For all major platforms it’s too narrow. The ‘To’ and ‘CC’
buttons already fills up the complete width. If the new hbox doesn’t wrap we
get an ugly scrollbar. On the other hand we shouldn’t enlarge the contact
sidebar anymore. It has still the correct width. IMO the ‘BCC’ button has to
wrap into the next line.

Frustrating as it is, I gather my composure and read the whole set of exchanges that have gone on in my absence. The whole “Wrap into the next line” thing really pushed my buttons since it does wrap, as designed. I specifically revised the patch before ever even posting it to include this behaviour, because, as noted, it’s ugly/unusable if it doesn’t wrap. I know this from testing, something that obviously isn’t happening anywhere else yet…

At this point, the bug officially gets assigned to me (since I’m working on the patch), and I post a reply indicating tested behavior of the patch (which no one else seems to have done thus far…):

2007.04.05 14:24:06 PDT — cwright

Woah, woah, calm down :) I’ll try to explain. In the contacts side bar, we
have the address book selector, the directory pane, and then the add-to and
add-cc button hbox. With the additional hbox for the add-bcc button, it simply
takes space from the directory listing. It does _Not_ change the width of the
side bar. It does _Not_ change the height of the side bar. It does _Not_
change the window dimensions in any way. Its modifications to the user
interface are just about the same as the ones for the extension that was
mentioned in Comment #22. Nothing radical.

I developed it and tested it on OS X, surely one of the “Major Platforms” you
reference. Normal window size is perfectly usable. Maximizing the window does
not wrap. Maximizing the window and then dragging the side-bar resizer all the
way to the right still doesn’t wrap. That’s what the spring tags are for I
believe (they’re in both hboxes, just in case).

Would it be more helpful to just fling up some before/after screenshots?

Hopefully someone will actually test the patch before it gets included in the release branch. Empirical evidence suggests that this won’t be the case though.

2007-04-09 05:36:22 PDT — (apparently, open source developers don’t work over Easter.) I get my first positive feedback: Any chance to get this into TB 2?

Don’t get your hopes up yet though. This school of thought is shot down in short order.

2007-04-09 09:14:32 PDT

(In reply to comment #40)
> add-cc button hbox. With the additional hbox for the add-bcc button, it simply
> takes space from the directory listing. It does _Not_ change the width of the
> side bar. It does _Not_ change the height of the side bar. It does _Not_

Especially on Mac OS X the default width of the sidebar is still too narrow.
The right side of the CC-Button is still cut-off. Can you provide a screenshot
after this changes were applied? Are the To and CC buttons smaller with that
patch?

> Would it be more helpful to just fling up some before/after screenshots?

Indeed. It would be.

(In reply to comment #41)
> Any chance to get this into TB 2?

No, it is too late for TB2.0 bcause we also have l10n changes here.

Once again, did anyone bother to test the patch? I upload a screen shot, and hope that gets around the “actually-build-and-look-at-the-screen” bug that these people are suffering from.

Four days pass, and all is silent. Is someone testing the patch? Did they fall asleep? Finally, someone chimes in. And, they tested and patch. And they have more to gripe about…

2007-04-13 23:52:22 PDT

Thank you Christopher. Meanwhile I got my Mac OS X to build my own builds.
After integrating your patch all looks fine. But there is one thing what I
would prefer. IMO putting the CC and BCC button on one line would make a bit
more sense due to both sending copies to given recipients. David what’s your
opinion?

It took 9 days to get someone else to test this patch. And verify it’s ok-fullness. But it only took 1 day to get review, and there’s still no word on superreview. It is now no wonder why there are so many stupidly trivial extensions out there; It seems to be the only way to actually change Thunderbird’s behavior.

The distinction between “copy” and “actual” seems very arbitrary. The only reason a To: field exists in E-mail is to mess around with reply versus reply-to-all scenarios. While BCC and CC have similar operation in this way, I feel that To: and CC: are better grouped together because both are published information, while those in the BCC field are unpublished. I don’t know that there’s a right answer to this, but I have a feeling there’s going to be some debate about it anyway.

I figured there would be some kind of discussion or suggestion period at this point. However, I was quite mistaken. Instead the list remained silent for another 7 days, and then there was another request for help applying the patch.

2007-04-20 11:01:25 PDT —

How do I apply the patch on comment #30? When I click on the link, I see the
code.

So basically, we can group our responses into two categories:

• People who want the patch applied, and have little or no experience with programming or patching
• People who like to drag their feet without qualified statements

2007-04-30 11:01:37 PDT —
Some more administrative action takes place: The QA Contact is changed from undefined to message-compose@thunderbird.bugs, a bogus-looking e-mail address.

2007.05.13 — When I initially started this entry over 40 days ago, I figured it’d be 4 or 5 weeks from start to finish. This would work out to be about 1 week per line of code, which is more than fair. After all, at that rate it would take thousands of years for the bigger patches that regularly make it in somehow.

This, unfortunately, is not the case. We’ll be pushing 6 weeks now, which is ridiculous for a non-branching, cosmetic patch. I can understand the volume of show-stopping bugs, the ones that crash Thunderbird, the “important” bugs and all that. I really do. But at rates like this, I can also see why nothing has changed significantly with the Thunderbird project in several years. Why the 1.5, 2.0, and 3.0alpha are all the same program, with minor alterations. And, most importantly, why the amount of contribution to this project leave a lot to be desired. I’ll continue another entry when/if this ever wraps up, but in the meantime, I think we may be better off switching to Horde.

Man, sometimes I hate open source…

Oracle, a database vendor, is pretty tight-lipped about its financing. Nowhere on the website is price listed. Today, we managed to breach this obfuscation. We had to call them.

Licensing for Oracle is offered on a per-CPU basis for the database servers, and on a per-machine basis for the user clients. They weren’t clear as to what constitutes a CPU — does Hyperthreading count as 2 CPUs? does Dual-Core?

Either way, the price per CPU is $40,000.

Per client machine, the license cost is $800.

Plus a support contract.

So, instead of spending well over $200,000 on this…

We’re replacing Yon Reptile Campaign’s ancient Oracle server with one running PostgreSQL.

However, Tyan — the motherboard manufacturer — saw fit to include only 1 of the typical 3 heat sink mounting tabs. Somehow, this 1 tab managed to suddenly break on Sunday, March 18, at about 4:45pm.

And now the processor has Opteron Cancer.

Opterons are not designed to run without a heat sink, so of course this lead to an almost instantaneous overheat — and the BIOS is configured such that the machine does a hard-power-off under high temperature conditions, so as to avoid thermal damage.

On the plus side, we were able to be on-site within a few hours of when this all happened, so we could assess the situation and make appropriate repairs. We replaced the mounting bracket with a spare, and continued on our way.

As the day progressed, our monitoring software showed some interesting behavior though.

The CPU temperature was abnormally high for mostly-idle conditions. In fact, it was as hot now as it used to be under full load. And to make it worse, the temperature wasn’t even stable, but instead erratically went up and down between about 50 and 70 degrees Celsius. We decided to replace the thermal paste, which binds the heat sink to the CPU, in the hopes that — because the heatsink’s been removed and replaced a few times — it was no longer operating at its optimum efficacy.

This thermal-paste-replacement procedure revealed an even more shocking discovery, however!

On the top of the CPU was some form of malignant cancerous tumor!

Opteron during server installation:

Opteron meltdown, after thermal paste was removed:

This tiny addition to the CPU adversely affects the transfer of thermal energy from the CPU to the heat sink. Because of this interruption, the temperature is now considerably higher, and more erratic. Basic attempts to pry the addition off, as if it were foreign-based, were to no avail.

Wherever this meltdown-bubble came from, and whatever it is, it is here to stay.

At least until we get that replacement CPU.

We were analyzing Yon Reptile Campaign’s IT budget, and discovered this expenditure:

This was probably a pretty big job, even compared to the task of getting this business’s IT situation under control…