Unfortunately the plugin only looked like it worked properly. It would report back correctly things like the number of volumes you had online, the number of disks, failed disks etc., but if you had a failed disk it would not actually return the proper error status. It just kept on going blindly saying OK : Bad Disks=3.

So I have written my own script to check the RAID controller status, check_megaraid_sas. It is somewhat similar to the work I did for the PERC3Di with afacli and Nagios quite a while back.

In order to use it you need to have LSI's MegaCli utility installed and the user executing the script will need to have sudo privileges (w/o a password) to execute it. Then you will end up with output like:
OK: 0:0:RAID-1:2 drives:68GB:Optimal 1:0:RAID-5:7 drives:2792GB:Optimal Drives:10 Hotspare(s):1
or (less good)
WARNING: 0:0:RAID-1:2 drives:74GB:Optimal 0:1:RAID-5:4 drives:1396GB:Optimal Drives:6 (3 Errors)

The warning is due to the detection of "other" disk errors on the drive. I am trying to find out from Dell if I can reset this count in the controller. Otherwise if it is cumulative I will probably modify my code to take a n argument for a threshold under which to ignore non-fatal errors. The output above is basically in the form:
<status> <controller #>:<volume #>:<RAID level>:<volume drive count>:<volume size>:<volume status> ... Drives:<total drives attached to controller(s)>

Having fell foul to a failed RAID container, I quickly went out and found the latest firmware updates which I knew existed but had yet to apply to the failed system. I applied it to the system and a couple of other which I knew were also in need. Installed on a bootable flash drive, the updates went by rather quickly. Yay.

I reboot the system and all seems well, that is until I notice that syslog is running full load on one CPU. So I look at my /var/log/messages and it is chock full of errors like this, at a rate like 100-line per second:

May 10 09:30:03 doomed kernel:   status = 1, message = 00, host = 0, driver = 08
May 10 09:30:03 doomed kernel:   <6>sd: Current: sense key: Illegal Request
May 10 09:30:03 doomed kernel:     Additional sense: Invalid command operation code
May 10 09:30:03 doomed kernel: FAILED

So this is very rapidly filling up my /var partition and making me unhappy. Very unhappy.

I thought this was perhaps related to a failed rebuild of my newly initialized RAID container... so I waited for it to finish 1.43TB of RAID building, and it still spewed away. Googling the error message didn't yield a whole lot in results. Eventually I looked on the Dell site to see if their were any other updates I was missing, and lo and behold the day after they released the PERC5/i firmware update there was also a driver update.

I downloaded the driver. It requires The Dell DKMS (Dynamic Kernel Module Support), so I download that as well. Install the DKMS rpm, no problem. Install the megaraid_sas rpm, no problem their either. It installs the module, does a mkinitrd, and waits for me to reboot. I reboot and all is hunky dory. No more crazy errors. I am happy and decide it is now time to rinse and repeat with my other two servers exhibiting the same problem.

No joy. When I go to the next box and install the megaraid_sas rpm I get:

Preparing...                ########################################### [100%]
   1:megaraid_sas           ########################################### [100%]

Loading tarball for module: megaraid_sas / version: v00.00.03.09

Loading /usr/src/megaraid_sas-v00.00.03.09...
Creating /var/lib/dkms/megaraid_sas/v00.00.03.09/source symlink...

DKMS: ldtarball Completed.

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area....
make KERNELRELEASE=2.6.16.27-0.9-smp -C /lib/modules/2.6.16.27-0.9-smp/build SUBDIRS=/var/lib/dkms/megaraid_sas/v00.00.03.09/build modules....(bad exit status: 2)

Error! Bad return status for module build on kernel: 2.6.16.27-0.9-smp (x86_64)
Consult the make.log in the build directory
/var/lib/dkms/megaraid_sas/v00.00.03.09/build/ for more information.

Error! Could not locate megaraid_sas.ko for module megaraid_sas in the DKMS tree.
You must run a dkms build for kernel 2.6.16.27-0.9-smp (x86_64) first.
error: %post(megaraid_sas-v00.00.03.09-1.noarch) scriptlet failed, exit status 4

WTF?!?

It ends up that the first box I updated was actually a pristine install of SLES10 as released. No kernel updates no nothing. So all it ended up doing, it seems, was unpacking a precompiled module from somewhere and stuffing it in the right place. Hell, I didn't even have the kernel-source rpm installed on it.

The referenced make.log looks like:

DKMS make.log for megaraid_sas-v00.00.03.09 for kernel 2.6.16.27-0.9-smp (x86_64)
Thu May 10 21:19:18 EDT 2007
make: Entering directory `/usr/src/linux-2.6.16.27-0.9-obj/x86_64/smp'
make -C ../../../linux-2.6.16.27-0.9 O=../linux-2.6.16.27-0.9-obj/x86_64/smp modules
  CC [M]  /var/lib/dkms/megaraid_sas/v00.00.03.09/build/megaraid_sas.o
/var/lib/dkms/megaraid_sas/v00.00.03.09/build/megaraid_sas.c: In function 'megasas_probe_one':
/var/lib/dkms/megaraid_sas/v00.00.03.09/build/megaraid_sas.c:2629: error: 'IRQF_SHARED' undeclared (
first use in this function)
/var/lib/dkms/megaraid_sas/v00.00.03.09/build/megaraid_sas.c:2629: error: (Each undeclared identifie
r is reported only once
/var/lib/dkms/megaraid_sas/v00.00.03.09/build/megaraid_sas.c:2629: error: for each function it appea
rs in.)
make[3]: *** [/var/lib/dkms/megaraid_sas/v00.00.03.09/build/megaraid_sas.o] Error 1
make[2]: *** [_module_/var/lib/dkms/megaraid_sas/v00.00.03.09/build] Error 2
make[1]: *** [modules] Error 2
make: *** [modules] Error 2
make: Leaving directory `/usr/src/linux-2.6.16.27-0.9-obj/x86_64/smp'

If I take a look in /var/lib/dkms/megaraid_sas/v00.00.03.09 I see that there is a patches directory, and in there a sles10-ga.patch. Part of the patch file says:

-       if (request_irq(pdev->irq, megasas_isr, IRQF_SHARED, "megasas", instance)) {
+       if (request_irq(pdev->irq, megasas_isr, SA_SHIRQ, "megasas", instance)) {

Well, that is odd, because the make log was complaining about IRQF_SHARED... so it seems the patch was not getting applied to the source at all. K-lame. Whatever detection mechanism there is for the distro you are running has obviously failed here. So a little of this:

 # patch < patches/sles10-ga.patch 
patching file megaraid_sas.c
# dkms build -m megaraid_sas -v v00.00.03.09
 
Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area....
make KERNELRELEASE=2.6.16.27-0.9-smp -C /lib/modules/2.6.16.27-0.9-smp/build SUBDIRS=/var/lib/dkms/megaraid_sas/v00.00.03.09/build modules....
cleaning build area....

DKMS: build Completed.

and presto, the module builds. Complete the task with a

# dkms install -m megaraid_sas -v v00.00.03.09
Running module version sanity check.

megaraid_sas.ko:
 - Original module
   - Found /lib/modules/2.6.16.27-0.9-smp/kernel/drivers/scsi/megaraid//megaraid_sas.ko
   - Storing in /var/lib/dkms/megaraid_sas/original_module/2.6.16.27-0.9-smp/x86_64/
   - Archiving for uninstallation purposes
 - Installation
   - Installing to /lib/modules/2.6.16.27-0.9-smp/kernel/drivers/scsi/megaraid//

/etc/modprobe.conf: added alias reference for 'megaraid_sas'
depmod.....

Saving old initrd as /boot/initrd-2.6.16.27-0.9-smp_old
Making new initrd as /boot/initrd-2.6.16.27-0.9-smp
(If next boot fails, revert to the _old initrd image)
mkinitrd.....

DKMS: install Completed.

and everything is good to go. Yay. Reboot. Doom averted. Go home.

A co-worker of mine came up with a subsequently much neater solution to this:

The fix is the make dkms aware of newer SLES 10 kernel versions by doing the following:
vi /usr/src/megaraid_sas-v00.00.03.09/dkms.conf

Change the following lines:

PATCH[6]="sles10-ga.patch"
PATCH_MATCH[6]="2\.6\.16\.21-0\.8"

to:

PATCH[6]="sles10-ga.patch"
PATCH_MATCH[6]="2\.6\.16\.2.-0\..*"

There is a Bertucci's not far from where I live in Cambridge, on Main St (they call it their Kendall Sq. restaurant, but it is really in Central Sq. ... I suspect it is wishful thinking on their part) . At one point in time, long long ago, it was your basic run of the mill Bertucci's. I.e. it served something along the lines of better-than-your-corner-pizza-shop pizza, salads, beer, and the service was OK. We, the young family, would go on occasion if we didn't feel like cooking and wanted something simple in a child-tolerant setting.

Then the service started to go down hill and the food quality suffered. Things would happen like your server forgetting about your order as it sat getting cold on the counter. Or pizzas that were under-cooked with raw dough in the middle. So we stopped going for a couple of years. We started going again a while back because we had activities in Central in the evenings, tried it again for a quick dinner, and it seemed to work OK (decent food, OK service at times).

Tonight though, it just completely sucked. I already wrote a bunch of feedback which I submitted to the Bertucci's website, but I figured I might as well dump it here as well to share with potential Googlers. Here is what I said:

When we arrived the person who seated us (who I later heard referred to as the Manager) neither greeted us nor even asked how many of us were dining. We just received a silent escort to a booth.

Shortly after we ordered we noticed that the restaurant was getting rather warm. We asked our waitress about this and were informed that yes, they were having problems with the AC and that the staff were taking turns in the walk-in cooler. Our food arrived, but it was so uncomfortable that we stopped eating 1/3 through.

By the time our waitress made it back to our table to check in on us the AC had come back on, but we were to overheated to enjoy our food anymore, which we said to her and asked for it to be wrapped up to go. This earned us a huff from the waitress as she whisked the pizzas away. We paid promptly and left.

To top it off, when we got home, no more than 5 minutes later, we discovered that we forgot a Nalgene water bottle in the booth. So we went immediately back to get it. When we got there and asked about it we were told "Oh, we'll have to see if we still have it." The we were told that "It is being washed right now" and could we "wait a few minutes for it"? Then we were told that they normally throw they find like this away. All of which together sounds like "We were hoping that you weren't coming back for this and planned on keeping it ourselves".

I have had my share of slow and inattentive service at this restaurant in the past, but returned because I considered this to be outweighed by the predictably reasonable quality food and environment for the price. This evening though has pretty much finished it for me there. I really have to wonder what is going on with the management of this Bertucci's. They must be able to get by on something other than their in-restaurant level of service.

I may be willing to give a different Bertucci's a try eventually if the need arises, but for now I have no plans on returning to one... especially this one.

On May 27th, 2006 (a Saturday) I took a motorcycle ride down to Nantasket Beach in Hull, MA and took a few photos. If all you want is to cut to the chase and the photos, just go to the gallery page.

Otherwise, read on.

The first has to do with a friend of mine, Jason. Now Jason doe lots of stuff. More than I could (or should) keep track of. One things that he does is film documentaries. He did one on BBSes (which I have still yet to watch all of... sorry Jason!). He is also somewhere in the process of doing one on arcades. He travels to arcades, films, snaps photos etc etc. Invites me too. Yay, I love arcades. Boo, I usually have some sort of family obligation to fulfill on the days I am invited along. Such is life.

It gets me thinking about arcade though.

I stopped going to arcades in the mid 1990's or so... when the games were starting to be just video games, and they were getting to be pretty tired ones at that. Especially when they were compared to computer and console games. Also, at least in the Boston area, arcades were disappearing. Probably all my fault, me and them computer games.

I started going to arcades though when I was around 5 or 6 years old (early/mid 1970's). I lived in Worcester MA with my dad, and, as best as I can remember, going out for entertainment meant playing pinball with him and some of his friends from school (Whoopie Tech). Pinball in Greek diners. Pinball in small arcades in malls with lots of flashy lights and flared trousers.

When we moved to the suburbs of Boston there were still trips to the arcade, but then it meant Fun and Games (which in high school we put down as "FaGs") in Framingham. Dad played pinball, I started to play more video games as well as the second player up on the ocassional game of pinball.

The best times though were when we would drive down to Nantasket Beach in the summer time in our Rabbit convertible. It was the mid-1980's, so video games were certainly king, but the arcades at Nantasket Beach were just so huge that they seemed to keep everything. Huge warehouse-like expanses with worn wood floors. Tons of old pinball games that still took dimes to play. Old mechanical baseball games. Machines that you could stamp a message into the border of souvenir coins with. Skeeball. Fortune-telling machines. Every kind of arcade game that probably every came out up until then there was an example of. We would spend hours there. Have some dinner from the food stand, get a box of salt water taffy, and then drive home when it was well dark.

What I didn't know then was that I was probably experiencing the final great days of Nantasket Beach with its arcades and amusement park. I remember going there sometime around 1991 or so on motorcycles with my dad, my girlfriend (now wife), and some friends. The arcades were still there, but some of them had shut down. It was no longer one huge continuous strip of arcade after arcade. The amusement park was closed, or on the verge of closing, so that condos could be built with big ocean views instead. I remember getting completely soaked riding back in the rain to top it all off.

This brings us back to the current day. I got to see the fruit born of my last visit to Nantascket Beach some 15 years ago. The condos are there, already looking dated... just another fugly example of the crap that sprang up all along the South Shore in the late 1980's and early 1990's. Only a small hint of what I remember remains of the arcades. Some names are still there, but the guts are gone.

I'm sure there are plenty of people who have connected with all sorts of other places similarly in the past. That place that you went to with your dad and had some meaningful activity the memories of stick with you to this day that has now disappeared, leaving you to wonder if you could ever have anything close to as meaningful somewhere else with your own child in the future that they would be able to look back upon the same way.

Life right now is in a wierd combination of the two: terribly busy in general, but not a whole lot going on. Or at least it seems this way. This is as likely due to the prolonged stretch of rainy weather that we have had here... I haven't seen any sunshine for a week and a half and the extended forecast doesn't offer much hope.

I have been able to set a bit of time aside though now and then and worked on my Triumph Speed Triple website, Speed Three. This evening, for example, I have started work on the "bits" section, a page well needed to help me retain the dribs and drabs of info I come across about bike accessories.

I just finished reading all 10+ volumes of Transmetropolitan, written by Warren Ellis with art by Darick Robertson and Rodney Ramos, and I would honestly have to rank it as one of the best comics I have read.

The same day I take one of my occasional lunchtime walks to Million Year Picnic to see if any of the usual stuff I read showed up. Of course, by the time I get there I have completely forgotten the title of the comic, and have a vague recollection of the author's name. Saying that I am looking for a comic by Warren Ellis who I have only just heard of gives the guy at the counter the sense that I have just crawled out of a cave... fortunately I can describe the chapter I read and he knows it and has it out in 30 seconds. I bought the first two volumes. If I had known just how fast I was about to consume it all I probably would have bought all eleven (1-10 + 0) at a time and tried to hit them up for a discount.

Anyways, amazing stuff.

...and the results are....

It sucks

It seems that when I save the entry to publish it goes as far as comitting it to the database, but it fails to actually rebuild the pages accordingly. This is probably some "The Sidekick web browser doesn't support Javascript" foolery. No big loss.

There isn't really much else to say here about it at this point.