This is somewhat related to my earlier posting about updating the megaraid drivers. I use Nagios at work for system monitoring and one thing that I like to check is the status of the volumes managed by the RAID controller. When I first started configuring the Nagios on my new PowerEdge 1950 and 2950 systems I found a check_perc5i over on Nagios Exchange.
Unfortunately the plugin only looked like it worked properly. It would report back correctly things like the number of volumes you had online, the number of disks, failed disks etc., but if you had a failed disk it would not actually return the proper error status. It just kept on going blindly saying OK : Bad Disks=3.
So I have written my own script to check the RAID controller status, check_megaraid_sas. It is somewhat similar to the work I did for the PERC3Di with afacli and Nagios quite a while back.
In order to use it you need to have LSI’s MegaCli utility installed and the user executing the script will need to have sudo privileges (w/o a password) to execute it. Then you will end up with output like:
OK: 0:0:RAID-1:2 drives:68GB:Optimal 1:0:RAID-5:7 drives:2792GB:Optimal Drives:10 Hotspare(s):1
or (less good)
WARNING: 0:0:RAID-1:2 drives:74GB:Optimal 0:1:RAID-5:4 drives:1396GB:Optimal Drives:6 (3 Errors)
The warning is due to the detection of “other” disk errors on the drive. I am trying to find out from Dell if I can reset this count in the controller. Otherwise if it is cumulative I will probably modify my code to take a n argument for a threshold under which to ignore non-fatal errors. The output above is basically in the form:
<status> <controller #>:<volume #>:<RAID level>:<volume drive count>:<volume size>:<volume status> ... Drives:<total drives attached to controller(s)>
Tags: nagios
Continual False Warnings after RAID Rebuild
Hi,
I’ve got a Server with a Raid 5 configuration installed with a Megaraid Raid Controller. I had to replace a faulty disk. It has rebuilt fine and the Raid array is back online.
I’ve got a Nagios Monitoring server checking this box. It has NRPE 2.12 installed. The Remote Servers have Nagios 3.12, NRPE 2.12, nagios-plugins-1.4.13, check_megaraid_sas and MegaCli-4.00.11.rpm installed.
Nagios is constantly reporting the LSI RAID STATUS as WARNING, even though all is OK. I assume it is referring to the history of the failure somehow…
Can anyone help ?
Thanks
Bozz
hello
i have to change line 132 in your script to
if ( m/Size:\s*(([\d\.]+)\s*(MB|GB|TB))/ ) {
because my megacli (MegaRAID SAS 8880EM2) returns size of logical disk like this:
Size:837.312 GB
origin regular expression does not like . and space between size and unit
regards Klaba
Ah, that is actually fixed in my latest version (rev 9) of the plugin, only it seems that I don’t have that latest version uploaded here on my own site (which I will fix). The latest version is (usually) up on the Nagios Exchange (or whatever they are calling it now, I keep forgetting) site as soon as I make any changes. I will dig up the links for them and edit this post for better future reference.
And here are the links to the two sites I try to keep it updated at:
MonitoringExchange (this seems to be the more popular source, it actually gets some comments there)
Nagios Exchange
-Jonathan
I am getting this error when trying to use this with one of the newer LSI 9260-8i cards.
This is the error
/usr/lib/nagios/plugins/check_megaraid_sas
Use of uninitialized value in concatenation (.) or string at /usr/lib/nagios/plugins/check_megaraid_sas line 163.
Use of uninitialized value in concatenation (.) or string at /usr/lib/nagios/plugins/check_megaraid_sas line 163.
Use of uninitialized value in concatenation (.) or string at /usr/lib/nagios/plugins/check_megaraid_sas line 163.
Use of uninitialized value in concatenation (.) or string at /usr/lib/nagios/plugins/check_megaraid_sas line 163.
Use of uninitialized value in concatenation (.) or string at /usr/lib/nagios/plugins/check_megaraid_sas line 163.
Use of uninitialized value in concatenation (.) or string at /usr/lib/nagios/plugins/check_megaraid_sas line 163.
Use of uninitialized value in concatenation (.) or string at /usr/lib/nagios/plugins/check_megaraid_sas line 163.
Use of uninitialized value in concatenation (.) or string at /usr/lib/nagios/plugins/check_megaraid_sas line 163.
Use of uninitialized value in concatenation (.) or string at /usr/lib/nagios/plugins/check_megaraid_sas line 163.
Use of uninitialized value in concatenation (.) or string at /usr/lib/nagios/plugins/check_megaraid_sas line 163.
OK: 0:0:RAID-: drives:: 0:1:RAID-: drives:: Drives:6
Here is the output of the commands the script appears to be running.
MegaCli -LdInfo -L0 -a0
Adapter 0 — Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name :
RAID Level : Primary-6, Secondary-0, RAID Level Qualifier-3
Size : 40.0 GB
State : Optimal
Stripe Size : 64 KB
Number Of Drives : 6
Span Depth : 1
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Access Policy : Read/Write
Disk Cache Policy : Disabled
Encryption Type : None
MegaCli -LdInfo -L1 -a0
Adapter 0 — Virtual Drive Information:
Virtual Drive: 1 (Target Id: 1)
Name :
RAID Level : Primary-6, Secondary-0, RAID Level Qualifier-3
Size : 516.875 GB
State : Optimal
Stripe Size : 64 KB
Number Of Drives : 6
Span Depth : 1
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Access Policy : Read/Write
Disk Cache Policy : Disabled
Encryption Type : None
Any help would be greatly appreciated.
Thank You,
Jason D.
Thanks for putting this together, having the same issue as Jason D.
LSI 9260-4i
MegaCLI 8.00.23
CentOS 5.5 x86_64
/usr/lib64/nagios/plugins/check_megaraid_sas
Use of uninitialized value in concatenation (.) or string at /usr/lib64/nagios/plugins/check_megaraid_sas line 163.
Use of uninitialized value in concatenation (.) or string at /usr/lib64/nagios/plugins/check_megaraid_sas line 163.
Use of uninitialized value in concatenation (.) or string at /usr/lib64/nagios/plugins/check_megaraid_sas line 163.
Use of uninitialized value in concatenation (.) or string at /usr/lib64/nagios/plugins/check_megaraid_sas line 163.
Use of uninitialized value in concatenation (.) or string at /usr/lib64/nagios/plugins/check_megaraid_sas line 163.
OK: 0:0:RAID-: drives:: Drives:4
Thanks for your help!
It seems the latest version of MegaCli has added new areas of whitespace in their output. Minor adjustments were made to the script to cope with this and it is now available as revision 10 from Nagios Exchange and Monitoring Exchange (linked to in an earlier comment).
Thanks Jonathan, much appreciated :)
Hi,
I got a similar error with MegaCLI Ver 5.00.10:
Use of uninitialized value $ldpdcount in concatenation (.) or string at ./check_megaraid_sas line 165.
I tried the newest available version of the script but the error is still there.
Maybe you have a solution for my problem.
thanks Mathias
Mathias: I have posted a newer revision (11) to the 2 different plugin sites. Please give it a shot. It contains a simple change to the pattern matching do address the occasion when MegaCli reports the “Number of Drives per span” and not just “Number of Drives”. I am not quite sure if this breaks some of the sense that the logical drive status/count is reported in, as I don’t have an example locally of that kind of configuration to refer to. Anyways, perhaps this newest update will help.
Thanks you very much for this fast support, the error is solved now.
best regards Mathias
A user of the plugin just pointed out to me that the newer versions of MegaCli will dump a copy of all output into a logfile named MegaSAS.log by default. I have updated the plugin so that it passes an additional “-NoLog” argument to MegaCli on all calls. From what I can tell, this additional argument has no influence on older version of MegaCli (which doesn’t seem to log). You can also just blank out the value if you don’t want it changed.
All of this and some other minor changes are available now in rev 12.
I have all set up. Run locally I get,
WARNING: 0:0:RAID-10:4 drives:930.5GB:Optimal Drives:6 (5 Errors)
Run through nrpe I get,
OK: Drives:0
What does the Drives:0 mean?
This is a common enough problem, so I guess it ought to be part of some FAQ that I never thought I would need to publish.
If the plugin runs just fine when you execute it manually, but fails like this with NRPE, then there is most likely a permissions issue for whatever user you NRPE daemon is running as on your local system. NRPE is trying to run MegaCli using sudo but failing to do so, and everything just sort of craps out from there.
Typically NRPE will be running as the “nagios” user. You will want to make sure that in your sudoers file you have an entry like:
nagios ALL=(ALL) NOPASSWD: /usr/sbin/MegaCliAdjust accordingly with the proper path to where MegaCli is installed. This, of course, assumes NRPE is actually running as the nagios user and that user is really defined in the system. On my SLES systems there wouldn’t be any nagios user in place by default, so I would need to do something along the lines of
# /usr/sbin/groupadd -r nagios# /usr/sbin/useradd -c 'Nagios Pseudo-user' -r -g nagios -d /etc/nagios nagios
before firing up NRPE.
All of that was performed. Before I set the user of check_megaraid_sas to nagios I got a message stating it could not understand nrpe output. After changing check_megaraid_sas user to nagios I recieved the text ‘OK: Drives:0′
Before my first post, the entry for nagios was in sudoer as you have shown. I just confirmed nagios is a user (on both local and remote). Any other ideas?
I found nrpe was running as user nrpe. I changed the nrpe.conf file and told it to run as nagios and restarted daemon, but still get same ‘OK: Drives:0′. A ps aux|grep nrpe showed it was running as nagios now.
This block of code is setting adapters to 0.
# Get the number of RAID controllers we have
open (ADPCOUNT, “$megacli -adpCount $megapostopt |”)
|| exitreport(‘UNKNOWN’,”error: Could not execute $megacli -adpCount $megapostopt”);
while () {
if ( m/Controller Count:\s*(\d+)/ ) {
$adapters = $1;
last;
}
}
When I execute MegaCli -AdpCount at commandline I get;
Controller Count: 1.
Exit Code: 0×01
If I execute MegaCli -AdpCount as a regular user I get,
Controller Count: 0.
Exit Code: 0×00
If it allowed regular user to execute, why did it not return correct value. Why did it return exit code of 0? Does a o mean an error?
It seems a normal user does not have access rights to the device which MegaCli is getting its information from. So I set the suid bit on MegaCli and since root is the owner I thought that would fix the issue, but it did not. Any ideas?
The check_megaraid_sas script is setup to call MegaCli via sudo (e.g. sudo /usr/sbin/MegaCli), so it should be getting called with root privileges. If MegaCli on your platform (which you haven’t mentioned) yields different results when executed directly as the root user compared to when executed via sudo, then I am not sure if there is a whole lot left you can do.
One possible solution, but a bad (really bad) practice, would be to have the nrpe daemon running as the root user and get rid of the whole sudo/non-root user issue.
I forgot to use sudo. Normal user does get correct output when using sudo.
What I have figured out is the ‘while’ never gets entered. Also, exitreport is not called.
# Get the number of RAID controllers we have
open (ADPCOUNT, “$megacli -adpCount $megapostopt |”)
|| exitreport(‘UNKNOWN’,”error: Could not execute $megacli -adpCount $megapostopt”);
while () {
open(HANDLE, “>>/tmp/check_megaraid_err.txt”);
printf HANDLE “HERE\n”;
close (HANDLE);
if ( m/Controller Count:\s*(\d+)/ ) {
$adapters = $1;
last;
}
}
The code below works when I login as nagios user. As nagios, I can also directly run check_megaraid_sas and get correct output. But if nrpe (which is running as nagios) runs it, it never enters the while loop as mentioned in earlier post.
===CODE===
#!/usr/bin/perl -w
my $megaclibin = ‘/usr/sbin/MegaCli’; # the full path to your MegaCli binary
my $megacli = “/usr/bin/sudo $megaclibin”; # how we actually call MegaCli
my $megapostopt = ‘-NoLog’; # additional options to call at the end of MegaCli arguments
my $adapters;
open (ADPCOUNT, “$megacli -adpCount $megapostopt |”);
while () {
if ( m/Controller Count:\s*(\d+)/ ) {
#if ( m/Controller Count:\s*(\S+)/ ) {
$adapters = $1;
last;
}
}
print “$adapters\n”;
close ADPCOUNT;
===END CODE===
I don’t know how the code above got pasted droping some text, but here it is again.
If I change the command to run in check_megaraid_sas to ‘sudo ls -l /tmp’ it also will not enter the while loop. If I do not use sudo, it works. I am puzzled.
===CODE===
#!/usr/bin/perl -w
my $megaclibin = ‘/usr/sbin/MegaCli’; # the full path to your MegaCli binary
my $megacli = “/usr/bin/sudo $megaclibin”; # how we actually call MegaCli
my $megapostopt = ‘-NoLog’; # additional options to call at the end of MegaCli arguments
my $adapters;
open (ADPCOUNT, “$megacli -adpCount $megapostopt |”);
while () {
if ( m/Controller Count:\s*(\d+)/ ) {
$adapters = $1;
last;
}
}
print “$adapters\n”;
close ADPCOUNT;
===END CODE===
Ok, I have nailed it. For future reference, here is what makes it work. They both are required. set the suid bit on MegaCli, and add nagios to ACL on raid interface device. The 2 commands I used are,
chmod u+s /usr/sbin/MegaCli
setfacl -m u:nagios:r /dev/megaraid_sas_ioctl_node
The setfacl will have to be executed after each reboot unless you find the option which makes it permanent.
@Durwin,
I have done what you recommended and still cannot get it to work on a RHEL5 x86_64 box. I was able to get it to work on one machine, but cannot get it to work on two others and I have verified that the configs are exactly the same on both machines. For me, NRPE runs as NRPE user. I get this when I try to set setfacl:
setfacl: /dev/megaraid_sas_ioctl_node: Operation not supported
–mike
Mike:
Check out this thread.
Also edit /etc/nagios/nrpe.cfg on remote machine and set user and group for nrpe to run as to nagios.
Opps,
http://www.linuxquestions.org/questions/mandriva-30/setfacl-test-operation-not-supported-266804/
@Durwin
For me, commenting out \Defaults requiretty\ in the sudoers file did the trick. It just could not have been the ACLs for me as NRPE worked fine on one of the identical machines with near-identical installs without the ACLs. Restarting the machine to enable the ACL support just was not something we could bring ourselves to do (db machine).
I decided to try to look at NRPE on a lower level, so I ran some packet captures using tcpdump and nrpe was indeed connecting. I then checked my /var/log/secure and that is where I noticed this:
\sudo: nrpe : sorry, you must have a tty to run sudo ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/MegaRAID/MegaCli/MegaCli64 -adpCount -NoLog\
This is what got me to thinking about the \requiretty\ option that I saw on some other plugins on Nagios Exchange. It was also one of the things that the working host did not have set, therefore it worked. Commenting that line out in my sudoers file automatically enabled NRPE to perform the check on that script.
Thanks though, hopefully if somebody else is having this issue, it will help them! :)
–Mike
I get the same issue as Durwin however my problem is more complicated:
I’m using a Dell PERC6/i RAID card.
The MegaCLI program is being executed as root.
Yet I’m still getting that error. It’s like the MegaCLI program doesn’t even see my RAID card. Any ideas?
“Controller Count: 0.
Exit Code: 0×00″
The script works great when executed on the local machine, but when I trying to execute from the Nagios server it just gives “(Null)”.
I changed the following line in the script to fit my install:
my $megaclibin = ‘/opt/MegaRAID/MegaCli/MegaCli’; # the full path to your MegaCli binary
I also added these lines to the commands.cfg and the services.cfg, I’m not really sure if it’s supposed to be done like this though. So any help would be appreciated.
Code in commands.cfg:
define command{
command_name check_megaraid_sas
command_line perl $USER1$/check_megaraid_sas -H $HOSTADDRESS$
}
code in services.cfg
define service{
use generic-service
hostgroup_name xen-servers
service_description check_megaraid_sas
check_command check_megaraid_sas
}
So far I can tell it the paths and permissions are correct, so I think it does execute something because otherwise my nagios web interface would’nt work.
If you are just using the plugin to check the RAID locally on your Nagios server, then you would want an entry like:
define command{
command_name check_megaraid_sas
command_line $USER1$/check_megaraid_sas
}
You might optionally pass flags to specify hotspare count, etc. There shoudn’t be a ‘perl’ there, and there is no -H flag for checking a remote host. If you need to check a remote host’s RAID status, then you need to have the pluging installed on the remote machine and check it via NRPE with a check command like:
define command{
command_name check_megaraid_sas
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c check_megaraid_sas -t 30
}
It works, it seems iptables was bugging as well so I just deleted all the rules. It blocked the connection despite the added accept rules.
Hi There,
after using your plugin for quite a long time I got the warning the first time:
WARNING: 0:0:RAID-10:4 drives:930.5GB:Optimal Drives:4 (180 Errors)
The number of errors counts up by the number of drives with each new check.
I used version 1.1 of the plugin and tried also 1.0.
So what may I do?
Thanks,
Matthias
I’m having the same issue as Matthias Fleschütz, number of errors increaes every time the script is called.
This is probebly not an error of the script, as
megacli -PdList -a0 -noLog
increases the “Other Errors” counter for each disk on every call.
I’m using the 8.01.06_Linux_MegaCLI.zip fro LSIs site with a “PERC 6/i Integrated” Adapter. On another machine with a “PERC H700″ Adapter I’m not seeing this issue.
I just “fixed” this by installing version 8.00.29 of the MegaCLI tool (instead of 8.01.06)
My guess is the newer MegaCLI versions issue some command the older controller does not understand, and thus makes them report an error.
So now, can I somehow reset that counter?
Hi,
We recently replaced a drive in our RAID array and about 2 weeks later I got a message similar to what you state in your description above:
WARNING: 0:0:RAID-5:4 drives:1394GB:Optimal Drives:4 (4 Errors)
What I am wondering is if those errors are listed out somewhere so I can see if it is something I need to address or not?
Thanks for a great plugin!
Ben
Hi, just found your plugin recently and started testing it on one of my systems, when I found a small gotcha- apparently MegaCli under vmware as of version 8.02.16 has a small change that puts the word “Size” in the ldinfo output several times, ie:
Adapter 0 — Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name :
RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3
Size : 271.899 GB
Parity Size : 135.949 GB
State : Optimal
Strip Size : 64 KB
Number Of Drives : 3
Span Depth : 1
Default Cache Policy: WriteThrough, ReadAhead, Cached, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAhead, Cached, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Enabled
Encryption Type : None
Is VD Cached: No
which was creating the output of:
OK: 0:0:RAID-5:3 drives:135.949GB:Optimal 0:1:RAID-1:2 drives:135.899GB:Optimal 0:2:RAID-1:2 drives:0GB:Optimal 0:3:RAID-5:3 drives:135.949GB:Optimal 0:4:RAID-5:3 drives:0GB:Optimal 0:5:RAID-5:8 drives:0GB:Optimal Drives:24 Hotspare(s):3
(yes, I have several disks, and I didn’t feel like putting out all the listing)
You can see that the first virtual disk isn’t correct, it should read 271.899GB, however the script output shows 135.949GB. it gets worse later, as disk sizes of 0 show up.
on line 144, change
if ( m/Size\s*:\s*((\d+\.?\d*)\s*(MB|GB|TB))/ ) {
to
if ( m/^Size\s*:\s*((\d+\.?\d*)\s*(MB|GB|TB))/ ) {
rerun script and get
OK: 0:0:RAID-5:3 drives:271.899GB:Optimal 0:1:RAID-1:2 drives:135.899GB:Optimal 0:2:RAID-1:2 drives:135.899GB:Optimal 0:3:RAID-5:3 drives:271.899GB:Optimal 0:4:RAID-5:3 drives:271.898GB:Optimal 0:5:RAID-5:8 drives:951.794GB:Optimal Drives:24 Hotspare(s):3
Thanks again for the great script, and sorry for the huge text wall!
Thanks for a useful plugin!
Did you ever learn more about what “other” errors are? I find mentioned elsewhere a suggestion that they often come from cable problems. But does anyone know a definition? Are they cumulative? Is there a way to reset them?
Also, what’s a reasonable number to ignore for media errors? I see in an older LSI doc that at least some of their controllers fail a drive at 32 media errors. I see in another doc advice to replace a drive with even a single media error.
Figuring out what the “Other” errors were never really ended up being much of a priority for me. The ones I see seem to indicate communications issues between the controller and the drive. They do seem to get reset at times, but I have not seen a way to directly reset the count via MegaCli or anything else without a reboot of the system.
I am also not sure what is a reasonable number for media errors. I have only seen a few drives where the media error has gone over a dozen, and in those cases the error count was low (or zero) and then exploded with the drive failing shortly thereafter. I don’t know if once the media error count hits a certain number the predictive failure flag is set, or is it really a drive-specific parameter. Media errors are, to an extent, to be expected and the drive should be invisible recovering from them up to a point. Are the ones being count unrecoverable errors or revoverable ones?
I don’t think the LSI is really providing any documentation on what these error types mean and how they ought to be interpreted.
Thanks for the plug in and this post, I just discovered that I have 15 “other errors” on one of my drives.
I’ll try contacting LSI tomorrow and see if I can squeeze any info out of them. I’ll also try powering down the server and changing out the cable in hopes of removing the errors.
I have installed Megacli on TWO server ,One server is working fine but other is giving below warning
WARNING: 0:0:RAID-1:2 drives:232.0GB:Optimal 0:1:RAID-5:5 drives:1.086TB:Optimal Drives:7 (1379 Errors)
I have updated the Megacli version to 8.00.29 still get the same error.
My RAID controller card is 02:00.0 RAID bus controller: LSI Logic / Symbios Logic LSI MegaSAS 9260 (rev 03)
Any help would be Appreciated.
I have returned small below script to disappear this errors.
MegaCli -AdpAllInfo -aAll | grep -A 8 “Device Present” | sed -e ‘s/^[ \t]*//’ | grep -E “Degraded|Offline|Critical\ Disks|Failed\ Disks” | \
while read line
do
i=`echo “$line”|awk ‘{print $NF}’`
j=`echo “$line”|awk ‘{print $1}’`
# echo “$line $test”
if [ "$i" == "0" ]
then
echo “$j : Status = OK”
else
echo “$j : Warning”
fi
done
}
status > /tmp/diskstatus
for stat in `cat /tmp/diskstatus | awk ‘{print $NF}’`
do
if [ "$stat" == "OK" ]
then
echo “OK: ” > /tmp/status
else
echo “WARNING: ” > /tmp/status
cat /tmp/diskstatus | grep -v “OK” >> /tmp/status
perl -p -e -i ‘s/\s+$/ /g’ /tmp/status
cat /tmp/status
exit
fi
done
cat /tmp/status
rm -rf /tmp/diskstatus /tmp/status
Is this OK? or script will not generate warning output when there is DISK error in RAID?
Appreciate your reply. Comments also welcome.
Thanks in advanced.