Here is my experience with HP SmartArray controllers with FreeNAS. Maybe this can help out others :).
Preface: Old HP server hardware is cheap! I have over 75TB of data and not much extra cash to drum up a proper solution. Keep in mind that this is for my home and the data isn't mission critical. I'm not a daily linux user. I know that you lose out on a lot of features by running each drive in an individual RAID0 logical volume. Here are some of the things I've done to mitigate potential issues.
Preface: Old HP server hardware is cheap! I have over 75TB of data and not much extra cash to drum up a proper solution. Keep in mind that this is for my home and the data isn't mission critical. I'm not a daily linux user. I know that you lose out on a lot of features by running each drive in an individual RAID0 logical volume. Here are some of the things I've done to mitigate potential issues.
- Update your HP SmartArray (SA) controller to start with. Originally, the older controllers did not support drives over 2TB but the new firmware does. I'm running 20x 5TB drives.
- As I stated in the preface, the HP SAs don't support JBOD mode, so you need to put each drive in its own RAID0 logical volume.
- After creating your first vdev and before loading any data in it, run a 'zpool status' and copy the outputs. This will give you a list of device names and gptid. If you attempt to view the hard disk serial from the web UI, you'll see the same serial number for all your drives because this is coming from the RAID controller. You will want to disconnect/remove one drive at a time and see what device goes offline, this way you can label your drives on your chassis/enclosure. This makes life a lot easier in the event of a drive failure.
- Since you lose out on some of the automatic SMART drive monitoring, you'll have to do this manually through the use of cciss_vol_status and smartctl. I got cciss_vol_status installed by:
- Create a jail.
- Download, compile, and install cciss_vol_status which is found here: https://sourceforge.net/projects/cciss/files/cciss_vol_status/ (look for cciss_vol_status-1.12.tar.gz)
tar zxvf cciss_vol_status-1.12.tar.gz
./configure
make
make install
(if you get permission errors, I found it easier to copy the archive to /tmp/ and working from there) - This will install to the jail's /usr/local/bin/, so if you were to access this from outside the jail, it would be at /{jail storage}/{jail name}/usr/local/bin/cciss_vol_status
- You can read up on the docs, but the typical usage would be something like: cciss_vol_status -s /dev/ciss0
note that /dev/ciss0 is your RAID controller, not a particular drive. In my box I have two: ciss0 which is a p410 and ciss1 which is a p812.
- Luckily, smartctl is already installed. So your typical command would be like: smartctl -d cciss,0 -a /dev/ciss0
where cciss,0 is the disk and again /dev/ciss0 would be your controller. So if you want to look at disk 2 on the controller, you would do: smartctl -d cciss,1 -a /dev/ciss0
Or disk 5 on controller 2: smartctl -d cciss,4 -a /dev/ciss1
- I then hacked together a daily health script to email me the output of some of these commands. Note that I used BiduleOhm's scripts (https://forums.freenas.org/index.php?posts/175548/) as a starting point.
Code:#!/bin/sh ### Parameters ### logfile="/tmp/smart_report.tmp" email="email@gmail.com" subject="Status Report for FreeNAS" drives="0 1 2 3 4 5 6 7 8 9 10 11" devdrives="da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11" cciss_vol_status="/mnt/data/jails/cciss/usr/local/bin/cciss_vol_status" tempWarn=40 tempCrit=45 sectorsCrit=10 testAgeWarn=1 warnSymbol="?" critSymbol="!" rm "$logfile" ###### zpool status ###### echo "########## zpool Status ##########" > "$logfile" zpool status >> "$logfile" echo "" >> "$logfile" ###### raid status ###### echo "" >> "$logfile" echo "########## raid Status ##########" >> "$logfile" "$cciss_vol_status" -s /dev/ciss0 >> "$logfile" echo "" >> "$logfile" ###### SMART status for each drive ###### for drive in $drives do brand="$(smartctl -d cciss,"$drive" -i /dev/ciss0 | grep "Device Model" | awk '{print $3, $4, $5}')" serial="$(smartctl -d cciss,"$drive" -i /dev/ciss0 | grep "Serial Number" | awk '{print $3}')" ( echo "" echo "########## SMART status report for ${drive} drive (${brand}: ${serial}) ##########" smartctl -d cciss,"$drive" -H -A -l error /dev/ciss0 smartctl -d cciss,"$drive" -l selftest /dev/ciss0 | grep "# 1 \|Num" | cut -c6- echo "" ) >> "$logfile" done sed -i '' -e '/smartctl 6.5/d' "$logfile" sed -i '' -e '/Copyright/d' "$logfile" sed -i '' -e '/=== START OF READ/d' "$logfile" sed -i '' -e '/SMART Attributes Data/d' "$logfile" sed -i '' -e '/Vendor Specific SMART/d' "$logfile" sed -i '' -e '/SMART Error Log Version/d' "$logfile" ### Send report ### cat "$logfile" | mail -s "$subject" "$email"
