Mirror the FreeNAS boot device

[Jarred Masterson]

I am setting up a FreeNAS box that will boot from CF cards which are attached to the host via a SATA to CF adapter. I have purchased two of these with the hope of creating a mirror so that if something were to happen to one of them then I wouldn't suffer any down time. This box will serve a host of iSCSI targets that will store VM images (or the iSCSI disk could be the virtual machine's physical disk, I'm going to do performance testing to decided if theres a benefit). Given this purpose I am working to minimize downtime as much as possible and mirroring the boot disk seems like the low hanging fruit of increasing my "9s". Right after redundant power supplies!

So my problem is this. I used the SuperMicro server board's bios to enable the intel RAID controller, and then configured it to place both of those CF drives into a mirror. This all seems good! I then shut off the FreeNAS box and then pulled the CF cards and used the physdiskwrite program to write them embedded image to them. I used a USB CF adapter to do this, don't know if it matters. I then replaced the CF drives and powered on the box. I checked the intel RAID config and it reports that still has one mirrored volume being exported to the BIOS and both CF cards display as being members of the "disk".

So everything looks great. But when in FreeNAS I can see a 8 GB disk reported as ada1. I know that FreeNAS isn't supposed to show the FreeNAS boot volume so I logged into the shell and ran the following command: egrep 'ada[0-9]' /var/run/dmesg.boot

Sure enough! As the result below depicts, the FreeNAS/FreeBSD subsystem sees both CF drives despite the config being set to export only one, and the BIOS reporting that it is booing off an Intel RAID volume. My question is now obvious, WTF is going on!!! Am I crazy?




ada0 at ahcich0 bus 0 scbus2 target 0 lun 0
ada0: <CF 8GB 20101008> ATAPI-0 device
ada0: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 512bytes)
ada0: 7647MB (15662304 512 byte sectors: 16H 63S/T 15538C)
ada1 at ahcich1 bus 0 scbus3 target 0 lun 0da3 at mps1 bus 0 scbus1 target 3 lun 0
ada1: <CF 8GB 20101008> ATAPI-0 device
ada1: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 512bytes)
ada1: 7647MB (15662304 512 byte sectors: 16H 63S/T 15538C)

 

[survive]

Hi Jarred,

No, you aren't crazy.

The problem is that the Intel chipset raid is still "fakeraid" that requires a proper driver to be loaded to actually do the raid functions. Unfortunately Intel has not released a driver that does this on FreeBSD so it also doesn't work of FreeNAS.

You only need the config database to restore your system so there's really only one file you need to really worry about. As long as you have a copy of that file you can be back up and going in the time it takes to install FreeNAS and upload your settings through the GUI. I realize that's not nearly as seamless as having the system boot from mirrored storage but that's what's available.

It would be pretty sweet to have the option to install the system to a ZFS mirror!

-Will

 

[Jarred Masterson]

You're right Will! Installing to a ZFS mirror would be very sweet! And I hear what you're saying about the config file. More on that in the last section of the post.

Do you happen to know if the "fakeraid" might be supported if it were in the Adaptec codebase instead of the Intel? The only reason I ask is it would seem the only reason for the adaptec codebase would be wider compatibility...

Meanwhile I have been beating my head against a wall trying to get FreeNAS installed inside of the adaptec codebase. On the off chance that it would work, even though I'm not confident that it would. I was trying to run physdiskwrite from within a Windows PE environment. Only to discover that Windows PE is missing something physdiskwrite requires. It seems I'll need to get a full OS booted on my box first and then write the image while running the mirror in adaptec mode. I might try this before making the decision between finding a cheap "trueraid" PCI card for my boot drive vs. backing up the config file. If I do go with a "trueraid" controller. Can you suggest one? I am very new to the FreeBSD/FreeNAS world and have a little difficulty with the hardware compatibility pages.

Another thought/question. If I choose to keep a spare on-hand. If I backup my config, can I restore it to a CF card on a different physical system? This way I can have a backup CF which already has the config loaded. Thereby, if I do have a failure the recovery time is as short as pop the new CF in a boot. Is that possible or would I have to have a replicated hardware config?

 

[survive]

Hi Jarred,

Do you have one of those Supermicro boards that has the option to use either the Intel "RAID" BIOS or the Adaptec BIOS to configure the on-board SATA ports? I've seen them on Supermicro's site but I honestly don't know how much real controller hardware there is on the board. For some reason I suspect the Adaptec BIOS is really just a different "front-end" for those who are more comfortable with that environment (it looks like every other Adaptec BIOS I've ever seen!). Of course you could also have a board with a proper dedicated Adaptec chip in addition to the Intel chipset controller....in that case disregard the above.

So I think the short answer is "it depends". Logically, if the board does have a "real" Adaptec controller chip on-board and FreeNAS has a driver for it I would expect that when you boot the install CD you should see the volume you created and be able to install to it, but obviously that is not the case here.

Here's what I think I would do....first get the make & model of the motherboard and go download the manual .pdf file. Search it for "Adaptec" and you should find a blurb either describing the feature or a section explaining how to configure it....that should help us determine exactly what sort of animal we are dealing with. If that doesn't yield anything useful then I would grab a flashlight & go stick my head in the case and actually look for an Adaptec chip on the board. Look in the space between the slots & the front edge of the board near the SATA ports, odds are the designers would want the traces between the chip & ports to be as short as possible so that's where it should be. If you find one google the chip model number and once again we should have a good idea what it actually is & what it can do. If you don't find a chip I would expect that the Adaptec code is simply a different "front-end" to the Intel chipset. If so you are out of luck trying to use it the way you do.

So what should you do if there is no "real" RAID onboard???

Keep in mind that FreeNAS is designed to minimize the write\wear on the flash media it's installed on. I don't know exactly what would happen to a running FreeNAS box if you physically removed the key from a running system, but I bet it wouldn't even notice till it tried to do something like save the .rrd files from the reporting page. The betting man in me says that even then it would continue to run just fine till you rebooted it (disclaimer: I don't know that any of tht is true, but I'm seriously considering going over to my filer, pulling out the key & finding out).

Just brainstorming...you could:

Invest in a "premium" USB key with the expectation that it would work better\be more reliable that some commodity $5.00 key.

Invest in a proper "DOM" module:

http://www.memoryx.net/mdom.html.
A DOM module is basically a bit of (NAND) flash molded into a block of plastic with a 40 pin PATA interface that can be plugged directly into the IDE header on your motherboard. These sort of devices are used in embedded gear like firewall boxes & commercial NAS devices. They are a but faster and "classier" than a USB key.

Since the config database is really what needs to be kept safe in case of failure I would think that a skilled unixman could whip up a script triggered by crscript on that could check for changes, and if found, copy or mail the config file somewhere. You could also look and see if you could make a script that could more or less "clone" your existing USB over to a second key. The only problem I see with that is what happens when you reboot.....

-Will

 

[Jarred Masterson]

Thanks for all the feedback. I really appreciate your time with this! I know that's just something that you "just say" but I really do mean it! Thanks for your help!

You are correct that I have a supermicro board that gives me the option of Intel vs. Adaptec. I too was wondering if the only difference between the two was the user-interface. The only reason I was thinking that might not be the case is that supermicro offers different drivers for the difference codebases on their website. If the only difference were the user-interface the the system wouldn't need a different driver would it? I also scoured the mobo and didn't find an adaptec chip so I am thinking there isn't an actual adaptec controller there.

Either way I tried to boot from the FreeNAS ISO with the controller configured as a RAID 1 under both codebases and confirmed that I could always see both drives.

BTW, I like you idea about pulling the drive and seeing what happens! Since I'm still testing I'm going to try it, and well... see what happens!!!

Also I wish I had seen SATA DOM modules before! That's essentially what I'm doing only I am using a SATA to CF adapter to accomplish it. I'm going to do some testing to see if I feel comfortable with swapping CF cards. On that note I do have a couple questions, since all of the configuration for the system is held in the config database can I restore that file to a CF card that's loaded in a different system with a different hardware config? The thinking here is could I potentially keep a spare CF card around that I can keep updated with the config file by booting it to a computer I have in my office.

The other option I've considered is could I write a shell script that copies the config file to my second CF card leaving it in the FreeNAS box? This way, I could configure the second card as the second boot priority and so if the first failed the system could boot to the second which already has a current config on it. Obviously, none of this is quite as good as a true RAID, but I'm just brainstorming at this point. I have seen a cheapo SYBA raid card that I think is supported by freebsd. It's only around $40 which isn't a lot of cost to add to this system. On the other hand, It's only 40 bucks! How long can I expect it to last? If it fails before the CF cards would then I've taken a step backwards!

 

[ProtoSD]

Another thought/question. If I choose to keep a spare on-hand. If I backup my config, can I restore it to a CF card on a different physical system? This way I can have a backup CF which already has the config loaded. Thereby, if I do have a failure the recovery time is as short as pop the new CF in a boot. Is that possible or would I have to have a replicated hardware config?

Jared,

You only need the config file, it can be restored on a different physical system, the only catch is that the device names may change and you'd need to Auto Import your pool again which is as easy as the click of an icon in the GUI. The disks have signatures so they can be in a different order if you needed to move them to different hardware and then Auto Import them. You can either backup the config from the GUI -> Settings -> General -> Save Config OR copying the /data/freenas-v1.db file. If you make any custom changes outside of the GUI to sysctl.conf, loaders.conf, or rc.conf etc., or any other files on your flash drive for that matter, then you would need to back those up as well.

 

[Jarred Masterson]

Thanks for the thought Will.

I hadn't thought of looking for an actual chip on the mobo but I did and didn't find one. You are correct that this is a supermicro board and that's why I have the option of intel vs adaptec codebase. I had thought there might be some substantial difference beyond just the user interface front-end. Mainly because supermicro provides a different driver for intel vs the adaptec. So there must be some difference or there wouldn't be a need for a separate driver. Either way it is all sort of academic right now since I took your advice and ran the ISO installer under the adaptec codebase and confirmed that it could still see both drives.

So on to now deciding what to do. I hadn't seen the DOM module before and would have gone that way in the beginning if I had. However, the SATA to CF adapters are essentially accomplishing the same thing for me. I was considering putting an el-cheapo SYBA raid card in the PCI slot. I found a SYBA card that looks like it is supported by FreeBSD but I have misgivings about it. If the card is that cheap, I'm introducing a new (very cheaply made) single point of failure. The RAID card's failure rate may in fact be higher that of the CF!

I have been testing things out and I have been able to config the system backup the config file and then restore it to a fresh load very easily! I think for now I'm going to proceed with keeping both cards in the machine and keeping a backup of the config file handy. I'll keep the secondary device with no config file for now. However, I'm going to attempt to write a script that will copy the config from one drive to the other. This way if I do lose the first drive I can pull it and reboot and be none the worse for the wear.

Thanks for that info protosd! It shouldn't be too tough to copy that .db file from one card to the other via an .sh script. I'll test things out and post back here if I come up with something cool like that

 

[bubulein]

Moin,
what about using a simple gmirror instead of fighting with the hardware/fake-RAID?
-Christian

 

[Jarred Masterson]

Thanks Christian! It looks like that may be the answer. gmirror is included in the freenas build so I think it might work. I don't know if it will survive upgrades to freenas but it's as easy as the documentation indicates then it wouldn't be too hard to reinstate after I upgrade. I'll try it out and let you know how it goes.