I've pounded a few versions of this out over the years, but I hate explaining over and over.
TrueNAS allows a ZFS boot mirror pool to be created to increase the reliability of the NAS. This sounds great in theory, but there's a flaw.
Due to the design of the average PC BIOS, a boot device that has become corrupted but not failed entirely may prevent booting. This means that your PC BIOS may not boot from the secondary device in a ZFS boot pool if the primary device is only partially failed. For example, if you have two El Cheapo 60GB SSD's and the one on SATA 0 develops an inability to read the boot LBA's, your BIOS will nevertheless see that SATA 0 is present and try to boot from it, fail, and hang with the dreaded "Disk Read Error" message or whatever your particular BIOS does. It will never attempt to use the valid data on SATA 1 which has the correct boot sectors. That sucks.
Some of us used to use "hardware" RAID 1 and IR mode controllers in the pre-ZFS days to make sure an SSD failure didn't impair boot. This is good but not perfect, since the "hardware" RAID controller cannot detect corruption. Additionally, this (alone) is no longer recommended with ZFS as it eliminates redundancy. If ZFS reads a block and it is clearly corrupt, ZFS has nowhere to pull redundancy from, even though it is there hidden on the other SSD obscured by the hardware RAID 1 controller. This is basic ZFS 101 "why we don't use hardware RAID". See this linked article for all the detail.
So is it possible to build a more reliable booting solution?
Yes, it is!
So, first things first.
LSI HBA's have up to three personalities.
1) IT mode -- the one everyone gets beat over the head with "must be crossflashed to IT" mode
2) MFI/MRSAS mode -- the highly UNDESIRABLE one that may cause data loss or weird problems because it is optimized for RAID
3) IR mode. This is "most basic RAID" mode, primarily capable of RAID1/RAID10 in addition to JBOD and passthrough modes.
IT mode happens to be the IR firmware with all the "RAID" functions stripped out. This makes the HBA somewhat faster because it isn't processing every I/O through that additional code.
However(!) -- you can use an IR controller, with the same approved firmware revisions as IT, and it works the same way, with the SAME driver as IT mode (IR fw 20.00.07.00 and MPS driver for 6Gbps, IR fw 16.00.12.00 and MPR driver for 12Gbps). Firmware versions current as of Feb 2022. The IR RAID virtual disks show up and are handled under MPR/MPS, although you do obscure the SMART data.
As noted above, though, using an IR RAID1 obscures the redundancy from ZFS, so even though you can make a highly reliable IR RAID1 boot device and offer that to the host system, corrupt data on the IR RAID1 SSD's can still hose you. That's bad.
Fortunately, SSD's are cheap. Buy three.
Use the first two to make an IR RAID1 virtual device, make sure that's the bootable device in the host BIOS and the card configuration.
Leave the third as a passthrough device.
Now, when you boot up the TrueNAS installer, use the IR RAID1 virtual disk *AND* the third SSD as your mirrored boot devices. Now if something goes wrong with the data on the IR RAID1, ZFS has a source for redundancy and can correct the error as it finds it, from the data on the third SSD.
Bonus awesomeness multipliers for the masochistic:
1) Set up patrol reads to verify the RAID1 disks
2) Set up a script to notify you if sas2ircu picks up on any problems
3) Add a fourth SSD to the mix as a spare in case any of the first three fail
TrueNAS allows a ZFS boot mirror pool to be created to increase the reliability of the NAS. This sounds great in theory, but there's a flaw.
Due to the design of the average PC BIOS, a boot device that has become corrupted but not failed entirely may prevent booting. This means that your PC BIOS may not boot from the secondary device in a ZFS boot pool if the primary device is only partially failed. For example, if you have two El Cheapo 60GB SSD's and the one on SATA 0 develops an inability to read the boot LBA's, your BIOS will nevertheless see that SATA 0 is present and try to boot from it, fail, and hang with the dreaded "Disk Read Error" message or whatever your particular BIOS does. It will never attempt to use the valid data on SATA 1 which has the correct boot sectors. That sucks.
Some of us used to use "hardware" RAID 1 and IR mode controllers in the pre-ZFS days to make sure an SSD failure didn't impair boot. This is good but not perfect, since the "hardware" RAID controller cannot detect corruption. Additionally, this (alone) is no longer recommended with ZFS as it eliminates redundancy. If ZFS reads a block and it is clearly corrupt, ZFS has nowhere to pull redundancy from, even though it is there hidden on the other SSD obscured by the hardware RAID 1 controller. This is basic ZFS 101 "why we don't use hardware RAID". See this linked article for all the detail.
So is it possible to build a more reliable booting solution?
Yes, it is!
So, first things first.
LSI HBA's have up to three personalities.
1) IT mode -- the one everyone gets beat over the head with "must be crossflashed to IT" mode
2) MFI/MRSAS mode -- the highly UNDESIRABLE one that may cause data loss or weird problems because it is optimized for RAID
3) IR mode. This is "most basic RAID" mode, primarily capable of RAID1/RAID10 in addition to JBOD and passthrough modes.
IT mode happens to be the IR firmware with all the "RAID" functions stripped out. This makes the HBA somewhat faster because it isn't processing every I/O through that additional code.
However(!) -- you can use an IR controller, with the same approved firmware revisions as IT, and it works the same way, with the SAME driver as IT mode (IR fw 20.00.07.00 and MPS driver for 6Gbps, IR fw 16.00.12.00 and MPR driver for 12Gbps). Firmware versions current as of Feb 2022. The IR RAID virtual disks show up and are handled under MPR/MPS, although you do obscure the SMART data.
As noted above, though, using an IR RAID1 obscures the redundancy from ZFS, so even though you can make a highly reliable IR RAID1 boot device and offer that to the host system, corrupt data on the IR RAID1 SSD's can still hose you. That's bad.
Fortunately, SSD's are cheap. Buy three.
Use the first two to make an IR RAID1 virtual device, make sure that's the bootable device in the host BIOS and the card configuration.
Leave the third as a passthrough device.
Now, when you boot up the TrueNAS installer, use the IR RAID1 virtual disk *AND* the third SSD as your mirrored boot devices. Now if something goes wrong with the data on the IR RAID1, ZFS has a source for redundancy and can correct the error as it finds it, from the data on the third SSD.
Bonus awesomeness multipliers for the masochistic:
1) Set up patrol reads to verify the RAID1 disks
2) Set up a script to notify you if sas2ircu picks up on any problems
3) Add a fourth SSD to the mix as a spare in case any of the first three fail
