HBA Card Firmware

[Ericloewe]

Early 870s, up to an unspecified date that may or may not be real, suffer from serious degradation that leads to premature death (both in time and writes). We were bitten by that at work, it was thanks to ZFS that we did not have a major outage on our hands.

 

[nabsltd]

So, consumer SSDs are not good or this specific Samsung 870 EVO/QVO or the Samsung itself?

If an SSD drive does not have power loss protection (PLP), you shouldn't use it with ZFS. There are very few consumer drives with PLP, and none of the Samsung drives have it, IIRC.

OTOH, the Samsung enterprise line (PM893 or PM863 for SATA, PM983 or PM9A3 for NVMe) have PLP and much better endurance than the consumer drives. The caveat for the M.2 drives is they are 110mm long instead of 80mm, so they won't work in some M.2 slots.

 

[Etorix]

PLP is only mandatory for SLOG devices. For data vdevs, L2ARC or boot duties it serves no purpose.

 

[Zedicus]

9400 based cards pretty regularly ARE in fact the cheapest PCIE 3.x based cards available. 16i cards pop up for $100 used, and the 8i cards are usually $50 give or take. and are the low power for the features available.

avago (broadcom, lenovo, lsi, whatever) 530 model number based things come with the RAID personality and NVME support standard. these MUST be flashed into SATA/SAS firmwares for reliable TrueNAS use.

avago (broadcom, lenovo, lsi, whatever) 430 model number based things come with the SATA/SAS firmware standard and are usually correct.

NOTE: the 530 line has a IT mode personality available, this is FAKE (more fake?) IT mode and while it should work, it is really untested. the preference is to flash the actual SATA/SAS only firmware.

the 9305 is also fine to use and is a SINGLE chip. the 9300 8i cards are fine to use and mostly support crossflashing so anything cheap is fine.

the 9300 16i is DUAL CHIP with pcie expander and should NOT be used only because it is hot and power hungry.

9500 and 9600 cards follow the same guidance as 9400 but are newer and fairly untested. i have no issue recommending a 9400 16i based card as long as you have time to wait for a 430 model to pop up or a 530 model if you are comfortable flashing things yourself.

the 'from china' variants also seem to be fine but even the 430 cards usually have ancient firmware so assume all 430 and 530 will need a flash.

 

[Ericloewe]

If an SSD drive does not have power loss protection (PLP), you shouldn't use it with ZFS.

That is very overstated. There is some perceived risk of losing "everything", but the fact that people are not losing "everything" left and right on bog-standard SSDs, ZFS or no ZFS, is clear indicator that there isn't much risk of actually losing "everything", but rather more likely recently-written data. It's not clear that this remaining risk is substantial enough for most users to care about that much, as the on-disk state is going to be rather recoverable, in the worst case, by rolling back a transaction or two.

530 model number based things

430 model number based things

You speak in riddles.

9600 cards

Yeah, those are basically completely different and really shouldn't be lumped in with the SAS3 gear. The whole software stack was redone and IT mode is gone.

 

[Patrick M. Hausen]

That is very overstated. There is some perceived risk of losing "everything", but the fact that people are not losing "everything" left and right on bog-standard SSDs, ZFS or no ZFS, is clear indicator that there isn't much risk of actually losing "everything", but rather more likely recently-written data.

I have 2 data centres worth of Samsung "prosumer" SSDs and have not yet had a single loss of a pool in 7 years. I did avoid the latest EVO/QVO line as far as possible, though, and where I absolutely could not get another device I used QVO and overprovisioned by a factor of 2 just to get the TBW value I wanted.

 

[nabsltd]

That is very overstated. There is some perceived risk of losing "everything", but the fact that people are not losing "everything" left and right on bog-standard SSDs, ZFS or no ZFS, is clear indicator that there isn't much risk of actually losing "everything", but rather more likely recently-written data. It's not clear that this remaining risk is substantial enough for most users to care about that much, as the on-disk state is going to be rather recoverable, in the worst case, by rolling back a transaction or two.

I never said that there would be a loss of "everything"...just that the risk of using a drive without PLP is not worth it with ZFS, which expects every piece of hardware to be honest about what it is doing, and SSDs without PLP will often claim that the data has been written even if it is still in some temporary buffer that won't be a problem the vast majority of the time.

Plus, the endurance of enterprise drives is worth it for the peace of mind.

Last, why buy a Samsung 900 Pro 4TB drive with 600TB lifetime write endurance for $320 when you can get a Samsung PM983 3.84TB with 6700TB write endurance for $185?

 

[Zedicus]

You speak in riddles.

lots of OEM cards go by the 'Lenovo' influenced model number. so you might see an Ebay auction for an Avago 530 16i.
like this or some combination
or this Avago 430 clone card

and the trouble is, the 430 and 530 cards are basically hardware identical, and all LSI 9400 based. but the 430 series is SATA/SAS default. the 530s are tri mode.

 

[Ericloewe]

I never said that there would be a loss of "everything"

That's true, but...

ZFS, which expects every piece of hardware to be honest about what it is doing

That's not true, that's half the reason for everything to be checksummed.

and SSDs without PLP will often claim that the data has been written even if it is still in some temporary buffer that won't be a problem the vast majority of the time.

Let's be honest: That ship sailed long before SSDs were a meaningful thing. All parts of the chain routinely lie about how persistent the latest write was. Hell, ZFS itself does that with sync disabled. Some things lie more than others, but if they're explicitly ignoring flush commands it's not likely that PLP is going to absolve them of their sins.

Last, why buy a Samsung 900 Pro 4TB drive with 600TB lifetime write endurance for $320 when you can get a Samsung PM983 3.84TB with 6700TB write endurance for $185?

That's valid, of course, but it's rarely that easy.

I have 2 data centres worth of Samsung "prosumer" SSDs and have not yet had a single loss of a pool in 7 years. I did avoid the latest EVO/QVO line as far as possible, though

Yeah, the second sentence is redundant. The early 870s (EVO and QVO alike) die like flies in DDT, apparently due to bad firmware, bad flash ICs or both. Fortunately, I only had two on hand, but it was a persistent issue across lots.

lots of OEM cards go by the 'Lenovo' influenced model number. so you might see an Ebay auction for an Avago 530 16i.

That's a new one to me. Figures they'd make it confusing.

 

[Fastline]

Samsung "prosumer" lines like "860 Pro" or "970 EVO Plus" have been very reliable for us in the past. We still have a lot in operation without any issues.

So no, not generally "not good". The new "QVO" line has been found by various reviewers to suffer from performance degradation due to the internal architecture. I am not current with respect to the state of controller firmware updates that might have fixed these issues.

You will probably have to do a little bit or research yourself - I'd start with e.g. Serve The Home. IMHO the most important metric is the guaranteed write endurance (TBW/PBW) value followed by any performance considerations.

In general you best go with a reputable vendor and I do not consider Samsung specifically unreliable - far from it. But as this market is moving so fast there really is no answer to questions like this that will not be outdated in another six weeks or so.

Got it.

As the speed is increased so the number of writes will be more. Therefore, endurance is the first factor for the SSD pools, yeah?

 

[Fastline]

Early 870s, up to an unspecified date that may or may not be real, suffer from serious degradation that leads to premature death (both in time and writes). We were bitten by that at work, it was thanks to ZFS that we did not have a major outage on our hands.

Also, i have heard that the TRIM is broken on most of the Samsung Drives. Is it due to that?

 

[Fastline]

If an SSD drive does not have power loss protection (PLP), you shouldn't use it with ZFS. There are very few consumer drives with PLP, and none of the Samsung drives have it, IIRC.

Well, i don't think PLP is necessary for data vdevs when it comes to SSD. But for SLOG i guess. Not sure if this is the case with L2ARC and/or Metadata. Can anyone confirm this?

OTOH, the Samsung enterprise line (PM893 or PM863 for SATA, PM983 or PM9A3 for NVMe) have PLP and much better endurance than the consumer drives. The caveat for the M.2 drives is they are 110mm long instead of 80mm, so they won't work in some M.2 slots.

Yeah, yeah. The enterprise ones are always better. They cost a lot more ;)

 

[Fastline]

PLP is only mandatory for SLOG devices. For data vdevs, L2ARC or boot duties it serves no purpose.

Thanks for confirming.

 

[Fastline]

9400 based cards pretty regularly ARE in fact the cheapest PCIE 3.x based cards available. 16i cards pop up for $100 used, and the 8i cards are usually $50 give or take. and are the low power for the features available.

avago (broadcom, lenovo, lsi, whatever) 530 model number based things come with the RAID personality and NVME support standard. these MUST be flashed into SATA/SAS firmwares for reliable TrueNAS use.

avago (broadcom, lenovo, lsi, whatever) 430 model number based things come with the SATA/SAS firmware standard and are usually correct.

NOTE: the 530 line has a IT mode personality available, this is FAKE (more fake?) IT mode and while it should work, it is really untested. the preference is to flash the actual SATA/SAS only firmware.

the 9305 is also fine to use and is a SINGLE chip. the 9300 8i cards are fine to use and mostly support crossflashing so anything cheap is fine.

the 9300 16i is DUAL CHIP with pcie expander and should NOT be used only because it is hot and power hungry.

9500 and 9600 cards follow the same guidance as 9400 but are newer and fairly untested. i have no issue recommending a 9400 16i based card as long as you have time to wait for a 430 model to pop up or a 530 model if you are comfortable flashing things yourself.

the 'from china' variants also seem to be fine but even the 430 cards usually have ancient firmware so assume all 430 and 530 will need a flash.

Yes, i got to know it recently.

 

[Fastline]

BTW guys,

Today, i was using one of the Supermicro board, which i use for some test purposes and i was testing something i noticed that the SAS3008 chip had 512MB metadata in the LSI Mega RAID Manager. My question is the metadata also present in the Intel/AMD PCH? If not, does this mean the LSI Cards will be a little faster for directory browsing when compared to the Intel/AMD PCH?

 

[Etorix]

Anything involving "MegaRAID" or cache indicates that it is NOT a 3008 (pure HBA) but a RAID controller, which should then be replaced and not used with ZFS.

 

[Fastline]

Anything involving "MegaRAID" or cache indicates that it is NOT a 3008 (pure HBA) but a RAID controller, which should then be replaced and not used with ZFS.

Well, I'm 100% sure that it is LSI3008 chip. However, this board is not being used for TrueNAS. I was using it for some other purpose. When I installed windows, i thought to check the temps of the chip so I used LSI Megaraid Storage manager software and when I checked the chip info, it says 512MB metadata. I have the same other board and on that it's IT mode. If you say 3008 is pure HBA and the fact we know that HBA does not have any kind of cache so why I see 512MB Cache?

 

[Ericloewe]

Anything involving "MegaRAID" or cache indicates that it is NOT a 3008

Could be a SAS3008 with MegaRAID firmware. Same concept as the once-ubiquitous M1015 that had a SAS2008 with MegaRAID firmware and thus could be crossflashed with IT firmware.

 

[Fastline]

Could be a SAS3008 with MegaRAID firmware. Same concept as the once-ubiquitous M1015 that had a SAS2008 with MegaRAID firmware and thus could be crossflashed with IT firmware.

Hmm. I see. So am I all safe to use this board? Provided that I will be flashing the IT firmware.

Secondly, I'm still confused about the 512MB metadata it shows.

Also, is LSI3008 is a True HBA or still a RAID controller?

 

[Ericloewe]

Neither, it's a SAS host controller IC that can take IT/IR firmware or MegaRAID, and its behavior will change accordingly. As for a mystery cache, it could be a reporting bug, host memory via DMA (in the vein of NVMe's HMB), DRAM (which the HBAs do include, PowerPC cores would not be useful without DRAM), or possibly something weird.

If you have IT firmware and the card is working, don't worry about the mystery cache.