Drives "Formatted with type 1 protection" eventually lead to data loss

[homelabber]

Long time lurker, first time poster who eventually set up his first FreeNAS system (11.3), ditching Windows Server.
I am not posting information about my motherboards etc. since none of this has anything to do with the problem that I encountered (if anyone is interested in the exact configuration, I will happily share the details):

I created my first pool (6 vDevs consisting of two mirrored SSDs each) without a problem and played with some iSCSI and SMB shares. A day or two later, I rebooted FreeNAS - and lost the pool (wasn't a big deal because no real data stored on FreeNAS at this point). Log file showed tons of CAM status: CCB request completed with an error for four of the six disks.

While trying to create a new pool, I watched the logs again and saw the same errors popping up. Again the pool would be created without any error messages showing in the GUI though! I already saw myself changing backplanes, SAS cables, HBAs, RAM modules, PSUs to find out what causes the problem...

Before bringing out the hardware tools, I wanted to rule out there is nothing wrong with the disks themselves. Therefore I used the very useful sol.net disk array test v2 which threw an !!ERROR!! dd: /dev/daX: Input/output error for each of the disks in question; the other two SSDs ran the tests just fine.

What struck me was that the culprits were all HGST SSDs (the others are from Toshiba and Samsung) that I bought on eBay a few months ago. They came with a 520 Byte block size and I sg_formatted them to 512 Bytes. After that, Windows Server would add them to a Storage Pool without a problem and I never had single issue using them in my now ex file server. Therefore I expected them to work in the FreeNAS box as well.

I checked each SSD with smartctl -a /dev/daX and the detail about the "Type 1 Protection" is what stood out:

Code:

=== START OF INFORMATION SECTION ===
Vendor:               HGST
Product:              HBSAC2DH2SUN3.2T
Revision:             C08H
Compliance:           SPC-4
User Capacity:        3,200,631,791,616 bytes [3.20 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
Formatted with type 1 protection
8 bytes of protection information per logical block

Apparently I forgot to add --pfu=0 to the command when formatting the drives with sg_format after I bought them on eBay. Duh.
After a round of sg_format -v --format --fmtpinfo=0 --pfu=0 --size=512 --six /dev/daX, the sol.net disk array test v2 reported no more errors and there were no relevant entries in the log during the pool creation either.

Why FreeNAS doesn't complain while creating the pool and would use the incorrectly formatted disks for some random amount of time, I do not know. But I now know that there is more than faulty SAS cables and funny PSUs causing CAM status: CCB request completed with an error messages.

A huge "Thank you" to this community! Because this forum delivered all the information and hints I needed to solve this problem.

 

[yorch]

Thanks a lot, your post was very helpful, got a couple of used HGST SSD SAS drives recently and even thought SMART was not showing a single error, kept getting the input / output error when trying to use them in unRaid. Spent several hours trying to figure what was wrong and was 5 minutes away from throwing them in the garbage until I found your post. Worked perfectly!