SOLVED SCSI Status Error, but eventually works?

[SageEdwards]

I am a first time user of both TrueNAS and enterprise server hardware so, I am sure I have probably made a stupid mistake somewhere in the install/build process.

To start I purchased the following build from ServerMonkey (I'm very happy with this company btw):

System Hardware:
HPE ProLiant DL380 Gen9
x2 Intel Xeon E5-2697 v3
128Gb (8x16GB) 2RX4 PC4-17000R
HP Smart Array P440ar 2GB FBWC Lvl 0-60 INT RAID
On-Board Quad Port 1GbE LOM
HPE Ethernet 546FLR-SFP+ Dual Port DA/SFP+ 10Gb FlexibleLOM Adapter
x2 HP G9 800 Watt PSU
x12 WD RED Pro NAS 16TB HDD
x1 Supermicro SATADOM 64GB

where I installed that HDDs and the SATADOM. However, to get the 5V power to the SATADOM, I made a cable to come off of the internal USB 3 port. Everything boots up fine with no errors. From here I created the boot drive using Balena and then successfully installed the most recent stable version of TrueNAS Core, 13.0-U1.1 in UEFI mode on the SATADOM, with SWAP storage and root password set. I rebooted the system but realized the system was set for legacy boot, so I changed it to UEFI in the BIOS and rebooted again. Once I booted it again and went through the POSTing process, and TrueNAS starts up, I eventually get the following error messages:
READ(16). cdb: 88 00 00 00 00 07 46 BF FF FE 00 00 00 01 00 00 CAM status: CCB request completed with an error Retrying command, 3 more tries remain READ(16). cdb: 88 00 00 00 00 07 46 BF FF FE 00 00 00 01 00 00 CAM status: SCSI Status Error SCSI status: Check Condition SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) ... ... ...

This then repeats for a very long time, but it does eventually work through it and enters the console setup, but with the following lines after the option menu:
The web interface could not be accessed. Please check network configuration. Enter an option from 1-11: ciss0: ADAPTER HEARTBEAT FAILED

I suspect that the latter error is simply due to the fact that I don't have the system plugged into any network at the moment (correct me if I am wrong), but it is the former error that concerns me. I suspect that this is due to my janky rigging I did for powering the SATADOM, but I really do not know.

I have attached three images showing the two error screen and the SATADOM powering method.

Any help and/or advice will be greatly appreciated!

-Sage

 

[HoneyBadger]

Welcome. The full spec dump is appreciated and helpful here - if you can also let us know what you'd like to do once this system is up and running (store your collection of Linux ISOs in your home, be a file share for a large office, act as shared storage for a hypervisor cluster) we can offer some further advice and pointers.

But unfortunately you've run up against a couple of FreeBSD/ZFS hardware requirements with that server.

HP Smart Array P440ar 2GB FBWC Lvl 0-60 INT RAID

This is the core of your problem. You have a RAID controller (and one using a particularly poor BSD driver) which is not supported.

Please read the resource below:

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Unfortunately there doesn't appear to be an HBA in the form-factor needed to directly replace your HP card, so you'll have to use a PCIe form factor one.

Thankfully, the magical decoder ring told me that your 10Gbe adapter:

HPE Ethernet 546FLR-SFP+ Dual Port DA/SFP+ 10Gb FlexibleLOM Adapter

does appear to be based on the Mellanox Connect X-3 Pro chip, so that will hopefully be supported under the mlx4en driver.

 

[SageEdwards]

Welcome. The full spec dump is appreciated and helpful here - if you can also let us know what you'd like to do once this system is up and running (store your collection of Linux ISOs in your home, be a file share for a large office, act as shared storage for a hypervisor cluster) we can offer some further advice and pointers.

But unfortunately you've run up against a couple of FreeBSD/ZFS hardware requirements with that server.



This is the core of your problem. You have a RAID controller (and one using a particularly poor BSD driver) which is not supported.

Please read the resource below:

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Unfortunately there doesn't appear to be an HBA in the form-factor needed to directly replace your HP card, so you'll have to use a PCIe form factor one.

Thankfully, the magical decoder ring told me that your 10Gbe adapter:



does appear to be based on the Mellanox Connect X-3 Pro chip, so that will hopefully be supported under the mlx4en driver.

Thanks for the info!

So it seems that I need to get a new sas HBA from LSI that is in a PCIe form factor.

Besides that, everything is fine.

With that, can you recommend me a HBA, and anything else I may need, to purchase to make this work.

Thanks again!

 

[SageEdwards]

Welcome. The full spec dump is appreciated and helpful here - if you can also let us know what you'd like to do once this system is up and running (store your collection of Linux ISOs in your home, be a file share for a large office, act as shared storage for a hypervisor cluster) we can offer some further advice and pointers.

But unfortunately you've run up against a couple of FreeBSD/ZFS hardware requirements with that server.



This is the core of your problem. You have a RAID controller (and one using a particularly poor BSD driver) which is not supported.

Please read the resource below:

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Unfortunately there doesn't appear to be an HBA in the form-factor needed to directly replace your HP card, so you'll have to use a PCIe form factor one.

Thankfully, the magical decoder ring told me that your 10Gbe adapter:



does appear to be based on the Mellanox Connect X-3 Pro chip, so that will hopefully be supported under the mlx4en driver.

Oh and the purpose for this NAS is to serve as storage for experimental data in the research lab I work in. It will hold large and small sized folders. In addition to this, it will hold backups from the five or so computers on our network, and also hold our docker images from different experiment setups.

 

[SageEdwards]

My thoughts on an upgrade, based on what I have read so far, is to get the LSI 9300-8i. It is a little overkill for the HDDs but I don't want to bottleneck myself and money really isn't an issue. I already have a sas expansion board, so I believe that this is all that I need to conduct the upgrade.

 

[HoneyBadger]

Thanks for the info!

So it seems that I need to get a new sas HBA from LSI that is in a PCIe form factor.

Besides that, everything is fine.

With that, can you recommend me a HBA, and anything else I may need, to purchase to make this work.

Thanks again!

The HP H220 can be reflashed into an LSI 9207-8i, but any card that uses the LSI SAS2308 chipset with the same SFF-8087 form factor connectors should be fine. You'll likely want a card that has the connectors on the "end" of the card as opposed to the "top" for cabling reasons though.

Based on the internal picture, it looks as if there's a SAS expander card already in a PCIe slot, and then cabling to go to the P440ar in the "flex slot" on the board - but there's only two SFF-8087 cables from the front trays, and then two going to the P440ar. So the expander is probably redundant at this point, and you could just remove both that and the onboard P440ar, and just take the cables from front of chassis and connect them to the new HBA. (Unless the expander also has external ports, and you want to leave those in play for later expansion via DAE/disk shelves.)

My thoughts on an upgrade, based on what I have read so far, is to get the LSI 9300-8i. It is a little overkill for the HDDs but I don't want to bottleneck myself and money really isn't an issue. I already have a sas expansion board, so I believe that this is all that I need to conduct the upgrade.

This is fine, and there's no kill like overkill, but you'll require new cables if you go with the 9300-8i as it uses SFF-8643 ("miniSAS HD") cables vs. the SFF-8087 ("miniSAS") of the older generation.

Oh and the purpose for this NAS is to serve as storage for experimental data in the research lab I work in. It will hold large and small sized folders. In addition to this, it will hold backups from the five or so computers on our network, and also hold our docker images from different experiment setups.

Backups and large files would do very well with a RAIDZ2 layout, small files not so much, but if it's just a lab then the performance requirements might still be satisfied. (I assume you're just going to save Docker images there and not mount NFS from running containers?) I'd personally build two vdevs of 6 drives in RAIDZ2 which will give you roughly 8x16=128T of usable space, and let you tolerate up to four drive failures (two per vdev)

 

[SageEdwards]

The HP H220 can be reflashed into an LSI 9207-8i, but any card that uses the LSI SAS2308 chipset with the same SFF-8087 form factor connectors should be fine. You'll likely want a card that has the connectors on the "end" of the card as opposed to the "top" for cabling reasons though.

Based on the internal picture, it looks as if there's a SAS expander card already in a PCIe slot, and then cabling to go to the P440ar in the "flex slot" on the board - but there's only two SFF-8087 cables from the front trays, and then two going to the P440ar. So the expander is probably redundant at this point, and you could just remove both that and the onboard P440ar, and just take the cables from front of chassis and connect them to the new HBA. (Unless the expander also has external ports, and you want to leave those in play for later expansion via DAE/disk shelves.)



This is fine, and there's no kill like overkill, but you'll require new cables if you go with the 9300-8i as it uses SFF-8643 ("miniSAS HD") cables vs. the SFF-8087 ("miniSAS") of the older generation.



Backups and large files would do very well with a RAIDZ2 layout, small files not so much, but if it's just a lab then the performance requirements might still be satisfied. (I assume you're just going to save Docker images there and not mount NFS from running containers?) I'd personally build two vdevs of 6 drives in RAIDZ2 which will give you roughly 8x16=128T of usable space, and let you tolerate up to four drive failures (two per vdev)

So, I reached out to Server Monkey's tech team and they are telling me that I will need to use the HP H240 12Gb Smart Host Bus Adapter "because the LSI 9300-8i has an SFF-8644 connection and the GEN 9 does not support that connection. Thoughts?

 

[jgreco]

HP H240

This does not use an LSI chipset and is not an IT mode LSI HBA. Please do go and re-read the previously linked pointer to my "What's all the noise" resource for further guidance. Your current round of SCSI errors and timeouts is the predictable and expected result of the use of your H240. Sorry.

 

[SageEdwards]

This does not use an LSI chipset and is not an IT mode LSI HBA. Please do go and re-read the previously linked pointer to my "What's all the noise" resource for further guidance. Your current round of SCSI errors and timeouts is the predictable and expected result of the use of your H240. Sorry.

ok, so what HBA do I buy? I am being told here to use either the HP H220 or the LSI 9300-8i, and Server Monkey is telling me I can't use the LSI 9300-8i. Is the HP H220 viable, given their reasons. I do not know.

 

[HoneyBadger]

Server Monkey is telling me I can't use the LSI 9300-8i

You would just need to obtain a set of SFF-8643 to SFF-8087 cables in the proper length in order to make this work, since the backplane/expander card uses the latter and the 9300-8i uses the former. Either long enough to reach the front of the case (including cable routing) or a short set that goes to the expander card (but then you have to leave said expander in a PCIe slot)

But the HP H220 is also a viable choice, since you are only using HDDs.

 

[SageEdwards]

You would just need to obtain a set of SFF-8643 to SFF-8087 cables in the proper length in order to make this work, since the backplane/expander card uses the latter and the 9300-8i uses the former. Either long enough to reach the front of the case (including cable routing) or a short set that goes to the expander card (but then you have to leave said expander in a PCIe slot)

But the HP H220 is also a viable choice, since you are only using HDDs.

So, something like this?

Amazon.com

 

[SageEdwards]

You would just need to obtain a set of SFF-8643 to SFF-8087 cables in the proper length in order to make this work, since the backplane/expander card uses the latter and the 9300-8i uses the former. Either long enough to reach the front of the case (including cable routing) or a short set that goes to the expander card (but then you have to leave said expander in a PCIe slot)

But the HP H220 is also a viable choice, since you are only using HDDs.

Sorry about all the messages, I just need to be absolutely sure that I am getting the correct items since this is not my money, but an institution's.

Thanks again for all of the help you have given thus far. It is greatly appreciated!

 

[SageEdwards]

I got the aformentioned LSI and cables. Everything now works perfectly. Thanks everyone!