SOLVED Inconsistent issue with some drives being detected

[USAFrenzy]

I've come across some issues with adding more storage to my configuration:

My build so far is using:

• Asus Tuff X570 Plus Wifi Motherboard (this was on sale when I bought it originally)
• Ryzen 5 2600x CPU
• EVGA 850 Watt PSU
• 64GB ECC A-Tech 3200MHz DDR4 Ram
• 8 Western Digital Red 10TB HDDs
• 4 Crucial 1TB SSDs
• 1 250GB SSD
• 1 500GB SSD
• EVGA RTX 2060 (hardware encoding and basic game server setup via a VM)

The motherboard has 8 sata ports and I originally had 8 drives all conencted to those ports without issue.
Since I needed more ports though, I bought a FebSmart PCIE 3.0 to Sata Host controller card and plugged in the remaining drives using those ports.

The storage devices are connected by:

• The 250GB SSD and 4 WD Red drives are connected via the PCIE card
• All Curucial SSDs and 4 WD Red drives are connected directly to the motherboard's SATA ports
• The 500GB SSD is a M.2 drive and is installed in the lower M.2 socket so as not to disable any SATA ports

The issue I'm running into is that mostly all drives are detected; the caveat being that only the 250 GB drive is detected using the PCIE card and not the other drives.

When running "lsblk", what I get back is:

Code:

NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda           8:0    0 465.8G  0 disk 
├─sda1        8:1    0     1M  0 part 
├─sda2        8:2    0   512M  0 part 
├─sda3        8:3    0 449.3G  0 part 
└─sda4        8:4    0    16G  0 part 
sdb           8:16   1   9.1T  0 disk 
sdc           8:32   0 223.6G  0 disk 
├─sdc1        8:33   0   100M  0 part 
├─sdc2        8:34   0    16M  0 part 
├─sdc3        8:35   0   223G  0 part 
└─sdc4        8:36   0   499M  0 part 
sdd           8:48   1   9.1T  0 disk 
sde           8:64   1   9.1T  0 disk 
sdf           8:80   1   9.1T  0 disk 
sdg           8:96   1 931.5G  0 disk 
├─sdg1        8:97   1     2G  0 part 
│ └─md127     9:127  0     2G  0 raid1
│   └─md127 253:0    0     2G  0 crypt [SWAP]
└─sdg2        8:98   1 929.5G  0 part 
sdh           8:112  1 931.5G  0 disk 
├─sdh1        8:113  1     2G  0 part 
│ └─md127     9:127  0     2G  0 raid1
│   └─md127 253:0    0     2G  0 crypt [SWAP]
└─sdh2        8:114  1 929.5G  0 part 
sdi           8:128  1 931.5G  0 disk 
├─sdi1        8:129  1     2G  0 part 
│ └─md126     9:126  0     2G  0 raid1
│   └─md126 253:1    0     2G  0 crypt [SWAP]
└─sdi2        8:130  1 929.5G  0 part 
sdj           8:144  1 931.5G  0 disk 
├─sdj1        8:145  1     2G  0 part 
│ └─md126     9:126  0     2G  0 raid1
│   └─md126 253:1    0     2G  0 crypt [SWAP]
└─sdj2        8:146  1 929.5G  0 part  

Is this an issue with the PCIE card or have I missed something relatively simple here? All storage devices are connected via the PSU supplied SATA power cables.

As a side note, two of the drives connected to the PCIE card hold backup data and are known to be working, however, running "zpool status" shows:

Code:

  pool: Fast-Pool
 state: ONLINE
  scan: resilvered 47.7G in 00:02:48 with 0 errors on Mon Jan  9 21:24:11 2023
remove: Removal of vdev 3 copied 518M in 0h0m, completed on Mon Jan  9 21:24:54 2023
        1.19M memory used for removed device mappings
config:

        NAME                                      STATE     READ WRITE CKSUM
        Fast-Pool                                 ONLINE       0     0     0
          mirror-2                                ONLINE       0     0     0
            e0c6fa8b-eaee-44d2-95f5-92fcba29df6c  ONLINE       0     0     0
            3fdc578f-ef3c-44c7-b67e-7ecc817c3fcd  ONLINE       0     0     0
          mirror-4                                ONLINE       0     0     0
            65f36084-8abe-466b-9f30-7018ab0482b7  ONLINE       0     0     0
            44ad82f4-a302-4b3c-97d0-2c0f8675246a  ONLINE       0     0     0

errors: No known data errors

  pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:00:17 with 0 errors on Sat Jan  7 03:45:18 2023
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          sda3      ONLINE       0     0     0

errors: No known data errors

which makes sense to me as to why the backup pool is not found given the disk detection not being correct here. The use case here was to reconfigure pool setup for Raidz2 configuration from a Raidz1 configuration, hence backing up the data. Any help would be greatly appeciated!

 

[artlessknave]

FebSmart PCIE 3.0 to Sata Host controller

these are universally garbage and cause; inconsistent issues....

you have unsupported hardware. at the very least, you should buy a supported HBA. and return that one.
when I look it up, it says 77$, which is insane. an HBA is around that, and the known good ones will work for basically ever.

 

[Redcoat]

Any help would be greatly appeciated!

@artlessknave has pointed you in the right direction. This from the resources section should carry you a bit further: https://www.truenas.com/community/r...bas-and-why-cant-i-use-a-raid-controller.139/

 

[Davvo]

Multiply your problems with SATA Port Multipliers and cheap SATA controllers

In the last year or two, we've had a resurgence of users asking about SATA Port Multipliers and cheap SATA controllers. Please, do NOT use port multipliers, and use cheap SATA controllers only after extensive research. SATA controllers and SATA...

www.truenas.com

 

[USAFrenzy]

First things first, thank you all for the ridiculously quick response.

I do remember seeing some posts floating about in regards to LSI controllers but was unsure if that would have been totally necessary given the mixed detection occurring on the PCIE card, but, after reading both articles linked, being told that the pcie card *tends* to be inconsistent, and hearing the opinions of all of you sharing the same conclusion, it appears I should do a bit more searching for the LSI controllers - thank you!

 

[artlessknave]

you are also running hardware that is actively discouraged. the specs for that mobo show a realtek NIC, which are only marginally more useful than the eWASTE SATA controllers. at the least, you should seriously consider getting a supported NIC; generally, intel, chesio, mellanox are the kings for this space.

ECC is good too see, though.

 

[Davvo]

Is my Realtek ethernet really that bad?

One of the longest running issues faced here in the forums may be people who come to these forums, wanting to recycle an old PC into a NAS. In general, this is a bad idea because typical PC hardware isn't optimized towards the task. However...

www.truenas.com

 

[USAFrenzy]

I do have plans on eventually expanding the NIC space soon as I would like to mess around with aggragating some of the network as well as having a network interface solely for VMs and throwing that together with a switch, so this is super helpful information too as I didn't even know Realtek was considered sub par for this kind of environment - thank you!

This motherboard only has 2 16x lanes and 2 1x lanes though, so I need to do some more reading into lane bifurcation to see if I can achieve lane splitting effectively for the controller card and NICs; the gpu I have running is in its full 16x lane for the game server side of things, but given my want to add more storage and expand the NIC space, that will eventually be moved to a dedicated machine.

 

[artlessknave]

sometimes, Davvo feels like my Link posting Bot.

 

[Davvo]

It's not much, but it's honest work.
Please look at my signature for more resources.

 

[artlessknave]

I never think to put the links because they are in my signature

......somewhere. Resources, usually.

 

[USAFrenzy]

Just logged in on desktop and I now wish they showed yall's signature readily on the mobile site -> bookmarked your guys' resources pages for future reading (amazing collection of page resources by the way).

I have to say it once more, thank you to the both of you!

I ended up purchasing a LSI 9240 card and a couple of sas-to-sata cables for a little under $80 (used) and will be returning the PCIE card.
I'll go ahead and mark this thread as solved as soon as I get those in this weekend and make sure all works as intended, or else I may be pinging the gurus for help once more

 

[artlessknave]

mobile site

ohhh. hmm. I didn't think of that. I only really only use mobile if anything better isn't available. like, on the bus. I can't type worth (^(^% on mobile so giving advise is impossible. might have to see what that my signature looks like on mobile....

LSI 9240 card

uuuhhhmmm. i guess it might be a bit too late to mention that, unfortunately, the LSI cards have an annoying amount of fakes.
also, that is a RAID card by standard, not an HBA. did you get one that specifically says it was flashed to IT mode? because if not, you will need to do that.

sas-to-sata cables

also, you might want to double check that you didn't get reverse sas breakout cables. they look the same but won't work (these are for going from SATA mobo ports to SFF-80xx on a backplane)

 

[USAFrenzy]

This is the LSI card in question with the cables:

https://www.amazon.com/gp/product/B08QJCQ76S/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1

If that's not correct, I can always cancel it as it hasn't shipped yet

 

[USAFrenzy]

This is the LSI card in question with the cables:

https://www.amazon.com/gp/product/B08QJCQ76S/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1

If that's not correct, I can always cancel it as it hasn't shipped yet

I should add that I'm now reading the reviews and it seems to be a mixed case so I might have naively jumped the gun here

 

[Redcoat]

Take a look at ebay Art of Server site https://www.ebay.com/itm/162958581156?hash=item25f1169da4:g:7sYAAOSwjMtcT8T8 for example. This is a guy who many of us here have had great experience with. Great products, great support (no dodgy stuff).

You need 2 off SFF8087 to 4 SATA connector cables to go with such a card (connect to 8 drives).

 

[artlessknave]

Actually, that card looks like it could be ok. It says it's IT mode, it should have the right cables, if it's crap Amazon's return is usually fairly painless, it doesn't appear to be a fakeLSI, and appears to have the right stickers and markings for a legit LSI card.

There is a decent chance of that just working.

The fakes are mostly eBay. I assumed that's where you would try first.

 

[Davvo]

About fakes, please read the following resource.

Fake server cards

Some time ago, forum frequenter @artlessknave kindly sent me a pair of LSI HBA cards that had failed to work out in a FreeNAS build. Having suspected that these were fakes, I asked for them to be shipped here for comparison. However, COVID...

www.truenas.com

 

[ChrisRJ]

I do remember seeing some posts floating about in regards to LSI controllers but was unsure if that would have been totally necessary given the mixed detection occurring on the PCIE card, [..]

That is not even the real issue here, because it becomes obvious very quickly. The true problem is that even if such a card seems to be ok initially, there is an increased risk of data corruption down the road. And usually this data corruption takes a while to become apparent. When that happens things have already going south for a while and you effectively lost data.

Or the phrase it differently: The challenge is not to assemble a machine that boots and passes a superficial test. Instead you need a machine that is rock solid and survives dodgy SATA cables, power brown-outs, dying HDDs, etc. Unless of course this is a playground machine.

 

[artlessknave]

And usually this data corruption takes a while to become apparent

ZFS wont allow data corruption to hide. it will report it immediately on read when the checksums dont match.
what can hide is hardware errors that can cause data corruption later, but, by design, any corruption should be obvious, with ZFS giving you notifications.

Instead you need a machine that is rock solid and survives dodgy SATA cables, power brown-outs, dying HDDs, etc.

this is the way. you build a bunker and let the world have at it.