HHD's missing under Disks, Drive Names skip

[Pray4Tre]

I am just getting into TrueNAS and this is my first implementation of it. I have 12 x 14TB drives installed.
When I installed TrueNAS from my flash drive and it asked what drive i'd like to install on, it showed all 12 of my hard drives plus my 2 m.2 drives.
When installed and booted in, I only saw 10 drives. I started wiping them back to zeroes and 2 drives errored out with this message and now I only see 8 disks.
"Error: [Errno 28] No space left on device"

That happened on disks sdg and sdh.
I also noticed my names on the drives skipped the 2 missing drives (like it recognized they were there but isn't listing them to add to a vdev).
Here is a pic for reference.

Can anyone help me discover what's going on?

Also for background, I had these drives in a previous install of TrueNAS that I ended up just wiping and installing fresh after some weird errors on all the new drives I got and TrueNAS taking 30+ minutes to boot up. This is all on new hardware, but the initial install of the OS was working flawlessly before weird stuff started happening.

 

[joeschmuck]

You should probably list all your hardware specs, along with what version of Scale you are running.

What kind of troubleshooting have you performed on this problem? Power off, check electrical connections, at Shell prompt did you look for /dev/sdg?, what does dmesg show?

 

[Pray4Tre]

Hardware:
CPU: Xeon W-1250
MOBA: GIGABYTE W480 Vision W
RAM: Crucial DDR4 2400 16GB (non ECC) - Have 128gb of ECC RAM coming soon
Boot drive: Samsung 980 PRO 1TB NVMe M.2
HHD's: 12 x 14tb Seagate Exos X14 7200 RPM

TrueNAS Version: TrueNAS-SCALE-22.12.2

I am waiting on wipes to finish before rebooting. Connections I've tested multiple times, all lights and drives spin up and like I said, when installing TrueNAS, it detected all 12 HHD's when asking for a boot device.

In shell I do see /dev/sdg, however I don't see sdh, sdi, or sdj. But I do so sdk and sdl.

How do I run dmesg?
When entering that I just get this:

Code:

admin@truenas[/dev]$ dmesg
dmesg: read kernel buffer failed: Operation not permitted

 

[Pray4Tre]

Is there also a way to check if a disk is connecting/reconnecting? sdg seems to be there sometimes and sometimes not, not sure if its just TrueNAS being weird or if the drive is disconnecting

 

[NugentS]

A few questions

2 * M.2 - are they NVMe or SATA? These are BTW a waste as boot disks. Way oversized

You have 8 * SATA Ports (SATA3 (0~7)
You have 2 * ASMedia ASM1061 SATA Ports - these may (note may) not be a good choice for use
This totals 10 - not 12.
You seem (at least at first glance) to be fitting a quart into a pint pot somehow
Please can you explain the 12 into 10 issue above? I could be looking at the wrong spec sheets
Are the drives causing issues on the GSATA3 (0~1) Ports?

Also you haven't really posted your hardware. What PSU are you using. 12 HDD's can cause wierdness on a low power / less well specced PSU.

 

[joeschmuck]

not sure if its just TrueNAS being weird or if the drive is disconnecting

TrueNAS is fairly stable so I would doubt that would be the issue.

How are the drives connected to the motherboard?

How do I run dmesg?

In the shell,

Code:

cd /var/log
cat dmesg.today | more

Is there also a way to check if a disk is connecting/reconnecting?

It should be in the dmesg file if it's happening.

One other thing you might want to try, install TrueNAS Core, see if that works fine, unless you really just want Scale. Core is more mature.

One other thing to note, you boot drive is 1TB, do you have something a lot smaller to use? And you could even try to boot from a USB Flash drive to see how the performance is. I hate saying that too, I just feel 1TB is some serious overkill.

Another question... So you install TureNAS and then it boots up. It will take a while for the first time. Once the GUI is available then see what drives the GUI can see.

The last thing here... The GUI is listing 10 drives. In the lower right corner of the GUI it says "items per" and then the rest is hidden by the Jobs screen. What does it say when you close the Jobs screen? Can you go to Page 2 and view the other drives? Just asking.

 

[Pray4Tre]

A few questions

2 * M.2 - are they NVMe or SATA? These are BTW a waste as boot disks. Way oversized

You have 8 * SATA Ports (SATA3 (0~7)
You have 2 * ASMedia ASM1061 SATA Ports - these may (note may) not be a good choice for use
This totals 10 - not 12.
You seem (at least at first glance) to be fitting a quart into a pint pot somehow
Please can you explain the 12 into 10 issue above? I could be looking at the wrong spec sheets
Are the drives causing issues on the GSATA3 (0~1) Ports?

Also you haven't really posted your hardware. What PSU are you using. 12 HDD's can cause wierdness on a low power / less well specced PSU.

I have both an PCIe LSI HBA card I used to connect the 2 additional drives.
I also have a PCIe Sata card with 4 SATA ports I used to connect the 2 additional drives.
Both of these have shown all 12 drives in bios and truenas install. Its just once booted into TrueNAS... that i'm not seeing drives.

I also understand the "how many drives are displayed per page" and how that works. I know the screenshot is hidden with the dropdown, but yes I have selected "show 25".

Out of nowhere I am now seeing 13 drives, only missing one is sdj now. No clue why they are taking so long to pop up in the gui... or if they will stay.

I understand the 1tb boot pool is oversized as well... But I got the drive on sale, it was like $5 more than the 500gb (smallest size they had for this model) soooo why not spend $5 for double the size?
The 2tb drive was intended to be a cahce drive... though through some research, it sounds like cache drives aren't worth it.

PSU is between 500 and 750w, gold 80+ rated. Power shouldn't be an issue.

 

[Pray4Tre]

TrueNAS is fairly stable so I would doubt that would be the issue.

How are the drives connected to the motherboard?


In the shell,

Code:

cd /var/log
cat dmesg.today | more

It should be in the dmesg file if it's happening.

One other thing you might want to try, install TrueNAS Core, see if that works fine, unless you really just want Scale. Core is more mature.

One other thing to note, you boot drive is 1TB, do you have something a lot smaller to use? And you could even try to boot from a USB Flash drive to see how the performance is. I hate saying that too, I just feel 1TB is some serious overkill.

Another question... So you install TureNAS and then it boots up. It will take a while for the first time. Once the GUI is available then see what drives the GUI can see.

The last thing here... The GUI is listing 10 drives. In the lower right corner of the GUI it says "items per" and then the rest is hidden by the Jobs screen. What does it say when you close the Jobs screen? Can you go to Page 2 and view the other drives? Just asking.

For dmesg, this is what I get:

Code:

admin@truenas[~]$ cd /var/log
admin@truenas[/var/log]$ cat dmesg.today | more
cat: dmesg.today: No such file or directory
admin@truenas[/var/log]$ 

As I replied to the other post, I understand 1TB is overkill... I purchased a drive that has high resilience for a boot drive vs a garbage tiny NVMe.
It was $5 more than the 500gb model... I'm not sweating the "overkill" in space.

I went with scale because everywhere I had seen online/youtube says why not use scale with more features and they said it ran better.
I suppose I could always give core a shot if all else fails, but I like the flexibility and scalability down the road.

For your other points, I also responded to the reply above

 

[Pray4Tre]

An updated screen shot, I'm assuming randomly sdj will pop up like the others did... but its weird...

 

[joeschmuck]

For dmesg, this is what I get:

Crap, sorry, you are using Scale, I gave you the command for Core.

Drives should not pop up randomly. Some add-on SATA cards may not has suitable drivers for them in the TrueNAS distribution. I'm not saying that is the cause, but it could be.

Again, I'd give TrueNAS Core a test run, see if the problem remains.

 

[morganL]

I have both an PCIe LSI HBA card I used to connect the 2 additional drives.
I also have a PCIe Sata card with 4 SATA ports I used to connect the 2 additional drives.
Both of these have shown all 12 drives in bios and truenas install. Its just once booted into TrueNAS... that i'm not seeing drives.

I also understand the "how many drives are displayed per page" and how that works. I know the screenshot is hidden with the dropdown, but yes I have selected "show 25".

Out of nowhere I am now seeing 13 drives, only missing one is sdj now. No clue why they are taking so long to pop up in the gui... or if they will stay.

I understand the 1tb boot pool is oversized as well... But I got the drive on sale, it was like $5 more than the 500gb (smallest size they had for this model) soooo why not spend $5 for double the size?
The 2tb drive was intended to be a cahce drive... though through some research, it sounds like cache drives aren't worth it.

PSU is between 500 and 750w, gold 80+ rated. Power shouldn't be an issue.

The variety of SATA cards is a concern.
Perhaps check to see if the drive issues are related to one of them.

 

[Pray4Tre]

I also get this message every time I try and make a pool. I’ve been trying to create raid z2 pools with 6 drives and I get this error:

 

[NugentS]

1. What LSI Card and is it in IT Mode - with what firmware?

2. "zpool status" in code blocks please - indents matter

3. Please post your hardware fully. Whats plugged into what - what LSI Card, what sort of cables are you using?

Also, I suggest unplugging any disks from the LSI card - and just use the 6 SATA Ports (not the Asmedia ones) and test setting up a single vdev with those. Then swap the disks for the other 6 and setup a vdev with those 6 disks. After setting up each vdev run some basic tests, including scrub to test for issues.

After you prove that that works, then take one of the vdevs and plug it into the LSI (hopefully a dual port card) running the same tests. This process might show up where the issue is.

 

[Pray4Tre]

I did some testing and disks all seem good after burn in testing except 1, which is RMA'd and i'm awaiting on a replacement.
Things were running fine but i'm getting some errors on sata links going down which is what I think was causing those other issues.
I tried finding this error online but just found another post from months ago with no replies.