Poor Pool Speed (Only Accessing 4 Drives Out Of 16)

[IrY100Fan]

Hi Everyone,

I setup a new server using TrueNAS SCALE 23.10.0.1 and sixteen 4TB SSDs. Originally I setup the pools as one RAID-Z2 VDEV of sixteen drives, then I tried two eight drive RAID-Z2 VDEVs but I have the same issue. The write performance is awful (~260MB/s) and watching the drive lights indicate only four drives get access at one time. I kind of expected all sixteen drive lights to be blinking and to have much greater performance (>1GB/s). (I mean, each drive is capable of 500MB/s each.) I even dabbled with dRAID but that was even worse performance. Any thoughts?

Thanks.
-Brian

 

[Arwen]

Hardware layout, including make & model. Sometimes people use something that is not really ideal for speed purposes.

 

[IrY100Fan]

The system is home-built.

AMD EPYC 7252 "Rome" 8-Core Processor
48GB DDR4-3200 ECC RAM
ASRockRack ROMED6U-2L2T Motherboard
(Onboard) Intel X710-AT2 10G NIC (Single port connected to 10G switch)
Intel Optane P1600X 58GB (Boot Drive)
(16x) SiliconPower A55 4TB SATA SSDs (Network Storage Drives)
TrueNAS SCALE 23.10.0.1

The sixteen SSDs are attached to the SATA ports directly connected to the processor.

The original config was all 16 as a single RAID-Z2 VDEV. (Most efficient use of drives vs capacity.)
I also tried splitting the drives up into two and three VDEVs. (No change.) I even tried a single 16 drive dRAID and also tried eight 2 drive mirrors. I get the exact same results with every config.

 

[IrY100Fan]

A little more info.

When I watch the drive LEDs when configured as RAID-Z2 or dRAID, only the first 4 drives flash. First 3 drives, the the fourth drive and repeats.

When I set it up as a bunch or mirrors, again only the first four drives flash. First 1 and 3, then 2 and 4 and repeat.

 

[enka]

Please describe your use case of datasets and a test case scenario for better understanding.
Pool(s) and dataset(s) configuration would also help.

 

[HoneyBadger]

What method are you using to test the write performance? Be aware that if you're using DD with an input file of /dev/random you may be limited by how fast your system can generate entropy, making the test invalid.

 

[IrY100Fan]

At this moment, because I have only begin to setup this server, I have one dataset for video files that is shared over NFS. The files I am transferring to the NAS are large 25-50G files. I left all of the defaults on for the pool and datasets (such as LZ4 compression).

This NAS will be the repository generally large files (greater then 1GB). There will be some smaller files, but the majority will be large. There is one main machine that will communicate with the NAS via 10G network and a couple of other media players that will connect via 1G. There will also be probably a single virtual machine running a TV tuner setup but that wont take much horsepower or memory. (It used to run fine on asystem with an AMD-A10 processor and 4G or RAM.)

 

[IrY100Fan]

To test the performance I am transferring a 35GB video file from my main desktop to the NAS over a 10G network. (The desktop has PCIe 4.0 storage that is capable of transferring over 3GB/s.)

 

[HoneyBadger]

I have one dataset for video files that is shared over NFS.

Your NFS client may be requesting synchronous writes. Since this dataset is for media files, share the dataset over SMB instead (defaults async) or disable sync on the dataset in the Datasets/Edit page, under the Advanced options.

 

[IrY100Fan]

Disabling sync does significantly increase the data throughput but it still only accesses the first 4 drives. I would think that having sixteen drives should allow full network saturation even with sync on.

 

[HoneyBadger]

Disabling sync does significantly increase the data throughput but it still only accesses the first 4 drives. I would think that having sixteen drives should allow full network saturation even with sync on.

That isn't normal or expected for it to only use 4/16 drives - I would suspect a signaling issue with the LEDs if anything. Do you see drive activity if you check the Reporting tab and query disk metrics, or look at the iostat values live while their is data being written?

 

[IrY100Fan]

According to the Disk Reporting Graphs, only 4 out the 16 have any data being written to them.

 

[HoneyBadger]

According to the Disk Reporting Graphs, only 4 out the 16 have any data being written to them.

Can you provide the output of zpool status -v in codeblocks?

 

[IrY100Fan]

Code:

  pool: main
 state: ONLINE
config:

        NAME                                      STATE     READ WRITE CKSUM
        main                                      ONLINE       0     0     0
          raidz2-0                                ONLINE       0     0     0
            82466379-5bf3-4deb-9d6f-05c93849bdd8  ONLINE       0     0     0
            927d7e46-821d-4e49-833e-5ac00e0cfe98  ONLINE       0     0     0
            b7dff8ad-9362-49e5-bc45-d97cf907e83e  ONLINE       0     0     0
            21f3c321-27de-433b-91af-7d2a319090ca  ONLINE       0     0     0
            ff0b5e08-6b4e-4e24-bc15-1ce4cb694ead  ONLINE       0     0     0
            d6ca3030-623a-4ddf-9d9b-fc992fb205f4  ONLINE       0     0     0
            863ac7b0-06ee-47d4-8059-9903eb81712e  ONLINE       0     0     0
            d40d02e7-bbae-463f-ba78-9ce798e404ac  ONLINE       0     0     0
            26593945-b536-4f4e-85f2-0280bbf0aecd  ONLINE       0     0     0
            f1e3367c-02c1-4b12-9910-4b2a98e01632  ONLINE       0     0     0
            9dd461c6-0dea-4023-b46f-d6bf29f0632f  ONLINE       0     0     0
            ed5b0e27-6d86-45dd-8404-0736b1a277df  ONLINE       0     0     0
            ea17a521-4b8c-480e-b8fb-25f49276b68e  ONLINE       0     0     0
            a90f9b95-7cb9-4ca6-9694-767feff953d2  ONLINE       0     0     0
            dd9b31c4-42a8-4b64-9c5b-2bb32b9cdeb7  ONLINE       0     0     0
            caa8e2ba-24e0-4cce-b914-c0bf5d20efdf  ONLINE       0     0     0

errors: No known data errors

 

[IrY100Fan]

Just for fun I installed an Areca ARC-1226 RAID card in my NAS, set it in JOBD mode and attached 8 of the SSDs to it as a test. Again only the first four SSDs were being accessed. Then I switched out the SSDs for mechanical hard drives (Seagate X20s). All 8 of those drives were accessed simultaneously. Now I'm really confused.

 

[PhilD13]

Check the motherboard documentation and see if the SATA controller onboard can only handle 4 SSD's.

 

[HoneyBadger]

Just for fun I installed an Areca ARC-1226 RAID card in my NAS, set it in JOBD mode and attached 8 of the SSDs to it as a test. Again only the first four SSDs were being accessed. Then I switched out the SSDs for mechanical hard drives (Seagate X20s). All 8 of those drives were accessed simultaneously. Now I'm really confused.

You're telling me, this is a head-scratcher for sure.

What happens if you make multiple separate pools, of four drives each?

 

[AlexGG]

I would have noted SMART raw values for "Total LBAs Written" or equivalent, written some significant amount onto the pool, and then checked the raw values against the notes; to see what the SSDs themselves think of write distribution. I am still thinking it may be some kind of measurement and/or indication problem.

After that, I would check for any difference in block sizes (ashift and recordsize) between HDD and SSD pools? If there is a difference, does forcing HDD values onto SSD pool light up the extra drives, even if the parameters themselves are not optimal?

 

[IrY100Fan]

Okay, I have found something...

It appears that only 4 of the drives I own actually work properly. (I actually bought 20 drives in total, only 4 work. That is 16 bad drives out of 20. 80% defective rate. Good grief.)

I found this by running each drive, one at a time, through my primary desktop system. Each of the bad drives don't always get detected first insert (sometime takes 5-6 re-plugs), have very slow and erratic performance (10-100 MB/s read speed) and never illuminate the drive activity indicator light. They all seem to hold data, just not in a very usable fashion. (FYI: The few working drives always get detected, have read speed of 500MB/s and blink the indicator light during access.)

All of the drives were purchased from NewEgg (sold by SiliconPower themselves) and not eBay or some other gray market. This is awful. I contacted the manufacturer for resolution. I'll see how they respond.

Thanks for everyone's help. Maybe some working drives will fix my problem.

 

[HoneyBadger]

I'm shocked that they weren't kicked out of the array for that kind of misbehavior @IrY100Fan - keep us posted on the Newegg status, and hopefully this gets resolved quickly for you!

Edit: Don't open any of the drives (would void the warranty) but as a curiosity ... is there a weight difference between a working and non-working drive? If so, I'd be concerned you got a counterfeit unit that's effectively a bunch of MicroSD cards in a trenchcoat.