Performance question

[Mom-frlm]

Hello,

I'm using an "old" supermicro server as a NAS. The specs are:

Intel(R) Xeon(R) CPU W3690 @ 3.47GHz
24 GB Ram (I know, should be more)
LSI 9750-16i4e DISK
16x Seagate IronWolf 8 TB HDD 7200 U/Min, CMR, 256 MB Cache, SATA 6 Gb/s
They are in RAIDZ2. 41TB are in use 50TB are free.

I tried to do a performance test using dd: (i did execute this in the shell on the web interface, not sure if that matters)
dd if=/dev/zero of=/mnt/FMUS-NAS-04/test1.img bs=1G count=10

The result I get is:
10737418240 byes transfered in 36.209469 sec (296536199 bytes/sec)

Well - that's not what I would have had expected.
Can someone please tell me if this is what I should be getting or is there a fundamental flaw in the setup.

Best regards Michael

 

[Mom-frlm]

No ideas?

 

[Rand]

Pool with compression?
Zeros compression very good

 

[Mom-frlm]

No - compression is turned off.

 

[Rand]

Performance is not actually good, sorry, didnt actually calculate.
First example were 280 MB, second is 50, so not the usual compression effect (huge numbers on old spinners)

RaidZ with a single write process will not be fast in the first place. Tried fio with multiple processes?

 

[blanchet]

16x Seagate IronWolf 8 TB HDD 7200 U/Min, CMR, 256 MB Cache, SATA 6 Gb/s
They are in RAIDZ2. 41TB are in use 50TB are free.

What is your pool layout? a single 16-disk wide raidz2 ?

 

[Mom-frlm]

What is your pool layout? a single 16-disk wide raidz2 ?

Yes.

 

[Mom-frlm]

Did some additional tests. We have a second - almost identical - server. There's less storage (16x 2TB) in it and less ram (12GB).
Nevertheless, this one performs much better (560 MB read/write). The main difference is:
In the faster system the hardware RAID controller handles the discs and TrueNAS sees just on disc.
In the slower one the 16 discs are managed by TrueNAS. I thought that was the recommended setup?

 

[blanchet]

Using a RAID controller is not the recommended setup, because everything is delegated to the RAID controller. TrueNAS cannot detect issues on disks. For example, if a disk fails, do you receive an email notification from the RAID controller ?

A 16-disk wide vdevs is a very large vdevs. It is ideal to maximize disk space but it ruins the performances

ZFS: resilver performance of various RAID schemas

A pool with 2x 8-disks wide raidz2 would have better performances (more IOPS).

 

[Mom-frlm]

I guess there's no elegant way to modify the existing pool with data on it to another layout?

 

[blanchet]

There is only one way to change the pool layout:

• backup the data
• destroy the pool
• recreate the pool with the new layout
• restore the data from the backup

You may be interested by this article, especially the Future expansion section

ZFS: You should use mirror vdevs, not RAIDZ. – JRS Systems: the blog

 

[ChrisRJ]

You may be interested by this article, especially the Future expansion section

ZFS: You should use mirror vdevs, not RAIDZ. – JRS Systems: the blog

Please note that at least in my opinion this article is to be read with a little bit of caution. It contains a lot of valuable information but also some bias when it comes to the risk additional disks dying while resilvering. So from my perspective it should not be used as the sole resource for making decisions about the pool layout. I consider myself to be really paranoid about data safety, basically at the level of STP systems in the telco or finance sector. For me the article in question puts a lot of emphasis on a really low risk, that is only applicable if you need the last bit of uptime. And then we enter the discussion of fault-tolerant vs. fault-resilient.