Hi, I recently get a HP Gen10 ProLiant Microserver to build a NAS. I bought 4 HDD to build a ZFS pool using RAIDZ, however the write performance is lower than expected. I also tried mirror/stripe, or RAID10, they all seem to have the similar performance, so I wonder the bottleneck might not be ZFS or RAID. (benchmark stats at the end)
CPU usage/load is low during during the IO operations, also the memory usage or ARC usage were high either.
Here is the hardware:
- CPU: AMD Opteron X3216
- Memory: 8GB DDR4 ECC
- Storage Controller: Embedded Marvell 88SE9230 PCIe to SATA 6Gb/s Controller
- Hard Drives: TOSHIBA MG04ACA400N (4TB) x 4 (ada0,ada1,ada2,ada3)
- Boot Drive: SanDisk Extreme USB Disk (32GB) (da0)
- Sytem Drive: TOSHIBA- TR200 SSD (240GB) (da1)
And here is the
- Here is how I benchmark for write performance:
As you can see here the total write speed on each drive is roughly 30M/s. Then I tried creating UFS directly on each drive and here's the benchmark result:
Here is more information if needed:
A lot of new things to me, RAID, ZFS, FreeBSD - so if I missed or mistaken anything please kindly point me out
CPU usage/load is low during during the IO operations, also the memory usage or ARC usage were high either.
Here is the hardware:
- CPU: AMD Opteron X3216
- Memory: 8GB DDR4 ECC
- Storage Controller: Embedded Marvell 88SE9230 PCIe to SATA 6Gb/s Controller
- Hard Drives: TOSHIBA MG04ACA400N (4TB) x 4 (ada0,ada1,ada2,ada3)
- Boot Drive: SanDisk Extreme USB Disk (32GB) (da0)
- Sytem Drive: TOSHIBA- TR200 SSD (240GB) (da1)
And here is the
zpool status
:Code:
$ zpool status pool: freenas-boot state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM freenas-boot ONLINE 0 0 0 da0p2 ONLINE 0 0 0 errors: No known data errors pool: storage state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM storage ONLINE 0 0 0 gptid/7aae6b35-e327-11e9-9bc5-5480284903b2 ONLINE 0 0 0 gptid/7c1218b6-e327-11e9-9bc5-5480284903b2 ONLINE 0 0 0 gptid/7d5d2fb3-e327-11e9-9bc5-5480284903b2 ONLINE 0 0 0 gptid/7eac58ef-e327-11e9-9bc5-5480284903b2 ONLINE 0 0 0 errors: No known data errors pool: system state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM system ONLINE 0 0 0 gptid/33c5fbcb-e294-11e9-9ddf-5480284903b2 ONLINE 0 0 0 errors: No known data errors
- Here is how I benchmark for write performance:
Code:
$ sudo dd if=/dev/urandom of=/mnt/storage/test bs=1M count=3000 3000+0 records in 3000+0 records out 3145728000 bytes transferred in 25.706836 secs (122369319 bytes/sec) (on another terminal)$ sudo zpool iostat -v 5 capacity operations bandwidth pool alloc free read write read write -------------------------------------- ----- ----- ----- ----- ----- ----- freenas-boot 767M 27.8G 0 0 0 0 da0p2 767M 27.8G 0 0 0 0 -------------------------------------- ----- ----- ----- ----- ----- ----- storage 826M 14.5T 0 418 0 130M gptid/7aae6b35-e327-11e9-9bc5-5480284903b2 203M 3.62T 0 107 0 32.2M gptid/7c1218b6-e327-11e9-9bc5-5480284903b2 208M 3.62T 0 106 0 33.0M gptid/7d5d2fb3-e327-11e9-9bc5-5480284903b2 211M 3.62T 0 94 0 33.2M gptid/7eac58ef-e327-11e9-9bc5-5480284903b2 205M 3.62T 0 109 0 31.8M -------------------------------------- ----- ----- ----- ----- ----- ----- system 13.8G 206G 0 0 0 0 gptid/33c5fbcb-e294-11e9-9ddf-5480284903b2 13.8G 206G 0 0 0 0 -------------------------------------- ----- ----- ----- ----- ----- -----
As you can see here the total write speed on each drive is roughly 30M/s. Then I tried creating UFS directly on each drive and here's the benchmark result:
Code:
$ sudo zpool destroy storage $ sudo mkdir /mnt/ada{0,1,2,3} $ for i in {0,1,2,3}; do sudo newfs -U "/dev/ada${i}p2"; done (output omitted) $ for i in {0,1,2,3}; do sudo mount /dev/ada${i}p2 /mnt/ada${i}; done $ for i in {0,1,2,3}; do sudo dd if=/dev/urandom of=/mnt/ada$i/test bs=1M count=2048; done 2048+0 records in 2048+0 records out 2147483648 bytes transferred in 16.578687 secs (129532796 bytes/sec) 2048+0 records in 2048+0 records out 2147483648 bytes transferred in 16.758181 secs (128145390 bytes/sec) 2048+0 records in 2048+0 records out 2147483648 bytes transferred in 16.818135 secs (127688572 bytes/sec) 2048+0 records in 2048+0 records out 2147483648 bytes transferred in 16.899451 secs (127074163 bytes/sec)
Here is more information if needed:
Code:
$ sudo lspci 00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Root Complex 00:01.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Wani [Radeon R5/R6/R7 Graphics] (rev 84) 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Host Bridge 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Root Port 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Root Port 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Host Bridge 00:08.0 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1578 00:09.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 157d 00:10.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 20) 00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 49) 00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 49) 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 4a) 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 4a) 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Function 0 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Function 1 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Function 2 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Function 3 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Function 4 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Function 5 01:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller (rev 11) 02:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe 02:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe $ sudo geom disk list Geom name: ada0 Providers: 1. Name: ada0 Mediasize: 4000787030016 (3.6T) Sectorsize: 512 Mode: r2w2e4 descr: TOSHIBA MG04ACA400N lunid: 500003984ba82788 ident: 18R8K6QUFSYC rotationrate: 7200 fwsectors: 63 fwheads: 16 Geom name: ada1 Providers: 1. Name: ada1 Mediasize: 4000787030016 (3.6T) Sectorsize: 512 Mode: r2w2e4 descr: TOSHIBA MG04ACA400N lunid: 500003984ba82756 ident: 18R8K6PTFSYC rotationrate: 7200 fwsectors: 63 fwheads: 16 Geom name: ada2 Providers: 1. Name: ada2 Mediasize: 4000787030016 (3.6T) Sectorsize: 512 Mode: r2w2e4 descr: TOSHIBA MG04ACA400N lunid: 50000397fbb021a6 ident: 87Q9K4QLFSYC rotationrate: 7200 fwsectors: 63 fwheads: 16 Geom name: ada3 Providers: 1. Name: ada3 Mediasize: 4000787030016 (3.6T) Sectorsize: 512 Mode: r2w2e4 descr: TOSHIBA MG04ACA400N lunid: 500003984b802ff8 ident: 18R3K6J5FSYC rotationrate: 7200 fwsectors: 63 fwheads: 16 Geom name: da0 Providers: 1. Name: da0 Mediasize: 31376707072 (29G) Sectorsize: 512 Mode: r1w1e2 descr: SanDisk Extreme lunid: 6001b4403fc164390000000000000000 ident: AA010119161727160443 rotationrate: unknown fwsectors: 63 fwheads: 255 Geom name: da1 Providers: 1. Name: da1 Mediasize: 240057409536 (224G) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r1w1e3 descr: TOSHIBA- TR200 lunname: TOSHIBA-TR200 lunid: 50001b8c00050609 ident: 000012345678 rotationrate: 0 fwsectors: 63 fwheads: 255 Geom name: da2 Providers: 1. Name: da2 Mediasize: 1000170586112 (931G) Sectorsize: 512 Stripesize: 4096 Stripeoffset: 0 Mode: r0w0e0 descr: WD Elements 25A2 lunname: WD Elements 25A2 WXK1A27DJJCS lunid: WD Elements 25A2 WXK1A27DJJCS ident: 57584B31413237444A4A4353 rotationrate: 5400 fwsectors: 63 fwheads: 255
A lot of new things to me, RAID, ZFS, FreeBSD - so if I missed or mistaken anything please kindly point me out