So Im running out of knowledge horsepower on this one and wanted to get some advice from the experts.
I've recently been optimising my system and am already blown away by the results but I suspect there is some more in it to give before I need to condemn it into production and stop fiddling.
Server system is x9srl-f, e5-1620v2, 32GB ECC, 2*9211-8i (R16), 10*WD 4TB Reds, Intel x520 10gbe, 9.2.1.7 on Crucial USB stick.
Dog old server used for testing connection, Intel Q6600 CPU @ 2.4Ghz, 2GB RAM, 16*1TB RAID6 Areca 1280ML on a shonky old Nvidia engineering sample motherboard...with a Intel x520 (way past retirement age)
To bring you up to speed with what Ive done so far with the new server,
Tested/Benched CPU - ok
Memtest+ RAM - 4 passes clean
Verified all disks with surface scans - all ok
Tested all disks (dd ) in parallel and individually and they are all performing within spec, circa 150MB/s
Connected two machines directly and tuned 10gbe (large frames/basic tunables)
EDIT: since posting the original thread Ive found my client is maxed out at 5gbps when receiving data due to CPU load @ 90%+.
client->server = stable 9.90gbps
server->client = circa 5gbps
Here's the disk perfs...
I was trying to figure out how to verify that my disks are performing over the network connection as expected. I saw a suggestion that ATTO would be a good test. Here's my result....
I ran a 'zpool iostat' along side this and it produced the following results....which are clearly 'odd'. I don't understand why the bandwidth read column is mostly 0's, I'm guessing its because everything is being retrieved from cache and the access isn't touching the disks? I'd welcome any feedback as to further understand and if possible improve the read performance.
Edit: I just ran a 'arcstat.py -f read,hits,miss,hit%,arcsz 1' and cache hits are 100%...
I've recently been optimising my system and am already blown away by the results but I suspect there is some more in it to give before I need to condemn it into production and stop fiddling.
Server system is x9srl-f, e5-1620v2, 32GB ECC, 2*9211-8i (R16), 10*WD 4TB Reds, Intel x520 10gbe, 9.2.1.7 on Crucial USB stick.
Dog old server used for testing connection, Intel Q6600 CPU @ 2.4Ghz, 2GB RAM, 16*1TB RAID6 Areca 1280ML on a shonky old Nvidia engineering sample motherboard...with a Intel x520 (way past retirement age)
To bring you up to speed with what Ive done so far with the new server,
Tested/Benched CPU - ok
Memtest+ RAM - 4 passes clean
Verified all disks with surface scans - all ok
Tested all disks (dd ) in parallel and individually and they are all performing within spec, circa 150MB/s
Connected two machines directly and tuned 10gbe (large frames/basic tunables)
EDIT: since posting the original thread Ive found my client is maxed out at 5gbps when receiving data due to CPU load @ 90%+.
client->server = stable 9.90gbps
server->client = circa 5gbps
Here's the disk perfs...
Code:
[admin@freenas] /mnt/RAID/share% dd if=/dev/zero of=tmp.dat bs=2048k count=50k 51200+0 records in 51200+0 records out 107374182400 bytes transferred in 127.213963 secs (844044001 bytes/sec) (804MB/s) [admin@freenas] /mnt/RAID/share% dd if=tmp.dat of=/dev/null bs=2048k count=50k 51200+0 records in 51200+0 records out 107374182400 bytes transferred in 132.921829 secs (807799465 bytes/sec) (770MB/s)
I was trying to figure out how to verify that my disks are performing over the network connection as expected. I saw a suggestion that ATTO would be a good test. Here's my result....
I ran a 'zpool iostat' along side this and it produced the following results....which are clearly 'odd'. I don't understand why the bandwidth read column is mostly 0's, I'm guessing its because everything is being retrieved from cache and the access isn't touching the disks? I'd welcome any feedback as to further understand and if possible improve the read performance.
Code:
[admin@freenas] /% zpool iostat 1
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
RAID 1.14T 35.1T 79 56 9.76M 3.32M
RAID 1.14T 35.1T 0 0 0 0
RAID 1.14T 35.1T 0 0 0 0
RAID 1.14T 35.1T 0 0 0 0
RAID 1.14T 35.1T 0 0 0 0
RAID 1.14T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 1.89K 0 173M
RAID 1.15T 35.1T 0 780 0 65.7M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 720 0 59.4M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 787 0 64.8M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 65 0 8.24M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 301 0 7.83M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 761 0 64.8M
RAID 1.15T 35.1T 0 48 0 6.12M
RAID 1.15T 35.1T 0 714 0 58.7M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 128K
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 138 0 15.8M
RAID 1.15T 35.1T 0 326 0 8.53M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 761 0 64.8M
RAID 1.15T 35.1T 0 763 0 64.8M
RAID 1.15T 35.1T 0 763 0 64.8M
RAID 1.15T 35.1T 0 763 0 64.8M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 276 0 1.21M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.14T 35.1T 0 241 0 999K
RAID 1.14T 35.1T 0 0 0 0
RAID 1.14T 35.1T 0 797 0 64.7M
RAID 1.15T 35.1T 0 5.16K 0 595M
RAID 1.15T 35.1T 0 5.64K 0 654M
RAID 1.15T 35.1T 0 5.58K 0 643M
RAID 1.15T 35.1T 0 604 0 43.0M
RAID 1.15T 35.1T 0 763 0 64.8M
RAID 1.15T 35.1T 0 0 0 128K
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 498 0 28.7M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 362 0 15.7M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 714 0 58.7M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 552 0 35.7M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 763 0 64.8M
RAID 1.15T 35.1T 0 426 0 53.3M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 637 0 49.3M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 761 0 64.8M
RAID 1.15T 35.1T 0 763 0 64.8M
RAID 1.15T 35.1T 0 763 0 64.8M
RAID 1.15T 35.1T 0 763 0 64.8M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 363 0 15.6M
RAID 1.15T 35.1T 0 399 0 50.0M
RAID 1.15T 35.1T 0 1.57K 0 139M
RAID 1.15T 35.1T 0 1.77K 0 135M
RAID 1.15T 35.1T 0 1.49K 0 130M
RAID 1.15T 35.1T 0 763 0 64.8M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 418 0 22.2M
RAID 1.15T 35.1T 0 885 0 80.3M
RAID 1.15T 35.1T 0 2.86K 0 244M
RAID 1.15T 35.1T 0 2.98K 0 259M
RAID 1.15T 35.1T 0 2.68K 0 250M
RAID 1.15T 35.1T 0 1.05K 0 73.9M
RAID 1.15T 35.1T 0 550 0 68.8M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 734 0 61.2M
RAID 1.15T 35.1T 0 1.60K 0 174M
RAID 1.15T 35.1T 0 4.17K 0 413M
RAID 1.15T 35.1T 0 4.35K 0 456M
RAID 1.15T 35.1T 0 3.92K 0 406M
RAID 1.15T 35.1T 0 2.81K 0 296M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 379 0 17.6M
RAID 1.15T 35.1T 0 2K 0 194M
RAID 1.15T 35.1T 0 4.22K 0 474M
RAID 1.15T 35.1T 0 5.38K 0 620M
RAID 1.15T 35.1T 0 5.07K 0 581M
RAID 1.15T 35.1T 0 5.42K 0 625M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 243 0 1.08M
RAID 1.15T 35.1T 0 2.32K 0 234M
RAID 1.15T 35.1T 0 5.35K 0 617M
RAID 1.15T 35.1T 0 5.33K 0 646M
RAID 1.15T 35.1T 0 5.73K 0 661M
RAID 1.15T 35.1T 0 5.32K 0 643M
RAID 1.15T 35.1T 0 1.28K 0 130M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 2.43K 0 248M
RAID 1.15T 35.1T 0 5.32K 0 612M
RAID 1.15T 35.1T 0 4.94K 0 596M
RAID 1.15T 35.1T 0 6.07K 0 740M
RAID 1.15T 35.1T 0 5.80K 0 702M
RAID 1.15T 35.1T 0 1.30K 0 125M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 500 0 31.8M
RAID 1.15T 35.1T 0 2.46K 0 252M
RAID 1.15T 35.1T 0 5.72K 0 667M
RAID 1.15T 35.1T 0 4.69K 0 561M
RAID 1.15T 35.1T 0 6.21K 0 757M
RAID 1.15T 35.1T 0 6.50K 0 789M
RAID 1.15T 35.1T 0 497 0 28.4M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 2.44K 0 250M
RAID 1.15T 35.1T 0 5.30K 0 611M
RAID 1.15T 35.1T 0 6.27K 0 767M
RAID 1.15T 35.1T 0 5.66K 0 655M
RAID 1.15T 35.1T 0 4.92K 0 586M
RAID 1.15T 35.1T 0 1.71K 0 187M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 2.52K 0 259M
RAID 1.15T 35.1T 0 5.81K 0 700M
RAID 1.15T 35.1T 0 4.54K 0 520M
RAID 1.15T 35.1T 0 6.30K 0 769M
RAID 1.15T 35.1T 0 6.15K 0 749M
RAID 1.15T 35.1T 0 5.77K 0 673M
RAID 1.15T 35.1T 0 3.36K 0 383M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 2.44K 0 250M
RAID 1.15T 35.1T 0 5.43K 0 627M
RAID 1.15T 35.1T 0 6.27K 0 735M
RAID 1.15T 35.1T 0 4.85K 0 577M
RAID 1.15T 35.1T 0 5.84K 0 710M
RAID 1.15T 35.1T 0 6.18K 0 752M
RAID 1.15T 35.1T 0 4.10K 0 453M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 2.53K 0 260M
RAID 1.15T 35.1T 0 5.60K 0 647M
RAID 1.15T 35.1T 0 6.60K 0 774M
RAID 1.15T 35.1T 0 5.43K 0 659M
RAID 1.15T 35.1T 0 5.81K 0 671M
RAID 1.15T 35.1T 0 5.80K 0 706M
RAID 1.15T 35.1T 0 3.58K 0 389M
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0
RAID 1.15T 35.1T 0 0 0 0Edit: I just ran a 'arcstat.py -f read,hits,miss,hit%,arcsz 1' and cache hits are 100%...
Code:
[admin@freenas] /% arcstat.py -f read,hits,miss,hit%,arcsz 1 read hits miss hit% arcsz 0 0 0 0 19G 4 4 0 100 19G 4 4 0 100 19G 4 4 0 100 19G 8 8 0 100 19G 4 4 0 100 19G 4 4 0 100 19G 4 4 0 100 19G 50 50 0 100 19G 34 34 0 100 20G 14 14 0 100 20G 14 14 0 100 21G 86 86 0 100 21G 202 202 0 100 21G 202 202 0 100 21G 199 199 0 100 21G 127 127 0 100 21G 8.8K 8.8K 0 100 21G 18K 18K 0 100 21G 17K 17K 0 100 21G 18K 18K 0 100 21G read hits miss hit% arcsz 10.0K 10.0K 0 100 21G 208 208 0 100 21G 451 451 0 100 21G 439 439 0 100 21G 409 409 0 100 21G 212 212 0 100 21G 9.9K 9.9K 0 100 21G 18K 18K 0 100 21G 17K 17K 0 100 21G 17K 17K 0 100 21G 7.5K 7.5K 0 100 21G 499 499 0 100 21G 865 865 0 100 21G 809 809 0 100 21G 820 820 0 100 21G 329 329 0 100 21G 8.4K 8.4K 0 100 21G 13K 13K 0 100 21G 13K 13K 0 100 21G 13K 13K 0 100 21G 5.8K 5.8K 0 100 21G read hits miss hit% arcsz 1.0K 1.0K 0 100 21G 1.6K 1.6K 0 100 21G 1.8K 1.8K 0 100 21G 1.6K 1.6K 0 100 21G 569 569 0 100 21G 8.3K 8.3K 0 100 21G 12K 12K 0 100 21G 12K 12K 0 100 21G 12K 12K 0 100 21G 4.2K 4.2K 0 100 21G 2.3K 2.3K 0 100 21G 3.2K 3.2K 0 100 21G 3.1K 3.1K 0 100 21G 3.3K 3.3K 0 100 21G 869 869 0 100 21G 9.7K 9.7K 0 100 21G 12K 12K 0 100 21G 12K 12K 0 100 21G 12K 12K 0 100 21G 6.1K 6.1K 0 100 21G 4.7K 4.7K 0 100 21G read hits miss hit% arcsz 5.6K 5.6K 0 100 21G 5.7K 5.7K 0 100 21G 5.7K 5.7K 0 100 21G 1.3K 1.3K 0 100 21G 9.0K 9.0K 0 100 21G 10K 10K 0 100 21G 10K 10K 0 100 21G 10K 10K 0 100 21G 2.3K 2.3K 0 100 21G 8.3K 8.3K 0 100 21G 9.6K 9.6K 0 100 21G 9.9K 9.9K 0 100 21G 9.7K 9.7K 0 100 21G 1.7K 1.7K 0 100 21G 9.1K 9.1K 0 100 21G 10K 10K 0 100 21G 10K 10K 0 100 21G 10K 10K 0 100 21G 1.3K 1.3K 0 100 21G 13K 13K 0 100 21G 14K 14K 0 100 21G read hits miss hit% arcsz 14K 14K 0 100 21G 14K 14K 0 100 21G 808 808 0 100 21G 4.0K 4.0K 0 100 21G 5.2K 5.2K 0 100 21G 5.3K 5.3K 0 100 21G 5.3K 5.3K 0 100 21G 345 345 0 100 21G 17K 17K 0 100 21G 17K 17K 0 100 21G 16K 16K 0 100 21G 17K 17K 0 100 21G 2.5K 2.5K 0 100 21G 5.3K 5.3K 0 100 21G 5.3K 5.3K 0 100 21G 5.3K 5.3K 0 100 21G 5.3K 5.3K 0 100 21G 2.1K 2.1K 0 100 21G 17K 17K 0 100 21G 18K 18K 0 100 21G 18K 18K 0 100 21G read hits miss hit% arcsz 18K 18K 0 100 21G 4.3K 4.3K 0 100 21G 5.2K 5.2K 0 100 21G 5.3K 5.3K 0 100 21G 5.3K 5.3K 0 100 21G 5.3K 5.3K 0 100 21G 5.2K 5.2K 0 100 21G 18K 18K 0 100 21G 17K 17K 0 100 21G 17K 17K 0 100 21G 16K 16K 0 100 21G 1.0K 1.0K 0 100 21G 5.3K 5.3K 0 100 21G 5.3K 5.3K 0 100 21G 5.3K 5.3K 0 100 21G 5.4K 5.4K 0 100 21G 5.8K 5.8K 0 100 21G 19K 19K 0 100 21G 19K 19K 0 100 21G 15K 15K 0 100 21G 16K 16K 0 100 21G read hits miss hit% arcsz 1.5K 1.5K 0 100 21G 5.2K 5.2K 0 100 21G 5.3K 5.3K 0 100 21G 5.3K 5.3K 0 100 21G 5.3K 5.3K 0 100 21G 5.3K 5.3K 0 100 21G 4.2K 4.2K 0 100 21G 6.0K 6.0K 0 100 21G 18K 18K 0 100 21G 17K 17K 0 100 21G 17K 17K 0 100 21G 17K 17K 0 100 21G 16K 16K 0 100 21G 5.0K 5.0K 0 100 21G 5.3K 5.3K 0 100 21G 5.3K 5.3K 0 100 21G 5.3K 5.3K 0 100 21G 5.3K 5.3K 0 100 21G 5.3K 5.3K 0 100 21G 3.6K 3.6K 0 100 21G 7.9K 7.9K 0 100 21G read hits miss hit% arcsz 17K 17K 0 100 21G 18K 18K 0 100 21G 17K 17K 0 100 21G 17K 17K 0 100 21G 17K 17K 0 100 21G 4.8K 4.8K 0 100 21G 5.3K 5.3K 0 100 21G 5.3K 5.3K 0 100 21G 5.3K 5.3K 0 100 21G 5.3K 5.3K 0 100 21G 5.3K 5.3K 0 100 21G 3.8K 3.8K 0 100 21G 9.4K 9.4K 0 100 21G 17K 17K 0 100 21G 18K 18K 0 100 21G 16K 16K 0 100 21G 17K 17K 0 100 21G 17K 17K 0 100 21G 4.3K 4.3K 0 100 21G 5.0K 5.0K 0 100 21G 5.3K 5.3K 0 100 21G read hits miss hit% arcsz 5.1K 5.1K 0 100 21G 5.2K 5.2K 0 100 21G 5.1K 5.1K 0 100 21G 3.9K 3.9K 0 100 19G 4 4 0 100 19G 4 4 0 100 19G 141 141 0 100 19G 7 7 0 100 19G 4 4 0 100 19G 4 4 0 100 19G 4 4 0 100 19G 4 4 0 100 19G
Last edited:
