Hyper-threading and Prefetch

[marcusmarcus]

Hyper-threading and Prefetch Performance

Decided to do some tweaking on my system to get as much speed as I can out of it. I used totusoft.com's LAN speed Test utility to write a 1GB file to my freenas machine from my desktop and read it back. I decided on 1 GB as most of what I'll be transferring to and from the machine are movies that are 700MB to 1GB in size. I ran the test 5 times after every change I made and averaged the results. Here is what I found out on my system

Hyper-threading enabled and prefetch enabled: 216Mbps write and 722Mbps read
Hyper-threading enabled and prefetch disabled: 174Mbps write and 708Mbps read
Hyper-threading disabled and prefetch enabled: 400Mbps write and 673Mbps read
Hyper-threading disabled and prefetch disabled: 389Mbps write and 642Mbps read

Prefetching setting did not give me much of a suprise but hyperthreading did. Apparently FreeNAS does take advantage of HT when reading but not when writing. Looks like I will be disabling my HT for now, hopefully future versions of FreeNAS make better use of HT but so far I am happy with the performance I am getting from my FreeNAS system.

Also to note: I did try enabled "Large RW" and "AIO" one at a time and both together. Large RW did improve my write speed but my read speed stayed the same. AIO slowed down both read and write, especially when Large RW was enabled with it. So my best performance was achieved by disabling hyper-threading and AIO and enabling Prefetch and Large RW. Hope this helps some of you.


EDIT: Here are the results of my CIFS settings with prefetching enabled and HT disabled

Large RW disables and AIO disabled: 225Mbps write and 688Mbps read
Large RW enabled and AIO disabled: 400Mbps write and 673Mbps read
Large RW disables and AIO enabled: 214Mbps write and 433Mbps read
Large RW enabled and AIO enabled: 364Mbps write and 445Mbps read

I've bolded my optimal settings

 

[matthewowen01]

The Atom's have a 16 stage, in order pipeline, which is not advantageous for heavy I/O, as demonstrated by your test. Nice find, but I will stress that this may be isolated solely to the Atom series and not a general rule. I'm very curious but sadly i don't have any hyper-threaded systems handy.

bringing back bad memories from my architectures class...

 

[ProtoSD]

I'm re-running my test from here: http://forums.freenas.org/showthread.php?981-Notes-on-Performance-Benchamarks-and-Cache.&p=3992&viewfull=1#post3992

I'll update here with my results.

Small improvement in WRITE (22 minutes vs. 23):

Code:

dd if=/dev/zero of=/mnt/tank/tmp.000 bs=2048k count=50k
51200+0 records in
51200+0 records out

107374182400 bytes transferred in 1340.643589 secs (75291364 bytes/sec)

BIG improvement in READ (7 minutes vs. 18)

Code:

dd if=/mnt/tank/tmp.000 of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out

107374182400 bytes transferred in 421.643001 secs (99580329 bytes/sec)

 

[Paktas]

protosd, what did you change for your read/write stats to improve?

 

[shewless]

Do you disable hyperthreading in the bios or via some sysctl?

When you say prefetch do you mean vfs.zfs.prefetch_disable: 0?

 

[marcusmarcus]

Yes, I disabled hyperthreading in the BIOS.
Yes, I added “vfs.zfs.prefetch_disable=0” to "/boot/loader.conf"

 

[Daisuke]

BIG improvement in READ (7 minutes vs. 18)

Not me. FreeNAS 8.0.3 on an Intel Atom D525 1.8GHz processor with 8GB RAM and ZFS RaidZ2 array of 6 disks + 1 64GB SSD cache disk (see complete setup):

Hyper-threading enabled:

Code:

$ dd if=/dev/zero of=/mnt/nas/media/tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 602.091800 secs (178335235 bytes/sec)

$ dd if=/mnt/nas/media/tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 363.791025 secs (295153467 bytes/sec)

Hyper-threading disabled:

Code:

$ dd if=/dev/zero of=/mnt/nas/media/tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 685.600273 secs (156613389 bytes/sec)

$ dd if=/mnt/nas/media/tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 401.614066 secs (267356628 bytes/sec)

I get better results with hyper-threading enabled.

 

[marcusmarcus]

Not me. FreeNAS 8.0.3 on an Intel Atom D525 1.8GHz processor with 8GB RAM and ZFS RaidZ2 array of 6 disks + 1 64GB SSD cache disk (see complete setup):

I get better results with hyper-threading enabled.

I haven't tried it with hyper-threading enabled since upgrading to 8.0.3, I may have to try and see if it improves or gets worse. You maybe getting different results because of the SSD Cache Drive.
Also noticed you said you have an Atom D525 with 8GB of RAM. Is your system really seeing and using all 8GB? I'm asking because AFAIK the Atom processors have a Max Memory of 4GB http://ark.intel.com/products/49490

 

[Daisuke]

Also noticed you said you have an Atom D525 with 8GB of RAM. Is your system really seeing and using all 8GB? I'm asking because AFAIK the Atom processors have a Max Memory of 4GB http://ark.intel.com/products/49490

Ya, you are not the only one noticing this, I got several messages related. :)
I think there is a myth stating that Atom processors cannot handle more than 4GB of RAM. Not sure if is related to Supermicro board on my system only or this applies to all D525 models. My system uses on a regular basis 6GB of wired RAM (default ZFS usage) and FreeNAS displays 8177MB or RAM, which is technically impossible based on Intel specs.