Performance Tuning

[fungus1487]

Hi All,

First off, great job with FreeNAS 8, i'm completely new to FreeNAS but had the OS up and running within minutes and had setup a zfs volume and shared with a windows network after just a few more.

Initial tests were good i'm getting on average 40MB/s read and 60MB/s write over gigabit lan.

Does that sound right? Writing being faster than reading that is. I understand there is some form of buffer which fills the RAM when writing whilst offloading it to disk as fast as it can but reading is limited by the disk speed? If so why am I getting such poor read speed with the following hardware...

Asus AT5NM10T-I Motherboard
Intel(R) Atom(TM) CPU D525 @ 1.80GHz
2GB DDR3 RAM
1TB F3 7200rpm SATA 3Gb/s

A suggestion I have seen all over this forum is to increase the RAM but I have been in situations before where throwing RAM at a system in an effort to make it perform better is painting over the underlying issue. If this isn't the case though I will get more RAM, cheers for any help.

 

[William Grzybowski]

I think writes are faster due to the feature of ZFS called COW(Copy-on-write).

Reading would be slower because ZFS relies a lot on RAM to perform well because of the ARC, storing metadata and prefetch. Using more RAM for ZFS is not covering the underlying issue.

One question though, how are you testing to get those speeds? Local benchmark? Samba/CIFS? NFS?

 

[fungus1487]

Hi Thanks for the response,

I am testing with the Intel NAS Performance Toolkit from another machine on my network as well as real-world usage (what windows reports on a read/write :P)

I am currently only running CIFS service which I understand will be slower than NFS. I have mostly professional Windows 7 copies on my network but one or two "Home Edition" (god-help-me) so NFS isn't going to be available to those. Still I can setup an NFS share and try this, just was hoping to push CIFS to the limit.

 

[William Grzybowski]

Just a comment, usually CIFS is CPU bound, samba implementation is single-threaded and with an atom CPU you cannot hope for much...

Either way i think more ram would definitely help, 2gb for OS+ZFS+applications is too low

 

[fungus1487]

Just a comment, usually CIFS is CPU bound, samba implementation is single-threaded and with an atom CPU you cannot hope for much...

First and foremost low power/noise is crucial for this build, and the read/write is perfectly acceptable (tbh I was expecting lower)

Either way i think more ram would definitely help, 2gb for OS+ZFS+applications is too low

I will chuck in another 2GB and see what results I get. Thanks for the help.

I'll post back once I find out what increase in performance (if any) this has.

 

[marcusmarcus]

I have an Atom D525 system but have 4GB of RAM. I get around 50-60MB/s read and 60-70MB/s write. The biggest improvement I got was disabling hyper-threading in the BIOS.
Check out my topic for some tweaks/testing I did: http://forums.freenas.org/showthread.php?1226-Hyper-threading-and-Prefetch

 

[fungus1487]

Well, i'm back and with some more helpful information hopefully.

My hardware spec has changed (due to now having several free drives available). Update hardware is as follows...

Asus AT5NM10T-I Motherboard
Intel(R) Atom(TM) CPU D525 @ 1.80GHz
2x2GB DDR3 RAM
2x2TB WD Caviar Green WD20EARX
2x2TB Seagate Barracuda ST2000DL003

I have a new install of the latest FreeNAS™ 8.0.3 p1 64 bit release on a flash drive.

The above is configured as one zfs raidz volume giving me 6TB usable and is shared via CIFS and FTP.

All 4 drives are connected via the motherboard sata connectors.

A local benchmark produces the following results.

Code:

dd if=/dev/zero of=/mnt/live/ddfile bs=2048k count=10000
20971520000 bytes transferred in 301.433680 secs (69572584 bytes/sec)
20971520000 bytes transferred in 306.356794 secs (68454561 bytes/sec)
20971520000 bytes transferred in 315.765097 secs (66414940 bytes/sec)

dd if=/mnt/live/ddfile of=/dev/null bs=2048k count=10000
20971520000 bytes transferred in 101.996352 secs (205610491 bytes/sec)
20971520000 bytes transferred in 94.844927 secs (221113776 bytes/sec)
20971520000 bytes transferred in 97.076618 secs (216030600 bytes/sec)

Does 66MBps write seem like reasonable local drive speeds for a zfs volume? They appear a pretty slow to me.

If so is there any tuning in FreeNAS that would directly impact (improve) read/write on a zfs volume?

Any other suggestions welcome, thanks.

 

[Daisuke]

ZFS = minimum 8GB of RAM, period. If you have less, forget performance. From documentation:

By default, ZFS disables pre-fetching (caching) for systems containing less than 4 GB of usable RAM. Not using pre-fetching can greatly reduce performance. 4 GB of usable RAM is not the same thing as 4 GB installed RAM as the operating system resides in RAM. This means that the practical pre-fetching threshold is 6 GB, or 8 GB of installed RAM. You can still use ZFS with less RAM, but performance will be affected.

 

[fungus1487]

ZFS = minimum 8GB of RAM, period. If you have less, forget performance. From documentation:

Ok yes, documentation says this, developer says that, hardware guy says another thing.

I wasn't asking and neither am I looking for enterprise level performance (if that was the case the spec would be 10 fold of what I have), just if performance can be improved from what I get out of the box from FreeNAS and my current hardware.

Anyway I have seen suggestions all over the forum to set the following loader variable.

Code:

vfs.zfs.prefetch_disable=0

Doing so has improved both write and read drastically.

Code:

dd if=/dev/zero of=/mnt/live/ddfile bs=2048k count=10000
20971520000 bytes transferred in 226.371328 secs (92642121 bytes/sec)
20971520000 bytes transferred in 215.055635 secs (97516719 bytes/sec)
20971520000 bytes transferred in 301.184295 secs (69630191 bytes/sec)

dd if=/mnt/live/ddfile of=/dev/null bs=2048k count=10000
20971520000 bytes transferred in 83.703656 secs (250544851 bytes/sec)
20971520000 bytes transferred in 81.177317 secs (258342118 bytes/sec)
20971520000 bytes transferred in 80.127334 secs (261727415 bytes/sec)

The change gives me 20% improvement on reading and a 27% improvement on writing.

Are there any gotcha's I should know about using vfs.zfs.prefetch_disable

Also are there any other gems to tweak which may directly affect read/write?

 

[fungus1487]

Back again, I have noticed the following when running dmesg.

Code:

ada0 at ata3 bus 0 scbus3 target 0 lun 0
ada0: <WDC WD20EARX-00PASB0 51.0AB51> ATA-8 SATA 3.x device
ada0: 33.300MB/s transfers (UDMA2, PIO 8192bytes)
ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)

ada1 at ata3 bus 0 scbus3 target 1 lun 0
ada1: <WDC WD20EARX-00PASB0 51.0AB51> ATA-8 SATA 3.x device
ada1: 33.300MB/s transfers (UDMA2, PIO 8192bytes)
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)

ada2 at ata0 bus 0 scbus5 target 0 lun 0
ada2: <ST2000DL003-9VT166 CC3C> ATA-8 SATA 3.x device
ada2: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)

ada3 at ata1 bus 0 scbus6 target 0 lun 0
ada3: <ST2000DL003-9VT166 CC3C> ATA-8 SATA 3.x device
ada3: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)

Does this suggest that FreeNAS sees both my WD20EARX drives as only being able to perform 33.300MB/s transfers?

 

[William Grzybowski]

Yes, looks like ada0 and ada1 are bound to UDMA2 (33MB/s) for some reason, maybe a bug with that sata controller in freebsd... Did you try change sata modes in the BIOS? From AHCI to IDE and vice-versa? Dunno if thats helps/is related....

 

[fungus1487]

Yes, looks like ada0 and ada1 are bound to UDMA2 (33MB/s) for some reason, maybe a bug with that sata controller in freebsd... Did you try change sata modes in the BIOS? From AHCI to IDE and vice-versa? Dunno if thats helps/is related....

Would changing this in the bios potentially break the raidz volume I have setup across the 4 disks? I have a lot of data on there currently. If so might try messing about with this on a spare drive

 

[William Grzybowski]

No. Data is not touched

 

[Daisuke]

Back again, I have noticed the following when running dmesg.

Code:

ada0 at ata3 bus 0 scbus3 target 0 lun 0
ada0: <WDC WD20EARX-00PASB0 51.0AB51> ATA-8 SATA 3.x device
ada0: 33.300MB/s transfers (UDMA2, PIO 8192bytes)
ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)

ada1 at ata3 bus 0 scbus3 target 1 lun 0
ada1: <WDC WD20EARX-00PASB0 51.0AB51> ATA-8 SATA 3.x device
ada1: 33.300MB/s transfers (UDMA2, PIO 8192bytes)
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)

ada2 at ata0 bus 0 scbus5 target 0 lun 0
ada2: <ST2000DL003-9VT166 CC3C> ATA-8 SATA 3.x device
ada2: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)

ada3 at ata1 bus 0 scbus6 target 0 lun 0
ada3: <ST2000DL003-9VT166 CC3C> ATA-8 SATA 3.x device
ada3: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)

Does this suggest that FreeNAS sees both my WD20EARX drives as only being able to perform 33.300MB/s transfers?

Ya. Even the other disks show connected as SATA 1 (150MB/sec). Are all your disks connected to a SATA port? These are my disks (ada0 is a SSD cache disk):

Code:

ada0: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 61057MB (125045424 512 byte sectors: 16H 63S/T 16383C)
ada1 at ahcich8 bus 0 scbus8 target 0 lun 0
ada1: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
...
ada6 at ahcich13 bus 0 scbus13 target 0 lun 0
ada6: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
ada6: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada6: Command Queueing enabled
ada6: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)

 

[marcusmarcus]

@fungus1487
I believe so. Try going through GUI into the storage tab > View Disks > then select Edit for those drives and change Transfer Mode from "Auto" to what the drives are supposed to be, Probably "SATA 1.5Gbit/s" or "SATA 3.0 Gbit/s"

 

[fungus1487]

A bit more delving has turned up a few things.

1. I have tried manually changing the disk settings suggested by marcusmarcus, this did not affect anything sadly

2. I have changed both ST2000DL003 drives to use AHCI and now when running dmesg I get the following output.

Code:

ada0 at ata3 bus 0 scbus3 target 0 lun 0
ada0: <WDC WD20EARX-00PASB0 51.0AB51> ATA-8 SATA 3.x device
ada0: 33.300MB/s transfers (UDMA2, PIO 8192bytes)
ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1 at ata3 bus 0 scbus3 target 1 lun 0
ada1: <WDC WD20EARX-00PASB0 51.0AB51> ATA-8 SATA 3.x device
ada1: 33.300MB/s transfers (UDMA2, PIO 8192bytes)
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2 at ahcich2 bus 0 scbus5 target 0 lun 0
ada2: <ST2000DL003-9VT166 CC3C> ATA-8 SATA 3.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3 at ahcich3 bus 0 scbus6 target 0 lun 0
ada3: <ST2000DL003-9VT166 CC3C> ATA-8 SATA 3.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)

It looks like FreeNAS can now see my ada2 and ada3 as SATA 2 devices that support 300MB/s transfers (which is all good).

3. The onboard controller for the first two drives ada0 and ada1 is a JMicron® JMB362 controller which is 2 x SATA 3Gb/s port(s) compatible. But it appears that FreeBSD does not support my controller as a search on http://www.freebsd.org/releases/8.2R/hardware.html#DISK turns up no SATA controllers by JMicron. It seems strange that their website indicates that FreeBSD and FreeNAS support this controller from earlier versions http://www.jmicron.com/PDF/OSSupport/OS_Support.pdf would support for this controller have been dropped?
If so where could I look to see why it was dropped or if possible find a way to make the OS support it?

Thanks for the help so far guys.

 

[fungus1487]

Me again,

Anyway, looks like there is and will be no support for the JMicron controller on my motherboard which is a shame but not the end of the world.

I have another SATA controller in the PCIe expansion port which adds a further 4 SATA connections, swapping over the connections from the JMicron controller to this now produces the following output.

Code:

ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <WDC WD20EARX-00PASB0 51.0AB51> ATA-8 SATA 3.x device
ada0: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <WDC WD20EARX-00PASB0 51.0AB51> ATA-8 SATA 3.x device
ada1: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2 at ahcich2 bus 0 scbus5 target 0 lun 0
ada2: <ST2000DL003-9VT166 CC3C> ATA-8 SATA 3.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3 at ahcich3 bus 0 scbus6 target 0 lun 0
ada3: <ST2000DL003-9VT166 CC3C> ATA-8 SATA 3.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)

Which now looks like FreeNAS see's both my WD drives as SATA I speed devices (which considering I can connect up to 4 on the controller through the PCIe will max out the PCIe bandwidth anyway)

Having re-run the performance testing I notice I am still getting very similar speeds so I have come to the conclusion that my limiting factor is CPU/RAM bound. As I am entirely happy with my setup as it is (read and write over network CIFS reaching acbout 60MB/s) I'm not going to consider upgrading any/all hardware until I max out on capacity for this box.

Theoretically I am looking at a 4 drive RAIDZ volume (live) giving me 8TB (6TB usable) and a second 4 drive RAIDZ volume (backup) giving me 8TB (6TB usable) which will be used to store snapshots on. The live volume will make use of my faster SATA ports whilst the backup can use the remaining slow ports (seeing as its a backup i'm not too bothered about read/write speeds).

Note for future, make sure your SATA controllers are supported under FreeBSD :)

Thanks for all the help guys.

 

[William Grzybowski]

Yeah, you cannot expect much using CIFS...
Did you test local speeds though? Any improvements?

 

[fungus1487]

My tests were all run locally, I only meant I am happy with CIFS performance overall.

I know bettering the hardware would improve transfer speeds internally which 'should' see an improvement when running network wide tests. But I have managed to make FreeNAS see my drives as supporting faster transfer rates than it did out of the box and so far the local benchmarking hasn't really moved so I am guessing its just the most i'm going to be able to push out of it with the Atom processor and 4GB RAM.

 

[fungus1487]

Just for clarities sake here are my local benchmarks I got after FreeNAS sees my 2 WD drives as 150MB/s and my 2 Seagates as 300MB/s devices.

Code:

dd if=/dev/zero of=/mnt/live/ddfile bs=2048k count=10000
20971520000 bytes transferred in 232.676777 secs (90131556 bytes/sec)
20971520000 bytes transferred in 308.367623 secs (68008177 bytes/sec)

dd if=/mnt/live/ddfile of=/dev/null bs=2048k count=10000
20971520000 bytes transferred in 118.784559 secs (176550893 bytes/sec)
20971520000 bytes transferred in 108.619242 secs (193073710 bytes/sec)