So, how do I test the performance of my system?

[keylevel]

I'm trying to work out how different drives, pool configuration, L2ARC, etc. affect the performance of a system to work out the optimal configuration.

I'm using OS X to talk to an AFP share, and I can get nearly line speed writes (108 MB/s) to the pool when copying files to it using Finder.

However, reading the same file back using Finder only gives about 50 MB/s. Using 'cp' I get more like 75 MB/s. This is in line with the write speeds I get using 'cp'.

The files I'm working with are much bigger than any cache sizes, so this shouldn't be giving a speed boost for later reads.

What I'm after is a means to test the read and write performance that's less dependent on OS behaviour. Does anyone have any recommendations, keeping in mind I have no real budget? ;)

 

[keylevel]

I found http://www.helios.de/web/EN/products/LanTest.html, but that is only reporting 61 MB/s read/write speeds (which ties in with the network traffic reported by FreeNAS).

 

[joeschmuck]

Here is a thread I wrote a long time ago and should still hold true today. Don't let the title fool you, it's all benchmarking and I go through several things. This should be a good starting place for you to setup some benchmarking tests.

For your situation above, it would have been nice if you included your current configuration as we could provide some direct feedback as to why your performance is not what you desire however I will point out the obvious reasons I feel it would be.

1) Writing will be fastest because the data flows into the cache and when it's ready it will write to the hard drive. Both of these are faster than your LAN.
2) Reading requires time to access the file that is all over the hard drives, this requires some time. You already stated that the file is larger than the amount of RAM you have so the file cannot be cached. An L2ARC will not help you here. You should read up on what an L2ARC is and how it works, there is a lot of data on the internet to read.
3) Reading being slow can be caused by poor pool design but can be significantly increased with a proper pool design.

Try other protocols, AFP might just be slow.
Try a different machine and make a direct connection to the FreeNAS server (no switches, routers, etc...).

 

[keylevel]

Thanks

I'm not actually unhappy with the performance I'm getting (I've just got an evaluation lash-up together for now with a J1900 and a single 2.5" FireCuda drive in the pool) - in fact, I think 108 MB/s writes to that is very good :)

What I don't like is the variability in read speed that I'm seeing - these tend to be pointing to an issue with my client. I'm not expecting the L2ARC to improve this, but I would expect its presence to affect things overall as it does take CPU and controller bandwidth to manage it with a benefit to more "real world" workloads.

 

[joeschmuck]

I'm not expecting the L2ARC to improve this, but I would expect its presence to affect things overall

It would and the outcome would likely be to slow your read speeds down, unless you have 64GB or more of RAM. Real world you would not see the speed slow down, benchmarking would show that. If you had a lot of files you read over and over again, the L2ARC would help you there, eventually. You need to read up on how an L2ARC works, don't assume adding it would make a positive outcome. Many people see these old ass Youtube videos where someone slaps together a system and adds an L2ARC and Cache with 16GB RAM and think it's all good. It isn't all good. I'm not saying you did that, but many people do.

For your situation, because you provided the motherboard model I can say with almost 100% certainty that your bottleneck is the Realtek LAN controller. I've been down this road myself and fought it tooth and nail, but it's due to the FreeBSD NIC drivers. If this is the problem then you can't fix this unless you add in an Intel NIC.

 

[keylevel]

Ah, good point. Forgot to mention that I had already booted the Realtek out and was using an Intel E1000. Speeds with the Realtek were "interesting" - started out ok, but dropped by more than 50% after a few minutes.

This is really just a "learning platform" that I've got at the moment. I like what I see, so I'm now looking to specify final hardware. One option is a Supermicro board with a J1900 and two Intel NICs.

 

[Chris Moore]

Ah, good point. Forgot to mention that I had already booted the Realtek out and was using an Intel E1000. Speeds with the Realtek were "interesting" - started out ok, but dropped by more than 50% after a few minutes.

This is really just a "learning platform" that I've got at the moment. I like what I see, so I'm now looking to specify final hardware. One option is a Supermicro board with a J1900 and two Intel NICs.

Before you run out and buy anything, you might want to post what you are thinking about getting and we will be happy to provide feedback or recommendations. Also, if you gave more detail about the type of data you are working with, video files for example, it might help the give you valid suggestions. There is a large pool of experience here that you are largely missing out on by posting only cryptic bits of information.

 

[keylevel]

Thanks - I've already got a few other threads opened where that's being discussed.

Basically, I'm looking to replace a very old ReadyNAS NV+ (they were still Infrant when I bought it). I've had this for over 10 years, will all of the original disks still working. This is basically just used as a backup target for all of the various home and office PCs and to store photos. Currently running in X-RAID (similar to RAID-5). This gets backup up to an off-site rotation (HDDs). It works great, but is slow...

Most of my systems are Apple (with some Linux), so AFP is useful (especially for TimeMachine). I plan to continue with off-site HDD rotation and have been using snapshots and replication for that (similar to what I use with the ReadyNAS where a snapshot is taken and copied to a USB-connected HDD).

My replacement needs to be low-power (J1900 is currently at about 20 W idle, 25 W active) and to offer a pool that can suffer one or two failures (either raids or mirror). J1900 boards (Supermicro have one with 6 SATA) seem to offer enough performance, but they don't support ECC (which is ok for my use-case). I'm also considering the Supermicro A2SDi-2C-HLN4F, but I'm not certain it's fast enough and there is a risk that the x533 networking may not work yet.

The system I've got here is mainly bits that I had lying about with a couple of extras (e.g. an ASM1061 mini-pcie card to add a couple of SATAs) so that I could evaluate FreeNAS.

 

[keylevel]

I've just run a series of tests using FTP (FileZilla), and that's giving much more reasonable results - 100 MB/s write, 99 MB/s read.

Just shows what some decent buffering in the client can do to help.

 

[keylevel]

Tracked down what was causing the differences between read and write speeds - for some reason the MTU on the client was set to over 7000. Reducing this to 1500 means that I now get read and write speeds of about 105 MB/s when using a single HDD in the pool.

I ran some comparative tests on the ReadyNAS and got 40 for read, 20 for write. I gave up with FTP as that peaked at 2 MB/s!!!

 

[Mike Guilmette]

So I have a question. For a Time Machine backup is it normal to see 25MB/s upload?
I followed This Post for setting up my apple shares and datasets. I set up 2 datasets, each with their own alleged quotas, and are currently uploading. According to Activity Monitor it is sending between 15 and 27MB/s.

Freenas 11
Supermicro X11SSH LN4F
Xeon 1230 v5 stock speeds
2x16gb SAMSUNG ECC RAM
LSI 9207 4i4e HBA
4x1tb 2.5" WD Red drives in an icy dock cage.

Fios 150/150Mbps

If you need anything else please let me know

 

[Philip Robar]

I found http://www.helios.de/web/EN/products/LanTest.html, but that is only reporting 61 MB/s read/write speeds (which ties in with the network traffic reported by FreeNAS).

For those who test with Helio Lan Test in the future, choose the 10 Gb Network option instead of the 1 Gb—even if you only have a 1 Gb network—and you'll probably get the speeds you're expecting for a properly functioning network and server. Just like the original poster, I was only getting a little over 60 MB/s in each direction which was much slower than I expected. This lead to some extensive testing with Bonnie++ [1] as a sanity test of local disk performance on the server. I got numbers that seemed reasonable, so I reran Helios Lan Test, but still got just over 60 MB/s. Then for the heck of it I ran Blackmagic Design's Disk Speed Test [2] (I was testing from a Mac client.) and got just over 100 MB/s which left me puzzled. Obviously each test is different, but I wouldn't expect simple tests like these to be so far apart in results. After some poking around on Helios's site and not finding anything useful it popped into my head to try Helios's 10 Gb test option which gave me results of a little over 100 MB/s both ways. Whew!


[1] ZFS Raidz Performance, Capacity and Integrity ("Calomel.org is a privately designed and maintained site dedicated to documenting open source packages and programs." They have several interesting articles on FreeBSD, ZFS, and SMB/Samba.)

[2] Black Magics Designs Disk Speed Test is designed to give an indication what type of video editing your disk system will support. It's available in the Apple Mac App Store.

 

[Chris Moore]

In case anyone's interested I'll be posting my Bonnie++ testing results some time in the next few days.

Here is a link to some useful resources I have accumulated:
https://www.ixsystems.com/community/resources/links-to-useful-threads.108/

Among those is one written by one of our forum moderators that is designed to performance test a storage pool:

solnet-array-test (for drive / array speed) non destructive test
https://forums.freenas.org/index.php?resources/solnet-array-test.1/

There are many others including a link to a git repository (or two) that has scripts for monitoring system health. Please explore what this forum has to offer.

 

[Chris Moore]

I'm not sure how to take this reply since you thank me for my post

I am providing additional information. If you find it helpful, great. If you don't, maybe someone else will. End of story.

I am not being condescending or trying to say that you did anything wrong. Please share anything and everything that you are willing to do.

 

[Chris Moore]

Also, how do you know that I don't regularly "explore what this forum has to offer" and haven't already read many of the articles and threads in your resource link?

I don't. Even though you have been a member since 2014, you only have a few posts, so I have no idea how much you have or have not used the forum.
It isn't a pissing contest.