What could be the bottleneck ?

[Mouftik]

Hi all,

After reading every post on FreeNAS hardware recommendation build and using my FreeNAS for years but mostly for cold storage, I really use it for most of my data fr a couple of month. For cold storage, it was working just fine, but things doesn't get well for a day to day use.

On the first hand, here is the current configuration of the Freenas box :
Intel G3220
32Gb of RAM
M1015 HBA flashed to IT mode
4 Port Intel quadport NIC

6*4Tb WD RED disks in RAIDZ2
2*512Gb Samsung Pro SSD in Mirror

The services used by my NAS are pretty simple : AFP/SMB to my desk computers (Serving files and Time Machine for Backup) and NFS for Linux server shared folders. Nothing more, just one random telegraf binary for reporting. I use the NFS shares on like 6-8 VM's to get docker configurations or files to allow docker to migrate from one VM to another for maintenance purposes.

The problem I have is that even with this configuration, when I want to write a single Zip file from my desktop machine to FreeNAS, I get like 30-50MB/s and when I want to read from it I get 60-70MB/s in the good days. Beeing the only user of this NAS at any time (without counting VM's but I store almost all my data on the RAIZ2 HDD and VM's use the SSD) it seems to me that's it's a little bit low, isn't it ?

Is there any probleme with my config ? I know that on small files I can't reach the full 125MB/s but on a single zip file of 2.5GB, it seems I may approach this speed.

 

[Yorick]

Test the speed locally, not over the network. How is it then? As in, find out whether this is an issue with drive / pool speed, or network speed.

Edit: Couple ideas on how to test
Write speed:
dd if=/dev/random bs=2048k count=10k | pv > /mnt/pool/path/testfile
Read speed:
pv /mnt/pool/path/largefile >/dev/null
Read and write speed:
pv /mnt/pool/path/largefile > /mnt/pool/path/testfile

That write speed /dev/random might tax your CPU. You could also do /dev/null but be sure to write to a dataset without lz4 in that case.

Give us an idea of the motherboard / NIC you are using, as in, exact model.

When you are writing from a desktop machine, is that a Mac or a Windows machine?
If a Mac, how does it behave when you force sync off on the FreeNAS side?

 

[Pitfrr]

In addition to @Yorick's local test, to test the network speed you can use iperf which works with a server and a client's side.

On the server's side:
iperf -s

On the client's side
iperf -c <IP address of the server>

 

[Mouftik]

I tested with iozone and get those results :
Writer speeds :

Reader report :

For the NIC, I am not using the integrated NIC which is a crappy realtek one, and use the 4 ports of the intel card (They are configured as Lagg). lspci gives me Intel 82580 controller.
I will test network performance afterwards, but I am pretty sure that iperf give me the full gigabit performance last time I tested.

Both of them, use SMB and AFP over the Mac and SMB via a Windows 10 machine. Also via NFS, I can not achieve more than 300-400MBits.

 

[Mouftik]

Ok something was wrong on my previous tests, I didn't disabled the compression and it seems that iozone tests with zeros so ... Benchmark was biased. I've created a Dataset for this and tested again. Checking IOStat I have those results :

And looking at FreeNAS reporting I have a 84MB/s per disk writing speed, which in my case of RAIDZ2 would be a usable 336MB/s ... Way above the gigabit I want to achieve.