Performance testing disk reads and writes. Where to begin?

[gallyjh]

Hi, I'm new to FreeNAS. Over the last couple weeks I've been tinkering with it and trying to tune my system. Let me describe my equipment:


Dell PowerEdge R720xd 12x LFFE5-2690V2NO
DUAL INTEL XEON PROCESSOR E5-2690 V2 TEN CORE 25M CACHE 3.0GHz
128 DDR3 ECC RAM
2X Seagate IronWolf 8TB NAS Hard Drive 7200 RPM 256MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive ST8000VN0022
2X 2 of Silicon Power 64GB SSD 3D NAND With SLC Cache Performance Boost SATA III 2.5"
Dell Perc H310 SAS/SATA Raid Controller (REMOVED)
Dell H310 6Gbps SAS HBA w/ LSI 9211-8i P20 IT Mode
Broadcom 5720 Quad Port Gigabit Daughter Card
Dell Dual SD Card
PCIe NVMe card with 256 GB Samsung NVMe SSD


I've installed ESXi on the bare metal to the dual SD cards. Created a VM for FreeNAS on the PCIe Samsung SSD. Made the LSI controller pass-through for the VM. The (2) 8TB NAS hard drives (Mirrored) and (2) 64 GB SSD's are managed by FreeNAS. Following instructions from this link, I setup the (2) 64GB SSD as ZIL and Cache (in my case 32 GB for ZIL and 32 GB for Cache).

If I'm forgetting some other important fact, I apologize. Just let me know.

Anyway I'm looking to run some performance tests, but not sure how to get a baseline - that is, probably finding what my max performance on the 8 TB drives when simply making a copy of a large file.

I've created an ESXi Windows VM on the Samsung SSD, a SMB share with FreeNAS and attached that SMB share to the Windows VM. I've copied a large 7GB file, and that seems to give me ~100 MB/s (I would expect this only to be limited by the write speed of the Samsung SSD and read speed of the Seagate HDD). That's making me think it's limited by the NIC (with it hovering at the number) - which doesn't make sense to me since everything is on the same machine. My understanding is that NIC/vSwitch will throughput as much as possible even though they are label 1GB/10GB.

I guess my question is how should I go about testing performance? I think I need to first do some baseline tests. Read/Write speed for the Samsung SSD and the Seagate HDD. Any advice?

 

[kdragon75]

Are you using pass through with HBA for the FreeNAS VM? Are you using vmxnet3 NICs in the VMs? What protocal are you using to transfer files? Are the souce and destination on the same phisical disks? Have you tried doing local benchmarks on FreeNAS?

Please search the forum for information on benchmarking.

 

[kdragon75]

I just re-read your post. When using smb you will NOT use the SLOG unless your dataset it's manually set to sync always. Try testing with iperf from the windows VM to the FreeNAS VM. This will rule out any network limitations.

Side nore, if you change your VM NICs, your MAC addresses will change and you will need to reconfigure the NICs.

 

[gallyjh]

Are you using pass through with HBA for the FreeNAS VM?

Yes.

Are you using vmxnet3 NICs in the VMs?

My NICs on all VMs are all vmxnet3

What protocal are you using to transfer files?

SMB and I've also tried iSCSI

Are the souce and destination on the same phisical disks?

No. The Windows VM is on the Samsung SSD and the SMB is on the Seagate HDD.

Have you tried doing local benchmarks on FreeNAS?

Nope, and that's the adivce I was looking for. AFAIK I don't think FreeNAS has a built in ability to perform a benchmark, or am I wrong? In the meantime, I'll search the forum for information on benchmarking, but I imagine there is a lot of noise to sift through to find something relevant...

I'll post my results when I find something.

 

[gallyjh]

I just re-read your post. When using smb you will NOT use the SLOG unless your dataset it's manually set to sync always. Try testing with iperf from the windows VM to the FreeNAS VM. This will rule out any network limitations.

In regards to the SLOG, that was something I read about when it comes to sync = always. I have this set. I'll post back results on the Window VM to FreeNAS VM. What is your advice on testting performance from the windows VM to FreeNAS? FTP (since that will avoid encryption and be raw)?

 

[gallyjh]

OK found this thread that may be useful:

https://forums.freenas.org/index.php?threads/basic-performance-tests.14338/

Will see what I find out...