My First Test Server - IOPS Question?

[riggieri]

Hi Everyone

I built a first test bed serve and got it running on my network after catching some first stupid newbie mistakes.

Here is the Build
Dell T7400
1x Xeon X5660
12GB ECC RAM
4X HGST 1TB in Raidz1
Chelsio 420-BT Bonded LACP

After running some copy tests over SMB from a iMac Pro and fixing a couple things, I was seeing 225MB/s Read / 375MB/s Write. Great for a single client.

While I was doing a 667GB test transfer write operation over SMB, from another client, I tried copying a 256KB folder with scripts and readme (NextCloud Script), and that literally took over 2 minutes.

Is that normal for ZFS/RaidZ1 in a single vdev or should I be looking at something else. Is this where a SLOG comes in?

Could it be the bonded ethernet?

 

[ezekiel.incorrigible]

Is this test repeatable? Did you do any other tests from the same client? How many individual files are there in your 256KB folder? (you can find out with find . | wc -l) what is the transfer time for a tarball of that same data?

 

[riggieri]

Thanks Ezekiel for the reply.

I will run some more tests when I have a chance early next week. I will say it wasn’t that many files at all. Maybe like 8-10. But I’ll give you an exact number when I run the tests.

 

[ezekiel.incorrigible]

Well, if it's just a few files then I doubt it's anything to do with the iops. Probably going to need a lot more info. to diagnose! I suggest seeing how long it takes for your 'other client' to transfer to your macbook pro - what about trying other protocols like rsync or scp? Just try to narrow down where the issue may lie - is it the client's network connection in general? Is the problem specific to SMB? can your macbook pro scp from your 'other client' in a reasonable amount of time?

 

[riggieri]

So some more info and tests.

Server (This is just a test server):
Dell T7400
1x Xeon X5620 @ 2.4GHZ
12GB ECC RAM
4X HGST 1TB in Raidz1 connected to LSI 1068e onboard
Chelsio 420-BT Bonded LACP
NextGear XS716T

Client1:
iMacPro
OnBoard 10GBe
Connected directly to Switch

Client2
Mac Pro Tower
OnBoard 1GBe Controller
Connected directly to Switch

So Here is what I have been doing to test. I am running SMB/CIFS because AFP is deprecated and we need SMB3 to run FCPX libraries. I am doing straight finder copies, because this is the real world scenario. Copying ProRes files between 50-100GB. All footage is being copied from client DAS RAID, each can support 1GB/s.

Single Client Test:
Client 1 Copy to Server: ~200MB/s
Client 1 Read from Server: ~350MB/s
Client 2 Copy to Server: ~116MB/s
Client 2 Read from Server: 116MB/s

Two Client Tests
Client 1 Copy to Server and then initiate Client 2 Copy to Server
Client 1: ~200MB/s
Client 2: SLOW and then Fail
Test 2:
Client 1: ~100MB/s
Client 2: 100MB/s
Test 3:
Client 1: ~200MB/s
Client 2: SLOW and then Fail

Client 2 Copy to Server then initiate Client 1 Copy to Server (Ran 4 Times)
Client 1: ~75-100MB/s
Client 2: ~75-100MB/s

Client 1 Read From Server While Client 2 Reads from Server (As Expected)
Client 1: ~175-200MB/s
Client 2: ~75-110MB/s

Client 2 Read From Server While Client 1 Copy to Server (As Expected)
Client 2 Read: 35MB/s
Client 1 Copy: 150MB/s

So is it expected behavior if the array is being written to at full speed to not be able to handle another request? It seems that when the array is writing data but not at full speed and then has a second write operation request, it is fine. Only when it is writing at its full bandwidth is it causing issues. Directory browsing at that time is slow and the first part of the copy seems to take a second. When the copies are started close to each other it seems to be ok, but when the first copy is started and then I wait a few minutes to start second copy. Or is this a SMB issue?

I also understand this is just a 4 drive RaidZ1, and my planned deployment is a 6 Vdev/6 Drive Raid Z1 pool, and that will handle multiple read/write operations far superior.