10GBe, stripe of mirrors, write is 2x faster then read ? Please help to sort this out.

[ysnk]

My system: FreeNAS-9.3-STABLE-201412090314
Xeon E3-1220V2, 16GB ECC, LSI 9220-8i in IT mode.
8 HDDS in 4 mirrors :
3 pairs of WD RE4 2TB and 1 pair of Hitachi 7K3000 2TB
CIFS share.
Direct 10GBe connection ( intel X540-t1, cat7 cable) to my PC workstation ( dual E5-2696V2, 64GB ram, windows7)
To test the performance I created a 48GB RAM drive with 2 folders:
1st one is about 20GB mix or large rar archives and mp4 videos, all files are larger than 3Gb
2nt one is also about 20GB, my typical video post project - mix of small files, mostly tga, png, jpg, some mp4. About 5000 files.
I copied folders to and from NAS and looked at transfer speed reporting by windows.
Results are:
1) large files - 500-550MB/s write to freenas , 280-300MB/s read
2) small files - 180-200MB/s write, 130-140MB/s read

Looks like something is wrong, I thought read speed should be faster then writes?
Also small files performance is lower than I expected.
Is there something I can tune up? What should I look into first?

Thank you in advance!
____________
Update: I just tested two more configurations - 8 drives in stripe, and two 4-drive raidz2 striped.
Stripe of 8 hdd gave me nodifference in speed, very close to the original stripe of mirors.
Two raidz2 is slower of course, but not much - read speed in fact is only about 10% lower. Write speed is about 25% lower.

 

[diehard]

I know this is probably not super helpful and not what you are looking for but... for starters, is it possible to use something besides CIFS? You have decent hardware and no matter what , CIFS will be a bottleneck.

 

[ysnk]

I know this is probably not super helpful and not what you are looking for but... for starters, is it possible to use something besides CIFS? You have decent hardware and no matter what , CIFS will be a bottleneck.

I'm not really familiar with this stuff - I thought CIFS is the only option for windows. What else can I use?

 

[diehard]

I'm not really familiar with this stuff - I thought CIFS is the only option for windows. What else can I use?

Windows comes with an iSCSI initator, and depending on the version of Windows you have, NFS support is available.

 

[L]

That's cool. ZFS was originally written to for write performance. Your writes should be sequential. I would like to see what zpool iostat is reporting. half a 10gbe for write to me is pretty good. ZFS is caching most of the writes. For the reads it can't. Read cache maybe filling and spilling.

 

[ysnk]

Code:

 zpool iostat -v                                                                                                  
                                           capacity     operations    bandwidth                                                    
pool                                    alloc   free   read  write   read  write                                                   
--------------------------------------  -----  -----  -----  -----  -----  -----                                                   
freenas-boot                            1.82G  13.2G      4      0  84.5K  7.32K                                                   
  gptid/d6106a4c-84a5-11e4-b2ee-94de805a2835  1.82G  13.2G      4      0  84.5K  7.32K                                             
--------------------------------------  -----  -----  -----  -----  -----  -----                                                   
volume1                                 14.9G  7.24T    143    140  10.9M  7.63M                                                   
  mirror                                3.72G  1.81T     36     35  2.72M  1.91M                                                   
    gptid/5410b38c-84a8-11e4-9a8e-94de805a2835      -      -     16     24  1.37M  1.91M                                           
    gptid/549d46bb-84a8-11e4-9a8e-94de805a2835      -      -     15     24  1.35M  1.91M                                           
  mirror                                3.72G  1.81T     35     34  2.71M  1.91M                                                   
    gptid/866e2f0a-84a8-11e4-9a8e-94de805a2835      -      -     16     24  1.38M  1.91M                                           
    gptid/86f1c8db-84a8-11e4-9a8e-94de805a2835      -      -     15     24  1.34M  1.91M                                           
  mirror                                3.72G  1.81T     36     34  2.72M  1.91M                                                   
    gptid/a753c3f1-84a8-11e4-9a8e-94de805a2835      -      -     16     24  1.35M  1.91M                                           
    gptid/a9e364c4-84a8-11e4-9a8e-94de805a2835      -      -     16     24  1.37M  1.91M                                           
  mirror                                3.72G  1.81T     35     35  2.71M  1.91M                                                   
    gptid/46f55973-852f-11e4-acff-94de805a2835      -      -     15     24  1.35M  1.91M                                           
    gptid/47549630-852f-11e4-acff-94de805a2835      -      -     16     24  1.36M  1.91M                                           
--------------------------------------  -----  -----  -----  -----  -----  -----             

 

[bestboy]

Windows comes with an iSCSI initator, and depending on the version of Windows you have, NFS support is available.

my2cents:

• Forget about NFS on Windows. It's lackluster implementation makes it even worse than CIFS in my experience.
• iSCSI in general is probably a very good idea for your workstation.
• People tend to forget the good old FTP. Its simple, fast and specifically made for this job. With concurrent transfers you can usually max out your hardware (either your disks or your network - whatever comes first with 10GbE)

 

[L]

Code:

 zpool iostat -v                                                                                                 
                                           capacity     operations    bandwidth                                                   
pool                                    alloc   free   read  write   read  write                                                  
--------------------------------------  -----  -----  -----  -----  -----  -----                                                  
freenas-boot                            1.82G  13.2G      4      0  84.5K  7.32K                                                  
  gptid/d6106a4c-84a5-11e4-b2ee-94de805a2835  1.82G  13.2G      4      0  84.5K  7.32K                                            
--------------------------------------  -----  -----  -----  -----  -----  -----                                                  
volume1                                 14.9G  7.24T    143    140  10.9M  7.63M                                                  
  mirror                                3.72G  1.81T     36     35  2.72M  1.91M                                                  
    gptid/5410b38c-84a8-11e4-9a8e-94de805a2835      -      -     16     24  1.37M  1.91M                                          
    gptid/549d46bb-84a8-11e4-9a8e-94de805a2835      -      -     15     24  1.35M  1.91M                                          
  mirror                                3.72G  1.81T     35     34  2.71M  1.91M                                                  
    gptid/866e2f0a-84a8-11e4-9a8e-94de805a2835      -      -     16     24  1.38M  1.91M                                          
    gptid/86f1c8db-84a8-11e4-9a8e-94de805a2835      -      -     15     24  1.34M  1.91M                                          
  mirror                                3.72G  1.81T     36     34  2.72M  1.91M                                                  
    gptid/a753c3f1-84a8-11e4-9a8e-94de805a2835      -      -     16     24  1.35M  1.91M                                          
    gptid/a9e364c4-84a8-11e4-9a8e-94de805a2835      -      -     16     24  1.37M  1.91M                                          
  mirror                                3.72G  1.81T     35     35  2.71M  1.91M                                                  
    gptid/46f55973-852f-11e4-acff-94de805a2835      -      -     15     24  1.35M  1.91M                                          
    gptid/47549630-852f-11e4-acff-94de805a2835      -      -     16     24  1.36M  1.91M                                          
--------------------------------------  -----  -----  -----  -----  -----  -----             

Are you running a read and write test at the same time?

 

[ysnk]

my2cents:

• Forget about NFS on Windows. It's lackluster implementation makes it even worse than CIFS in my experience.
• iSCSI in general is probably a very good idea for your workstation.
• People tend to forget the good old FTP. Its simple, fast and specifically made for this job. With concurrent transfers you can usually max out your hardware (either your disks or your network - whatever comes first with 10GbE)

I have few other computers on the network I want to connect to the same share. Can I use iSCSI to connect? I never used it before.
I want to keep it as compatible with Windows as possible.
I'm not sure if FTP will work for me, I don't need to transfer files - I want to mount the drive and use it as folder to have my project files etc. Or am I missing something?

 

[L]

Code:

 zpool iostat -v                                                                                                 
                                           capacity     operations    bandwidth                                                   
pool                                    alloc   free   read  write   read  write                                                  
--------------------------------------  -----  -----  -----  -----  -----  -----                                                  
freenas-boot                            1.82G  13.2G      4      0  84.5K  7.32K                                                  
  gptid/d6106a4c-84a5-11e4-b2ee-94de805a2835  1.82G  13.2G      4      0  84.5K  7.32K                                            
--------------------------------------  -----  -----  -----  -----  -----  -----                                                  
volume1                                 14.9G  7.24T    143    140  10.9M  7.63M                                                  
  mirror                                3.72G  1.81T     36     35  2.72M  1.91M                                                  
    gptid/5410b38c-84a8-11e4-9a8e-94de805a2835      -      -     16     24  1.37M  1.91M                                          
    gptid/549d46bb-84a8-11e4-9a8e-94de805a2835      -      -     15     24  1.35M  1.91M                                          
  mirror                                3.72G  1.81T     35     34  2.71M  1.91M                                                  
    gptid/866e2f0a-84a8-11e4-9a8e-94de805a2835      -      -     16     24  1.38M  1.91M                                          
    gptid/86f1c8db-84a8-11e4-9a8e-94de805a2835      -      -     15     24  1.34M  1.91M                                          
  mirror                                3.72G  1.81T     36     34  2.72M  1.91M                                                  
    gptid/a753c3f1-84a8-11e4-9a8e-94de805a2835      -      -     16     24  1.35M  1.91M                                          
    gptid/a9e364c4-84a8-11e4-9a8e-94de805a2835      -      -     16     24  1.37M  1.91M                                          
  mirror                                3.72G  1.81T     35     35  2.71M  1.91M                                                  
    gptid/46f55973-852f-11e4-acff-94de805a2835      -      -     15     24  1.35M  1.91M                                          
    gptid/47549630-852f-11e4-acff-94de805a2835      -      -     16     24  1.36M  1.91M                                          
--------------------------------------  -----  -----  -----  -----  -----  -----             

Can you run for a couple a few increments during testing? #zpool iostat -v 1

 

[ysnk]

Are you running a read and write test at the same time?

I don't know, all I did is typed "zpool iostat -v"

 

[L]

I don't know, all I did is typed "zpool iostat -v"

Is the workload running? zpool iostat will give you the numbers for writes and bandwidth from the freenas's perspective. It is interesting to look at so that you have measurements on both sides. From the windows point of view and from inside the freenas.

 

[ysnk]

Can you run for a couple a few increments during testing? #zpool iostat -v 1

How long should I let it run?

Is the workload running? zpool iostat will give you the numbers for writes and bandwidth from the freenas's perspective. It is interesting to look at so that you have measurements on both sides. From the windows point of view and from inside the freenas.

So I should run iostat while copying files to/from freenas?

 

[ysnk]

I ran iostat in the shell while copying folders with small files from and to nas.
I stopped it with ctrl-C

Code:

Shell
  mirror                                3.16G  1.81T     97    254  8.62M  12.4M                                                  
    gptid/46f55973-852f-11e4-acff-94de805a2835      -      -     56    135  4.75M  12.4M                                          
    gptid/47549630-852f-11e4-acff-94de805a2835      -      -     40    136  3.87M  12.4M                                          
--------------------------------------  -----  -----  -----  -----  -----  -----                                                  
                                                                                                                                  
                                           capacity     operations    bandwidth                                                   
pool                                    alloc   free   read  write   read  write                                                  
--------------------------------------  -----  -----  -----  -----  -----  -----                                                  
freenas-boot                            1.82G  13.2G    142      0   273K      0                                                  
  gptid/d6106a4c-84a5-11e4-b2ee-94de805a2835  1.82G  13.2G    142      0   273K      0                                            
--------------------------------------  -----  -----  -----  -----  -----  -----                                                  
volume1                                 12.7G  7.24T    257  1.43K  20.0M  93.5M                                                  
  mirror                                3.18G  1.81T     71    406  5.40M  23.1M                                                  
    gptid/5410b38c-84a8-11e4-9a8e-94de805a2835      -      -     32    300  2.79M  23.1M                                          
    gptid/549d46bb-84a8-11e4-9a8e-94de805a2835      -      -     36    300  2.62M  23.1M                                          
  mirror                                3.19G  1.81T     51    371  4.92M  23.6M                                                  
    gptid/866e2f0a-84a8-11e4-9a8e-94de805a2835      -      -     23    306  2.26M  23.6M                                          
    gptid/86f1c8db-84a8-11e4-9a8e-94de805a2835      -      -     27    307  2.67M  23.6M                                          
  mirror                                3.19G  1.81T     56    375  4.35M  23.9M                                                  
    gptid/a753c3f1-84a8-11e4-9a8e-94de805a2835      -      -     24    304  1.77M  23.9M                                          
    gptid/a9e364c4-84a8-11e4-9a8e-94de805a2835      -      -     31    304  2.58M  23.9M                                          
  mirror                                3.19G  1.81T     76    315  5.30M  22.9M                                                  
    gptid/46f55973-852f-11e4-acff-94de805a2835      -      -     38    261  2.39M  23.0M                                          
    gptid/47549630-852f-11e4-acff-94de805a2835      -      -     37    264  2.91M  23.0M                                          
--------------------------------------  -----  -----  -----  -----  -----  -----                                                  
                                                                                                                                  
                                           capacity     operations    bandwidth                                                   
pool                                    alloc   free   read  write   read  write                                                  
--------------------------------------  -----  -----  -----  -----  -----  -----                                                  
freenas-boot                            1.82G  13.2G     19      0  14.8K      0                                                  
  gptid/d6106a4c-84a5-11e4-b2ee-94de805a2835  1.82G  13.2G     19      0  14.8K      0                                            
--------------------------------------  -----  -----  -----  -----  -----  -----                                                  
volume1                                 12.8G  7.24T    174  1.36K  15.1M  78.2M                                                  
  mirror                                3.20G  1.81T     40    408  3.84M  19.4M                                                  
    gptid/5410b38c-84a8-11e4-9a8e-94de805a2835      -      -     17    298  1.75M  19.4M                                          
    gptid/549d46bb-84a8-11e4-9a8e-94de805a2835      -      -     22    297  2.09M  19.4M                                          
  mirror                                3.21G  1.81T     37    380  3.27M  20.1M                                                  
    gptid/866e2f0a-84a8-11e4-9a8e-94de805a2835      -      -     22    291  1.81M  20.1M                                          
    gptid/86f1c8db-84a8-11e4-9a8e-94de805a2835      -      -     14    291  1.47M  20.1M                                          
  mirror                                3.21G  1.81T     43    337  3.59M  20.1M                                                  
    gptid/a753c3f1-84a8-11e4-9a8e-94de805a2835      -      -     18    288  1.76M  20.1M                                          
    gptid/a9e364c4-84a8-11e4-9a8e-94de805a2835      -      -     24    286  1.83M  20.1M                                          
  mirror                                3.20G  1.81T     53    262  4.36M  18.6M                                                  
    gptid/46f55973-852f-11e4-acff-94de805a2835      -      -     26    241  2.13M  18.6M                                          
    gptid/47549630-852f-11e4-acff-94de805a2835      -      -     26    241  2.23M  18.6M                                          
--------------------------------------  -----  -----  -----  -----  -----  -----             

 

[bestboy]

I have few other computers on the network I want to connect to the same share. Can I use iSCSI to connect?

Nope, that's not possible. iSCSI is no share. It provides a block device/virtual hard disc to 1 computer. You could still use it for a scratch drive or something.

I'm not sure if FTP will work for me, I don't need to transfer files - I want to mount the drive and use it as folder to have my project files etc. Or am I missing something?

Yeah, in this case FTP is probably also not the best choice. It shines at copying data from A to B (which is what you tested in OP), but sharing is a bit cumbersome. It could be useful in a scenario, where you work locally and then at the end of the day copy stuff over to the NAS for sharing or backup.

 

[ysnk]

Nope, that's not possible. iSCSI is no share. It provides a block device/virtual hard disc to 1 computer. You could still use it for a scratch drive or something.

Yeah, in this case FTP is probably also not the best choice. It shines at copying data from A to B (which is what you tested in OP), but sharing is a bit cumbersome. It could be useful in a scenario, where you work locally and then at the end of the day copy stuff over to the NAS for sharing or backup.

Yes, I thought so. I'm using SSD on my workstation as a scratch drive. Now I just need to get the proper speeds out of FreeNAS.

 

[ysnk]

Update: I just tested a few different configurations - 8 drives in stripe, and two 4-drive raidz2 striped
Stripe of 8 hdd gave me no no difference in speed, very close.
Two raidz2 - slower of course, but not much - read speed in fact is only about 10% lower. Writes are about 25% slower.

 

[L]

This tells me more that the writes are getting cached into memory and not running out to disk right away. The numbers on the cifs client are incremented the minute the data gets into cache on the freenas, not when they get to disk

 

[ysnk]

This tells me more that the writes are getting cached into memory and not running out to disk right away. The numbers on the cifs client are incremented the minute the data gets into cache on the freenas, not when they get to disk

That makes sense, but isn't it applies to the write performance? The write speed is fine, it's the read speed from freenas to workstation seems slow - is there a way to fix it?

 

[L]

so striping has no effect on read either? more vdev should give better perf.

Typically the way to improve write performance is to try and cache your working set.