Writing speed depends on install location

[Hitmoky]

Hey FreeNAS!

Yesterday I installed FreeNAS 11 on two thumb drives attached to a server that handles 12 sata disks behind a storage controller configured there as a raid0 array and in FreeNAS itself configured as an raidZ3.

Somehow the writing speed that it produces was around 40 mb/s, the same writing speed that the USB2 produces. After which I installed FreeNAS again on a mini-hdd, which produces a writing speed of around 100mb/s.

In short my question is: Why is FreeNAS it's writing speed changed as soon as the used medium on which FreeNAS is installed is changed.

 

[garm]

Don’t run ZFS on a hardware RAID array

 

[Hitmoky]

Don’t run ZFS on a hardware RAID array

What method would you recommend as to present the drives to FreeNAS?

 

[IceBoosteR]

In a nutshell: Just read a little bit around in the forum. Present the disk as they are, physical, not logical or virtual. Use flashed SAS-HBAs in IT mode to do this.

 

[Hitmoky]

In a nutshell: Just read a little bit around in the forum. Present the disk as they are, physical, not logical or virtual. Use flashed SAS-HBAs in IT mode to do this.

Ah I think I understand. I shall look into the subject some more, thanks for giving me a push in the right direction ^^

 

[IceBoosteR]

No problem mate, afterwards, if you have a straight overview of this, we can help you more on this.
To go with a starter, have a look in the resource section.
https://forums.freenas.org/index.php?resources/
And normally the installation media should not slower your disk speed, this is independant from it.

 

[kdragon75]

And normally the installation media should not slower your disk speed, this is independant from it.

If the system dataset is stored on the boot pool and they are testing with samba, some samba components would be read/written to the boot pool. I doubt it would have that effect but it's worth mentioning.

The system dataset stores debugging core files and Samba4 metadata such as the user/group cache and share level permissions.

FreeNAS Doc - System Dataset

 

[IceBoosteR]

If the system dataset is stored on the boot pool and they are testing with samba, some samba components would be read/written to the boot pool. I doubt it would have that effect but it's worth mentioning.
FreeNAS Doc - System Dataset

Ok yeah agree to that. Normally my thumb drives are havin writes/read at about 500k/s so nothing really heavy.
Good that you mentioned that.

 

[Stux]

In short my question is: Why is FreeNAS it's writing speed changed as soon as the used medium on which FreeNAS is installed is changed.

How are you measuring this?

 

[Hitmoky]

How are you measuring this?

Two ways:

• Through bash we run dd and write a file in the pool it's folder (locally on the FreeNAS server)
• Through a network share, copying a large file over the network (gigabit connection from a Windows computer to the FreeNAS server).

Pardon my late reply.

 

[Hitmoky]

If the system dataset is stored on the boot pool and they are testing with samba, some samba components would be read/written to the boot pool. I doubt it would have that effect but it's worth mentioning.
FreeNAS Doc - System Dataset

Although I believe (could be wrong) Samba is also responsible for the Windows share that we used for testing, we are also testing it locally on the FreeNAS itself.

 

[kdragon75]

can you provide the commands used for testing?

 

[Hitmoky]

can you provide the commands used for testing?

FreeNAS: dd if=/dev/zero of=/mnt/mainstorage bs=1024 count=1000000

Windows desktop: drag and drop 10gig file to a network share of /mnt/mainstorage.

Result: ~ 30-40 mb/s.

Edit
I have experimented with different block sizes, although most of them gave the same results, one in particular was ruling out that the speed was being limited by the thumb drives. Block size 4096 had a speed of around 200mb/s. This could indicate that perhaps FreeNAS is getting wrong values by the raid controller

 

[kdragon75]

When talking about block size do you mean 4096k or just 4096? Also when you say 200mb/s do you mean 200MB/s?

raid controller

Thats most of your issue. When you setup the "RAID0" drives did you disable write cache and read ahead?

Try your DD with a block size of at least 512k check you raid controller settings and report back. Please be sure to use correct units

 

[Hitmoky]

When talking about block size do you mean 4096k or just 4096? Also when you say 200mb/s do you mean 200MB/s?

Thats most of your issue. When you setup the "RAID0" drives did you disable write cache and read ahead?

Try your DD with a block size of at least 512k check you raid controller settings and report back. Please be sure to use correct units

The block size notation is always in bytes so I did not include it, pardon. The speed of dd is in megabytes.
All the drives are configured as raid0, and write cache and read ahead has been disabled so that FreeNAS won't get all confused.

I just checked with DD and a block size of 512kb, which gave a speed of ~ 1.5 / 2 GB/s. I guess we can conclude from that that the raid controller is making things difficult for FreeNAS.

 

[kdragon75]

I just checked with DD and a block size of 512kb, which gave a speed of ~ 1.5 / 2 GB/s. I guess we can conclude from that that the raid controller is making things difficult for FreeNAS.

Performance wise, this sounds normal. With DD each IO operation is set of syscalls to when writing 10GB at 1024 byte chunks takes a lot more work than working with 512 KB chunks. I (from here) looks like your storage is performing adequately.

Next we need to test the network performance. Take a look at using iperf to test network throughput.

Once we know the storage and network are adequate, we can look into why CIFS is so slow.

 

[kdragon75]

Also, we will need the exact specs of your FreeNAS system including model numbers for items like the CPU, Motherboard (unless Dell or similar OEM), HBA (in your case the RAID card), and drives.