Nas unbelievably slow... needs 10 Minutes to write 100MB

[afmobiwx]

Hello Forum.

I built a Freenas for Backups in an old Fujitsu server to try it. New to Freenas.

	FreeNAS-11.1-RELEASE
Platform	Intel(R) Xeon(R) CPU E3110 @ 3.00GHz
Memory	4056MB

I removed the very restricted SAS Raid controller and used 4 sata Disks, each 4 Terabyte. one volume RaidZ over this 4 disks.
System is on a 160GB disk. system dataset on the raidZ volume.
dedup off, writing to a dataset without lz4 comp.
i have the backup server connected with nfs to the freenas, rsync on the backupserver to the nfs mount.
The write speed is unbelievably slow, even getting slower after a while. at the moment it need 10 minutes to write 100MB.
i tried network speed with iperf3, but it shows gigabit ok.
When i had the Nas as Linux server with SAS Raid5 it was flying, this is very disappointing.
can somebody put me on a track where to look? i am really at the end of my wisdom.

 

[toadman]

I suppose it could be a great many things. Can you be more specific about the use case?

Are you writing locally to the pool, or over the network? If from the network, what client and what network protocol? (I assume it's not rsync over NFS as I am guessing that workflow goes FreeNAS > backup which would be a read on FreeNAS vs a write.)

Then, what are you writing? large files, small files?

RaidZ is going to be slower than other topologies.

You've got very little RAM in the system. While it obviously works, most folks here will point to the minimum recommendation of 8GB.

 

[afmobiwx]

I am writing over the network, and it is a write on freenas. the freenas is my backup, i test it in that role before i would use it somewhere else.

topology: 3 xen servers with a NFS Disk repository. the NFS Repo ( NAS ) is a dell T420 with Ubuntu 18.04 Server, 4x 7200rpm Sas disks in Raid10 with together 8TB. All VMs are Linux. VMs are all on this repo.

The Freenas is the Repository for the Backups. I set up the Xen servers to use the Freenas as a Repo for daily XVA exports, that works. Not very fast but it works, 30GB in 20 Minutes. weekly export 240GB in 3 hours, that are only the productive small servers and Xen is having a lot of work to do as well, it actually is slower but not much slower than exporting from xenserver local disks to my other storage.

The big file server is a VM, as is the Backup server. Until now i had a bacula on this backup VM writing into File Volumes to VDIs and a USB Disk on the xen server the backup vm was residing on, but that is too unflexible.

so what the Backup VM does now is mounting the Freenas share via NFS, and writing to the freenas. I have a share on the freenas that works sort of ok for the daily bacula backups, but i also wanted to make a 1:1 copy of user data, svn, Git, jenkins and docker repo from file and build server to the Freenas, to use the snapshot feature.

This data is rsynced daily from the various VMs to the Backup VM VDI ( pulled by the backup server ), works fast. now PUSHING the data to a Freenas share via NFS starts slow, slows down and then times out. it´s of course quite a lot of large, small, smaller and smallest files.

As the freenas is only in test i used an old fujitsu server i found in the heap of old hardware, and 4GB is all there is.
i dont expect it to be very fast, it works sort of ok for that in most cases, only this case is impossible... and that would be the case freenas would be interesting.

 

[Patrick M. Hausen]

Turn off sync writes for the NFS volume and see if that improves things by a great margin. Then in case you value your data get a fast and reliable SLOG device.

Search the forum for "NFS VMware" and you will find lots of threads with the probable reasons for your performance issues.

 

[afmobiwx]

the nfs backup shares for the xen run ok, what does not run is user data rsync via nfs.

nfs options for the xen shares: rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountvers=3,mountport=898,mountproto=tcp,local_lock=none

using fairly similar options for the connections from the backup server, it is so slow it is unusable. at the moment i am trying to delete data on the backup share via rm -r * from the backup server, it delets about 1GB in 10 minutes.

 

[Patrick M. Hausen]

Again: does it dramatically improve when you turn off sync writes? Just for a test ...

 

[jgreco]

The forum rules, conveniently linked at the top of every page in red, have some guidelines for posting a problem summary that is more likely to get a higher quality answer.

As it is, we have no idea about your hardware, and while sync writes could be the problem, so could things like SMR HDD's, or using a RAID controller instead of an HBA, or a poorly supported network card, etc.

 

[afmobiwx]

Again: does it dramatically improve when you turn off sync writes? Just for a test ...

i did not turn on sync in NFS options when mounting, if that is what you mean, async should have been the default as far as i knew.
I also do not have the VMs on the Freenas, just the ISO repo and the Backup, so i would not use sync at all.

But i have been setting the mount options now async to noauto,nolock,async,rw,noacl,relatime,vers=3,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountvers=3,local_lock=none,mountproto=tcp

The first 50GB write quite fast ( 10 Minutes ) ( file sizes are about the same, around 500MB to 4 GB files ), then the rsync descends into another directory with a git repo and transfer rates slow down drastically. ~100MB in 5 Minutes now.
so it is sure a problem with small files.

 

[afmobiwx]

The forum rules, conveniently linked at the top of every page in red, have some guidelines for posting a problem summary that is more likely to get a higher quality answer.

As it is, we have no idea about your hardware, and while sync writes could be the problem, so could things like SMR HDD's, or using a RAID controller instead of an HBA, or a poorly supported network card, etc.

i did write that i threw out the RAID controller, so i am using the HBA now. Using also the Board network gigabit card ( Fujitsu TX150 S6 Server ). also wrote the model of the processor and how much RAM.
iperf3, as i wrote, from and to the Freenas shows no problem and gives gigabit rates, so it is not the network or network card. also latency is perfect.

i do, in fact, use SMR disks, i just had a look.
But the problem is not that ALL writes are slow, its only the very small files where the speed drops almost to a halt.

 

[jgreco]

RAIDZ is optimized towards large sequential files and doesn't do as well on small files.

A lot of the information in the block storage sticky is also at least somewhat relevant to storing small files on HDD.

https://www.ixsystems.com/community/threads/the-path-to-success-for-block-storage.81165/

Doing small writes on SMR drives is *catastrophically* slow with a copy-on-write filesystem such as ZFS. There may be no fixing it if you happen to have SMR drives.

 

[afmobiwx]

RAIDZ is optimized towards large sequential files and doesn't do as well on small files.

A lot of the information in the block storage sticky is also at least somewhat relevant to storing small files on HDD.

https://www.ixsystems.com/community/threads/the-path-to-success-for-block-storage.81165/

Doing small writes on SMR drives is *catastrophically* slow with a copy-on-write filesystem such as ZFS. There may be no fixing it if you happen to have SMR drives.

Thank you. That sorts a lot of what i read in the last two days, i did not have the SMR problem on my radar... So if I would TAR and use the Freenas share like a Tape storage it would work with speed, but then snapshotting would be pointless.

i will change to Raid10 on the FreeNAS now, see what that gives me. Maybe it´s enough performance gain for this to work out at least fast enough for the daily backups.

 

[woods]

Haven't read everything, but I had some very slow write speeds in comparison with my reads last week. It was a drive with bad sectors causing it. I didn't replace it (because I'm a noob) because freenas didn't take the disk offline I figured the slow writes were caused by something else.

But after a lot of troubleshooting I bit the bullet and replaced the drive. And what do you know, it's now writing at 10Gbps.

So perhaps in your case, if even a single drive smells a bit fishy, you might try replacing it, even if it's just a test.

EDIT:

also: I have pool of two striped vdevs of 6x drives in Z2

My writes/reads are both saturating my 10G network from 1MB unit sizes and up. But as you can see in the pic, at 512 it starts to drop.


I've been reading and yes, small files do slow down the system. I think one way of bypassing this is to compress your folders into a single item? Not sure though.

 

[jgreco]

Thank you. That sorts a lot of what i read in the last two days, i did not have the SMR problem on my radar... So if I would TAR and use the Freenas share like a Tape storage it would work with speed, but then snapshotting would be pointless.

i will change to Raid10 on the FreeNAS now, see what that gives me. Maybe it´s enough performance gain for this to work out at least fast enough for the daily backups.

Even if you're doing "sequential" writes, SMR is going to be slow. ZFS has a tendency to do little writes all over the place for metadata updates, and this really tears through SMR's ability to cope. SMR is really just not designed for any sort of CoW filesystem.

 

[afmobiwx]

well to be honest i did not pay attention to smr vs. pmr when buying the disks, as i kind of had in mind they are only cheaper because they are low RPM. Did lose track to the changes in architecture during the last couple of years, i fear.
It does not have to be very fast, though... just not THAT slow.
will try with raid10 first, see if it runs just fast enough, before getting different disks.