Writes faster than reads

Status
Not open for further replies.

Doubleslash

Cadet
Joined
Mar 12, 2013
Messages
5
Hi community,

I am running the famous ProLiant N36L Microserver with 8GB RAM and 4x2TB HDDs (2xSamsung F4 and 2xWD20EARX). I have a large volume created from a ZFS RAID0 stripe over 2 ZFS RAID1 mirrors, no RAIDZ. I have not create a dataset or something
Using xdd or dd I can get around 200 MB/s read and write from empty ARC caches (after fresh boot) from a 5G file using sequential access patterns. Also monitoring the network performance with iperf between my i7 iMac and the FreeNAS box is around 116 MB/s in each direction, using a 1GbE network connection and a switch. So far so good from the hardware side.

The problem arises when I am attempting use any kind of networking server to read and write large files sequentially. No matter if it is FTP, SMB or AFP... I always top out at around 60 MB/s read. Writes are very fast, with SMB being the fastest at constantly 100 MB/s write testing from a Windows 8 machine using an SSD. The iMac also has an SSD and achieves 100 MB/s write as well with some bumps inbetween.

Anyhow: using both Windows 8 (SMB) and Mac OS X 10.8 or generic protocols (FTP)... reads are limited to 60 MB/s. Any idea why that is? What could it slow down from raw 200 MB/s to 60 MB/s over the network?

I use FreeNAS 8.1.3-RC1 x64 and have the following sysctls set:

Variable Value Comment Enabled Available actions
kern.ipc.maxsockbuf 16777216
kern.ipc.nmbclusters 32768
kern.ipc.shmall 32768
kern.ipc.shmmax 67108864
kern.ipc.somaxconn 8192
net.inet.tcp.delayed_ack 0
net.inet.tcp.hostcache.expire 1
net.inet.tcp.inflight.debug 0
net.inet.tcp.inflight.enable 0
net.inet.tcp.path_mtu_discovery 1
net.inet.tcp.recvbuf_auto 1
net.inet.tcp.recvbuf_inc 524288
net.inet.tcp.recvbuf_max 16777216
net.inet.tcp.recvspace 262144
net.inet.tcp.rfc1323 1
net.inet.tcp.sack.enable 1
net.inet.tcp.sendbuf_auto 1
net.inet.tcp.sendbuf_inc 8192
net.inet.tcp.sendbuf_max 16777216
net.inet.tcp.sendspace 262144
net.inet.udp.maxdgram 65535
net.inet.udp.recvspace 65536
net.local.stream.recvspace 65536
net.local.stream.sendspace 65536
kern.maxfiles 65536
kern.maxfilesperproc 32768
kern.maxvnodes 400000
net.inet.icmp.icmplim 300
net.inet.icmp.icmplim_output 1
net.inet.tcp.mssdflt 1452
vfs.zfs.prefetch_disable 1

Any ideas?
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,973
Your results are normal for the system setup you have. If you want faster read transfers without a hardware change then you would need to ditch ZFS and go with UFS. Maybe someone else could offer a different solution.
 

Doubleslash

Cadet
Joined
Mar 12, 2013
Messages
5
Doesn't make a lot of sense to me. On dd/xdd test I can achieve multiples of this performance and also network performance is good. I cannot go to UFS since FreeBSD does not support RAID10 with UFS.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I think joeschmuck is right. The N36Ls seem to be great low power reliable machines, but they are not powerhouses. They work decently for most people's use and don't require alot of power(most people would be fine with 60MB/sec). I'm not sure why you need that kind of speed but you could try UFS and see if that helps. Aside from that you'd have to start trying to find where the bottleneck is(there's lots of places.. you'd have to narrow it down yourself) and see if the bottleneck can be fixed without buying a whole new machine.

If your dd tests are producing 200MB/sec read and write then I'd see what NIC you have on-board. An Intel NIC may help if the bottleneck is with the NIC itself or the CPU usage is going very high when copying data to/from the server.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,973
dd is purely an internal transfer copy between the hard drives, nothing more. Now think about adding the overhead processing to process the data you are reading and place it into the proper protocol format and then send it out over your NIC.

Your problem comes down to CPU horsepower to be honest. Do you really need over 60MB/sec or it is just nice to have? I can read from 80 to 90 MB/sec but my CPU is a few steps higher than yours. I can write data at up to 108 MB/sec but typical is 101 MB/sec.

One other issue on your system might be your NIC, a separate Intel NIC can achieve better performance from a machine with an integrated NIC on the MB as it uses a lot of CPU processing where an Intel NIC card offloads that work. I'm not saying you should go spend $20.00 on a new NIC, it's just a possibility for you to consider.

Have you looked at the NAS4Free project? Maybe that would help you out.

EDIT: CyberJock, you beat me to the posting, damn you man! :)
 

paleoN

Wizard
Joined
Apr 22, 2012
Messages
1,403

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Have you tried enabling prefetch? It's usually a win with mirrors.

I noticed that earlier but I forgot it's a double negative. So either it was deliberately disabled or the 8GB of RAM isn't available. Are you sure you are using the 64 bit version? Also, did you mean to type 8.3.1 RC?
 

Doubleslash

Cadet
Joined
Mar 12, 2013
Messages
5
Guys,

thanks for the comments. Here are some insights I gathered:

* CPU is not an issue... both smbd and afpd don't exceed 38-45% CPU utilization during 60 MB/s of transfer, load is equally distributed on both CPU cores
* 60 MB/s read might be good enough for most of us... since I recently upgraded to an SSD however I had to swap out the vast majority of my 'production' data (DSLR RAW photos and videos, virtual machine images, and other large binary files) to my homebrew NAS - say you are trying to open a a panoramic shoot of the Grand Canyon being stitched together from 7 raw images (20 MB each) with several adjustment layers and preview data - 60 MB/s vs. 100 MB/s is quite a difference in loading time if the complete TIFF is > 1GB
* I had tried both prefetch disabled/enabled, it didn't cause any difference - and as I said raw disk speed using dd/xdd is good

That said I went to a more radical approach and disabled ALL my sysctls. No immediate change but after a reboot (or probably restart of afpd, not sure) here we go: 95-100 MB/s read and write. Although sometimes, especially on writes after a fresh boot for a short periods of time transfers slow down to 25 MB/s, nearly come to a halt and go up again to 90 MB/s.
Now I am trying to figure out what parameter set caused this difference and how can make the connection more stable... but as you can see there is quite a bunch of parameter I've had enabled :)

Any suggestions what was the silver bullet?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
(This is not a personal attack.. but something I see many newbie to FreeNAS do)

As soon as I read this...
I had tried both prefetch disabled/enabled...

I thought "Oh great.. someone making sysctls that didn't do the proper homework." Then what do I read?

That said I went to a more radical approach and disabled ALL my sysctls. No immediate change but after a reboot (or probably restart of afpd, not sure) here we go: 95-100 MB/s read and write. Although sometimes, especially on writes after a fresh boot for a short periods of time transfers slow down to 25 MB/s, nearly come to a halt and go up again to 90 MB/s.
Now I am trying to figure out what parameter set caused this difference and how can make the connection more stable... but as you can see there is quite a bunch of parameter I've had enabled :)

Any suggestions what was the silver bullet?

This is exactly what I was about to recommend. 99.99% of the time, if you are putting in sysctls and asking the forum for support you do not have the required knowledge to fully appreciate the consequences of what you are changing. I always tell people that unless its broken don't try to fix it. Do you know how many tweaks my system has? Zero.

I have no recommendations except to not try to tweak the system. The default values work very well for more than 99% of situations. I have not and do not try to tweak the system unless something is actually wrong. The problem is that if something is wrong and you aren't devoted to spending as much time as it takes to find the issue(perhaps days..) then people start throwing in tweaks out of desperation to just "get it working". This is a big mistake and generally can only make things worse(see the shock in my face that you posted with even lower numbers than the defaults provide). Even worse, some tweaks can put your data at risk, and people have done lost data because they didn't understand what they were doing.

If you actually have something not working right, THEN lets look at what is broken and how to fix that individually. If I had noticed(and more than likely other senior posters had noticed) that those were custom added sysctls we would have told you to delete those as the first step. I'm not sure why you disabled prefetch, or even thought it might be a good idea... from the manual:

NOTE: by default, ZFS disables pre-fetching (caching) for systems containing less than 4 GB of usable RAM. Not using pre-fetching can really slow down performance. 4 GB of usable RAM is not the same thing as 4 GB of installed RAM as the operating system resides in RAM. This means that the practical pre-fetching threshold is 6 GB, or 8 GB of installed RAM. You can still use ZFS with less RAM, but performance will be effected.

So many people think they can tweak their system for more performance and then create threads when performance is in the toilet I added this to the noobie guide back in Oct 2012...

Manually optimizing the system settings should not normally be necessary as long as basic system requirements are met. In particular plenty of RAM for ZFS users.

So stop tweaking your system and just use it. See how it performs. If you want more speed then I'd try:

1. Buying an Intel NIC.
2. Consider buying more powerful hardware.
3. Switch to UFS.(of course, you said this isn't an option for you...)

Honestly though, if you are getting over 50MB/sec from that hardware I'd be happy you have what you have. ZFS can be a CPU hog, and is definitely a RAM hog. Neither of those are in abundance on your system.

My advice, if you are playing with 1GB TIFFs regularly as a career, is to consider going with 10Gb ethernet. It is very expensive($300 per network card) and will require you to buy new server hardware to support those speeds, but you you will be able to load up those TIFFs as fast as the zpool can read it.
 

Doubleslash

Cadet
Joined
Mar 12, 2013
Messages
5
@cyberjock

I agree with most parts of your post. Apart from that I am pretty much aware what these sysctl's are doing. I have done some Linux network tuning exercises in my professional career already and most of these settings actually do make sense. They work so well that most modern Linux distribution now ship with these as a standard. They also made sense on the system I was coming from: FreeNAS 0.7.2. I used them there to get decent stable network bandwidth around 90 MB/s where the factory settings only gave 45-50 MB/s. Since I thought FreeNAS 8 would be based on the same set of standard system settings I was importing them just right away after migration.
Apparently in the meantime the settings for the TCP/IP stack where changed to more reasonable defaults, so I don't need them anymore. Still, non of my values should cause the connection to cap out at 60 MB/s - it's apparently non-expected behavior somewhere.

ZFS caching/pre-fetching makes no sense on strictly sequential access patterns - that's why I wasn't seeing any difference either. Still I gave it a try to make sure I am without any side effects.

To your suggestions:

* I actually do have an Intel NIC, running the 82574L chipset. It's a nice piece of technology and serves me well.
* I don't think I need more powerful hardware - the components alone deliver expected performance when tested separately. Also nothing is showing up being contended during testing. If you look at top (1) you will see that neither smbd nor afpd nor any of the CPUs run at 100%.
* and yes, UFS makes no sense without RAID10 functionality.

Now that I have peak performance I have been investigating what the cause for the periodic drops in throughput were. Looking at the process list at the same time I see some python scripts kicking in based on cronjobs which for a period of 2-5 secs eat 20-25% CPU. I looked at them and came to the conclusion that autosnap.py and alert.py in /usr/local/www/freenasUI/tools/ cover ZFS periodic snapshots and the alert GUI element of the web-frontend. I am not sure if it is save to just disable the cronjob and have everything still working in the management interface.
Nevertheless I decided that this is the price to pay for something freeware developers built and for the cheap hardware I am using. Given that my box came 199€ with 50€ for the management card and 25€ for the Intel NIC EXPI9301CT I pretty much outperform every 4-Bay Synology consumer box which are >450€ :)
 

scurrier

Patron
Joined
Jan 2, 2014
Messages
297
For what it's worth I am seeing this strange slow read, fast write on my machine too with a 4 disk raidz2 and a 30 gig file. In the process of doing some tests and research to figure it out.
 

Kymmitsu

Cadet
Joined
Jan 24, 2014
Messages
6
I have a HP 54L Microserver. 8 Gig RAM. I too was gaving similar issues with very slow sometimes fast read / write speeds. Stalling during transfers 4 X 3T WD Greens (Striped Mirror ZFS), and a 2TB 204UI Samsung UFS.

I use the Mod bios to enable 3GBs SATA speeds and the CD port to run at that speed as well.

CyberJock is always saying Autotune is for 16G RAM so my sysctls and Tunables are empty. No entries are disabled.

But the biggest difference for me was to disable Powerd + not allow the HDD's to spin down.

So with Autotune and Powerd enable I was transfering 1.4TB in 20 -24 hrs. Rsync - 2TB UFS --> Smirror

Removing entries and turning off Powerd the 1.4TB transfers in ~ 4 - 4.5 hours. Rsync - 2TB UFS --> Smirror

Sorry no MB/s stuff but if I use FTP it transfers around ~ 80 -100MB/s accross the network the same 1.4TB in the very similar time.

BTW I am using 9.2 ...... I simply cant get CiFS working correctly for my simple setup on the newly released 9.2.1.X
 
Status
Not open for further replies.
Top