Writes incredibly slow over CIFS/AFP. Reads are fine.

[cyclerider]

I've been having issues transferring files over the network to my FreeNas server from a mac while using AFP or SMB/CIFS. Reading speeds are decent, but writes are absolutely unusable. Using a windows machine or FTP does not seem to be affected as far as I have been able to determine. I've been researching it for a couple days, but I wasn't able to find anything. I've included as much information as I could think of, but if you need more let me know.



How I want to use the server:

I used to have Ubuntu on this server, but a hard drive started going bad so I decided to order a new one to replace it and then try out FreeNas since I want to setup a ZFS server in the future once I have the funds for better hardware. When I had Ubuntu, I would use Carbon Copy Cloner on my mac (I believe it uses rsync) to backup the photos and videos I took for work on a nightly basis over AFP. However, now it takes forever to transfer files over AFP or SMB via OSX 10.8.5. We're talking a gig or less per hour. Using the same computer booted into Windows 7 seems to be fine over SMB. Originally I thought it might be not enough RAM because that seems to be one of the first solutions people suggested when I was researching the issue, so I upgraded to the motherboard max of 4GB but it was still slow. So I tried FTP on a whim, and found it going about the speeds I found reasonable. I ran a couple tests with different configurations that I have for you guys to look at.



Server Hardware:

MSI Wind Board D510 Intel Atom D510 BGA559 Intel NM10 Mini ITX Motherboard

4 GB of ram (FreeNas only reads 3GB)

1x3 TB WD RED WD30EFRX

2x1.5 TB Seagate Barracuda 7200.11 ST31500341AS 1.5TB 7200 RPM 32MB Cache SATA 3.0Gb

Seagates are connected to a PCI Sata Card:
Koutech IO-PSA420 32-bit PCI SATA II



The drives are each setup individually (no mirrors or raid or anything) using UFS.

32 bit freenas on a 32 GB PNY flashdrive that had never been used before



Network:
Gigabit Linksys router, all computers are hooked up via gigabit ethernet.

AFP Share Details:
/mnt/cyclerider
allow list: cyclerider
read-write access: pathum
Disc Discovery: checked
Disc Discover Mode: default
enable .AppleDouble: checked
AFP3 UNix Privs: checked

Default File Permission:
Owner - RWE
Group - RE
Other - RE

Default Directory Permission:
Owner - RWE
Group - R
Other - R

Drive Setup
Each hard drive is setup as a separate UFS Volume.

Owner (user): cyclerider
Owner (group): cyclerider
Mode:
Owner - RWE
Group - RWE
Other - RE
Type of ACL - Unix
Set permission recursively - unchecked



Test Configuration:

I connected my laptop to my Linksys gigabit router using an ethernet cable and turned off the wireless. The server is also hooked up to the same router. My laptop is Boot Camped with Mountain Lion and has Windows 7 installed. The laptop has 8GB ram, a samsung SSD and a 2.66 ghz core2duo. I created a folder that contained files that I felt would be somewhat representative of what a backup might consist of. My FTP client was FileZilla, and was set to allow 10 simultaneous file transmissions. I would keep an eye on the network speed reported by the computer and wrote down the max speed I saw, and levels that it stayed around while going through the transfer. The results are below:



9.31 GB folder as reported by OSX, 197 items, consisting of Canon Raw Files (CR2), XMP files, and MOV files.

Mac - Speeds from iStat Menus


SSD>FreeNAS FTP

Time: 4 minutes 50 seconds

Max Speed Seen: 60 MB/sec, stayed in the 50s to 60s. Slowed down as it was transferring less files.

FreeNAS FTP > SSD
Time: 5 minutes, 10 seconds

Max Speed Seen: 65 MB/sec when it had 10 files going at once, 80 MB/sec when it had 1 file, stayed in the 20-35 range for the most part

SSD > FreeNAS AFP
Time: Didn't bother letting it finish, but it estimated 45 minutes

Max Speed Seen: 12 MB/sec. Most of the time it would slow to nearly 0, then spike up to 4 or more MB/sec for about 10 seconds then go back down.

FreeNAS AFP > SSD
Time: 6:20

Max Speed Seen: 80 MB/s near the end. Usually stayed in the 20-25s. stayed in the 30s to 40s near the end.

FreeNAS SMB > SSD
Time: 6:15

Max Speed Seen: 50 MB/s near the end. Usually stayed in the 15-20s at first. then stayed in the 25s to 40s near the end.

SSD > FreeNas SMB
Time: Didn't bother letting it finish, but it estimated 6 hours

Max Speed Seen: 1.1 MB/sec


Windows:
Same folder, but Windows reports it as 8.67 GB

Speeds looked at through Resource Monitor


SSD > FreeNas FTP

4 minutes
Max Speed Seen: 
62.5, 44-55 usual

FTP > SSD
Time: 4:30

83MB./sec max, 15-25, 50-60 for a few seconds


Mapped Freenas as a Network Drive:

Freenas SMB > SSD

Time: 6:15

Speed: 37.5 MB/sec max, 15-19 MB/sec at first, then 25s to 30s, then 35s to 40s near end

SSD > FreeNas SMB

Time: 4:12
66MB/sec max. 37-50, 50-62, 18-30


 

[cyclerider]

a small update:
I realized I had misread some documentation on the motherboard and that it was 64bit compatible, so I made a fresh install of 64bit freenas. It now reads more of the RAM, but it seems I still have the same issues when it comes to writing over the network while using AFP and SMB/CIFS. I didn't redo any of the tests, but I can if that will help troubleshoot my issue.

 

[cyberjock]

So I deleted my post because you said you used UFS, I just didn't see it above.

I will say that your NIC is a Realtek. Not the most recommended by any stretch. PCI SATA isn't recommended either because its a parallel bus architecture and is limited to 133MB/sec for all devices combined.

Might try dropping in an Intel NIC just to see if things change. Other than that, fresh out of ideas.

 

[cyberjock]

I don't believe in UFS for FreeNAS because most people have zero FreeBSD experience, so their data is probably safer with their native OS and native file system. I'd have expected at least 20MB/sec from every one of those tests. Could be an issue with your client machine and not the server too.

 

[cyclerider]

So I deleted my post because you said you used UFS, I just didn't see it above.

I will say that your NIC is a Realtek. Not the most recommended by any stretch. PCI SATA isn't recommended either because its a parallel bus architecture and is limited to 133MB/sec for all devices combined.

Might try dropping in an Intel NIC just to see if things change. Other than that, fresh out of ideas.

I've been doing my testing with the WD Red, which is directly connected to the SATA port on the MB so the PCI card shouldn't be affecting the tests too much.

I think I might be able to snag an intel NIC from an old server at my friends place, but then I'd have to lose one of the 1.5 TB drives since I only have 2 SATA ports. I'll try it though.

I don't believe in UFS for FreeNAS because most people have zero FreeBSD experience, so their data is probably safer with their native OS and native file system. I'd have expected at least 20MB/sec from every one of those tests. Could be an issue with your client machine and not the server too.

I tested it with two different clients, and I had the issue with both. But Windows on the Mac seemed to be fine, so it could be an issue with OSX but I don't know what it could be. I'll rerun the tests later tonight if I can get a hold of that Intel card.

 

[cyclerider]

Would the NIC have that big of an influence depending on protocol? Because I get great speeds over FTP, just not with AFP/CIFS.

 

[cyberjock]

Would the NIC have that big of an influence depending on protocol? Because I get great speeds over FTP, just not with AFP/CIFS.

Normally I would say no. But Atoms are horribly underpowered which adds an extra layer of potential problems. Realteks do all of their processing in the driver, so a CPU that is underpowered can be crippled with a Realtek. Intels do everything on the NIC, so regardless of the situation you're always covered. It could be a mixture of how MacOSX is accessing things during certain file copy operations coupled with an underpowered CPU. Realteks are also known to do very very bizarre things that make no sense. They'll perform slowly for no reason, they may disconnect and reconnect randomly, jails sometimes have problems, etc. But as soon as they go Intel the problems magically goes away.

There's not much to go on, but as a rule we really shy away from Atoms here because people complain about their performance because they are underpowered and limited on RAM. And when we answer you with "no sh*t its underpowered" %randomuser% generally doesn't like that because they're now looking at either accepting the poor performance you get or buying all new hardware when they just bought this hardware.

 

[cyclerider]

wellllllllllll.... Looks like it was the Realtek. First AFP test with the Intel NIC completed the test in 2 minutes and 53 seconds. I'm going to test my Carbon Copy Cloner overnight to see if it works the way I want. I'll update in a couple hours.

Now I just gotta figure out a way to plug 3 harddrives into 2 SATA ports...

 

[cyclerider]

Sorry for taking so long to follow up. I wanted to let it run for a while to make sure it kept working the way I wanted to. Speeds seem to be good, and backups complete in a reasonable amount of time.

I'll probably just pick up a second 3 TB or a 4TB around Black Friday or Cyber Monday to replace the 1.5 TB drives and just keep those around until I find a use for them, perhaps mirrored in a future server.

One question I have is why is there roughly a .3 TB discrepancy between my 3TB external and the 3TB Red? My external reports that it has 1.48 TB free, while the mounted share only has 1.14 TB (1 TiB as reported by FreeNas web panel). Doing the conversion from TB to TiB, the used capacity seem to match up but not the amount free.

While I have you here, I also have a question about ZFS. I always hear 1 GB of RAM per 1 TB of space. But does that refer to physical space or volume space? if you had 2x3TB drives mirrored, would that be 3 GB of ram, or 6?

 

[mszpaku]

hi,
i got same problem, small transfer via network, i got two computers tested
1. intel atom d525mw with realtek rtl 8111e gigabit - mainboard Intel D525MW Intel Atom D525@ 1.8GHz (Dual Core)
2. optiplex desktop with e5200 intel procesor with broadcom 1gb

its strange beacause i instal freenas 911 on intel with one hdd 500GB samsung and two 320hdd in stripe and the transfer its about 50-60MB/s
oki nice - but i move pendrive with freenas (installed on atom motherborrd)
and hdds to optiplex only dont reinstal freenas !!!!

on optiplex i got 102-108 MB/s on both hdd's

even i try instal on that intel atom !!!!!! windows 7 x32 install netowrk drivers and it was same
100-105 MB/s transfer

i dont know why freenas doesnt faster than win 7 LOL

i think problem is at freenas drivers od realtek network card....

maybe that info could help someone

 

[Knowltey]

hi,
even i try instal on that intel atom !!!!!! windows 7 x32 install netowrk drivers and it was same
100-105 MB/s transfer

i dont know why freenas doesnt faster than win 7 LOL

i think problem is at freenas drivers od realtek network card....

maybe that info could help someone

FreeNAS and ZFS are more processor heavy during network transfers than Windows 7 is. There is a lot more going on when FreeNAS is transferring files than when doing it in Windows 7 because FreeNAS is doing various integrity checks while transferring the data. As a result a processor that can acheive a full gigabit transfer in Windows 7 may or may not actually be able to do it in FreeNAS.

Honestly I'm surprised you're able to get a full 100+ from the E5600.

Also yeah, Realtek cards apparently are known to not play nice with FreeNAS, I haven't personally had any issues with mine, but there are multiple posts around the forum saying to avoid using Realtek cards with FreeNAS for various reasons.

 

[mszpaku]

FreeNAS and ZFS are more processor heavy during network transfers than Windows 7 is. There is a lot more going on when FreeNAS is transferring files than when doing it in Windows 7 because FreeNAS is doing various integrity checks while transferring the data. As a result a processor that can acheive a full gigabit transfer in Windows 7 may or may not actually be able to do it in FreeNAS.

Honestly I'm surprised you're able to get a full 100+ from the E5600.

Also yeah, Realtek cards apparently are known to not play nice with FreeNAS, I haven't personally had any issues with mine, but there are multiple posts around the forum saying to avoid using Realtek cards with FreeNAS for various reasons.

i dont agree with that freenas is doing various integrity check more than windows , (sorry i dont speak well en)
i insert to optiplex 330 with e5200 (mistake not e5600 but it slower than e5600 - i think its not important here) and broadcom 1Gb network card ...
second network card to pci dlink with rtl8169sc 1Gb
on start was near 40 and rise to 65 Mb/s after 10 seconds

the transfer on than is on broadcom reading 110-112Mb/s and writing - 107-111 Mb/s it start with no rising speed just start 105 Mb

i checked that transfer on big 30GB file

oki i know understand difference between type of network card but its on the same freenas system
but my test shows that the freenas allow so high transfer...
i write here because i got more problem on the e5200 should be the same problem with slower transfer,
why on freenas was problem on intel atom but not on intel with e5200

oki i try check procesor load a betwen intel atom and e5200

 

[Knowltey]

i dont agree with that freenas is doing various integrity check more than windows , (sorry i dont speak well en)

There really isn't any agreement or disagreement to be had. The filesystems that FreeNAS uses DO more when working with data than the filesystems that Windows uses. Period. Given everything else being the same, transferring a file with FreeNAS is simply going to require more processor power than transferring a file in Windows. That's just how it is.

why on freenas was problem on intel atom but not on intel with e5200

Well on big reason that the Atom might not be performing as well as the E5200 is because an E5200 is almost twice as powerful as an Atom. Also Samba is single-threaded and Atom's have a low per-core speed, so their performance really drops when it comes to trying to handle single-threaded applications as Samba. The Atom simply doesn't meet the requirements of FreeNAS it isn't powerful enough.

 

[gpsguy]

As Knowltey has already explained to you, I believe your performance issue is due to the Atom CPU in combination with your Realcrap NIC. Do a search on the forum (or Google - site: forums.freenas.org atom d525 performance).

In addition, you haven't mentioned how much RAM you have in the machines, nor which version of FreeNAS you're using. That also can impact performance - especially, if they don't meet the minimum requirements.

 

[cheongi]

Suggest review TCP settings. I have 2 freenas boxes to play with (HP microserver N36L), one at work and one at home. The home one seems very speedy to write. The work one is slow, but I haven't bothered looking at why - haven't yet tweaked TCP settings on new windows servers. But I did some more research on it a couple of years ago trying to isolate the problem. See http://forums.freenas.org/threads/slow-share-mount-speeds-afp-smb.167/#post-11272