Excellent download / Extremely poor upload

[NasCon]

I have FreeNas 10.2 on a HP Microserver, with 2 pcs of 1TB discs on embedded minisas in mirror mode (through FreeNas, not through the embedded RAID). Also I have a 120GB (consumer) SSD configered as a cache. It is probably not my best choice to put the SSD on a regular SATA connection, instead of on the minisas, but I wanted to keep the 2 remaining bays free for a second mirror or a Raid5 with 4 discs.

I am getting nice speeds getting files FROM the NAS to alocal SSD drive in my PC (steady at 112 MBit), but I am getting TERRIBLE speeds of fluctuating between 0.5 and 7 MBit) getting files from my local harddrive TO the NAS (same share, same local SSD). I would have guessed to get decent speeds, since SATA SSD to SATA SSD can pull the Gigabit connection completely full, and FreeNas would find a way to get everything to the mirror afterwards on it's own? I am uncertain about how to troubleshoot this. Any suggestions?

 

[Basil Hendroff]

FreeNAS 10.2? ...Corral?

Are you getting the same up/down speeds using a different client for the test?

MicroServer model (& CPU if not GEN 7)?

You do mean MB/s?

 

[NasCon]

Excuse me, typo: 11.2 (just updated, I was having the same problem with the RC1 before).
It is a HP Proliant Microserver N54L with a Turion Processor @2.2KHz, 2 cores, I pimped memory to 8GB
Not the best configuration I bet, but I don't think that explains the enormous difference between read and write performance.

 

[Basil Hendroff]

Hmm.... I'm getting above 100 MB/s read and write between a Win 10 PC and an N54L. So, there's some issue writing back to your N54L.

Have you repeated the test from a different client? What is the network that sits between your client and the N54L?

 

[HoneyBadger]

I am getting nice speeds getting files FROM the NAS to alocal SSD drive in my PC (steady at 112 MBit), but I am getting TERRIBLE speeds of fluctuating between 0.5 and 7 MBit) getting files from my local harddrive TO the NAS (same share, same local SSD).

What's the network protocol you're using here? Is it NFS, by any chance?

I would have guessed to get decent speeds, since SATA SSD to SATA SSD can pull the Gigabit connection completely full, and FreeNas would find a way to get everything to the mirror afterwards on it's own? I am uncertain about how to troubleshoot this. Any suggestions?

If you added that SSD through the UI as "cache" then what you have is a "read cache" which obviously won't help your writes. But let's get an answer to that first question about the network protocol you're using.

 

[NasCon]

The Protocol I use is SMB.
Oh, I did not know the SSD cache was read-only. Can I get some caching done also for writing? I know about the risc of loosing power etc. I make it a habit to have reasonable backups, and for some data the performance outweighs the security. I have everything behind a UPS also.

The odd thing is...
I just took the server to my office for closer examination and networked it from there. I got upload speeds of over 90 MB/s. Close to Seymour Butt his experience. Now I placed it back in the basement, and the upload/write is slow as h0rs3 sh!t again... Download/Read is equally good at stable 112 MB/s in both cases.

So I guess I will have to troubleshoot my network instead of my FreeNas server ;-)

=== EDIT =============================
Yes, on a different port of the switch I was getting about 30 MB/s, and when I plugged it in another switch I was getting the full 90 MB/s again!
So, I seem to have a rotten switch, time for a new one...
Thanks for the quick replies!
====================================

 

[Basil Hendroff]

Glad you managed to sort this out.

 

[SweetAndLow]

You should remove the SSD cache it's not helping you in any way. It's actually hurting your performance.

 

[NasCon]

You should remove the SSD cache it's not helping you in any way. It's actually hurting your performance.

Can you please elaborate on that?

 

[Constantin]

Glad you got it sorted out. FWIW, consider specifying what type of cache the SSD is being used for - L2ARC or ZIL. The benefits of either cache depend on a number of factors (use case, protocol used, # & config of disks, etc.). My pool is somewhat larger (Z3 with 8 disks total) and I use two SSDs in mirror mode as a SLOG and a single SSD as a L2ARC.

However, my system has a lot more memory than yours and that's where I would start adding resources if you want to keep using that motherboard. Lastly, I would suggest the use of a Z2 pool if the data is important to you. You'll need more disks though.

Also, I'm guessing your HP is running at 2.2GHz not kHz.:) I doubt a Turion running at kilohertz clock speeds could transfer 100MB/s.

 

[gpsguy]

btw, I've got 16GB in my N54L. :)

I pimped memory to 8GB

 

[NasCon]

FWIW, consider specifying what type of cache the SSD is being used for - L2ARC or ZIL.

Can't remember FreeNAS asking me questions about this when I added the SSD. Honestly it is a bit above my knowledge too. I will delete the SSD from the pool and add it again, to see if some choices pop up.

However, my system has a lot more memory than yours and that's where I would start adding resources if you want to keep using that motherboard. Lastly, I would suggest the use of a Z2 pool if the data is important to you. You'll need more disks though.

I have 8GB, that should be good for home-lab/home-use purposes, no? If I can get stable speeds around 100MB/s I am more than happy. I will look into a Z2 pool. I have 2 more 1TB drives of exactly the same kind laying around. So I have 4 1TB drives now, what do you think is the best approach. I am looking for safety over capacity, but would like to be able to use about 2TB.

Also, I'm guessing your HP is running at 2.2GHz not kHz.:) I doubt a Turion running at kilohertz clock speeds could transfer 100MB/s.

You are quite right about that! :)

 

[gpsguy]

If you post the output of zpool status in code tags, we'd be able to see which cache you configured.

Can't remember FreeNAS asking me questions about this when I added the SSD. Honestly it is a bit above my knowledge too

I would follow the disk replacement process and upgrade the size of one of your mirrors.

Some users have squeezed 6x3.5" drives in their NxxL servers. To do so, you'd need to run one of the hacked BIOS's that allow AHCI on the SATA connections in the ODD bay. And, a bracket to mount 2x3.5" drives in the 5.25" bay. As I recall one brand might have used "butterfly wing" in the name/description - but I can't find it now.

Two way mirrors don't give you the redundancy of a RAIDz2 pool, but for "safety", I'd prefer keep the mirrors, use the stock bios and not overload the power supply.

If you outgrow this platform, I'd right-size it with a case that can hold at least a micro-atx sized motherboard and sufficient disk bays for your needs.

I am looking for safety over capacity, but would like to be able to use about 2TB

 

[Constantin]

Use all 4 drives then - a Z2 array would result in two drives data, 2 drives of parity, resulting in an available capacity of 2TB.

 

[NasCon]

If you post the output of zpool status in code tags, we'd be able to see which cache you configured.

Here it is:

Code:

root@fn1:~ # zpool status
pool: CachedMirror
state: ONLINE
scan: scrub repaired 0 in 0 days 00:19:36 with 0 errors on Sun Feb 24 09:19:42 2019
config:

NAME STATE READ WRITE CKSUM
CachedMirror ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/464c4448-d517-11e8-a639-c8cbb8cc3db8 ONLINE 0 0 0
gptid/4b92fb6d-d517-11e8-a639-c8cbb8cc3db8 ONLINE 0 0 0
cache
gptid/4d4b25be-d517-11e8-a639-c8cbb8cc3db8 ONLINE 0 0 0

errors: No known data errors

I can see no info about the kind of cache that is being used...

I do feel a bit worried about this:

Code:

  pool: freenas-boot
state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub repaired 0 in 0 days 00:09:10 with 3887 errors on Mon Mar  4 12:54:14 2019
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  DEGRADED     0     0     0
          da0p2     DEGRADED     0     0     0  too many errors

errors: 3891 data errors, use '-v' for a list
root@fn1:~ # root@fn1:~ # zpool status
          da0p2     DEGRADED     0     0     0  too many errors

errors: 3891 data errors, use '-v' for a list

I am getting no error-notifications about this, and also no mention of degradation of the boot volume (USB stick) in the dashboard or anywhere else. This "zpool status" command is the first that shows this...

 

[jgreco]

I pimped memory to 8GB

You "pimped memory" to the bare minimum requirement for FreeNAS?

I have 8GB, that should be good for home-lab/home-use purposes, no?

For a home fileserver, probably, but for a home lab, which suggests more demanding use, probably not. If you're using the term home lab to mean someplace where you do virtualization work, the low end is really 32GB and mirror pairs of HDD. If you're just using it for random file storage, you can get by with less, but the performance may not be "awesome".

The NxxL's can do 16GB of RAM. This never hurts.

You should remove the SSD cache it's not helping you in any way. It's actually hurting your performance.

Can you please elaborate on that?

If you're trying to put L2ARC on a system with 8GB, that's a train wreck. We normally don't suggest L2ARC until you have 64GB of RAM or more, though each case is its own unique thing. Even with 16GB, though, you will find that the ARC has trouble gathering sufficient knowledge of where the hot spots are, so it is just pushing mostly random crap out to L2ARC. You want L2ARC to be populated with useful data, so your ARC needs to be a significant fraction of your working set size.

 

[NasCon]

You "pimped memory" to the bare minimum requirement for FreeNAS? :)

Is it? I remember having ran FreeNAS with 1GB in the past. But I'll admit that was long time ago, and pre-ZFS. But the "pimping" part was more targeted to the default availability of 1GB for these micro servers.

Maybe I should leave the idea of using ZFS, it seems to be a third layer on a second layer of disc abstraction, and to my feel does pose more questions and limitations than solutions for a home-use NAS...

A'll leave out the SSD and see what is does to the performance.

 

[Constantin]

I wouldn’t leave it. Bit rot is real. ZFS is one of the few file systems that detects it.