New Build HP ProLiant N40L Ultra Micro Tower Server

[andyl]

If I come up with anything I'll let people know, but in the mean time simple observations about single disks working better are meaningful too. There's a clue somewhere.

Thanks for the response. Not trying to diss people's hard work here, but I'm not looking to spend too long fiddling with my NAS :)

 

[ProtoSD]

I agree, it's interesting to a point, but I'd rather just not have to worry about it. It's possible that when ZFS v28 is integrated in June (May was announced, but...), anyway, that might help things but I'm sure it'll introduce others.... My advice about v28 is AVOID DEDUPE...

 

[andyl]

I agree, it's interesting to a point, but I'd rather just not have to worry about it. It's possible that when ZFS v28 is integrated in June (May was announced, but...), anyway, that might help things but I'm sure it'll introduce others.... My advice about v28 is AVOID DEDUPE...

Righto - I'll do a bit of reading and see if I can figure that one out :)

Would you think that the upgrade to v28 will be easy and safe? I'm really a bit of a newbie when it comes to ZFS....

 

[andyl]

Just for reference, I setup a couple of NFS exports to my backup disks.

Copying from the backup disk which is a 2Tb NTFS formatted USB external desktop HDD shared over NFS from a Core2 Duo desktop with 4Gb of RAM to my media box's local home dir runs very smoothly at ~30Mbps.

So an overall write speed of ~70Mpbs to the FreeNAS starts not to look too shabby. The problem is that when the throughput stalls, I would assume all access to the NAS stops. So any media access occuring will stop.

 

[ProtoSD]

Righto - I'll do a bit of reading and see if I can figure that one out :)

Would you think that the upgrade to v28 will be easy and safe? I'm really a bit of a newbie when it comes to ZFS....

I think it should go pretty smooth. There are always exceptions, but if people do a scrub and make sure there are no other problems before upgrading I think it should pretty straight forward. The thing with Dedupe is it needs LOTS of RAM, there is no playing around to see if you can get by with 2GB. The other problem is if you enable dedupe and later decide to disable it, any files that have been deduped will still suck up and use the extra memory and cpu. The only way to get around it is to copy everything to a non-deduped volume and then move it back. Anyway, I don't want this thread to get off topic...

 

[Hexland]

Now, this is interesting.

I finally received my IBM M1015 SAS controller, so I installed it and flashed the on-board BIOS with the LSI IT firmware.

Using an identical setup from my previous experiments -- I created the following.

HP Microserver N40L
8GB RAM
FreeNAS-8.2.0-BETA3-x64
4 x 2TB drives (SAS Port 0)
2 x 2TB drives (SAS Port 1)
1 x 60Gb SSD (SAS Port 1)
No drives attached to the N40L main board

6 x 2TB drives, formatted 4K (GNOP) -- RAID-Z2

Copying a 15Gb file to an iSCSI device volume resulted in this...

Copying the same file across a CIFS share resulted in this

So... removing the drives from the onboard controller and connecting them to the M1015 controller appears to have removed the 'breathing' network stalling problem that I was seeing before.

Write speed to the RAIDZ2 array also appears to be faster

Code:

ZFS, All drives formatted 4096K (gnop)
ZFS-RAID-2, 6x2TB (gnop) -- New BIOS, IBM M1015 Controller
----------------------------------------------------------------------------------
dd if=/dev/zero of=test.dat bs=2048k count=50k
107374182400 bytes transferred in 528.229085 secs (203272000 bytes/sec) -- 193.86Mb/s

compared to the onboard write speed

Code:

ZFS, All drives formatted 4096K (gnop)
ZFS-RAID-2, 6x2TB (gnop) -- New BIOS with SB settings
----------------------------------------------------------------------------------
dd if=/dev/zero of=test.dat bs=2048k count=50k
107374182400 bytes transferred in 628.539118 secs (170831344 bytes/sec) -- 162.92Mb/s

 

[andyl]

Code:

ZFS, All drives formatted 4096K (gnop)
ZFS-RAID-2, 6x2TB (gnop) -- New BIOS, IBM M1015 Controller
----------------------------------------------------------------------------------
dd if=/dev/zero of=test.dat bs=2048k count=50k
107374182400 bytes transferred in 528.229085 secs (203272000 bytes/sec) -- 193.86Mb/s

From these speeds it looks like you are getting about twice the write bandwidth that I am.
I'm seeing about a sustained transfer of 100Mbps being passed to the Microserver by the switch when copying to the NAS.
Even your 163Mbps before the RAID controller is about half as fast again as my sustained writes.

Any ideas why?

 

[Hexland]

I'm not sure... I know that I got a bit of a speed bump by going from the hacked Russian BIOS I was using (from back in November) to the latest one I got from bios-mods.com.

You're definitely running with 4K formatting? I think only one of my drives in the vdev was the older advanced format type (emulating 512K sectors) and it was enough to bring the whole vdev to a crawl

Code:

ZFS, All drives formatted 512K
ZFS-Raid-2, 5x2TB
-----------------------------------------------------------
dd if=/dev/zero of=ZFSTank01/test.dat bs=2048k count=50k
107374182400 bytes transferred in 1354.490325 secs (79272757 bytes/sec) -- 75.600Mb/s

The other thing to note is that I'm not using a switch between the Windows Server machine and the FreeNAS machine -- they're connected directly via an ethernet crossover cable, and both NICs have MTU 9000 set.

I'm still not happy with the performance though -- so I'm going to play around with it a bit more when I get time... I'm sure it should be faster.

 

[andyl]

I'm not sure... I know that I got a bit of a speed bump by going from the hacked Russian BIOS I was using (from back in November) to the latest one I got from bios-mods.com.

I want to keep mine as generic as possible. This is meant to be a super reliable storage solution, I want to be able to swap these disks into another box if this one blows up.

You're definitely running with 4K formatting? I think only one of my drives in the vdev was the older advanced format type (emulating 512K sectors) and it was enough to bring the whole vdev to a crawl

I think so. How do I check? I presume there's a command that'll show me, but I can't find it.

The other thing to note is that I'm not using a switch between the Windows Server machine and the FreeNAS machine -- they're connected directly via an ethernet crossover cable, and both NICs have MTU 9000 set.

I hate jumbo frames. Non-generic again....
I'll do some testing this afternoon and see if MTUs of 9000 make any difference. I think the bottleneck is in the disk write, since when I copy from the NAS to a local disk over NFS, I get a smooth transfer at ~800Mbps.

 

[Hexland]

I want to keep mine as generic as possible. This is meant to be a super reliable storage solution, I want to be able to swap these disks into another box if this one blows up.

The BIOS change doesn't change anything to do with the disk formats or anything like that... all it does is enable certain options which the motherboard/chipset supports (which are hidden by HP by default). If you installed the BIOS, the disks will still be usable if you move them into another FreeNAS based unit.

If you have a drive connected to the onboard Optical Drive SATA connector without the BIOS mod installed, then you are almost certainly running at a reduced speed.

I think so. How do I check? I presume there's a command that'll show me, but I can't find it.

SSH into your box, or use the shell web interface (or the shell option on the TTY head) and type

Code:

zdb -C | grep ashift

If the answer is 9 -- you have 512K sector alignment. If it is 12, you've got 4096K alignment

 

[andyl]

Looks like 4k....

Code:

[root@freenas] ~# zdb -C | grep ashift
                ashift=12
[root@freenas] ~#

 

[andyl]

Well, I bit the bullet and having verified the backups I killed my old NAS. I needed to reuse the disks in the FreeNAS.

I took the three Seagate 1.5Tb and single Hitachi 2Tb out of the old Drobo and installed them in the Microserver and then put a fresh install of FreeNAS-8.0.4-RELEASE-p1-x64 (11059) on the USB key.

I setup a single ZFS RAID-Z volume, which came out at about 4Tb.

I'm now copying the data back over rsync and NFS. Copies to the NAS are running at ~300Mbps - which is a great improvement on the disks I had in there before.

Pretty pleased, actually.

 

[hpnas]

I have 3 Western Digital Caviar Green WD20EADS 2TB drives and 8 GB of RAM in my N40L. I am running FreeNAS 8.0.3 from a USB drive embedded. I just moved off of OpenMediaVault to take advantage of ZFS; however, I've noticed that when it comes to smaller files <5 MB FreeNAS is slow. If the file is large >500 GB I can easily get speeds of 880 Mbps vs 200 Mbps of smaller files. I've read through a lot of ZFS tweaking and noticed some are configurations for 32 bit or 4 GB of RAM and some don't state the configuration of their system. In other words, I believe not all ZFS tweaks are a one size fits all so I haven't tweaked mine until I find a configuration more like mind. Anyhow, I realize small files are going to lag somewhat; however, I just wanted to see if anyone else here noticed major differences in speed with smaller files vs large ones? Any good ZFS tweaks for my configuration on the HP N40L?

 

[ProtoSD]

I've answered this in so many places I don't remember which settings I posted where, but I usually try to take into consideration the different memory configurations. This post below has values that should be correct for your system. I know I posted some other settings a few pages back, but I can look at those while I'm editing this message. They should be very similar.

http://forums.freenas.org/showthread.php?5338-Browsing-directories-slow

These settings should make a significant difference with your setup.

 

[slth]

I came across this thread whilst looking for more information about performance figures with the N40L and ZFS.

From what I read, it seems that there is no solution for the transfer stalls (yet)?

Has anyone tested if this problem still occurs with a JBOD configuration, e.g. reading / writing to a CIFS share on a zpool of a single 2TB disk?

 

[maynardo]

I am also interested in finding the solution for this problem, however I did not buy the server yet. I want to set up a RAID-Z1 with 3 disks on N40L, but if the problems persist I should think of some other solution (some other hardware) for my NAS server.

 

[dvpatel]

I have all the hardware. A N40L, 4GB ECC RAM, 2GB thumb drive 4 x 3TB seagates. I have actually installed NexentaStor on the OEM disk but have yet to break open my 3TB disks to set up my first ZFS array. Sad to read all this post purchase. Any thoughts on if the ZFS breathing is a issue on Nexenta as well? I can use either OS but decided to check out FreeNas as I still have my old H/W RAID 5 freeNAS 0.69 I believe alive and kicking. Any solutions to these slowdowns?

 

[dvpatel]

Ah well. I suppose NexentaStor wins due to lack of support for the nice folks here. I had already installed Nexenta which was waiting the drives. This weekend the project will be completed with Nexenta on the stock 250GB N40L HDD.