Raid-Z2 speed Problems CIFS and FTP

[Archaeopteryx]

Hi,
first, let me thank you for your knowledge base (especially thanks to cyberjock) with helpful tips and tutorials to configure FreeNAS. At the moment everything runs (nearly;)) smoothly.

I built my system after various recommendations (certainly with ECC:D) and tested it now like a month with only a single HDD (1.5 TB):

My system:

Intel Xeon E3-1230 v3, 4x 3.30GHz

2x Kingston ValueRAM DIMM 8GB, DDR3-1333, CL9, ECC

Supermicro X10SLM+-F bulk

Fractal Design Define Mini

be quiet! Straight Power E9 400W


Everything works fine. Only the space is already full ;)

So my buying list is at the moment:
4 x HGST Deskstar NAS 4TB 7200rpm, SATA 6Gb/s (a 137,75€) (continuous operation ready)

My first intention was to use it as RAID-Z2 (RAID6), because I read an article that RAID-Z1 isn't sufficient for HDD > 3TB (long rebuild times etc)

But with Raid-Z2 I could only use 8TB from the 16 TB :/

To get more space, I want to split my Data. One big chunk is data1, which could easily restored from other sources(Like movie trailers etc.). So if data1 would go bankrupt, it wouldn't be a problem. The other chunk are backups, my personal picture library and personal video files which shouldn't go bad.

I would make two pools(?), the first is a RAID-0 or JBOD configuration, the second one would be a simple mirror. (To have a normal backup for the private stuff).

I would now have for data1 4TB+4TB=8 TB
and for the private data (4TB+4TB)/2 = 4 TB

accessible. So 4 TB more than the RAID-Z2 configuration.

(Additionally I have an extern 4TB drive as a backup for the private data.)

Now to my questions:

1) If I want to add after some time 2 6TB drives to the “pool” data1, would it be possible?

2) Could I eject one HDD (from data1) of the server and (without maybe one file at the border) access the data in an extern USB case? (Because Wikipedia shows that the data in JBOD is written sequentially)

2) Is the configuration above okay or would you want to warn me about it?

3) Is the mentioned HGST HDD okay? I would, as an alternative, choose the WD RED, but the HGST is a little bit faster.

Thank you very much,

Archaeopteryx

Ps: I hope you can understand my English text. I haven't written english for quite a long time. Feel free to correct me :)

 

[danb35]

When you place multiple disks into a pool, ZFS does not under any circumstances use a JBOD configuration. You can stripe them, in a way similar to RAID0. Thus, to your questions:

(1) Yes, you could add any number of drives at any time to your striped zpool. By doing so, you're simultaneously increasing its capacity and increasing your risk of total data loss on that pool--if any one disk fails, you'll lose everything in that pool.

(2) No--because your drives would be striped, not in a JBOD configuration, you would not be able to remove a disk and have the array continue to function. Additionally, you wouldn't be able to get any usable data from a single disk anyway.

(3) 7200 RPM drives are louder, hotter, and draw more power than 5400 RPM drives.

 

[L]

So let me take a crack at this. Most hardware recommendations you read about are for commercial production. If this is home use, then I wouldn't necessarily worry about long rebuild times. Having some redundancy is better than the majority of home systems, or lowend nas boxes. I would probably put all drives into a big raidz pool.

You can continually add disks to a pool, but you can't remove them.

I have heard very good things about the HGST drives, but haven't used them myself.

 

[Dennis K.]

You can't add additional drives to an existing vdev (raid-z for example) after creation. You can only add a new vdev and add it to the pool. ZFS will always stripe the data over all vdevs of the pool. But be aware: if you new vdev only consists of one device and you somehow loose that device, your whole pool is gone. Bottom line is, that you cannot add additional devices to your raid vdev.

See also here: http://en.wikipedia.org/wiki/ZFS#Limitations

 

[Archaeopteryx]

Thank you for you answers. Sadly (even if ZFS is a super system) I realise that you need to have everything from the beginning.
I think I'll go for a simple Raid-Z2 Pool.

@Linda Kateley Yes, this is private use, but I said to myself, that after all these failed disks by my friends, that i want a "secure" solution. (Even if I don't get 100% ;) )

I made an table with the prices from my country to find the best bang for the buck:

I wanted to spend about 500€. The 80€ (or 65€) from the Seagate (Green) to the Red is not worth it, or?
The other solutions habe 12 TB space in Raid6, but are a bit expensive for my needs.

 

[L]

If I am doing a raidz2, I am not as worried about the quality of individual drive. Since I can lose 2. The one thing that is sure about disk that they will be bigger and cheaper next year. One of the great parts of zfs is that if you need more, just add more. Buy them when you need them at the price they will be .. in the future

 

[D4nthr4x]

So let me take a crack at this. Most hardware recommendations you read about are for commercial production. If this is home use, then I wouldn't necessarily worry about long rebuild times. Having some redundancy is better than the majority of home systems, or lowend nas boxes. I would probably put all drives into a big raidz pool.

Oh so if your data isn't a bunch of word docs, .pst files with inane email messages, and excel files generated by a company it isn't important? I see, might as well just buy a synology then since I just have "home" data.

--advice below--

I would probably not buy 4x4tb and go with 6x3tb the price should be similar. If you want to add capacity to the array you should ideally stick to adding another set of drives (so if you have 4x4 then adding another 4x4 would be optimal) then you have 2 raidz2 pools that are in a striped configuration (similar to "raid 0" but not really the same). That initial 4x4 set of drives you add is called a vdev once you make a vdev you can't add more drives to it, but you can add more drives to the pool so if you had a 4x4 vdev and added 2x6 to the pool they would show up as a second vdev (and I'm assuming you would mirror/raid 1 them) which means you would have 8tb capacity form the original + 6tb from the new one to make your pool size = 14tb. BUT the new 2x6 vdev would only have 1 drive redundancy and if you lose your 2x6 vdev you also lose your the data on your 4x4 vdev since they are striped in the same pool. This is explained here: http://forums.freenas.org/index.php...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

As for what drives to get, I would get the cheapest Red or Green WD drive per TB, I believe 3TB drives are the cheapest per TB and you should be able to get like 6 for the price of 4 4tb and that will give you around 12TB useable if you shove all 6 into a raidz2. WD Green Drives are the exact same drives as the Reds EXCEPT you have to follow the guide in the forum to change a setting on them (posted by cyberjock) that adjusts the idle time head parking.

 

[L]

All I was saying is that for most home systems there is a period of time when people sleep, eat, work, live, where the resilver time might not be a bother. For me I run my business at home, but there is still a period of time where I sleep and can have a little longer resilver. I would prefer to have the extra space.

 

[Mr_N]

Thank you for you answers. Sadly (even if ZFS is a super system) I realise that you need to have everything from the beginning.
I think I'll go for a simple Raid-Z2 Pool.

You dont need everything from the beginning, but you do need to start with a RaidZ2 or Z3 vdev, so later you can add more similar vdev's to the pool if you want to maintain a level of redundancy. If you want simple mirrors or striping i wouldnt be using freenas ;)

 

[FlexibleToast]

My first intention was to use it as RAID-Z2 (RAID6), because I read an article that RAID-Z1 isn't sufficient for HDD > 3TB (long rebuild times etc)

But with Raid-Z2 I could only use 8TB from the 16 TB :/

To get more space, I want to split my Data. One big chunk is data1, which could easily restored from other sources(Like movie trailers etc.). So if data1 would go bankrupt, it wouldn't be a problem. The other chunk are backups, my personal picture library and personal video files which shouldn't go bad.

I would make two pools(?), the first is a RAID-0 or JBOD configuration, the second one would be a simple mirror. (To have a normal backup for the private stuff).

I would now have for data1 4TB+4TB=8 TB
and for the private data (4TB+4TB)/2 = 4 TB

For only a 4 disk system, I would think RAIDZ would be sufficient. Sure, it's not optimal and sure it can only tolerate a single disk failure, but your proposed work around can also only tolerate a single disk failure. How much important data do you really have that will be in "private data?" If it is really important data, you need three copies anyway, two local, one offsite. You have a 1.5TB drive now, could you use that to back up to? You could create a RAIDZ pool, then have your private data in its own dataset. Set a cronjob to rsync incremental backups to an external source. If you can use ZFS on both ends, you can use its built in replication. A simple suggestion would be to put your 1.5TB in an enclosure, connect it to your router (most have a USB these days), share the 1.5TB, and rsync your private dataset over to it. If it runs out of space, upgrade. Granted this is not offsite so it is still not ideal.

One of the great parts of zfs is that if you need more, just add more. Buy them when you need them at the price they will be .. in the future

In our emailings back and forth you said to let you know about differences between using zfs and btrfs. To start with, this is actually a limitation of zfs compared to btrfs. Sure zfs you can add a vdev, or replace all devices in a vdev for more storage, but that's it. Granted that is better than previous filesystems and is a huge step for filesystems, but btrfs takes it a step further. Btrfs doesn't see raids as device level, it sees them as block level (I think I'm using the correct terminology here). What that means is that it can do all kinds of neat things like: add/remove a disk from a raid while online, have different raid levels for data and meta data, rebalance the raid striping on the fly, even change raid levels on the fly. Really, the only reason I have a particular interest in FreeNAS and ZFS is because it is more mature and the use of jails. Btrfs still doesn't have full support for RAID5/6, only RAID0/1/10.

Oh so if your data isn't a bunch of word docs, .pst files with inane email messages, and excel files generated by a company it isn't important? I see, might as well just buy a synology then since I just have "home" data.

This a common misunderstanding. There is a difference between important data and critical data. A home user has very important data. Things you've created that can't be recreated; pictures/video of your child's first steps, etc... The difference is, if you don't have access to this data, you aren't losing business or money. So, you can suffer a long down time as long as your data is restored. RAID is not a backup and your FreeNAS is only one part of what should be your three copies of anything important. As a home user you can take advantage of the longer rebuild times and optimize your size per cost by using less redundant techniques. There is no harm in waiting for your offsite backup to restore. Business critical data on the other hand, can incur significant "harm" in the way of lost sales, customers, you name it. It makes more sense for them to spend less money maintaining a more redundant system than it would cost them in losses if their system went down and had to be restored from a backup. I believe that is what Linda was trying to say, in not so many words.

 

[D4nthr4x]

Damn capitalists that think business is the most important thing in the world. Businesses aren't people they don't have feelings, they have profits. I'd much rather deal with a business that lost data than people that lose data. The human reaction to losing personal data is far more troubling to deal with than some company that doesn't matter that much.

 

[FlexibleToast]

Damn capitalists that think business is the most important thing in the world. Businesses aren't people they don't have feelings, they have profits. I'd much rather deal with a business that lost data than people that lose data. The human reaction to losing personal data is far more troubling to deal with than some company that doesn't matter that much.

Yes, that we can agree on, trust me. I didn't mention data loss at all. I said a home user doesn't typically care about down time when recovering from a offsite backup. A business on the other hand well may have significant profit loss so monetarily it makes more sense to have more local redundancy. The difference between "important" and "critical" is the speed at which it needs to be recovered, not the fact that it needs to be recovered.

 

[Archaeopteryx]

Hi,

I'm back and bought 5 x WD Green 3 TB Drives. I already changed the idle timer to 200 seconds and made a RaidZ2 Configuration.
Everything runs smoothly, my only problem is at the moment the LAN Speed with this setup.

My client PC is a Windows 7, Q6600 system and it should be sufficient to saturate 1Gig.

FreeNAS is an encrypted RAIDZ2 system with Intel NIC and Intel Xeon E3-1230 v3, who should support AES-NI and should also saturate 1Gig Ethernet.

The problem is that Windows File transfer over CIFS and FTP, too, are at 25-40 Mbytes/s.
They are connected over an TP-Link switch (TLSG1008D), but I already tested a direct Connection with a CAT6 cable -> same speed.

After that, i tested iperf:

bin/iperf.exe -c 192.168.1.60 -P 1 -i 1 -p 5001 -f M -t 20
------------------------------------------------------------
Client connecting to 192.168.1.60, TCP port 5001
TCP window size: 0.01 MByte (default)
------------------------------------------------------------
[156] local 192.168.1.100 port 50745 connected with 192.168.1.60 port 5001
[ ID] Interval Transfer Bandwidth
[156] 0.0- 1.0 sec 19.1 MBytes 19.1 MBytes/sec
[156] 1.0- 2.0 sec 23.6 MBytes 23.6 MBytes/sec
[156] 2.0- 3.0 sec 24.2 MBytes 24.2 MBytes/sec
[156] 3.0- 4.0 sec 23.9 MBytes 23.9 MBytes/sec
[156] 4.0- 5.0 sec 25.0 MBytes 25.0 MBytes/sec
[156] 5.0- 6.0 sec 25.0 MBytes 25.0 MBytes/sec
[156] 6.0- 7.0 sec 19.6 MBytes 19.6 MBytes/sec
[156] 7.0- 8.0 sec 24.0 MBytes 24.0 MBytes/sec
[156] 8.0- 9.0 sec 24.6 MBytes 24.6 MBytes/sec
[156] 9.0-10.0 sec 23.3 MBytes 23.3 MBytes/sec
[156] 10.0-11.0 sec 24.4 MBytes 24.4 MBytes/sec
[156] 11.0-12.0 sec 19.5 MBytes 19.5 MBytes/sec
[156] 12.0-13.0 sec 24.2 MBytes 24.2 MBytes/sec
[156] 13.0-14.0 sec 23.8 MBytes 23.8 MBytes/sec
[156] 14.0-15.0 sec 24.2 MBytes 24.2 MBytes/sec
[156] 15.0-16.0 sec 25.2 MBytes 25.2 MBytes/sec
[156] 16.0-17.0 sec 23.4 MBytes 23.4 MBytes/sec
[156] 17.0-18.0 sec 17.0 MBytes 17.0 MBytes/sec
[156] 18.0-19.0 sec 25.1 MBytes 25.1 MBytes/sec
[156] 19.0-20.0 sec 25.1 MBytes 25.1 MBytes/sec

but the intern transfer rates look fine (to me):

Any ideas how to solve it?

 

[SweetAndLow]

Does your windows 7 box have gigabit? your processor is 7 years old iI think. If the computer is that old also it probably is only 100mbit.

When testing local write/ speed make sure you have turned off compression otherwise your test is not accurate.

Also why did you encrypt your drives? Makesure you have the passphrase saved as well as the recovery key.

 

[Dennis K.]

Your internal test is flawed, as you have compression enabled. Disable compression (e.g. on a new separate dataset) and repeat the dd test. Anyway, that won't be the cause of your low transfer speeds, as iperf doesn't touch your pool at all. Have you tried IPerf the other way round with your windows box as server? Just to check the transfer speed from the FreeNas server to the windows box. Also, what brand/model is your NIC on your windows box?

 

[Archaeopteryx]

Hi,
i checked my Windows PC, it has 1Gig enabled.

I took my Windows Notebook, which is a little bit newer (i5 420M CPU), and got the following result:

bin/iperf.exe -c 192.168.1.60 -P 1 -i 1 -p 5001 -f M -t 10
------------------------------------------------------------
Client connecting to 192.168.1.60, TCP port 5001
TCP window size: 0.01 MByte (default)
------------------------------------------------------------
[156] local 192.168.1.171 port 49802 connected with 192.168.1.60 port 5001
[ ID] Interval Transfer Bandwidth
[156] 0.0- 1.0 sec 33.8 MBytes 33.8 MBytes/sec
[156] 1.0- 2.0 sec 31.5 MBytes 31.5 MBytes/sec
[156] 2.0- 3.0 sec 32.0 MBytes 32.0 MBytes/sec
[156] 3.0- 4.0 sec 31.2 MBytes 31.2 MBytes/sec
[156] 4.0- 5.0 sec 33.0 MBytes 33.0 MBytes/sec
[156] 5.0- 6.0 sec 33.4 MBytes 33.4 MBytes/sec
[156] 6.0- 7.0 sec 33.0 MBytes 33.0 MBytes/sec
[156] 7.0- 8.0 sec 32.8 MBytes 32.8 MBytes/sec
[156] 8.0- 9.0 sec 32.9 MBytes 32.9 MBytes/sec
[156] 9.0-10.0 sec 33.9 MBytes 33.9 MBytes/sec

Doesnt't look great :/
@Dennis
The speed with compression was probably too good to be true ;)
This is the speed test with compression off:

I only found Tutorials to make a speed check with FreeNAS as server. Which commands do I need for the other way round?

Thanks :)

 

[Dennis K.]

The speed with compression was probably too good to be true ;)
This is the speed test with compression off: [..]

These match the expected speeds for your build. So far so good.

I only found Tutorials to make a speed check with FreeNAS as server. Which commands do I need for the other way round?

Just swap the commands and the target IP address. The most basic are "bin/iperf.exe -s" on your windows box and "iperf -c <IP of your windows box>" on the freenas server.

 

[Archaeopteryx]

Here is the test with FreeNAS(192.168.1.60) as Client and WinPC(192.168.1.100) as server:

Edit:
I tested the speedtest with my WinLaptop(192.168.1.171) as server and FreeNAS as Client, too:

 

[cyberjock]

You should be getting over 900Mbits/sec with iperf tests. The fact you aren't is your indicator you have a networking problem.

 

[Archaeopteryx]

I would like get a possible network solved to get around 900 MBits/s

I drew the map of my network get a possible answer what I could do to solve this network issue.
Do you need any more information?

(The Laptop can be connected through W-Lan and Lan)

Besides I made a tailing of /var/log/messages and I have some message about winbindd, but I can't pin it down for my problem:

[root@freenas ~]# tail /var/log/messages
Sep 1 19:08:39 freenas notifier: Performing sanity check on Samba configuration: OK
Sep 1 19:08:39 freenas notifier: Starting nmbd.
Sep 1 19:08:39 freenas notifier: Starting smbd.
Sep 1 19:08:39 freenas nmbd[8055]: [2014/09/01 19:08:39.230225, 0] ../lib/util/become_daemon.c:136(daemon_ready)
Sep 1 19:08:39 freenas notifier: Starting winbindd.
Sep 1 19:08:39 freenas smbd[8059]: [2014/09/01 19:08:39.264630, 0] ../lib/util/become_daemon.c:136(daemon_ready)
Sep 1 19:08:39 freenas winbindd[8063]: [2014/09/01 19:08:39.287113, 0] ../source3/winbindd/winbindd_cache.c:3196(initialize_winbindd_cache)
Sep 1 19:08:39 freenas winbindd[8063]: initialize_winbindd_cache: clearing cache and re-creating with version number 2
Sep 1 19:08:39 freenas winbindd[8063]: [2014/09/01 19:08:39.288998, 0] ../lib/util/become_daemon.c:136(daemon_ready)
Sep 1 19:12:15 freenas winbindd[8683]: STATUS=daemon 'winbindd' finished starting up and ready to serve connectionsidmap range not specified for domain FREENAS