Controllers, Performance and ZFS help appreciated

[jim82]

Hi all,

First post on the forums, and a VERY new FreeNAS user. I'm loving it so far, but I have a couple of questions for the more seasoned FreeNAS users :)

1) With 14 x 2TB drives(WD Green EARS), how should I configure ZFS? As one large RAIDZ2 or another setup?

8 drives is connected internally on the motherboard and the last 6 drives will be connected to a Dell SAS Sata HBA 6i controller on the PCI-E bus, once they arrive.

My primary usage will be media streaming over LAN, but as I rip all my Blurays as images on my primary workstation, there will also be a great deal of writing to the NAS, so write speed should also be fairly good.

2) I tried making a RAIDZ2 array with the first 8 drives I got yesterday(6 more coming), ALL drives are connected to the onboard SATA controller. and with a DD write/read test I got these figures:

Code:

nas01# dd if=/dev/zero of=/mnt/VOL-16TB-Z2/ddtestfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 46.331798 secs (452637734 bytes/sec)
nas01# dd of=/dev/zero if=/mnt/VOL-16TB-Z2/ddtestfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 37.618446 secs (557479699 bytes/sec)

IMHO a very nice performance!

BUT, today I recieved my Dell SAS controller(bought very cheap), tried hooking a single 320GB SATA2 drive on it and created the disk as ZFS, just as that one single drive, making a volume for it self. There after I ran the DD test on that single volume.
The figures I got was horrifying :( I don't have the exact numbers now, but read was around 90MB/s and write was 55MB/s.

Will these numbers get much better when attaching 6 drives and configuring RAIDZ2 or do I need to look elsewhere for the bottleneck?

3) The Dell SAS Sata 6i controller I have flashed with LSI Logic firmware as Dell just copied a LSI HBA(LSI SAS 1068E chipset). The controller is running fine without problems, I tried the DD test both before and after I flashed the firmware on it, same speed. What I don't understand is that when running as a HBA, letting ZFS control the "raid" part. The controller really shouldn't be a bottleneck in anyway, right? I checked with "mptutil" and the drive are infact showing up a "physical", so it seems that it's working as intended(HBA).

4) My drives, being WD 2TB EARS, I guess they have the 4K "problem", it that something to be worried about? I don't see the "4K option" when creating volumes?

I know this is a wall of text, but I'm hoping someone can assist?

Thanks in advance
Jim

 

[sjieke]

The speeds you are getting for the single disk are very normal. These are the maximums a disk can perform. Putting more disks in Raidz2 will indeed improve the throughput. You could estimate read speeds using the following rule:
Say you have a Raidz2 of 6 disks. That is 4 data disks and 2 disks for redundancy. When reading the system can read from the 4 disks simultaniously. So the maximum read you will get is 4 times the max read of single disks. In reality this will be a little lower because of some overhead. For writing the same applies, except that the overhead will be bigger because the parity needs to be calculated and written to the 2 disks for redundancy.
If the number of disks keep increasing you will encounter other bottlenecks, for example the limits of the pci or pci-e bus etc.

 

[jim82]

Thanks for your reply :) I have now recieved my last 6 disk and are ready to configure ZFS with 14 drives, any idea of the best config?

Also working on getting a 2nd controller, so I'll run 4 drives on each SAS controller for improved throughput.

/Jim

 

[matthewowen01]

1. I'd do a pool consisting of 2 vdevs, each 7 disk raidz2's, 20 TB. but that's me.

2. 3. which pci-e slot did you plug it into. there are 2 diffrent modes your board can be configured in, 3 x PCIe 2.0 x16 (x16, x8, x8 or x16, x16, x1). do you have it in the 16 wide but 1x slot?

4. search the forums, the 4k issue has been addressed many times (and I'm not up to speed as my 4k disks are not in my FreeNAS box)

 

[jim82]

1. I'd do a pool consisting of 2 vdevs, each 7 disk raidz2's, 20 TB. but that's me.

2. 3. which pci-e slot did you plug it into. there are 2 diffrent modes your board can be configured in, 3 x PCIe 2.0 x16 (x16, x8, x8 or x16, x16, x1). do you have it in the 16 wide but 1x slot?

4. search the forums, the 4k issue has been addressed many times (and I'm not up to speed as my 4k disks are not in my FreeNAS box)

Thanks for your reply, I've set my motherboard to x8 <-> x8 in the BIOS for both the 2nd and 3rd slot, so that should be ok. First slot has my passive graphics card in.

Can you please explain how I set my disks in 2 vdevs, each consisting of 7 disk RAIDZ2?
Should I split up my disk over the 2 controllers so:

1st vdev = 7 drives on onboard SATA controller
2nd vdev = 7 drives on the Dell SAS Sata 6i

Or?

I'll search for the 4K issue..

/Jim

 

[freeflow]

There are 'sweet' spots for zfs. For zsf1 its (2+1), (4+1), (8+1). For ZFS2 (2+2), (4+2), (8+2). Try searching in this forum and elsewhere. There are no problems with other disk configurations. Try searching withing this forum or googling. You might find that 3 vdevs of 4+1 suit your needs. Yes that means buying another disk.

 

[jim82]

There are 'sweet' spots for zfs. For zsf1 its (2+1), (4+1), (8+1). For ZFS2 (2+2), (4+2), (8+2). Try searching in this forum and elsewhere. There are no problems with other disk configurations. Try searching withing this forum or googling. You might find that 3 vdevs of 4+1 suit your needs. Yes that means buying another disk.

Thanks for your reply,

I actually just installed another 2TB disk, so now I have a total of 15, with the option of adding another one for a total of 16.

Using 3 x 4+1 vdevs would be like running 3 x RAID5 arrays with 5 disks, correct? So if 1 of my 5 disks go, then the 2 other arrays will NOT be affected, right?

With 16 disks, I was thinking of making 2 arrays with RAIDZ2 (7+1), but I guess that's not in the "sweet spot" :(

Hmmm, very confusing what to do...

/Jim

 

[freeflow]

For an enterprise user the sweet spots may be important considerations. For single use home installation its probably not so critical.

 

[sjieke]

Using 3 x 4+1 vdevs would be like running 3 x RAID5 arrays with 5 disks, correct? So if 1 of my 5 disks go, then the 2 other arrays will NOT be affected, right?

You are indeed correct that it is the same as 3 x RAID5. But as far as I know, ZFS stripes the vdev's in 1 pool. So if you have a pool consisting of 3 vdevs in raidz, this is similar as RAID0 + RAID5. So if you lose 2 disks in 1 vdev the entire pool is lost.

With 16 disks, I was thinking of making 2 arrays with RAIDZ2 (7+1), but I guess that's not in the "sweet spot"

With 16 disks and 2 vdev's in RAIDZ2 you do hit the "sweet spot", 2 times 6+2
I would go with this setup (you would have something similar to RAID0 + RAID6). This allows 2 disk losses in each vdev.

 

[jim82]

You are indeed correct that it is the same as 3 x RAID5. But as far as I know, ZFS stripes the vdev's in 1 pool. So if you have a pool consisting of 3 vdevs in raidz, this is similar as RAID0 + RAID5. So if you lose 2 disks in 1 vdev the entire pool is lost.


With 16 disks and 2 vdev's in RAIDZ2 you do hit the "sweet spot", 2 times 6+2
I would go with this setup (you would have something similar to RAID0 + RAID6). This allows 2 disk losses in each vdev.

Would this be correct?

Code:

nas01# zpool status -v
  pool: JL-32TB
 state: ONLINE
 scrub: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        JL-32TB       ONLINE       0     0     0
          raidz2      ONLINE       0     0     0
            gpt/ada0  ONLINE       0     0     0
            gpt/ada1  ONLINE       0     0     0
            gpt/ada2  ONLINE       0     0     0
            gpt/ada3  ONLINE       0     0     0
            gpt/ada4  ONLINE       0     0     0
            gpt/ada5  ONLINE       0     0     0
            gpt/ada6  ONLINE       0     0     0
            gpt/ada7  ONLINE       0     0     0
          raidz2      ONLINE       0     0     0
            gpt/da0   ONLINE       0     0     0
            gpt/da1   ONLINE       0     0     0
            gpt/da2   ONLINE       0     0     0
            gpt/da3   ONLINE       0     0     0
            gpt/da4   ONLINE       0     0     0
            gpt/da5   ONLINE       0     0     0
            gpt/da6   ONLINE       0     0     0
            gpt/da7   ONLINE       0     0     0

errors: No known data errors

 

[matthewowen01]

that's what i was talking about. looks good. how does it perform?

 

[Brand]

Code:

nas01# zpool status -v
  pool: JL-32TB
 state: ONLINE
 scrub: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        JL-32TB       ONLINE       0     0     0
          raidz2      ONLINE       0     0     0
            gpt/ada0  ONLINE       0     0     0
            gpt/ada1  ONLINE       0     0     0
            gpt/ada2  ONLINE       0     0     0
            gpt/ada3  ONLINE       0     0     0
            gpt/ada4  ONLINE       0     0     0
            gpt/ada5  ONLINE       0     0     0
            gpt/ada6  ONLINE       0     0     0
            gpt/ada7  ONLINE       0     0     0
          raidz2      ONLINE       0     0     0
            gpt/da0   ONLINE       0     0     0
            gpt/da1   ONLINE       0     0     0
            gpt/da2   ONLINE       0     0     0
            gpt/da3   ONLINE       0     0     0
            gpt/da4   ONLINE       0     0     0
            gpt/da5   ONLINE       0     0     0
            gpt/da6   ONLINE       0     0     0
            gpt/da7   ONLINE       0     0     0

errors: No known data errors

I'm just curious why you labeled the volume JB-32TB when the volume will not have 32TB of storage that is usable. I realize 32TB is the sum of all of the drives but you will be loosing 8TB for parity and then what you will loose because 1KB equaling 1024B instead of 1000B.

 

[jim82]

that's what i was talking about. looks good. how does it perform?

Code:

nas01# dd if=/dev/zero of=/mnt/JL-32TB/ddtestfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 29.974016 secs (699656658 bytes/sec)
nas01# dd of=/dev/zero if=/mnt/JL-32TB/ddtestfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 29.043847 secs (722064130 bytes/sec)

 

[jim82]

I'm just curious why you labeled the volume JB-32TB when the volume will not have 32TB of storage that is usable. I realize 32TB is the sum of all of the drives but you will be loosing 8TB for parity and then what you will loose because 1KB equaling 1024B instead of 1000B.

Looks better than writing 23,5GB :P

 

[jim82]

Now testing with 4 x raidz1 vdevs in a zpool.

Code:

nas01# zpool status -v
  pool: z1-x4
 state: ONLINE
 scrub: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        z1-x4         ONLINE       0     0     0
          raidz1      ONLINE       0     0     0
            gpt/ada0  ONLINE       0     0     0
            gpt/ada1  ONLINE       0     0     0
            gpt/ada2  ONLINE       0     0     0
            gpt/ada3  ONLINE       0     0     0
          raidz1      ONLINE       0     0     0
            gpt/ada4  ONLINE       0     0     0
            gpt/ada5  ONLINE       0     0     0
            gpt/ada6  ONLINE       0     0     0
            gpt/ada7  ONLINE       0     0     0
          raidz1      ONLINE       0     0     0
            gpt/da0   ONLINE       0     0     0
            gpt/da1   ONLINE       0     0     0
            gpt/da2   ONLINE       0     0     0
            gpt/da3   ONLINE       0     0     0
          raidz1      ONLINE       0     0     0
            gpt/da4   ONLINE       0     0     0
            gpt/da5   ONLINE       0     0     0
            gpt/da6   ONLINE       0     0     0
            gpt/da7   ONLINE       0     0     0

errors: No known data errors

A bit faster, but I'm not sure which way to go

Code:

nas01# dd if=/dev/zero of=/mnt/z1-x4/ddtestfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 23.603073 secs (888508039 bytes/sec)
nas01# dd of=/dev/zero if=/mnt/z1-x4/ddtestfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 22.307733 secs (940100913 bytes/sec)
nas01#

 

[jim82]

Ooops forgot to enable compression and set atime off, now check this out

Still 4 x RAIDZ1 vdevs in the zpool.

Code:

nas01# dd if=/dev/zero of=/mnt/z1-x4/ddtestfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 8.091965 secs (2591647330 bytes/sec)
nas01# dd of=/dev/zero if=/mnt/z1-x4/ddtestfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 6.377146 secs (3288543179 bytes/sec)
nas01#

That cant be right? 2,4 GB/s write and 3,0 GB/s read?




EDIT: Now also with RAIDZ2 test


2 x RAIDZ2 vdevs in zpool:

Code:

nas01# dd if=/dev/zero of=/mnt/z2-x2/ddtestfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 8.372625 secs (2504772436 bytes/sec)
nas01# dd of=/dev/zero if=/mnt/z2-x2/ddtestfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 6.859870 secs (3057130839 bytes/sec)
nas01#

Approximately the same speeds now, what to choose?

EDIT2: From Win7 to FreeNAS I'm only reaching ~85MB/s, would that be Samba slowing me down?

 

[jim82]

I went with the RAIDZ2 option, as I feel thats safer. So now they're running in the same zpool and I'm transferring my data.

 

[Brand]

From Win7 to FreeNAS I'm only reaching ~85MB/s, would that be Samba slowing me down?

A gigabit network should be capable of delivering a theoretical maximum transfer of about 125 MB/s. The keyword there is theoretical. The speed you are getting is very reasonable.

 

[matthewowen01]

writing a bunch of compressed zero's, that's right for that. try real world data and you'll see a dramatic drop.

you went with the 2 raidz2, that's what I'd have done. 4 raidz's performs better, but you've only got 1 drive of redundancy to loose 16 drives worth of data, too scary for me. if you've got the money, I'd get a hot spare or 2 in the near future. if one of your drives tanks, it will immediately get swapped in.

and on the original performance, holy cow, to bad you've only got a gigabit nic.

 

[sjieke]

Very nice setup. And I also think the 2 raidz2 vdev's is the better choice. As for your 85MB/s maximum I think you are hitting your networks limits. Just as Brand set, gigabit network has a theoritical maximum of 125MB/s. But the quality of your kabels, switches, routers, etc. also has its influence.