having trouble creating the array I want

[silvergoat]

I have:

4x WD Raptors I want to create a set of 2 mirrors with

and

5x WD Red 2TB drives that I want to create a set of 2 mirrors with, while having 1 available as a spare


The problem is that the GUI doesn't seem to allow me to do what I want to do, and the CLI only lets me create the first set of mirrors, afterwhich I don't know the correct commands to get what I want.

I have read about half of the manual, but there is not enough detail, especially without knowing the CLI commands. I am also willing to create the faux RAID 10 array with the Raptor drives and just create a Raidz1 array with the WD Red drives, but the most important factor is the speed so that I can stream movies to at least 2 users with the Reds (which turn out to be pretty slow and may be limited to one user), and I want the zfs file integrity to remain intact.

I'm open to suggestions. I also can build and rebuild the array as needed since I have not yet transcoded my DVD collection.

Edit: I'm also not sure I was able to force 4096k when I created the pool through the CLI; nor am I sure that I can create 2 zpools without issue.

 

[cyberjock]

Spares aren't "hot". They won't auto-takeover in the event that a disk fails. It won't be fixed until the ZFS daemon is implemented(FreeBSD 9). Don't ask when that will be.. nobody knows.. but alot of people want to see FreeNAS on FreeBSD 9. So, in short, I wouldn't worry about having a spare installed. Just do a SMART long test to make sure the disk isn't bad right now and put it on a shelf out of sight.

 

[silvergoat]

I know the spares don't work like that, but I really don't want to dig it out of the case I have all of this stuffed in (Antec 300)......which is why I don't mind just letting them go as raidz1, but the problem is that even with one user, there was a noticeable stutter in the video for the first few minutes. I don't know how it would react if there were two clients being served different information, but I think I would have better results with 2 mirrors in one group (raid 10?) than with raidz1. As of now, it looks like it will be impossible to stream full HD content, which I want to eventually be able to do......for this setup, I'd be happy with streaming standard def uncompressed to 2 clients at the same time. I just need to make sure I configure this correctly before I start storing data on the server.

 

[survive]

Hi silvergoat,

I think this will answer your question about how to add another vdev to your pool:

http://doc.freenas.org/index.php/Volumes#Using_Volume_Manager_After_a_Volume_Has_Been_Created

That said, I don't think converting your storage over to mirrors from raidz will solve your problem, though it's certainly worth trying to see if it does. I can't imagine a 5 disk raidz vdev on a Q6600 couldn't serve data up fast enough to feed a pair of standard definition media player clients.

When you say streaming what do you mean? Are the clients reading directly from a CIFS share (like XBMC would) or are you using something like minidlna to stream media over to an XBox, PS3 or something else? If you are doing the later I would certainly look at the transcoding service as well.....I guess you will know as soon as you try the "RAID-10ish" configuration.

If you verify that the transcoding process is what's really causing your problems (not the disk config) I would certainly encourage you to give a 5 disk raidz2 a whirl. Considering you are ready to burn up 3/5ths pf your disk space right now with between the mirrors & the spare you would certainly get more value out of your disk dollar with raidz2. If you have the space in the case and the cash in the wallet it might be worth getting a 6th 2Tb drive for your big pool now so you have your disks configured with one of the more ideal number of drives.

-Will

 

[cyberjock]

Q6600 are great with multi-threaded applications that don't need alot of power. But they are pretty lacking in the single-threaded world and total computational power. I always recommend any i3/5/7(although i3 should save some power if electricity usage is a concern). I have an i3-530 system and I have personally seen it stream 4 movies over a RAIDZ2 simultaneously without a hitch. It can saturate 2xGb LAN.

The Q6600 chipsets also are limited to 8GB of RAM total if memory serves me right. If you have 4GB or less that would kill your performance too.

 

[silvergoat]

Hi silvergoat,

I think this will answer your question about how to add another vdev to your pool:

http://doc.freenas.org/index.php/Volumes#Using_Volume_Manager_After_a_Volume_Has_Been_Created

That said, I don't think converting your storage over to mirrors from raidz will solve your problem, though it's certainly worth trying to see if it does. I can't imagine a 5 disk raidz vdev on a Q6600 couldn't serve data up fast enough to feed a pair of standard definition media player clients.

When you say streaming what do you mean? Are the clients reading directly from a CIFS share (like XBMC would) or are you using something like minidlna to stream media over to an XBox, PS3 or something else? If you are doing the later I would certainly look at the transcoding service as well.....I guess you will know as soon as you try the "RAID-10ish" configuration.

If you verify that the transcoding process is what's really causing your problems (not the disk config) I would certainly encourage you to give a 5 disk raidz2 a whirl. Considering you are ready to burn up 3/5ths pf your disk space right now with between the mirrors & the spare you would certainly get more value out of your disk dollar with raidz2. If you have the space in the case and the cash in the wallet it might be worth getting a 6th 2Tb drive for your big pool now so you have your disks configured with one of the more ideal number of drives.

-Will

I'm going to stick with the raidz1 for the WD Red drives, and use mirrors for the Raptors....it should give me a good comparison as to what kind of speeds are possible. I plan on using lzjb compression on both pools.

By streaming I just mean a file server from which my PC media player will read the files. There will be no video processing on the server. The next thing to play with will be the sharing and my options for serving those files to other PC.....CIFS, I've read can be slower, so I would like to be able to share to a Windows PC without, but I will have CIFS enabled for an xbox in another room. I did originally run this in raidz1, but my transfer speeds were half of what I was used to with my previous configuration, so I scrapped it.

I made a list of things that may have caused the slower transfer rates......obviously, switching from striped Raptors to raidz1 Reds might be the biggest difference. Even with gzip=6 and a slower CPU and less RAM, the striped Raptors were almost 2x as fast.

 

[silvergoat]

Q6600 are great with multi-threaded applications that don't need alot of power. But they are pretty lacking in the single-threaded world and total computational power. I always recommend any i3/5/7(although i3 should save some power if electricity usage is a concern). I have an i3-530 system and I have personally seen it stream 4 movies over a RAIDZ2 simultaneously without a hitch. It can saturate 2xGb LAN.

The Q6600 chipsets also are limited to 8GB of RAM total if memory serves me right. If you have 4GB or less that would kill your performance too.

The low RAM limit of my first board is why I switched to the Q6600. I doubled the RAM to 8GB, but it's not enough.....maybe the enabled prefetch and slow drives may play a role.

 

[silvergoat]

I've been searching and reading stuff all morning and here's what I've come up with:

Raptor mirror + Raptor mirror (in one group of disks) + WD Red x5 in raidz1

I don't know if they have to be in the same zpool or a different pool for each set. I tried to create two zpools but I got an error message that I will have to look at when I try again tonight.

For the raptor mirrors, the first commands I used were:

Code:

CLI: zpool create raptors mirror ada0 ada2

return: some error message about mounting

CLI: zpool add raptors mirror ada1 ada3

then I exported the zpool and auto imported from the GUI

However, I just came across this really good resource with a ton of commands that instructs the following:

Code:

zpool create testpool mirror /pool/disk1 /pool/disk2 mirror /pool/disk3 /pool/disk4

It pretty much does the same thing as I did the first time, but creates them all at once. Is there any difference?


I do want to force 4096 and enable lzjb encryption at the same time since I'm not sure I will be able to modify these through the GUI once the pool(s) are formed. I'm going to try one more time to create two different zpools through the CLI, and I will take down the error I get when I try to create the second pool.

 

[cyberjock]

Enabling compression is a major performance limiter. Also, if you enable compression and do a dd /dev/zero you'll get amazingly high(and impossible) speeds because zeros compress so well.

I wouldn't even consider compression with that CPU. And enabling compression on video streams is a waste of CPU cycles since video files don't compress at all. If you have compression enabled that would likely explain why you can't even get 2 video streams.

Keep in mind that just like dedup disabling(or enabling) compression only affects new files. Files already on the server will NOT be changed.

 

[silvergoat]

The server is blank as of now- I've only played with the creation and destruction of test zpools.

There was no compression enabled on the zpool that had slow transfer rates.

 

[cyberjock]

You should be able to create the zpool from the GUI. If you can't you are probably doing something wrong(or you're the only person on the planet that's found it). I have no idea if running commands from the CLI aligns the partitions to the 4k boundary and stuff, but I'd be leery that you can't get the GUI to work.

As for your performance, if you are already that slow then something is wrong somewhere. Could be too slow of hardware(Q6600 is 6 years old as of this quarter), a single failing disk can slow down the whole zpool, or you have a bottleneck somewhere. For instance, if your raid controller is PCI, you shouldn't expect anything above about 50MB/sec. You didn't say if your speeds were from dd or network speeds, but Intel NICs give the best performance.

 

[silvergoat]

All disks were tested with the manufacturers extended test and passed. Transfer speeds are slow through the network, measured from my desktop the same way as when I was getting 65MB/s speeds. It could be the hardware, but unless it's slower than the E3300 that I was using before, it should not be slower. Controller is the Intel SATA 2 Controller on the motherboard. My NIC card is the Intel Gigabit CT, same I used when I was getting the faster speed. The only changes have been the CPU, doubled the RAM, and the hard drives are Reds instead of Raptors, Raptors were striped and Reds are raidz1.

 

[silvergoat]

I just found one potential reason for slower than expected speeds. All of my disks are advanced format, however none of them will let me change sectors to 4096- even if I force it through the GUI.

I used this to try to change it to 4096

Code:

geom nop create -v -S 4096 disk

I used this to check status after I created a zpool in the GUI and in the CLI

Code:

zdb poolname | grep ashift

None of them returned ashift=12


Edit: I figured out how to force this in the CLI and then successfully importing the zpool, but I have the same problem as last time.

I have one zpool created through the CLI and running, but I can not create another zpool in the CLI- This is what I entered:

Code:

zpool create reds raidz ada5.nop ada6.nop ada7.nop ada8.nop ada9.nop cache ada4

The error code I get is

Code:

cannot create 'reds : no such pool or dataset

 

[silvergoat]

Here is my current configuration:

Code:

pool: raptors
state: ONLINE
scan: none requested
config:

raptors
-mirror-0
--ada0.nop
--ada2.nop
-mirror-1
--ada1.nop
--ada3.nop

What I want to create is:

Code:

raptors
-mirror-0
--ada0.nop
--ada2.nop
-mirror-1
--ada1.nop
--ada3.nop

reds
-raidz
--ada.nop9
--ada.nop8
--ada.nop7
--ada.nop6
--ada.nop5
---(cache drive) ada4

 

[silvergoat]

I created the second zpool through the GUI and this time the 4096k sectors remained.

Will they disappear when I reboot, or is the sector size set for good?

 

[silvergoat]

Testing right now and I'm satisfied- turns out that the 4096k sectors made a small difference.

Transfer speeds are now right around 60MB/s, where they were before- I'll just assume the write speeds are slower because of the raidz configuration on the Red drives, over 4 striped Raptor drives. I also added a spare 128GB SSD drive as my L2ARC for the zpool composed of the Red drives.....hopefully it will mitigate the slowness of the drives- if not, I don't mind since that drive has been sitting for months.

Edit: maybe I should add another L2ARC SSD for my Raptors, on which I plan to store my HD content.

 

[cyberjock]

Adding an L2ARC is only helpful for situations like database use. Plus, the file has to be read once, so unless you are doing lots of small reads of the same files over and over you'll see no benefit.

Plus, using an L2ARC will require more RAM. Feel free to spend the money, but there's a reason why I don't use one... they don't work like you think. Same story with the ZIL. They are not dumb caches and only cache certain types of read and writes.

 

[silvergoat]

Adding an L2ARC is only helpful for situations like database use. Plus, the file has to be read once, so unless you are doing lots of small reads of the same files over and over you'll see no benefit.

Plus, using an L2ARC will require more RAM. Feel free to spend the money, but there's a reason why I don't use one... they don't work like you think. Same story with the ZIL. They are not dumb caches and only cache certain types of read and writes.

Can I can remove it after the fact without affecting anything of the data/permissions/folders?