Couple Hardware questions

[louisk]

Okay, my first FreeNAS box is alive and I will post all the specs soon. I have a question. We talked about Pools of Vdevs and I thought I knew how that worked but now I am not sure. I have created 3 Vdevs of 4 disks each in Z2. Now I am not seeing how the next layer works that gives the clients a single target that is a stripe of the vdevs. Can you enlighten me?

Thanks,
Stephen

When you add vdevs to the pool, the striping is taken care of. ZFS stripes across all vdevs in a pool.

 

[Stephen J]

Thank you for the verbose and highly detailed answer louisk. I feel like I have stepped out of a cave into glorious light.
In all seriousness, I have really appreciated your input but I need a little more explanation. In the FreeNAS GUI, I used Create Volume to create vdevs and named them Vdevice_1,2 and 3.

Vdevice_1 /mnt/Vdevice_1
Vdevice_2 /mnt/Vdevice_2
Vdevice_3 /mnt/Vdevice_3

What is the next step? I used Create ZFS Volume and created one for each of the Vdevs and used the same name because I had heard that is what you did. That just creates this

Vdevice_1 /mnt/Vdevice_1
Vdevice_2 /mnt/Vdevice_2
Vdevice_3 /mnt/Vdevice_3
Vdevice_1 /mnt/Vdevice_1/Datastore
Vdevice_2 /mnt/Vdevice_2/Datastore
Vdevice_3 /mnt/Vdevice_3/Datastore

and that doesn't look right. I also tried Create ZFS Dataset and that didn't look right. Maybe I am just missing it in the manual but nothing is looking straightforward enough to look right. Or maybe I am just dumb, blind or both.

Thanks for putting up with me,
Stephen

 

[louisk]

Ahh, I don't know if it's not documented, or just not obviously.

If you want to have multiple vdevs in the same pool, you use the same pool name when you create them.
For example
create raidz vdev with ada0, ada1, ada2 -> pool raidz
create raidz vdev with ada3, ada4, ada5 -> pool raidz (you now have a single pool called raidz, mounted on /mnt/raidz, with 2 vdevs, each containing 3 spindles)
create raidz vdev with ada6, ada7, ada8 -> pool raidz (you have no added another raidz vdev to the existing pool called raidz...)

Make sense?

 

[Stephen J]

Perfectly. Thank you Louis. That is exactly what I needed.

Stephen

 

[Stephen J]

Hi Stephen,

If you have the time watch these:

https://blogs.oracle.com/video/entry/becoming_a_zfs_ninja

-Will

Thank you Will. Definitely looking forward to watching it.

Stephen

 

[Stephen J]

The Specs

I said I would post my hardware setup so here it is.

Case
Supermicro CSE-836TQ-R800B (3U 16 3.5" drives and 7 full height PCI slots) I kind of migrated to 3U from 2U. The primary deciding factor was the full height slots.
Supermicro X8DTH-6F-O (Dual Xeon 1366, Dual Chipsets, integrated LSI 2008 8port SAS, 7 x8 PCIE slots) At first I was going to use the board with 4 NICs but decided dual Chipsets would be good to have.
(1) Intel Xeon E5606 Westmere-EP 2.13Ghz CPU (cheap and I can always pop a 6 core monster in the other socket if I am severely under powered)
(1)LSI Internal 9211-8i 8-port 6Gb/s HBA
Kingston 6x8GB DDR3 1333 ECC Registered Server Ram
(16) Seagate Barracuda ST2000DM001 2TB 7200RPM HDDs (I talked my rep at Tigerdirect down to $149 each since I was buying so many, 2 weeks later regular price is $129 :( )
(1) Intel 2port Gigabit Server NIC (more bandwidth!)
(2) Patriot Swing 4GB USB flash drives ( I set these in a dual boot setup. A guy I know recommended I replace these with Transcend SLC USB DOMs, ?? what do you think?)
(2) Startech S25SLOTR 2.5" Removable Hard Drive Bay for PC Expansion Slot (3U gave me room for these to have Cache or ZIL SSDs)
(2) 3ware CBL-SFF8087OCF-05M SAS breakout cable (Nice cables but the cables that came with the MB have each individual broken out into 2 smaller ones. You can see from the picture. Not sure which one I prefer although the ones included with the MB have sideband connectors. Not that I know how do do anything with them.)

Everything went together without a hitch. Connecting all 16 breakout cables to the backplane was kind of fun but I decided to not get the 836E1 case with the expander because I didn't want a single point of failure that would be so hard to replace, nor did I want the bottleneck. They do have a case with dual expanders (836E2) that you can use with dual controllers for redundancy but would require SAS drives.
I set the unit up with 4 vdevs of 4 drives in RaidZ2. This gives me 14.2TB of space with decent speed. I split each vdev up between the two controllers. 2 disks of each vdev are on each controller so that if a controller goes south, the pool should stay online, dangerously vulnerable but online.

Input and ideas for improvement welcome.

Stephen

 

[louisk]

Hmm, I think for that kind of setup, I would've gone with mirrors. I think it would actually be faster (more striping).

Are you going to be doing any link aggregation?

 

[Stephen J]

I am going to try 3 disk mirrors and see what the performance looks like. I pulled all but one dimm of ram out of it. It seemed to me like having the 48gig in there was skewing my dd read write tests.

Stephen

 

[louisk]

I expect you'd find it more useful to do some more real-world tests. Simulating multiple client connections at the same time should be sufficient. It shouldn't take much to saturate a few gig-e links.