BUILD Hardware selection - 1st FreeNAS build

[Bidule0hm]

AFAIK a vdev will have the same IOPS as the IOPS of one disk of this vdev regardless of the number (<-- if we keep it reasonable of course) of disks in the vdev.

 

[Robert Trevellyan]

AFAIK a vdev will have the same IOPS as the IOPS of one disk of this vdev regardless of the number (<-- if we keep it reasonable of course) of disks in the vdev.

According to the article:

For best performance on random IOPS, use a small number of disks in each RAID-Z group. E.g, 3-wide RAIDZ1, 6-wide RAIDZ2, or 9-wide RAIDZ3 (all of which use ⅓ of total storage for parity, in the ideal case of using large blocks). This is because RAID-Z spreads each logical block across all the devices (similar to RAID-3, in contrast with RAID-4/5/6). For even better performance, consider using mirroring.

My interpretation of the last sentence is, "mirroring can deliver the best IOPS because it has the fewest disks per vdev."

Later he says:

For performance on random IOPS, each RAID-Z group has approximately the performance of a single disk in the group. To double your write IOPS, you would need to halve the number of disks in the RAID-Z group. To double your read IOPS, you would need to halve the number of “data” disks in the RAID-Z group (e.g. with RAIDZ-2, go from 12 to 7 disks). Note that streaming read performance is independent of RAIDZ configuration, because only the data is read. Streaming write performance is proportional to space efficiency.

EDITED a lot ...

 

[Bidule0hm]

Ah, now I understand where this comes from: to double the IOPS you need to double the number of vdevs, and to do that with the same total number of drives the only solution is to halve the number of drives per vdev.

For example: 2 vdev of 8 drives --> 4 vdevs of 4 drives ;)

But if you just do: 2 vdev of 8 drives --> 2 vdevs of 4 drives then you won't have more IOPS...

 

[Robert Trevellyan]

Sorry, I edited my reply again.

 

[DataKeeper]

So went to install a dozen drives today and had a D'OH moment!

I don't have any flathead screws to work in the supermicro caddies!

Anyone have a source for these that doesn't take a week or so to get?

 

[Darren Myers]

Bahahahaha! I almost did that too, this seller is AWESOME! For supermicro stuff. Here's what you need. http://pages.ebay.com/link/?nav=item.view&alt=web&id=321127593861

 

[DataKeeper]

@Darren Myers Thanks for that. My 4 year old was there helping so "Golly Gee Darnit DOH!" spewed out of my mouth which got him giggling uncontrollably for a few minutes while saying it back at me :D

So I also received my 2 64GB Sata DOMs today. Man they are SOOOO much smaller then I thought! Just tiny and there is TONs or room on the X10SRL-F board for these things. So small in fact that when I installed them there is a good 3/8" between them with the spacing of the yellow powered ports on this board. Thumbs up to Supermicro for rethinking their placements!

 

[pclausen]

Looking good! Wow, those Sata DOMs sure are tiny. I've been looking for the 16GB myself, but they are impossible to find right now for some reason and I don't know what's up with the 32GB costing twice that of the 64GB units. I might just end up getting the 64GB units in the end like you did.

Lol on the screws. Glad Darren got you a link. If for some reason you strike out, I can send you my spare bag. That would at least get your 1st 12 drives in with 4 screws each. :D Each tray/drive will actually take 6 screws, but all you really need are the 4 outside ones.

So are you still planning to start out with a single 12 drive VDev RAIDZ2? Or go with the recommended max of 8 data drives plus parity (for a max total of 9, 10 or 11 drives)?

 

[marbus90]

16 and 32GB DOMs have been phased out. 64GB ones are the only ones sold now.

 

[pclausen]

Ah, that would explain it. Thanks.

If I start out with a single USB stick, would it be a big pain to convert to DOMs later, or would I just need to copy the USB content to both DOMs?

 

[marbus90]

You can just add the DOMs to the freenas-boot pool as mirror disks and remove the old stick from said pool.

 

[Darren Myers]

@Darren Myers Thanks for that. My 4 year old was there helping so "Golly Gee Darnit DOH!" spewed out of my mouth which got him giggling uncontrollably for a few minutes while saying it back at me :D

So I also received my 2 64GB Sata DOMs today. Man they are SOOOO much smaller then I thought! Just tiny and there is TONs or room on the X10SRL-F board for these things. So small in fact that when I installed them there is a good 3/8" between them with the spacing of the yellow powered ports on this board. Thumbs up to Supermicro for rethinking their placements!

No problem there, trust me...i ran into that too before i got the actual hdds i was going to use i went to throw in some spares, and me not thinking grabbed my box of misc screws, started sorting through it and realized i had 8 screws to get me started, but why the hell did they make them so special.....until i started poking around some other forums and it seems supermicro squeezed every millimeter out of the case to fit the most drives (hence the tiny screws + recessed head) and i was went "those bastards....making me spend another 10$", but i also picked up a 2U supermicro case that takes 24 2.5 drives and i knew supermicro would do something similar to i went to that sellers page and typed in "supermicro 2.5" and sure enough there the screws were, special yet again.

You can just add the DOMs to the freenas-boot pool as mirror disks and remove the old stick from said pool.

This...this right here is important to know!

 

[pclausen]

Yeah, that's great it is that easy. So would the steps be as follows:

1. Power down server
2. Add DOM(s)
3. Power back up and go into the BIOS to make sure it can see the DOM(s)
4. Go somewhere in the FreeNAS GUI to add the DOM(s) to the boot pool
5. Wait for the data to be mirrored to the DOM(s) Can this process be monitored somehow?
6. Power down server
7. Remove USB stick
8. Power back up and go into the BIOS and make the DOM(s) boot priority 1 (and 2)
9. Done

 

[Ericloewe]

Do note that GRUB needs to be manually installed after a device is replaced, at least for now.

 

[marbus90]

5. zpool status freenas-boot
6. remove USB stick from boot pool instead of power down
8. do that during the next bootup (update or something)

 

[DataKeeper]

Ok so I got hit with the Java 8 BS and couldn't use the ipmi interface. REALLY!!!!!! I didn't have Java installed before this and it updated to 8 the other night. Anyways, cleared Firefox, added the systems IPs, restarted both the servers and my desktop. I can login and start the Java app for the new X10 system but continue to get the security alert from the older X8 system. :( Anyone else run into this? Can you download an older version?

@pclausen .. Yeah I just decided to order the 64GB DOMs to get them. It's funny since I was on the phone with Supermicro last week and when talking about the DOMs they never stated the 16/32s were being phased out. If I had known that I'd have ordered the 64s sooner. Marbus90 must have an inside source :D

I haven't made up my mind on the vdevs yet. No, I wouldn't do 18 drives as cj did nor would I do 12 in a business environment. I've read the "rules" on the limits and while 6/8/10 are the typical selections 12 isn't likely to be much worse then 10. The system is in my home, I have spare drives on hand for quick swaps and will have backups of the important data. The bulk of the data on the server will be media which can simply be ripped from source again if it were to be lost. That said, I've also given thought to a few other set ups and separating the important data to separate vdevs from the bulk of the data. Still undecided at this time. Might setup a 12 drive vdev, automate an 80% population, pull a drive and rebuild/re-sliver to see for myself. I'm in no rush so I have time to play. :D

The Screws... So it seems there is a shortage on these currently. I'll order some from one of the online vendors but I'm told it's still 2-3 weeks before they arrive on the east coast from California for some reason. I stopped in 3 places who sell used racks looking for a 36" and no one had any screws, or 36" used enclosures. :( Took my name and number saying if any came in they'd call me. Usually get some every few months so hopefully I get a call soon. They had 42s but I don't need anything that large and I can't pick it up myself whereas the 36 still gives me plenty of room and I can put it in the back of the SUV.

 

[marbus90]

What? an american SUV which can't take on a 42U rack? aren't you afraid of being run over? ;) and it isn't 36" but 36U.

 

[DataKeeper]

Lol yes 36U.. Been in the workshop with a tape measure all day. :D I drive a 2003 Ford Explorer Sport and with the rear seats down 6' is the absolute length with the front seat all the way forward. I know well as I drove 1200 miles like this with a 6' curio cabinet in the back from Canada.

I do need a new vehicle soon and with a kid and two mid sized dogs I'm really thinking about getting something larger :)

 

[Darren Myers]

@DataKeeper not to be too personal, but where on the east coast are you? I'm in PA, and i have an extra baggy of 3.5 supermicro screws that i'd be more then happy to send you

 

[pclausen]

I too have a baggy I'd be happy to contribute to the cause. Send a PM regarding those. Its the baggie pictured above. :) I'm in VA.