extending Storage with external RAID-CASE, ZFS

[niclas197]

Hey,
I'm running a litte bit short on storage here on my HP Microserver and I thought of using one of the many available 4-Bay RAID cases but I'm not sure what setup I have to use in order to use ZFS on it.

I think that it would be required to adress each disk individually (i guess that is what JBOD means). Is this possible with USB 3.0 and are there any (half size, half height) PCIe Controller that are supported with FreeNAS? Are there any recommendations on what Raid Case to use?

The HP Microserver also has an eSATA port but it does not support port multiplication and my understanding is that FreeNAS does not support it well anyways (or is there a PCIe Controller (half size, half height) that supports eSATA port multiplication)

Are there any alternatives?

 

[matthewowen01]

I think that it would be required to adress each disk individually

yes

i guess that is what JBOD means

no, jbod means it presents all the disks as one large block device, basically 'tapped' together end to end. so FreeNAS will see one large disk, not several small ones.

Is this possible with USB 3.0 and are there any (half size, half height) PCIe Controller that are supported with FreeNAS?

USB 3 is not supported. you may be able to find a half height PCIe card with some external connections, here are 2 options among many.
http://www.newegg.com/Product/Product.aspx?Item=N82E16816103095
http://www.newegg.com/Product/Product.aspx?Item=N82E16816115082

Are there any alternatives?

Not really, you got a very small footprint device and have no room for expansion. this is why i try and talk people away from the mini itx tiny foot print devices. they are elegant, but as soon as you need to expand, you're sunk.

 

[niclas197]

hmm thanks for now.

My idea was that I can access every disk individually with a supported port multiplier and a case like this: http://www.fantec.de/html/en/2/artId/__1401/gid/__1990901990352901990352490/article.html ?

 

[jgreco]

no, jbod means it presents all the disks as one large block device, basically 'tapped' together end to end. so FreeNAS will see one large disk, not several small ones.

That's incorrect. JBOD means "Just a Bunch of Disks". In the old days, this was typically implemented with a SCSI bus.

Now where this gets all screwed up is where manufacturers get stuck trying to figure out how to make a case that the Average User(tm) can leverage; your average user is expecting to take a single cable from his little drive tower and hook it up to a single port on his PeeCee; this led to Badness(tm) with manufacturers of USB arrays (and later, other arrays) who really didn't understand what "JBOD" meant and also had to deal with stupid implementation issues in Windows, etc., and so they decided to Conveniently Redefine(tm) the term.

So some devices that claim to offer "JBOD" do no such thing but instead offer concatenation, etc., as Matthew suggests. This doesn't mean that that's what JBOD means, it just means some moron called it JBOD. You can call a Toyota Prius a Hummer but it doesn't make it one. No matter how loudly you scream it.

A true JBOD array will allow for independent access to the component disks. Current SATA technology may implement this over eSATA with a port multiplier. USB and Firewire also both allow for JBOD. If you're an old-timer, your existing SCSI bus is probably JBOD too.

 

[niclas197]

That's incorrect. JBOD means "Just a Bunch of Disks". In the old days, this was typically implemented with a SCSI bus.

Now where this gets all screwed up is where manufacturers get stuck trying to figure out how to make a case that the Average User(tm) can leverage; your average user is expecting to take a single cable from his little drive tower and hook it up to a single port on his PeeCee; this led to Badness(tm) with manufacturers of USB arrays (and later, other arrays) who really didn't understand what "JBOD" meant and also had to deal with stupid implementation issues in Windows, etc., and so they decided to Conveniently Redefine(tm) the term.

So some devices that claim to offer "JBOD" do no such thing but instead offer concatenation, etc., as Matthew suggests. This doesn't mean that that's what JBOD means, it just means some moron called it JBOD. You can call a Toyota Prius a Hummer but it doesn't make it one. No matter how loudly you scream it.

A true JBOD array will allow for independent access to the component disks. Current SATA technology may implement this over eSATA with a port multiplier. USB and Firewire also both allow for JBOD. If you're an old-timer, your existing SCSI bus is probably JBOD too.

Thank you. I've seen some manufactures that also claim support for the "Raid" Mode "BIG" which looks a lot like what matthew described. My investigations lead me to conclude that I should avoid USB since it seems not to work very well with ZFS.

FreeBSD 8.2 is said to extend the support for port multiplication but I have yet to find an affordable one.

(BTW. what I really don't understand is: Is a port multiplier a feature of some sata controllers or is this a additional device? The information I can find is not really helpful :-( )

 

[jgreco]

Thank you. I've seen some manufactures that also claim support for the "Raid" Mode "BIG" which looks a lot like what matthew described. My investigations lead me to conclude that I should avoid USB since it seems not to work very well with ZFS.

FreeBSD 8.2 is said to extend the support for port multiplication but I have yet to find an affordable one.

(BTW. what I really don't understand is: Is a port multiplier a feature of some sata controllers or is this a additional device? The information I can find is not really helpful :-( )

The important part is to find out whether or not the gizmo you're pondering supports useful modes, of course, which is hard since your average FreeNAS hacker won't necessarily have a Fry's in the area, meaning that there's probably a lot of mail-order purchasing going on. Once the wrong thing is bought, things become harder.

A "port multiplier" is essentially an external board that acts a bit like an ethernet switch or something like that; it allows several drives to share a single SATA connection. If you know what SCSI LUN's are, kinda like that. Each drive is addressed separately, but all the data goes over the single SATA connection (and of course is limited to the SATA connection's speed). Now of course, your cheap SATA silicon might not support such complex concepts; some of the stuff out there seems lucky to be able to even talk to a single SATA drive successfully. Complex SATA RAID controllers might or might not support PM's. What you really need is silicon designed for the task.

A basic description of PM's is on Wikipedia.

http://en.wikipedia.org/wiki/Port_multiplier

But a more interesting read is available if you go looking for Backblaze Pods.

http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/

For about $7,000, you too could have a 135TB FreeNAS box.

 

[jgreco]

this is why i try and talk people away from the mini itx tiny foot print devices. they are elegant, but as soon as you need to expand, you're sunk.

I think the reality is complicated. There aren't really any great solutions that I've found for some applications. It's easy for us datacenter geeks to find massive storage enclosures. We had solutions for 24 drives in 4U a really long time ago, but they're big rack-mount enclosures. The power supplies are inefficient and they're meant for large server boards. I scaled that down and we have some really nice "low power" (~100 watt) 1U, Opteron 240EE, 4 hotswap bay SATA storage servers that are quite nice. We also have some 2U 12-bay servers. They all address various needs.

The thing is, though, to just serve files is mostly not a CPU-intensive task. We didn't have great options mid-2000's, but now there's a whole bunch of stuff. For the high-volume IOPS in a commercial virtualization environment, it's still not a problem to build a Big Storage Server. Great! But there's a whole 'nother branch of the storage world, and too-few good solutions.

There are the old-timers like me, who've been slowly generating increasing amounts of data for years, and we want to perpetuate this forward. Might not access any given bit of it for years at a time. But maintaining it on off-line storage has become problematic. The pace of growth of hard drives makes it fine to consider hard drives as a solution. Something like the Microserver is ideal.

But what I'd really like is a Microserver with 6 or 8 or 10 drives. I don't want an energy-hungry device. I don't need a heavy CPU or to be able to saturate a 10GE (or even a gigE). I just want to be able to store and retrieve things.

Realistically, I think I could live with the performance of a USB enclosure of some sort for some of the applications where I'd like a storage server. Long term archival storage. CDROM image storage. iTunes library storage. None of them are demanding.

I've given some thought to maybe picking up a Microserver plus a SANS DIGITAL TowerRAID TR4M. Looking to hit the low power goal, it isn't clear what the best value will be. There might be a better argument for seeing if FreeNAS will cooperate with a Mac Mini and a pair of TR4U's, or something. It'd be nice for there to be more obvious combinations of things out there. I'd definitely like to remain in the 100W-or-less power budget while moving up in flexibility of how much storage is attached (which is why four-bay expanders are modestly attractive).

 

[matthewowen01]

i should have been more specific, my real point was people should buy with the intent of expansion in mind. they can buy a micro atx AMD board with 4-6 on board sata ports, a pci-e 16x and a 1x slot. they can easily cram 8 gigs of ram in it and a 45 watt dual core cpu. They will have plenty of options to expand down the road. they make some good 4 port 1x cards and 8 or more port cards that will fit into a 16x slot.there are plenty of cases that can handle a lot of drives. Fractal Design has a $100 mid tower case that holds 10. NZXT has a sub $50 mid tower that holds 8 with 3 additional 5 1/4 inch bays that can be converted to 5 more 3 1/2 with more parts.

the SANS DIGITAL TowerRAID TR4M looks promising.

 

[niclas197]

Thanks for the input.

I don't know about this but SATA and PMP looks like a minefield to me.
I've looked for some USB alternatives and found this: http://www.fantec.de/html/en/2/artId/__1430/searchStr/__qb-35us3/article.html which costs only around 160€ and ONLY supports JBOD and is designed for software raid use. I'm not really sure about USB 3 per se but it does not look that bad to me since I could use 2-4 of this beauties with one singe controller card.

I've been searching around a bit and only found out that freebsd 8.2 added usb 3.0 support but I have yet to find a pcie controller of which I am positive that the driver does not suck.

What do you guys thing of this alternative?

 

[jgreco]

they can buy a micro atx AMD board with 4-6 on board sata ports, a pci-e 16x and a 1x slot. they can easily cram 8 gigs of ram in it and a 45 watt dual core cpu. They will have plenty of options to expand down the road. they make some good 4 port 1x cards and 8 or more port cards that will fit into a 16x slot.there are plenty of cases that can handle a lot of drives. Fractal Design has a $100 mid tower case that holds 10. NZXT has a sub $50 mid tower that holds 8 with 3 additional 5 1/4 inch bays that can be converted to 5 more 3 1/2 with more parts.

I started seriously preferring not to have to rip apart machines in order to swap out drives many years ago, so some of those options aren't as great.

The HP Microserver remains interesting because it's got remote management capabilities, but it's also worth looking at other solutions alongside it. Kind of wish HP had a few more Microserver options.

 

[jgreco]

I've been searching around a bit and only found out that freebsd 8.2 added usb 3.0 support but I have yet to find a pcie controller of which I am positive that the driver does not suck.

What do you guys thing of this alternative?

I think it's great that new hardware and new drivers get tested ... as long as I'm not the one suffering the data corruption.

 

[matthewowen01]

I started seriously preferring not to have to rip apart machines in order to swap out drives many years ago, so some of those options aren't as great.

The HP Microserver remains interesting because it's got remote management capabilities, but it's also worth looking at other solutions alongside it. Kind of wish HP had a few more Microserver options.

i got 3 stitches in my thumb at work from a case. no feeling in a small little patch. i do feel your pain. i wish HP had a macroserver option.

 

[jgreco]

i got 3 stitches in my thumb at work from a case. no feeling in a small little patch. i do feel your pain. i wish HP had a macroserver option.

Geez, you're kidding, right? HP has all the good stuff. Their mid-level stuff like the Proliant DL3xx rocks.

What's hard to find, though, is something that offers the convenience of having the files on-line without also being very noisy and power-hungry. You know, kind of like how you stick a 1TB HDD in grandma's PC just to make sure that you never have to revisit the capacity issue. Those of us with collections of data greater than a TB or two might not need to access any given file more than once every decade, if even at all, but storing off-line is inconvenient - what's a good format? a reliable format? etc.

Our Proliants take a few hundred watts each, which is fine given that they're pretty busy, but they take near that idle as well. So while there's no doubt that one would be a set-and-forget solution, it eats power up for no ROI, and makes lots of noise too. If I look at our current fileserver, which does more than just serving files, the load average is still 0.00 most of the time.

I do really like the idea of having some horsepower available just in case I decide to do something intensive, but a system that was capable of idling in the few-dozen watts range would be sweet. Like I'd be just fine with a Mac Mini-equivalent.

 

[matthewowen01]

Yes mostly a joke. I know they make good stuff, but there is no small step above the microserver. The microserver is designed for a small file server. I'd like to see something similar with a 45 watt amd that can hold 6 or 8, drives nice and cleanly without being overkill with dual quad core xeons.