How to expand and how far can I expand storage?

[Toast]

Hello! I've stalked the forums for a bit. I've had my freenas up for about 2 years now and I am still a complete noob at this. I mainly follow tutorials and what others have done.

My setup:
Asrock c2750d4i
6x4tb reds in a raidz2 plugged into the intel sata ports
32 ram

My pool is 83% full.
How should I continue to expand and how far can I expand? I've read the "don't be afraid to be sasy" but I don't think I fully understand it all. I understand you can get a sas card like the IBM 1015. If I wanted to do 2 seperate 20 bay chassis, how would I go about doing that with my setup? Should I upgrage my platform?

 

[nojohnny101]

maybe a little more information from you would be helpful in determining the best way forward.
- how much raw data do you have?
- do you have some place this can be stored off the current pool?
- do you have proper backups of your data that is on FreeNAS?
- at your current rate of data growth, how much raw storage should you add so you don't have this same problem in 6 months.
- what is your budget?

EDIT: a bit confused and content was removed so as to not spread misinformation.

also some others might come on here and tell you to avoid the marvell points on that board you have like the plague. I have a very similar board (c2550D4I) and have had no problems with my marvell ports. It is true they are not as reliable as the intel ports and ASRock themselves say in the manual to use the intels first and then use the marvell for the reliability reason. But before you spend money on extra cards, I would try to use the marvell ports for a couple of weeks or months when you get your new disks added and then see how they behave. if no problems then you saved yourself some money. if not, then you can go the expansion card route.

my 2 cents!

 

[Toast]

raw data: 12.1TB used
I dont have a place to unload it to. I was wanting to add another raidz2 in the mean time, but I was thinking that while I was at it, if I should buy a new case with more bays than just 12? Like a 20 or 24 bay case. If I did, I'm guessing I would need a sas controller that can connect 20 to 24 drives. Then how do I add more drives after that? I need some sort of external sas port to another box. Then in the new box, do I need a motherboard setup in it as well? I havnt planned a budget, but Ive been saving in the mean time until I figure out how to do all of this. I think as long as the marvell controllers are updated to the 1063 firmware it should be fine. I guess I can fill up the case that I have now then get a controller that has an external port to connect to an additional box with more drives maybe? Also I have no backup... Thanks

 

[nojohnny101]

Also I have no backup... Thanks

yikes do you not care about your data? it is replaceable then correct?

how much do you anticipate your data needs over the next couple of years? is this storing data for a company or something where you can accurately predict a growth rate? or is it just for personal use? how much space do you project you are going to need over the next couple of years?

I think as long as the marvell controllers are updated to the 1063 firmware it should be fine. I guess I can fill up the case

IMHO this is what I would do. Unless, as I asked above, your storage needs necessitate 20-24 drives. that is a serious setup. I mean to each his own but don't forget to take into account all that comes along with a setup like that:

- likely a new motherboard
- louder
- greater energy demands
- new PSU
- is most likely going to take up more space

only you can know, i'm just going off the information you have provided. if you want to go the rack way, by all means! if so, please post pictures!:D

 

[melloa]

raw data: 12.1TB used ... Also I have no backup... Thanks

My two cents? Build a 2nd FN with the configuration you'd like to have and back-up your data. I'm kind of old school when we used to have three tapes for each database, so I do have two FNs syncing daily ;)

 

[Toast]

I know, I should have a backup :(. Is it advisable to create a backup in freenas with a separate pool? Like could I get 3 8tb drives in my nas as a backup? Or is that frowned upon?

This is for home use. I plan to build a server closet in my basement. So noise shouldn't be an issue. I'm not sure I the heat part yet. It stays very cold down there year round.

I filled 12.1 TB in two years. I don't know if I'll be filling it up the same rate or if I'd fill up faster knowing I have more room. I want to know how to go beyond my 1 case as well. I think I'm ready to buy, I just want to buy in the right direction. I don't have the know how to be ready for future expansions if need be. Is my platform not suitable for 50TB? How do you create a 100TB setup? I can't afford a 100 TB setup but I would like to be able to do it one day. I just don't know how to do it and what it takes to do it.

 

[SweetAndLow]

maybe a little more information from you would be helpful in determining the best way forward.
- how much raw data do you have?
- do you have some place this can be stored off the current pool?
- do you have proper backups of your data that is on FreeNAS?
- at your current rate of data growth, how much raw storage should you add so you don't have this same problem in 6 months.
- what is your budget?

if you want to expand your current raidz2, you can not without destroying it and starting over. this would require you to temporarily move the data off the pool then reconfigure, load it back on.

so you have two options:

1) expand the current radiz2 you have by however many more disks (maybe 2) you want to add (i believe it is not recommended to go beyond 12 disks in raidz2 but that is even stretching it). this would require you to offload the data to some external drives, add the disks, setup a new pool with raidz2 x however many disks you want, then load the data back on. the number of disks you add depends on your budget and your needs into the future. this will depend on your budget and how many disks you can purchase right now

2) probably the slightly better option if you can afford it is to just create another raidz2 pool with 6 newly purchased disks with the greatest capacity you can afford. this will give you 2 6disk raidz2 pools. this avoids having to deal with any extra cards or anything else. this also has the benefit that you would not have to move your data off the box.

also some others might come on here and tell you to avoid the marvell points on that board you have like the plague. I have a very similar board (c2550D4I) and have had no problems with my marvell ports. It is true they are not as reliable as the intel ports and ASRock themselves say in the manual to use the intels first and then use the marvell for the reliability reason. But before you spend money on extra cards, I would try to use the marvell ports for a couple of weeks or months when you get your new disks added and then see how they behave. if no problems then you saved yourself some money. if not, then you can go the expansion card route.

my 2 cents!

This information is not correct. I want to make sure to clarify so someone doesn't get the wrong idea.

To expand a zfs pool you can either replace all the drives in a single vdev with larger drives and that vdev will grow in size or you can add a new vdev to the pool. In your option 1 you are just telling them to destroy their pool and create it with more drives, yes this would make the pool bigger but not something most people want to do. In your option 2 you are telling them to create a second pool along side their first pool. This also isn't something most people want to do since it adds administration overhead and is kind of silly. Adding those drives as a second vdev would give you the same storage capacity and also give you better performance for that single pool.

If you are looking at getting a 24bay case you have lots of options. First you need to find a case with a backplane hopefully. This backplan with either have a sas connection or sata connections. Depending on what interface it has will affect how you connect the drives to the motherboard. The simplest way is to get a backplane with a sas interface and you can use a single sas cable and usually a hba card to connect all 24 drives. If you got a backplane with sata interfaces you will need lots of sata ports on your motherboard and probably multiple HBA cards with forward breakout sas cables. And lastly adding a external chassis can be done but a little out of knowledge domain. But I think you basically just use a HBA with external connections and connect your external box via magic ;).

 

[melloa]

I know, I should have a backup :(. Is it advisable to create a backup in freenas with a separate pool? Like could I get 3 8tb drives in my nas as a backup? Or is that frowned upon?.

You can create multiple pools in one FN, one with the MB ports, others with SATA controlers, etc, but consider Murphy's Law ... You will be putting all your eggs in one basket.

This is for home use.

Same here. I have one FN with 10 x 4tb using my MB ports and a 2nd with 10 x 2tb plus 4 x 4tb on a IBM M1015.

I plan to build a server closet in my basement. So noise shouldn't be an issue. I'm not sure I the heat part yet. It stays very cold down there year round. .

Ventilation is key for the TBF of the equipment, specially the HDs. Consider exhaust the heat our of your closet or go with a open rack. I went with an open rack with a fan trail on top as I do have several servers running, but my FNs are on cases with multiple fans. I didn't get a hot swap as for a home user like me, it is not important to keep it running to replace a failed drive, but you will find several folks here that will disagree :)

I filled 12.1 TB in two years.

The question here is: What do you really need to store?

I don't know if I'll be filling it up the same rate or if I'd fill up faster knowing I have more room. I want to know how to go beyond my 1 case as well. I think I'm ready to buy, I just want to buy in the right direction. I don't have the know how to be ready for future expansions if need be.
Is my platform not suitable for 50TB? How do you create a 100TB setup? I can't afford a 100 TB setup but I would like to be able to do it one day. I just don't know how to do it and what it takes to do it.

The above will be beyond simple replies in a forum. You will need to do lots of researches to get to a point you will be able to do it. Possible? Yes, for sure. Back in May a user was building one with 270 x 8TB HDs for his company.

 

[nojohnny101]

@SweetAndLow thank you for the correction. i think i was mixing up terms (pool and vdev) and that is why it is wrong. I will edit above post to not spread misinformation. I meant to say option 1 to be creating a new, larger vdev with >6 drives which would require to destroy the current 6 drive vdev that he has it and start from scratch. option 2 i meant to say that he could add a 2nd vdev in his current pool and keep the first/current vdev which wouldn't require him to destroy his current vdev.

sorry!

 

[Toast]

I think I'm confused on the part of having an hba with an external port connecting to another box with just hdd's and no motherboard. Am I going to need more that one pcie slot on my main board if I want a 24 bay then another 24 bay? Is there a a tutorial I can read on this?

 

[Ericloewe]

I think I'm confused on the part of having an hba with an external port connecting to another box with just hdd's and no motherboard. Am I going to need more that one pcie slot on my main board if I want a 24 bay then another 24 bay? Is there a a tutorial I can read on this?

There's a lot of stuff, but no tutorial per se, because it's not a question best answered with a tutorial.

You might want to start with jgreco's SAS guide, which takes you through the basics. In essence, a single HBA can theoretically support up to 1024 SAS devices, including expanders.

 

[Mirfster]

Am I going to need more that one pcie slot on my main board if I want a 24 bay then another 24 bay? Is there a a tutorial I can read on this?

If this helps, I am messing around with a LSI 9200-8E. It has two external ports and while I am currently attaching both connectors to another system it technically should be able to work fine with just one connection. That would/should leave the other available to use on a third JBOD if desired.

 

[depasseg]

If this helps, I am messing around with a LSI 9200-8E. It has two external ports and while I am currently attaching both connectors to another system it technically should be able to work fine with just one connection. That would/should leave the other available to use on a third JBOD if desired.

Or you can likely daisy chain your JBOD enclosures (they usually have an In and out). That is the way the large SAN providers do it.

 

[depasseg]

I think I'm confused on the part of having an hba with an external port connecting to another box with just hdd's and no motherboard. Am I going to need more that one pcie slot on my main board if I want a 24 bay then another 24 bay? Is there a a tutorial I can read on this?

You are on the right track and as was mentioned above there isn't a tutorial on this. But, yes, you can use 1 HBA in one PCI slot with 1 cable going to a JBOD enclosure (like I have with my 45 drive sc847 JBOD). Most decent enclosures also include an outgoing/downstream connector, so that you can connect another enclosure, and so on. Eventually however, you will reach the saturation point of the SAS connections or the PCI bus and need to add another card. As you add more storage, you will also need to re-evaluate the amount of RAM you have. And finally, having an enclosure with lots of extra drives is a nice way to make a backup.
Here's a decent picture of what I'm talking about (except disregard the right column and the rows 2,4,6):

from http://www.stornews.com/?p=243

 

[Toast]

Thank you all for your replies. Ive been waiting to respond until Ive read some more. I think Im going to go with the 846 that has a sas2 backplane http://www.ebay.com/itm/SUPERMICRO-...493540?hash=item58b9fd1724:g:LnoAAOxyThVTaXxj . I picked the higher power supply one because I hear that they are quieter. Then Im going to get the ibm m1015 http://www.ebay.com/itm/IBM-46C8933...147373?hash=item41ac925cad:g:E2UAAOSw~OVW1fH7 . Then try to crossflash into IT mode. If I do fill up the case I believe that I can get an internal sas cable to converts into an external female port http://www.newegg.com/Product/Product.aspx?Item=N82E16812153067 and then add another 846 case with the same cable inside of it to its backplane sas2. Is this correct? Or should I be worried about speed limitations? Also I see different versions of the IBM 1015 such as 46m0861 and 46c8933. Should I be careful selecting which IBM 1015? Thank you all again and sorry it takes a while to repond.

 

[depasseg]

If I do fill up the case I believe that I can get an internal sas cable to converts into an external female port http://www.newegg.com/Product/Product.aspx?Item=N82E16812153067 and then add another 846 case with the same cable inside of it to its backplane sas2. Is this correct?

Yes and no. It could work, but a more elegant solution is to get a PCI female (internal) to female (external) adapter (http://www.newegg.com/Product/Product.aspx?Item=N82E16816133055). It is passive. This way you don't have a loose cable coming out of your server.

 

[Toast]

Thank you for that. I'll probably do that then.