Running out of space

Status
Not open for further replies.

Grantp

Contributor
Joined
Feb 26, 2013
Messages
111
I currently have 8x3TB WD Red hard drives configured as Raidz2 and mirrored giving me just over 6TB of actual space. Unfortunately I've vastly underestimated the amount of space my data would eventually take, I am down to 2.1TB free and only about halfway through importing my data. My question is how do I go about adding more disk space. My chassis has no more room to add drives. Can I somehow drop the mirror and use those drives (or am I talking total crap with that statement). I've attached a couple of screen shots showing my setup any help/advice would be gratefully received
Storage.png

Volume Status.png

Many thanks Grant.
 

aufalien

Patron
Joined
Jul 25, 2013
Messages
374
Well, I've been there many times, not with ZFS but in general :)

Ponder this article (pretty ez flow);

https://www.dan.me.uk/blog/2012/11/14/increase-capacity-of-freebsd-zfs-array-by-replacing-disks/

It may be the ticket for you. Although as always and the article emphasizes this; backup your data. And I would ensure your server is on UPS as hope is not a plan. In other words what if the power goes out or you trip a ckt? Actually its more like when will the power go out but you get the jist.

And old sys admin adage;

Tech is like a hard drive, not a question of if it will fail but when.

Good luck.
- aurf
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
Read my presentation(link in my sig). It explains how to increase disk capacity.

Also, you are not in a mirrored RAIDZ2. You are in a zpool with 2 vdevs. Each vdev has 4 disks in a RAIDZ2. There is no reason to use the word "mirror" at all as that doesn't apply to you.

What I would consider if I were you is destroying your pool and going with an 8 disk RAIDZ2. That will give you about 4TB more space with your current disks. Keep in mind that you will have to backup all of the data on your pool again. But if you do that PLEASE setup automatic SMART monitoring, testing, and the FreeNAS email feature so you are warned the instant anything goes wrong. Additionally, keep a spare hard drive on the shelf so that at the first sign of issues you can do a disk replacement while you wait for the RMA.
 

aufalien

Patron
Joined
Jul 25, 2013
Messages
374
Very nice,Cyber"removed" colored manual.

Page 127 of the FreeNAS manual rocks this info. So scrap my link and sleep with the manual under your pillow.

- aurf
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
Very nice Cyber Duck, colors even.

Page 127 of the FreeNAS manual rocks this info. So scrap my link and sleep with the manual under your pillow.

- aurf

Uh, what? Did I miss something? What's with the attitude? Cyber Duck? and colors?
 

Grantp

Contributor
Joined
Feb 26, 2013
Messages
111
Thanks for reply's guys much appreciated. CyberJock I wish I'd setup my zpool as you are suggesting but thought I needed 6 or 10 drives for that. My concern is still I do what you suggest getting an extra 4TB(approx) then I'm still back down to 2TB free space after importing the rest of my data, that space won't last me very long 12 months maybe before I'm going to have to expand again. As my case only holds 8 hard drives am I better off getting a larger case now and installing more Hard Drives, this is a home server sat in my lounge, so can't go for anything to huge or noisy (at the moment using Fractal Design Define R4) and system running virtually silent.

I suppose what I'm asking is what's my best option for future expansion.

Another thought I have 4x2TB drives sat around if I can squeeze them in the case somehow can I create another vdev and the add this to my zpool .
 

aufalien

Patron
Joined
Jul 25, 2013
Messages
374
Well, 8x4TB would get you 25% more storage and like you said, thats not much.

However those 4x2TB would get you the same gain.

Some math for us so that we are all on the same page;

8x3TB = 24TB raw
8x4TB = 32TB raw
= 8TB raw diff

4x2TB = 8TB raw

Same gain weather you add a vdev or replace drives.

The question you must ask is what data of yours is worth the money and go from there.

Also perhaps add a bit more RAM and enable dedup to see. I think you can find out how much space can be freed by running dedup in some kind of simulated mode?

- aurf
 

Grantp

Contributor
Joined
Feb 26, 2013
Messages
111
Thanks for reply aufalien so the answer to my question is YES I can create another vdev from the 4 spare 2TB drives if I can somehow get drives into my case. £1200 is out of question at the moment to change all drives to 4TB. The bad news appears to be in 12 months time I am going to be out of expansion options.

Your statement 'perhaps add a bit more RAM' - As far as I'm aware 32gb of memory is maximum supported by my SuperMicro X9SCM-F motherboard and again the expansion would have to be to 64gb so quiet a large outlay. It looks like I totally under speced my new server
 

Johhhn

Explorer
Joined
Oct 29, 2013
Messages
79
Why not just expand externally?



Sent from my iPhone using Tapatalk
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
The RAM thing is not hard and fast. I have about 50TB of disk space with 32GB of RAM and its fine. For home use you will be okay with 32GB to a fairly large amount of data. For business use you should try to stick to the 1GB per TB of storage whenever possible.

And always be ready to oversize your RAM if you plan to use an ESXi datastore on ZFS.

Dedup needs about 5GB of RAM per TB of data. And that's just a good guestimate. There is no upper limit. If you ever end up trying to mount a dirty pool with dedup enabled and it won't your only option will be to give it as much ram as it needs. If it needs 128GB of RAM, so be it. As you can kind of tell, that's not a good option at all.
 

Grantp

Contributor
Joined
Feb 26, 2013
Messages
111
Why not just expand externally?

I have no problem with expanding externally, didn't know I could, how do I attach new hardware to existing hardware, what do I need. As I said in one of my earlier post

Grantp said:
I suppose what I'm asking is what's my best option for future expansion.

Do I just need some sort of drive caddy holding external drives that attatches to my FreeNAS box or do I need to build another server and link the 2 together in some way?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
The usual way is to use an expansion chassis, an SAS expander, and an SAS controller. For rackmount gear this would be something like a Supermicro SC826 with the SAS expander backplane (BE16 or BE26 models). SM sells it in a config designed for the purpose but I don't know the partno off the top of my head.. Then you add an SFF8088 controller (or if you have slotblanks free on the server an M1015 plus an int-to-ext adapter).

For home use you can get a second matching chassis to your existing one. You have to figure out how to get the power supply locked on without a MB present, plus you get to figure out lots of other little details. It is actually equivalent to the SuperMicro route but the downside is, it may also be almost as expensive(!!).

Suggestion: it is less hassle and also safer just to get a bigger chassis and move your existing system's guts.
 

Johhhn

Explorer
Joined
Oct 29, 2013
Messages
79
Would a port multiplier work in this situation?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Yes, but port multipliers are fraught with peril. Plus often buggy, and/or nonoptimal in other ways. They are typically cheap chips for the mass market. SAS expanders are typically targeted at enterprise, so they tend to work better.
 

Johhhn

Explorer
Joined
Oct 29, 2013
Messages
79
How about using esata ports? I've setup boxes with external enclosures with esata using mirror raid 10 with no issues.
 

Grantp

Contributor
Joined
Feb 26, 2013
Messages
111
The usual way is to use an expansion chassis, an SAS expander, and an SAS controller. For rackmount gear this would be something like a Supermicro SC826 with the SAS expander backplane (BE16 or BE26 models). SM sells it in a config designed for the purpose but I don't know the partno off the top of my head.. Then you add an SFF8088 controller (or if you have slotblanks free on the server an M1015 plus an int-to-ext adapter).

For home use you can get a second matching chassis to your existing one. You have to figure out how to get the power supply locked on without a MB present, plus you get to figure out lots of other little details. It is actually equivalent to the SuperMicro route but the downside is, it may also be almost as expensive(!!).

Suggestion: it is less hassle and also safer just to get a bigger chassis and move your existing system's guts.

Ok let me see if I've got this correct. Are you saying I need something like this

http://www.ebay.co.uk/itm/221306207736?ssPageName=STRK:MEWAX:IT&_trksid=p3984.m1438.l2649

or this

http://www.ebay.co.uk/itm/221305763511?ssPageName=STRK:MEWAX:IT&_trksid=p3984.m1438.l2649

and also this controller

http://www.ebay.co.uk/itm/IBM-Serve...ards&clk_rvr_id=539764036645&autorefresh=true

I can then use my existing MotherBoard / Memory etc (see my signature).

I am totally out of my depth here so a really NOOB question the M1015 is 8 port so how do I connect the other 8 / 16 drives that these chassis can hold?

Is this the chassis you were referring to?

http://www.supermicro.co.uk/products/chassis/2u/826/sc826be26-r1k28w.cfm
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Yeah although at least that 846E1 is the old 3Gbps version; I wouldn't recommend it. 24 drives on 4 x 3Gbps = 12Gbps, not really a lot for modern drives.

The 826be26 is the 12-drive version of the 6Gbps, I believe, and that would be a sweet unit. The downside there is the 1k28 power supply ... it is just way too big.

So you are kind of looking at the right things. But you don't understand how it attaches. Great. That is easily rectified. :smile: ;-)

The M1015 is not an "8 port". It is a two port. See, look. Count them. Two. Two SFF8087's. Which are four lanes. Which you COULD use as one lane per disk. Which is what you're thinking of, but it is kind of non-elegant.

With SAS, you can connect a quad-lane cable (SFF8087) from your controller to your expander. The cheapest way to get an expander is a chassis that has one built-in, IMO. This gives you a 24Gbps (4 x 6Gbps) pathway from the controller to the expander. Now the trick is, you DO NOT NEED 1 CHANNEL PER DRIVE. The four channels are sufficient to support many drives, and the controller and expander will manage this for you without any effort on your part. However, as you increase the number of drives, you could find yourself limited. So figure 12 drives that can do 150MBytes/sec = 1800MBytes/sec = about 16Gbps. So 12 drives is the sweet spot for a SFF8087 - there should be minimal-to-no contention. If you stick 24 drives on, then you COULD be pulling about 32Gbps at full blast, which would not work. In practice you can't pull that anyways :smile:

So the thing is, if you get a 12 drive chassis and fill it, then need more space, you get ANOTHER chassis and just hook that second chassis up to the second SFF8087 on the M1015. You need some different cabling (SFF8088 external stuff) but the idea is the same.

Now the one odd thing is that you might be tempted to get a 16- or 24-drive chassis. If you think you'll need it, I think it is better to just do it and live with the possibility that you'll run into bus contention. Because I don't *think* it'll happen, and if it does, it can probably be tuned-around.

And if it comforts you any, know that my most recent FreeNAS box was a SC846BE26-R920B, 24 drive. I only have 12 drives in it. I speculated that I might someday need to expand to 24. 11 4TB drives in RAIDZ3 gives ~30TB of usable space. Very pleasant. But realistically I am guessing by the time I fill it, I'll be upgrading the drives not adding more.

Now the thing is, you often see people in the forum trying to cobble together some 24-port RAID controller (~$1200 all by itself) or combine a M1015 ($100) with a 24-port SAS expander ($300) and then still having to buy a 24-port Norco chassis ($400) and power supply ($100) and by the time you're all done screwing around you're past $1000. The 846 can be had for like $1200 new, and if you surf eBay a few weeks you can often score one for $400-$500 (possibly a slightly different configuration). Seems to me like the way to go.
 

aufalien

Patron
Joined
Jul 25, 2013
Messages
374
I would highly suggest using serious hardware like what jgreco suggests. Its seems many on this forum use FreeNAS at home but its a serious biz app. I mean ZFS dev'd by Sun was for serious biz needs and filled a badly needed requirement for this decade.

FreeNAS is simply ZFS with a great GUI etc.. Its more in terms of an embedded approach which is simply brilliant but I would really suggest you apply best practices in IT.

All in all, KISS this thing (keep it simple... stupid...) and get a bigger chassis and transplant the guts, as jgreco suggests. Otherwise I fear you will be spending good money after bad.

And I'm being sincere on this.
 

Grantp

Contributor
Joined
Feb 26, 2013
Messages
111
jgreco thanks for taking to the time to write such a detailed reply. The largeish outlay is out of question at the moment but I'll keep an eye open on E-Bay and if I see SC846BE26-R920B for half price then I'll defo get one. At the moment best price here is around £900 and I've only just spent £1300 approx on current set up.

The 836BA (linked above) does appear to be 6Gb reading the spec

Backplane
line_yellow.gif

3U Direct Attached Backplane, features:
• 3Gb/6Gb support
• SES-2 Enclosure Management Support
• SAS/SATA support
• Four SFF 8087 connectors

I do have one slight concern about going this way and that is where do I put the chassis I have 2 choices 1) in my lounge next to Tv 2) In bedroom I am assuming these chassis don't run silent so noise could become an issue. At the moment I have PC on one side of TV and current FreeNAS Server on the other and both are virtually silent. jgreco how much noise does your setup make?

aufalien I am only a home user trying to set up a Media Server, I just appear to have a lot of Media to serve :)
 
Status
Not open for further replies.
Top