20 bay build suggestion/opinions

[Allan_M]

Hi there

First of all; English is not my native language. So I'd like to excuse any misunderstandings because of that. I'd like it very much, if you'd just ask me any questions to make any misunderstandings any clearer.

With that out of the way... prepare for wall of text.

For the last 8 years my girlfriend and I have been very happy with our 4 bay NAS (Qnap TS-409). This summer, however, it became very clear that time had caught up with it and a replacement was necessary. I then began to research for a possible replacement but wasn't very satisfied with what I found. Long story short; I'm considering FreeNAS as a possible solution to our use case.

So let me try to break the contents of this thread into separate pieces:

1. Description of our use case
2. Shares/Users
3. Hardware
4. Setup (vdev/pools)
5. Overall
   1. Backup
6. Last words

1. Use case
My girlfriend and I like to take a lot of pictures. I've been using my/a dSLR almost every week for the last 9 years. We have two kids; so, of course there are a lot of pictures and videos.
My girlfriend have a Windows PC [0], that I back up whenever she uses it (which is about 2-3 times a year. She has a smart phone, as do I). It's equipped with a 128 GB SSD, so it's not taking up a lot of space.
I have; a MacBook Pro [1], Mac Mini [2] and Windows PC ('workstation')[3].

I'd like to have around 60-100 MB/s between the NAS and computer 2 and 3. Those two, are connected to the network via a Gb-switch which is capable of around 108 MB/s (sustained transfer; think documents/footage [SSD<->SSD]) between the two. I switch between internal and external (USB3/FW800) and would like to keep those speeds - hence the 60-100 MB/s performance floor/cieling.

I read, somewhere, about the possibility of combining ZFS/FreeNAS and Shadow Copy - that could be cool.
Also, I'd very much like to be able to Time Machine to the NAS.

Right now, the NAS is/was used for:

• Backup
• Documents
• Images/Video
• iTunes media (our entire DVD-collection is ripped and encoded in mpeg4)
• Older files [some date around 20 years back]

2. Shares/Users
Like mentioned above; the only users are my girlfriend and me - but I have three computers connected to the network and those are Mac OS X as well as Windows (8.1 Pro). That raises some questions - at least in my head - about the combination of SMB and AFP? As I understand it - without 'hacks' - AFP is the only way of making Time Machine'ing to a NAS a reality. SMB, for obvious Windows-reasons. Right now, I'd reckon we only need two users [our kids are 3.5 and 0.5 years old].

3. Hardware
As mentioned in the title, we are considering a 20 bay device. The premise is: It should last us, at least as long as the old NAS did - which equates to at least 8 years. This also spreads out some of the monetary considerations - I'll get back to that in a moment.

For specific hardware, I've been looking at the suggestions/advices given on this forum and came up with this: [prices are from a local vendor I'd like to support]

CPU:

• Single core price/performance*: Pentium G3258/3460 ~$65
• Multi core price/performance*: E3-1231v3 ~$235
• 'All-round' price/performance*: i3-4160/4170 ~$105-$110

Motherboard [ both with IPMI]:

• Intel DBS1200V3RPS ~$160
• X10SLL+-F-O ~$200

Memory [on the QVL]:

• Samsung DDR3-1600: ~$8/GB

Harddrive [WD]:

• RED NAS HDD ~$43/TB
• RED Network NAS HDD ~$48/TB
• RED Pro NAS HDD ~$54/TB

*Performance source: Passmark.com

4. Setup (vdev/pools)
I used some time to contemplate the whole raidz/2/3 thing, and I think I came up with a solution/suggestion that would solve two problems - but also introduce some concerns. This relates, some way, to the shares [#2] and how to set things up.

Let's first forget about the multiple computers, Shadow Copy/Time Machine and all that. Let's first consider the financials. To fill the 20 bay NAS with 3 or 4 TB drives, would impose quite a monetary problem to the little family of four. My plan was to slowly fill the NAS with drives - like; 1 a month or every two months - to spread out the cost. Within 1-2 years, the NAS would be filled and at its full capacity.

This is to say, that [drive] cost is not a concern - since it is spread out over a longer period of time. After doing som reading on ZFS (primers, powerpoints/PDFs and whatnot), I think I settled on something like this:

Taking two drives, mirror them into one vdev and create a pool. Then later, take another two drives, mirror them into a new video and add those to the pool. Again, later, take another two new drives, mirror them into a vdev that gets added to the pool - so on and so forth.

That means, I could start with 2x 1TB [vdev] + 2x 1TB [vdev] = 2 TB [pool]
Then expand/add; 2 TB pool + 2x 2TB [vdev] = 4 TB [pool]

Maybe I have misunderstood ZFS and how it works. But I thought it looks quite clever. I a drive should fail, I'd just substitute it with a newer - possibly larger capacity drive.
The problem, of course, is if two drives in the same vdev should happen to fail simultaneously ...

I know, the actual capacity wouldn't be 2 TB or 4 TB for that matter. But I think you get the idea of how I saw it; a pool of multiple mirrored vdevs.

5. Overall
I have tried to describe our use case [1]. Right now, our combined contents is spread out onto five external harddrives [1 + 1 + 2 + 2 + 2 = 8 TB]. So that puts some sort of storage floor on our future NAS solution. We need, at least 6-8 TB of storage - preferably at least the double of that ~16-20 TB*. I'd like, the performance interval around 60-100 MB/s.

60 MB/s = My external FW800 drive.
100 MB/s = My external USB3 drive and also the current bandwith between my Mac Mini and Windows 'workstation'.

I know, that random I/O is another ballgame entirely.

I read a lot of the threads concerning choice of CPU/motherboard/RAM and to be honest. I'm not that bothered with the price of 'server grade' hardware. As mentioned earlier - this solution is supposed to last at least 8-10 years. A $5-10 difference, right now, doesn't matter that much in the long run.

I've suggested three types of CPU's. One focused on single core performance (SMB), one focused on multi core performance [NFS?/Multiple SMB's?/Awesomeness?] and at last one focused on single- as well as multi core performance.

Two motherboards; SuperMicro and Intel.
Harddrives - to be honest; I'm a bit lost on this one. I'll pick up any of the three suggested drives you recommend.

This brings me to the last points/questions:

1. I've bought a Norco 4220 case ~$45 (used]
2. I've also bought a MegaRAID SAS 9260-8i controller ~$115

The case was a steal. When I was collecting it, another guy in the neighborhood was selling the 9260-8i and I just couldn't pass it by. Later, when I got home, I realized it wasn't recommended - at all. However, I'll find use for it - perhaps in a future 'workstation'-project, but I can try to sell and raise cash for more drives.

Regarding the 20 bays, I considered buying two M1015's and a '4x SATA to SFF8087' cable. I am, however, open to suggestions.

* Regarding the pool and shares; would you set up one big share or multiple pools (for instance; a pool pr. share?)

5.1 Backup
Last section, is regarding backup. I thought about some sort of 'rotating' backup consisting of multiple external drives. When creating shares, I would set them up to match the capacity of the external backup drives.

To be honest, I got a little lost when trying to plan/visualize the backup scheme.

6. Last words
Finally; I'd like to thank you all for getting this far. I know, it's a wall of text but I've been trying to wrap my head around this the last couple of months.

To sum it all up: high capacity and redundancy. Ease of mind. Set and forget. Should last us, at least 8-10 years.

 

[Mirfster]

I've also bought a MegaRAID SAS 9260-8i controller ~$115

I don't believe this card can be flashed to IT mode and is only a Hardware Raid Controller. So that can't be used for FreeNas. Others can correct me if I am wrong.

 

[Allan_M]

I don't believe this card can be flashed to IT mode and is only a Hardware Raid Controller. So that can't be used for FreeNas. Others can correct me if I am wrong.

That I am aware of. That's what I meant by "Later, when I got home, I realized it wasn't recommended - at all. However, I'll find use for it - perhaps in a future 'workstation'-project, but I can try to sell and raise cash for more drives." Sorry for the confusion.

The reason I bought it, in the first place, was because I did a search for raid controllers supported [by FreeBSD, I suppose] and the 9260-i8 turned up. Of course, that's my mistake, for not doing proper research, regarding the use of ZFS/FreeNAS before buying this particular RAID-controller. I now know I was supposed to buy a dedicated HBA capable of target/JBOD-mode.

 

[Mirfster]

Sorry, not sure how I missed the part where you did mention that you were aware of it.

 

[Allan_M]

Sorry, not sure how I missed the part where you did mention that you were aware of it.

That's totally ok. I know I tend to write a lot.

 

[Glorious1]

3. Correct, you'll need to use an AFP share for the Time Machine backup. Make that a separate dataset and it should work fine. For other things, if you want to access the same files with both the Mac and PC, you'll have to use CIFS. Don't use two different shares for the same data, unless one is read-only.

4. You spotted one problem in your vdev/pool growth idea - that with multiple vdevs made of two mirrored disks each, if you lose two disks in the same vdev, you LOSE EVERYTHING in the pool. But another problem is that it is not efficient - you've got 50% parity in what is ultimately a large pool. I would wait until I could buy 5 or 6 disks and make a pool with RaidZ2. Later, when you can buy 5 or 6 more, you can add another vdev to it, or another pool.

Regarding shares, you can be flexible there. Within a pool, make multiple datasets for things you might eventually want to have separate shares for. But you can change that later. One share for a pool doesn't seem very realistic or useful. Certainly the Time Machine needs to be a separate share. And you may want to have a share for things you access alone, a similar share for your partner, and another share or more for the two of you together, for things you "share".

 

[Allan_M]

3. Correct, you'll need to use an AFP share for the Time Machine backup. Make that a separate dataset and it should work fine. For other things, if you want to access the same files with both the Mac and PC, you'll have to use CIFS. Don't use two different shares for the same data, unless one is read-only.

Just to be sure; a 'dataset' constitutes what I'd usually recognize as a 'shared volume' - To my best understanding, the terms are kinda switched when dealing with ZFS/FreeNAS. I'd create a pool first, then a dataset to which I'd assign a 'size', file system and which users to share it with (and, of course, protocol)?

I reckoned I'd have to make som SMB-volumes [datasets?], simply because there are some Windows computers on the network - but does that mean I can/should simply focus on a dual core CPU with a high single core performance? What about NFS? Isn't that multi-core 'aware'?

4. You spotted one problem in your vdev/pool growth idea - that with multiple vdevs made of two mirrored disks each, if you lose two disks in the same vdev, you LOSE EVERYTHING in the pool. But another problem is that it is not efficient - you've got 50% parity in what is ultimately a large pool. I would wait until I could buy 5 or 6 disks and make a pool with RaidZ2. Later, when you can buy 5 or 6 more, you can add another vdev to it, or another pool.

To be honest. I don't care that much about efficiency - that's one of the reasons I considered a +10 bay solution in the first place - but I appreciate that you point it out. Also, it wasn't my plan to make it into one large pool - though I can understand that it might have come of that way. Sorry about that.

Your suggestion regarding RAIDZ2 and 5 or 6 disks, has been my idea all along until recently. I read a blog post, about why one would consider mirrored vdevs and adding them to a pool instead of doing RAIDZ2 or -3. From a performance standpoint, it seemed quite reasonable but also somewhat risky.

If we consider the RAIDZ2 solution; I thought about doing 2 or 4 pools with 5 drives in each. Populated solely with 2 TB drives, that would equate to around;

5 x 2 TB drives@RAIZ2 -> 5*2TB-2*2TB = 6 TB usable

With 20 bays, we could do that four times, which equates to around 24 TB space. If we go by the <80% 'rule', that would give us about 19 TB 'usable' space before swapping out the 2 TB drives with - for instance; 3- or 4 TB drives [36 TB ~ 28 TB usable or 48 TB ~ 38 TB usable]. That's plenty, and also why 50 % storage efficiency is tolerable/just fine.

My only concern with RAIDZ/2/3 is performance. As mentioned, I'd like to have a level of performance around 60-100 MB/s for sustained reads/writes. I honestly have no idea if that's possible with this kind of hardware/solution.

Regarding shares, you can be flexible there. Within a pool, make multiple datasets for things you might eventually want to have separate shares for. But you can change that later. One share for a pool doesn't seem very realistic or useful. Certainly the Time Machine needs to be a separate share. And you may want to have a share for things you access alone, a similar share for your partner, and another share or more for the two of you together, for things you "share".

That was never a part of the plan. You've read that right on. I indeed intended to create multiple shares/datasets. But I have no idea of how that would affect the performance - besides SMB and single core performance.

In one of the threads (about suggested hardware), it says something along the line of: "As a home user this is about the most powerful CPU you can reasonably use." - regarding a E3-1230V2/3 CPU. Though I do understand the statement, I'm not entirely sure what to make of it. Does that mean, it's beyond 'reason', am I future proofing or just showing off - if I buy one of these?

When looking purely at the performance (source: Passmark.com) and local prices, an E3-1231V3 CPU - though pricier - is still a very reasonable CPU to consider. On the other hand; If our use case mostly will be limited to single/dual core performance, there is absolutely no reason to spend $235 on the CPU when a $65 or $105-110 CPU are up to the task. Those additional $125-170 are then spent better elsewhere.

 

[Glorious1]

I hope someone else will respond to you about the hardware. If not you should just make a new post in the Hardware forum and focus on that alone. This thread is a little too diffuse.

Regarding datasets and shares. You will create one or more datasets in a pool (pool=volume). Datasets have a lot of properties you can set. Actually the user's guide says it pretty well:

Permissions, compression, deduplication, and quotas can be set on a per-dataset basis, allowing more granular control over access to storage data. A dataset is similar to a folder in that you can set permissions; it is also similar to a filesystem in that you can set properties such as quotas and compression as well as create snapshots.

Shares allow certain people to access certain parts of a volume. Shares don't have to correspond to datasets, but I guess they often do.

The confusing thing maybe is that when you open a share from another computer, that computer treats it as a volume. But it's not a volume on the FreeNAS end.

9.3 has a wizard that create shares for you and creates datasets corresponding to the shares. I've never used it because I got everything set before 9.3, but that could be an easy way to get started.

 

[Allan_M]

I hope someone else will respond to you about the hardware. If not you should just make a new post in the Hardware forum and focus on that alone. This thread is a little too diffuse.

I understand and totally agree. Just wanted some sort of general response to whether or not I was totally off in my reflections.

The confusing thing maybe is that when you open a share from another computer, that computer treats it as a volume. But it's not a volume on the FreeNAS end. 9.3 has a wizard that create shares for you and creates datasets corresponding to the shares. I've never used it because I got everything set before 9.3, but that could be an easy way to get started.

I've actually read the entire manual, but that's not the same as me being 100 % sure about anything. But it helped a lot, in the workings of ZFS/FreeNAS and why it could be the solution for my girlfriend and me. This thread is supposed to iron out some misunderstandings or other things I might have got the wrong way around. I read something, about volumens and iSCSI, but I concluded it wasn't essential knowledge for what I'm trying to accomplish. Perhaps that could also be a source of confusion - I'm sorry.

I'll follow your suggestion and post a thread in the hardware section - but I think the only thing I'm trying to get my head around is which of the CPUs listed is the 'right' choice. Pentium G3258/3460, E3-1231v3 or i3-4160/4170.