Complete newbie tasked with setting up a NAS solution for a small business, need advice.

[nj4ck]

Hi there,

I've been tasked with completely overhauling our NAS system at work, even though I have no idea what I'm doing when it comes to this kind if thing. It must be the fact that I occasionally build new PCs whenever needed that led my boss to believe I was the right guy for the job.
All whining aside, we're currently running a 12TB RAIDZ2 NAS with no backup and one drive already dead, so I really do need to get this done ASAP.

We use our NAS constantly, it is our main storage for all work performed on individual workstations (we're a 3D animation and visual effects studio). That means we are reading and writing large amounts of data pretty much all the time (projects can be several hundred GB that are created, moved, updated and removed constantly). There can be up to about 10 users accessing the NAS at the same time, usually the number will be more like 6 or 7.

What we need is a fast, secure and not entirely wallet-melting solution that should include a >12TB main NAS, an on-site mirror and an off-site backup that will likely be located at my boss's house. I was thinking possibly a RAID 10 main NAS with 6 4TB drives, a mirror using 6 3TB drives in RAID 6 and an off-site backup likely just consisting of two 6TB drives to keep the cost down. I'm thinking of using rsync (or maybe there's something better?) to keep the mirror up to date, but I haven't yet figured out a way to keep the off-site backup running.
Then there's the question of hardware. How much CPU do I need? How much RAM? Do I need a dedicated RAID controller? Our current NAS is running one of those ASRock ITX boards with a quad-core Atom and 16GB of RAM. Just looking at the graphs, it seems our CPU usage is fairly low, but RAM is pegged at 100% and we recently lost one of the 8GB DIMMs, leaving us with 8GB of RAM and noticably crippled read-write speeds.
I was looking into Synology earlier and noticed people saying things like "incredibly fast" about them, but even the high-end ones don't seem to have much RAM, so I'm not sure why our NAS seems to be needing so much of it.
Speaking of which, there seems to be a massive bottleneck somewhere in our current setup. Take a look at this:

Apart from one of the drives being dead, aren't those read/write speeds incredibly slow? Maybe I'm reading this wrong, but it seems like the drives are never exceeding 4.5MB/s read/write speeds? The only bottleneck I can spot is the RAM usage, but maybe there's a network issue too?

In conclusion, I have more questions than I can even remember right now, but the main ones are:
1. What kind of NAS setup would you recommend for our purposes?
2. How important is CPU/RAM and could it be causing the slow speeds we're currently experiencing?
3. How would you recommend setting up an off-site backup for a 12TB NAS?

If anyone could help me out, I'd really, really appreciate that. I'm under quite a bit of pressure to get this figured out before our current NAS craps out any more and causes data loss. Essentially, I've got very limited time to learn how to NAS and do it properly on the first try.
Any help would be greatly appreciated.

 

[nojohnny101]

Whoa. Ok well you have come to the right place. I'm going to request more information from you so we can help you and hopefully clear you up on some things.

More information needed:
- can you list hardware specifications of your current box (you mentioned bits and pieces but one collective place would be nice). Make sure to include motherboard model, ram (ECC or non-ecc), I guess current OS if it isn't freenas, if it is freenas then what version and build, disk layout (raid or raidz type), any additional HBA's or card you have installed, boot drive, etc.
- how do the users currently access the box and have you thought about how they will access the new box?
- is your network 10GBe currently?
- what is the makeup of your environment (windows, mac, linux, mixed?)

All whining aside, we're currently running a 12TB RAIDZ2 NAS with no backup and one drive already dead, so I really do need to get this done ASAP.

Yikes. This is a disaster waiting to happen. Glad you are planing properly to have an off-site backup going forward.

Then there's the question of hardware. How much CPU do I need?

My guess would be a Xeon at minimum but I'm not personally familiar with the different levels, someone else can chime in on this.

How much RAM?

FreeNAS is very very very very RAM hungry. RAM is one of the best ways to spend your budget beyond necessities like CPU and motherboard. You can never have enough RAM. The veterans on here generally recommend 1GB of RAM per 1TB of raw storage. So if you have 12TB of storage (raw) then logic would say 16GB of RAM. But it sounds like you have a lot of users accessing and need speed so I would possibly get 32GB. Others can chime in on this if opinions differ.

Do I need a dedicated RAIDcontroller?

No. FreeNAS is a strictly software RAID solution. Adding a RAID card will actually make your life way more difficult.

Our current NAS is running one of those ASRock ITX boards with a quad-core Atom and 16GB of RAM. Justlooking at the graphs, it seems our CPU usage is fairly low, but RAM is pegged at 100% and we recently lost one of the 8GBDIMMs, leaving us with 8GB of RAM and noticably crippled read-write speeds.

If you have one one of the c2550 or 2750 boards from ASRock then you should search on the forums for these boards dying prematurely. It is a known problem and be prevented by disabling something in the BIOS. Look into it. Regarding the RAM usages, FreeNAS will always saturate near 100% of the RAM in any configuration. I don't have if you have 16GB, 32GB or 64GB in that box, it would almost always show near 100% utilization.

I was looking into Synology earlier and noticed people saying things like "incredibly fast" about them, but even the high-end ones don't seem to have much RAM, so I'm not sure why our NAS seems to be needing so much of it.

You will be able to get much more value for your money by building rather than buying. You're going the correct route if you're willing to put in the work and not rushing.

Apart from one of the drives being dead, aren't those read/write speeds incredibly slow? Maybe I'm reading this wrong, but it seems like the drives are never exceeding 4.5MB/s read/write speeds? The only bottleneck I can spot is the RAM usage, but maybe there's a network issue too?

Hard to diagnose without knowing about your setup. I would suggest though that you don't even deal with that if your'e building a new box anyways.

1. What kind of NAS setup would you recommend for our purposes?

People may recommend a system but I think it would be beneficial if you first listed what you were thinking based on your research. This was specific feedback can be given rather than us doing it for you. This will also force you to understand more about FreeNAS and how it works with hardware.

2. How important is CPU/RAM and could it be causing the slow speeds we're currently experiencing?

CPU usually comes mostly into play with VMs and if you are accessing the box via CIFS shares. This is highly dependent on the speed of your CPU. In sufficient RAM can cause instability and will definitely slow down everything on the system.

3. How would you recommend setting up an off-site backup for a 12TB NAS?

I would recommend repurposing your old box (the ASRock board) as the backup target off-site. The back-up box does not need near the power of the primary and if you do have the c2250 or c2750 then that should be plenty. FreeNAS has great built in support for replication over SSH which will make backing up to another FreeNAS box (as I suggested repurposing your old NAS would set this up nicely for you) very easy and will give you snapshots for restoring deleted data.

Essentially, I've got very limited time to learn how to NAS and doit properly on the first try.

I hate to be the one to say it, but that attitude, timeline, and pressure is setting you up to make mistakes. I know you have a boss and everything, but if he wants it done right, then you need to explain to him/her how properly researching and planning your build is the only way to ensure you don't make mistakes that will cost more in the long run or building a system that performs poorly.

Before recommendations can be made about your storage setup, you need to provide a bit more information regarding how your users are accessing and exactly what they are using it for. For example you mentioned it is a 3D animation and visual effects studio but people don't edit live from the NAS do they? Do they just use the NAS to store files and share them but not edit them live?

I'm going to help you out a bit and provide you with the following links to information that you at the very basic level, should read and understand.
FreeNAS manual
Cyberjock Essential Information Presentation
ZFS Terminology and Abbreviations Primer
PSU Sizing Guide

Good luck.

 

[danb35]

If you have a failed hard disk, in the short term, replace the disk. You're already running FreeNAS on a FreeNAS Mini*, so follow the manual's click-by-click instructions. The manual is linked at the top of this (and every) page in yellow; follow the link for the version you're using and use the table of contents to find the section on replacing a failed disk.

As to RAM usage, it's perfectly normal for close to 100% of your RAM to be used. If you've had a stick die, you should probably replace that too; as @nojohnny101 said, FreeNAS (and ZFS in general) really likes RAM.

As to a hardware recommendation, it seems like I'm pimping these a bit lately (paging @Mirfster, as he's the one that really likes them), but this system is well-equipped for FreeNAS, has plenty of room to expand the RAM if needed, has 12 bays for disks, and is only about $350 plus shipping with the coupon code "freenas": http://www.dfwpartsdepot.com/?produ...ing_wp_cron=1474766508.6439230442047119140625

For the use case you describe, two of them onsite, and possibly a third at your boss's house, would work well. You'd set up ZFS replication to do the backups (that's described in the manual too).

* Edit: This is a guess, based on your hardware description, your RAIDZ2 pool, and the usage graphs you posted, but it'd be good to confirm it.

 

[SweetAndLow]

If you use 12TB now you probably need to build a server with twice that much usable storage. You should also take growth rate into consideration.

Sent from my Nexus 5X using Tapatalk

 

[melloa]

Let me start by saying I'm not in any shape or form a FreeNAS SME/Guru/etc. Just a regular user, so only want to give you an idea how my system runs as I'm also using an ASRock with 40TB pool/32 GB RAM/i5 and I don't see that must memory been utilized and performance is good for my home use.

'course I use as a NAS and all my servers are hosted somewhere else in VMs, but:

As you can see the system is running great even with 3x your storage.

The point here is CHANGE YOUR HD ASAP! You don't want to be in a situation that you will lose your data. That's by the way, the one of the most important factor that made me stick with FreeNAS after testing all others appliances out there. Had HDs failing, reinstalled my system multiple times to test and learn, and my data was always safe.

 

[SweetAndLow]

Let me start by saying I'm not in any shape or form a FreeNAS SME/Guru/etc. Just a regular user, so only want to give you an idea how my system runs as I'm also using an ASRock with 40TB pool/32 GB RAM/i5 and I don't see that must memory been utilized and performance is good for my home use.

'course I use as a NAS and all my servers are hosted somewhere else in VMs, but:

View attachment 13837

View attachment 13838

As you can see the system is running great even with 3x your storage.

The point here is CHANGE YOUR HD ASAP! You don't want to be in a situation that you will lose your data. That's by the way, the one of the most important factor that made me stick with FreeNAS after testing all others appliances out there. Had HDs failing, reinstalled my system multiple times to test and learn, and my data was always safe.

Why is your wired memory so low? It should almost be maxed out. Did you use autotune

Sent from my Nexus 5X using Tapatalk

 

[melloa]

Why is your wired memory so low? It should almost be maxed out. Did you use autotune

Sent from my Nexus 5X using Tapatalk

No, autotune it is not on. Should it be? :)

 

[melloa]

Why is your wired memory so low? It should almost be maxed out. Did you use autotune

Sent from my Nexus 5X using Tapatalk

Went up after rsync kicked on ... but still see a share of green and no swap :)

 

[Stux]

I'll reply in more detail later, but in the meantime, replace the failed HD.

It sounds like your current NAS can become the onsite backup or the offsite backup.

Decide if you want to go to 10gbe. You should probably consider it.

 

[Stux]

Firstly, now that you've replaced that failed disk ;)

How many and what size disks are in the current NAS?

Since you've already got 12TB. You should probably aim for circa 24TB in the new NAS. Once you get to 80% full it's time to plan your next capacity expansion. And it should happen before you get to 85-90%.

I'm assuming your not editing off this NAS, but rather pulling files, editing jobs and pushing assets around. Please let us know if that's not the case.

Which would mean sequential speed is key. RaidZ2 actually has faster sequential write performance than mirrors.

I would suggest either 6 way or 8 way raidz2, possibly 2 by 6 way raidz2 in one of those 12 bay chassis Mirfster likes.

8 x 4TB in Z2 will get you 24TB.
6x 6TB in Z2 will also get you 24TB
6x3 x 2 will also get you 24TB and the best performance.
6x4 in Z2 will get you 16TB, and a growth path for another 6 disk vdev in a 12 bay chassis.

You'd already be over 80% in a 6x4TB Z2 array

If you're building your own, you probably will want a SuperMicro X11 motherboard with 8 SATA ports, and 2 16GB ECC DIMMs.

Pentium, i3 or Xeon depending on expected load.

Another option is a XeonD. They have six SATA ports, can come with 10gbe, m2 and in mini itx guise a PCI slot for additional drives. And they support 128GB of RAM. 2-8 core options.

Repurpose the existing NAS. You can setup automatic replication between the two NASes.

You can do the same thing offsite too.

 

[nj4ck]

Whoa. Ok well you have come to the right place. I'm going to request more information from you so we can help you and hopefully clear you up on some things.

More information needed:
- can you list hardware specifications of your current box (you mentioned bits and pieces but one collective place would be nice). Make sure to include motherboard model, ram (ECC or non-ecc), I guess current OS if it isn't freenas, if it is freenas then what version and build, disk layout (raid or raidz type), any additional HBA's or card you have installed, boot drive, etc.
- how do the users currently access the box and have you thought about how they will access the new box?
- is your network 10GBe currently?
- what is the makeup of your environment (windows, mac, linux, mixed?)

Ok, I'll try to list everything I know about our current system:

-ASRock C2550D4I motherboard with quad-core Atom SoC
- 16GB of DDR3 ECC (currently 8GB since one of the DIMMs died)
-6x WD Red Pro 3TB in RAIDZ2
-450W Silverstone gold PSU
-Silverstone DS380B case
-Freenas 9.3
-SSD boot drive (can't tell you exactly which one, I'm currently not in the office)

The NAS is accessed through our 10Gb network by all people at the office. We run a windows enviroment and mainly use Cinema4D, the Adobe suite, various network rendering clients and some other 3D applications. I'll give you a typical use case scenario: 1 person will create an animation in C4D, which will then be saved on the team render server, all other PCs that are not currently in use will download the assets from the team render server and begin rendering a large PNG sequence, that is rendered directly to the NAS. These PNG-sequences can easily reach 10-20 GBs in size. A second user will then import the PNG-sequence directly from the NAS into his After Effects composition and start working on it from there. Several people may be doing this at the same time. All work is performed directly from the NAS, people only work locally when it's absolutely necessary.

In its current configuration, a single user can get speeds like this:

Devide that by 6-8 users and you'll begin to understand our suffering ;)

If you have one one of the c2550 or 2750 boards from ASRock then you should search on the forums for these boards dying prematurely. It is a known problem and be prevented by disabling something in the BIOS. Look into it.

Thanks for the adive, we actually arleady had one die once already.

I would recommend repurposing your old box (the ASRock board) as the backup target off-site. The back-up box does not need near the power of the primary and if you do have the c2250 or c2750 then that should be plenty. FreeNAS has great built in support for replication over SSH which will make backing up to another FreeNAS box (as I suggested repurposing your old NAS would set this up nicely for you) very easy and will give you snapshots for restoring deleted data.

I was planning on using the old NAS as an on-site mirror and then using a third one as an off-site backup. My reasoning for this was that we can keep an on-site mirror synced perfectly over the local network, while only updating the off-site backup at night when nobody's using the network. Our connection is decently fast, but not fast enough to be constantly sending huge amounts of data during working hours. If all we had was an off-site backup that is synced once a day, we'd still risk losing a day's worth of work in the event of a failure. That's my theory at least, feel free to point out any holes in it.

 

[nj4ck]

If you have a failed hard disk, in the short term, replace the disk. You're already running FreeNAS on a FreeNAS Mini*, so follow the manual's click-by-click instructions. The manual is linked at the top of this (and every) page in yellow; follow the link for the version you're using and use the table of contents to find the section on replacing a failed disk.

As to RAM usage, it's perfectly normal for close to 100% of your RAM to be used. If you've had a stick die, you should probably replace that too; as @nojohnny101 said, FreeNAS (and ZFS in general) really likes RAM.

As to a hardware recommendation, it seems like I'm pimping these a bit lately (paging @Mirfster, as he's the one that really likes them), but this system is well-equipped for FreeNAS, has plenty of room to expand the RAM if needed, has 12 bays for disks, and is only about $350 plus shipping with the coupon code "freenas": http://www.dfwpartsdepot.com/?produ...ing_wp_cron=1474766508.6439230442047119140625

For the use case you describe, two of them onsite, and possibly a third at your boss's house, would work well. You'd set up ZFS replication to do the backups (that's described in the manual too).

* Edit: This is a guess, based on your hardware description, your RAIDZ2 pool, and the usage graphs you posted, but it'd be good to confirm it.

Wow, that's incredibly cheap. I was looking into getting a proper server rack, this looks just right. How's the noise on those? We don't have a dedicated server room, so I wouldn't want it sounding like a jet engine.

 

[nj4ck]

If you use 12TB now you probably need to build a server with twice that much usable storage. You should also take growth rate into consideration.

Sent from my Nexus 5X using Tapatalk

That's something I've been trying to convince my boss of, but he's fairly certain that we'll be fine with 12TB. We have additional archive storage that project folders are moved to once they've reached a certain age, so he may be right about that.

 

[nj4ck]

Let me start by saying I'm not in any shape or form a FreeNAS SME/Guru/etc. Just a regular user, so only want to give you an idea how my system runs as I'm also using an ASRock with 40TB pool/32 GB RAM/i5 and I don't see that must memory been utilized and performance is good for my home use.

'course I use as a NAS and all my servers are hosted somewhere else in VMs, but:

View attachment 13837

View attachment 13838

As you can see the system is running great even with 3x your storage.

The point here is CHANGE YOUR HD ASAP! You don't want to be in a situation that you will lose your data. That's by the way, the one of the most important factor that made me stick with FreeNAS after testing all others appliances out there. Had HDs failing, reinstalled my system multiple times to test and learn, and my data was always safe.

I've been told that resilvering an array without having a backup is risky, which is why I've been holding out on replacing the dead drive, even though I've got a replacement already waiting to be installed. Since we still stand to lose one more drive in our raidz2 array before the situation really becomes critical, I've decided not to replace the drive until the new NAS is up and running and we have a backup. Let me know if you'd recommend otherwise.

 

[nj4ck]

Firstly, now that you've replaced that failed disk ;)

How many and what size disks are in the current NAS?

Since you've already got 12TB. You should probably aim for circa 24TB in the new NAS. Once you get to 80% full it's time to plan your next capacity expansion. And it should happen before you get to 85-90%.

I'm assuming your not editing off this NAS, but rather pulling files, editing jobs and pushing assets around. Please let us know if that's not the case.

Which would mean sequential speed is key. RaidZ2 actually has faster sequential write performance than mirrors.

I would suggest either 6 way or 8 way raidz2, possibly 2 by 6 way raidz2 in one of those 12 bay chassis Mirfster likes.

8 x 4TB in Z2 will get you 24TB.
6x 6TB in Z2 will also get you 24TB
6x3 x 2 will also get you 24TB and the best performance.
6x4 in Z2 will get you 16TB, and a growth path for another 6 disk vdev in a 12 bay chassis.

You'd already be over 80% in a 6x4TB Z2 array

If you're building your own, you probably will want a SuperMicro X11 motherboard with 8 SATA ports, and 2 16GB ECC DIMMs.

Pentium, i3 or Xeon depending on expected load.

Another option is a XeonD. They have six SATA ports, can come with 10gbe, m2 and in mini itx guise a PCI slot for additional drives. And they support 128GB of RAM. 2-8 core options.

Repurpose the existing NAS. You can setup automatic replication between the two NASes.

You can do the same thing offsite too.

I haven't replaced the drive yet, since we don't have a backup yet and I've been told doing so without a backup would be too risky. Should I replace it anyways? The situation at the moment is that we have one drive down and one more to go before we risk losing data, so my plan was to hold out on replacing the drive unless a second one fails, in which case I'll have no choice.

I've been trying to convince my boss that we need more storage, but he seems convinced that we'll rarely run into situations like right now where we're utilizing nearly 90% of available storage. The cost of adding more storage is probably another major turnoff, especially since that would require the two backups to have equally large amounts of storage.
6x4TB in Z2 sounds like a viable option though, I might get HGST drives instead of WD red pros, they seem to be cheaper and have lower failure rates from what I read.
How would the speed of such an array compare to, say, a 16TB RIAD 10 array with 8x4TB?

We are editing off of the NAS, although things have become so painfully slow since we lost half of our RAM most of us are now working locally and copying the work back to the NAS when it's finished.

 

[danb35]

Let me know if you'd recommend otherwise.

I'd recommend otherwise. I don't at all agree that it's risky to resilver an array, and any risk there would be would be the same risk you'd run in creating a backup--either way you're going to have to read all the data on all the disks. Replace the failed disk and bring your array back to a healthy state. Then replace the failed DIMM as well.

On whatever you end up with as a server, I agree with the recommendation for lots of drive space. ZFS is a copy-on-write filesystem, and you're doing a lot of IO on that system. That's going to result in a lot of fragmentation, and that's going to kill your performance. Try a test--go to the shell on your machine and type zpool list. What does it report in the FRAG column?

 

[nj4ck]

I'd recommend otherwise. I don't at all agree that it's risky to resilver an array, and any risk there would be would be the same risk you'd run in creating a backup--either way you're going to have to read all the data on all the disks. Replace the failed disk and bring your array back to a healthy state. Then replace the failed DIMM as well.

On whatever you end up with as a server, I agree with the recommendation for lots of drive space. ZFS is a copy-on-write filesystem, and you're doing a lot of IO on that system. That's going to result in a lot of fragmentation, and that's going to kill your performance. Try a test--go to the shell on your machine and type zpool list. What does it report in the FRAG column?

It reports 32%. How bad/good is that?

 

[danb35]

Wow, that's incredibly cheap. I was looking into getting a proper server rack, this looks just right. How's the noise on those? We don't have a dedicated server room, so I wouldn't want it sounding like a jet engine.

I haven't used them, but @Mirfster says they're pretty reasonable. Here's a thread discussing them: https://forums.freenas.org/index.php?threads/ode-to-the-dell-c2100-fs12-ty.43665/.

It reports 32%. How bad/good is that?

Better than I would have expected, actually. And you're 90% full?

 

[danb35]

...and on the memory thing, here's what mine looks like:

As you see in my .sig, I have 128 GB of RAM, and a pool capacity a little over 40 TiB, half of which is used. Where's that RAM being used? Mostly in the ARC:

 

[Ericloewe]

It reports 32%. How bad/good is that?

Not very good, but it's not a very quantifiable number (which is an odd thing to say, I know, but it's just an indication).