Planning a build, need some clarification

[Wallybanger]

Hey guys,

I'll start off by saying that this is an awesome community and it's pretty crazy how many knowledgeable individuals are kicking around in here. I've been reading a lot of the guides and equipment posts and I've learned a lot in the last few weeks. I appreciate the work you're putting in!

About me: I'm a computer guy for sure but I have zero experience with Linux/Unix aside from hacking my android phones and entering commands I've found on the web. That said, I've built lots of windows machines and have a good understanding of the hardware.

What I'm after: I would like a slightly future proof NAS that I can use for work backup & file storage, movie/tv/music server (just playing video files over the network, don't think I need any transcoding ability), possibly a webserver, and I recently got it in my head that I would also like to use it for a 4 camera PoE IP security system (because why not?). I was actually just going to buy one of those Synology 1815+ disk systems until I found out about them selling "licenses" to run extra security cameras That sort of stuff pisses me off. If I'm going to spend $1000+ the thing had better do what I want right out of the box without limiting the hardware with a cash grab.

At this point I'm looking at the Lian-Li PC-Q26 and I'd like to run 8x 4TB WD Red drives in a RAID6 array for a total of 32TB. I think that should be a decent amount of future proofing. That said, I get the feeling that Mini-ITX may be a limiting factor. IPMI sounds awesome so I would like that and I would probably shoot for 32-64gb of ram. Can I get away with 32 and have decent performance?

Soo....

question 1) What are your opinions of the ASrock mobos? I know you guys suggest Supermicro but their Mini-ITX options are super limited. I have seen lots of ASrock builds and haven't see any threads with a full on denunciation of ASrock. If I absolutely have to go with Micro-ATX I guess I'm ok with that but I haven't seen any nice Micro-ATX Hot Swap cases.

question 2) Is RAIDZ2 (which is raid 6, correct?) the only option with FreeNAS? Is there a standard RAID6 software raid option? I ask because the lack of array recovery tools really bothers me. I'm totally on-board with ECC ram after reading Cyberjock's thread but it seems like there may be other things that could destroy your pool and if that were to happen it would be nice to be able to recover at least some of your files. Have there been any recent developments in the area of RAIDZ2 recovery? Google didn't tell me much.

question 3) The OS software worries me slightly. I would like to run FreeNAS with a webserver, torrent client and security cam suite but I'm afraid I may be biting off more than I can chew as a Linix noob. Is this going to be a nightmare or can I reasonably expect to get a good, solid system with a little work and few headaches?

Thanks in advance for the help :D

 

[SweetAndLow]

Asrock motherboards are just fine. Make sure to not use the marvel sata ports in the board since that don't have good support under BSD.

I wouldn't worry about recovery tools for zfs. The problems you run into with other raids and filesystems don't really happen on a well maintained zfs system. If something bad happens you can roll back the filesystem and a couple other things that are last ditch effort type things.

 

[nojohnny101]

welcome to the forums! you are absolutely correct, this is a great community! I have built two system but have only been working with freenas since december but i'm loving it!

1) I have built both of my system using ASRock motherboards. i think from the veterans here you will here this answer about them "while what they offer looks good on paper, they are too new in the motherboards for servers space to be trusted. time will tell". I went ahead despite that advice because as you said, they seem to have much more affordable option with a very strong feature set for the price. i also had some minor issues with one of the boards but it ended up just taking a simple CMOS reset which ASRock support walked me through very patiently. here are my setups:

Main FreeNAS BOX:
ASRock C226M WS
Intel G3258 3.2Ghz dual core
2 x 8GB DDR3 ECC Kingston RAM (unbuffered)
6 x 3TB Seagate NAS drives
SeaSonic G Series 550-Watt 80 PLUS Gold
Fractal Design NODE 804
1 x 16gb SanDisk Cruzer Fit (for FreeNAS OS)

Backup FreeNAS BOX
ASRock C2550D4I
1 x 8GB DDR3 ECC Kingston RAM (unbuffered)
4 x 2TB WD Reds
Silverstone ST45SF 450W 80 PLUG Bronze
Silverstone DS380B
1 x 16gb SanDisk Cruzer Fit (for FreeNAS OS)

either one of those boards would be great. see my post here for how I decided which board to use at my primary: https://forums.freenas.org/index.ph...el-avoton-quad-core-2-4ghz.42466/#post-275763

2) as far as recovery, keep in mind (and the veterans here will tell you) that RAIDz2 is not a backup! RAIDz1 is obviously not recommend at all anymore so you're smart for choosing RAIDz2. Yes, it does allow 2 drives to fail before you lose anything but that is why I ended up building a second freenas box because my data was too important to lose it to a disaster (fire, flood, power spike, etc.). If you have have RAIDz2 and ECC RAM you are most of the way there. But to be 100% secure, build a cheap backup box and then locate that offsite.

And I'm not really sure what you are worried about as far as "recovery tools". If one drives fails, then you just replace it ASAP and the pool rebuilds itself. Also with snapshots (read up on them in the manual), this prevents against accidental file deletion. ZFS is very good with correcting errors (and having ECC RAM helps that) so I wouldn't worry about recovering when things go bad.

3) I also knew almost nothing about linux before I started and I now know how to completely manager permissions, user accounts, groups and install jails from the CLI (command line interface). You learn as you go. Don't overwhelm yourself and think you're going to have everything up and running the 2nd day you boot your box. Take it one step at a time, read a lot!, post any questions you can't find information on, and then go from there.

Hope this helps!

 

[Wallybanger]

Asrock motherboards are just fine. Make sure to not use the marvel sata ports in the board since that don't have good support under BSD.

I wouldn't worry about recovery tools for zfs. The problems you run into with other raids and filesystems don't really happen on a well maintained zfs system. If something bad happens you can roll back the filesystem and a couple other things that are last ditch effort type things.

Yeah, I had heard about the Marvell SATA port issue. I'm looking at the Asrock board with LSI SAS ports. Apparently they get hot but I guess it isn't an issue(?). May upgrade the heatsink as someone suggested. It's surprisingly hard to find boards that support more than 6 SATA drives. Maybe I'm crazy but I really dislike the idea of running software raid on a bunch of drives with half of the drives on a different controller....

How do you roll back the file system if the pool goes bad? Do the snapshots get stored somewhere else?

welcome to the forums! you are absolutely correct, this is a great community! I have built two system but have only been working with freenas since december but i'm loving it!

1) I have built both of my system using ASRock motherboards. i think from the veterans here you will here this answer about them "while what they offer looks good on paper, they are too new in the motherboards for servers space to be trusted. time will tell". I went ahead despite that advice because as you said, they seem to have much more affordable option with a very strong feature set for the price. i also had some minor issues with one of the boards but it ended up just taking a simple CMOS reset which ASRock support walked me through very patiently. here are my setups:

Main FreeNAS BOX:
ASRock C226M WS
Intel G3258 3.2Ghz dual core
2 x 8GB DDR3 ECC Kingston RAM (unbuffered)
6 x 3TB Seagate NAS drives
SeaSonic G Series 550-Watt 80 PLUS Gold
Fractal Design NODE 804
1 x 16gb SanDisk Cruzer Fit (for FreeNAS OS)

Backup FreeNAS BOX
ASRock C2550D4I
1 x 8GB DDR3 ECC Kingston RAM (unbuffered)
4 x 2TB WD Reds
Silverstone ST45SF 450W 80 PLUG Bronze
Silverstone DS380B
1 x 16gb SanDisk Cruzer Fit (for FreeNAS OS)

either one of those boards would be great. see my post here for how I decided which board to use at my primary: https://forums.freenas.org/index.ph...el-avoton-quad-core-2-4ghz.42466/#post-275763

2) as far as recovery, keep in mind (and the veterans here will tell you) that RAIDz2 is not a backup! RAIDz1 is obviously not recommend at all anymore so you're smart for choosing RAIDz2. Yes, it does allow 2 drives to fail before you lose anything but that is why I ended up building a second freenas box because my data was too important to lose it to a disaster (fire, flood, power spike, etc.). If you have have RAIDz2 and ECC RAM you are most of the way there. But to be 100% secure, build a cheap backup box and then locate that offsite.

And I'm not really sure what you are worried about as far as "recovery tools". If one drives fails, then you just replace it ASAP and the pool rebuilds itself. Also with snapshots (read up on them in the manual), this prevents against accidental file deletion. ZFS is very good with correcting errors (and having ECC RAM helps that) so I wouldn't worry about recovering when things go bad.

3) I also knew almost nothing about linux before I started and I now know how to completely manager permissions, user accounts, groups and install jails from the CLI (command line interface). You learn as you go. Don't overwhelm yourself and think you're going to have everything up and running the 2nd day you boot your box. Take it one step at a time, read a lot!, post any questions you can't find information on, and then go from there.

Hope this helps!

Thanks :)

I have no problem paying for a supermicro board but they just don't seem to have the features I want. If they made one of those Mini-ITX boards with SAS ports or 8 SATA ports I would be all over it.

Yeah, I'm aware that it's not "backup". I have 5x WD 2TB mirror drives that I'm using now and if I build this NAS I will take them over to my buddies place and run them at his house for my "backup". The only problem is that 24TB of NAS won't backup to 5TB of extHD. That's not a huge deal as 5TB is way more than enough room for my work stuff but all of my movies, music and random junk would be susceptible to disappearing if the pool went bad. Not the end of the world but it would be nice to be able to recover some of it. Aside from bad ram I'm not really sure what could crash it... mobo failure? Power outage? All I know is that stuff happens and something could eventually fubar the whole setup.

Well it's good to know that I'm not alone! :P I've been reading about a bunch of the stuff and I'm just learning about jails and trying to figure out how to protect the whole thing from being attacked on the internet. To be honest, I really don't know nearly enough about internet security.... protocols, routers, ports, etc is still a bit of a mystery but I'm learning.

Thanks for the replies, guys!

 

[nojohnny101]

Yeah, I had heard about the Marvell SATA port issue. I'm looking at the Asrock board with LSI SAS ports. Apparently they get hot but I guess it isn't an issue(?). May upgrade the heatsink as someone suggested. It's surprisingly hard to find boards that support more than 6 SATA drives. Maybe I'm crazy but I really dislike the idea of running software raid on a bunch of drives with half of the drives on a different controller....

From your description earlier, it sounds like you want to use the box for basic file sharing and some media playing, don't think you will have to be changing heatsinks to accomplish that.

How do you roll back the file system if the pool goes bad? Do the snapshots get stored somewhere else?

There are two things that could happen, the OS or the device it is on goes bad or gets corrupted and that is an easy fix. If this happens your pool of data (your files that live on the hard drives in Raid2z) are untouched are are fine. You simply re-image a new USB and then upload your config file. Your config file contains all the user accounts, shares, services, etc. Upload that onto your clean install, reboot and everything back to normal!
I guess the other thing that could happen would be some type of failure with your pool (hard disks go back, RAM goes bad, etc.). This is a bit more complicated and I can't say how to recover from it because it depends on what is happening but most times snapshots will do fine. Read more about who they work here: link Snapshots are generally stored on the pool but can also be replicated to an off-site backup like I outlined earlier (read about replication in the manual).

I mean what you could do is if you have a bunch of files that don't change much just put them on USB drives that you can keep offsite (such as movies that you want to keep but don't change). Replicating to an offsite FreeNAS box can be selective as it is not just an "copy the entire pool" or nothing. But your needs may vary, you will have to either invest the money or just decided what is really worth saving if the worse case scenario happens.

here is some good reading about security as far as internet goes:
Setting remote access to FreeNAS box
Discussion on VPN

I used the first link to setup SSH into my box (also into my backup box). Using a non-standard public facing port for SSH is a good start but best case scenario is to use VPN.

 

[Wallybanger]

From your description earlier, it sounds like you want to use the box for basic file sharing and some media playing, don't think you will have to be changing heatsinks to accomplish that.


There are two things that could happen, the OS or the device it is on goes bad or gets corrupted and that is an easy fix. If this happens your pool of data (your files that live on the hard drives in Raid2z) are untouched are are fine. You simply re-image a new USB and then upload your config file. Your config file contains all the user accounts, shares, services, etc. Upload that onto your clean install, reboot and everything back to normal!
I guess the other thing that could happen would be some type of failure with your pool (hard disks go back, RAM goes bad, etc.). This is a bit more complicated and I can't say how to recover from it because it depends on what is happening but most times snapshots will do fine. Read more about who they work here: link Snapshots are generally stored on the pool but can also be replicated to an off-site backup like I outlined earlier (read about replication in the manual).

I mean what you could do is if you have a bunch of files that don't change much just put them on USB drives that you can keep offsite (such as movies that you want to keep but don't change). Replicating to an offsite FreeNAS box can be selective as it is not just an "copy the entire pool" or nothing. But your needs may vary, you will have to either invest the money or just decided what is really worth saving if the worse case scenario happens.

here is some good reading about security as far as internet goes:
Setting remote access to FreeNAS box
Discussion on VPN

I used the first link to setup SSH into my box (also into my backup box). Using a non-standard public facing port for SSH is a good start but best case scenario is to use VPN.

Yeah, you are probably right. I'm no network wiz so I really have no idea if I'd even be using this thing to it's full potential. At this point I'm not really even sure what to expect.

As for the other stuff... sounds like i have more reading to do :) Thanks for the links, I'll check em out.

Ugh, so that Lian-Li case only has 2 hot-swap drive bays lame. I thought all 10 were hot-swap. I may be back to a U-NAS case but those things are insanely expensive! Probably $350 by the time it lands on my doorstep. That's crazy.

 

[Ericloewe]

Is RAIDZ2 (which is raid 6, correct?) the only option with FreeNAS?

Well, ZFS is the only option.
That means RAIDZ1, RAIDZ2, RAIDZ3 and arbitrary numbers of mirrors.

Is there a standard RAID6 software raid option?

RAIDZ2 is software RAID6, but better in every way.

I ask because the lack of array recovery tools really bothers me.

Don't let it bother you. ZFS is more reliable than pretty much anything else out there. A spontaneous catastrophic loss of the pool is essentially unheard of - they seem to always be caused by things that would destroy a non-ZFS equivalent (failing drives, physical damage, ...).

Even if you end up with some corrupted data (for whatever reason), you're generally still able to save most of the pool's contents.

The OS software worries me slightly. I would like to run FreeNAS with a webserver, torrent client and security cam suite but I'm afraid I may be biting off more than I can chew as a Linix noob. Is this going to be a nightmare or can I reasonably expect to get a good, solid system with a little work and few headaches?

Unless you do something very stupid, you can run a lot of stuff in jails - assuming the software is available (web servers and torrent clients certainly are).
That said, I wouldn't run a web server on my NAS server, even jailed. The other two applications make sense, though.

 

[nojohnny101]

Ugh, so that Lian-Li case only has 2 hot-swap drive bays lame. I thought all 10 were hot-swap. I may be back to a U-NAS case but those things are insanely expensive! Probably $350 by the time it lands on my doorstep.That's crazy.

What about the Silverstone DS380B? also you will find many people on point out that hot swappable isn't really that much of a benefit in a FreeNAS box. Think about it, 98% of the time your drives are just going to sit in the box. When that 2% rolls around to replace a drive then some simple thumb screws usually do it. Most don't put it high on their priority list, especially if it comes with a higher price, because of the minimal advantage. In fact many veterans on here never recommend "hot swapping" on the fly. FreeNAS doesn't particularly like that and things could go wrong. If I am replacing anything, shut her down properly and do what you need to do. I don't run a production environment server that needs to be on 24/7 with mammal uptime. The couple of people that I give access to my box won't die if it is down for 30 minutes. ;) Your usage case may vary though.

 

[danb35]

FreeNAS doesn't particularly like that

I don't think this is correct--FreeNAS handles hot-swapping drives just fine. There are other issues (or potential issues), but lack of FreeNAS support really isn't the problem.

 

[nojohnny101]

@danb35 you're right, I worded that poorly. Not every motherboard supports it so I wanted to simply caution against it and as I said, most people's use cases for a FreeNAS box (primarily at home) gains no advantage to having hot swappable drives. But you are correct, it is supported by FreeNAS since FreeBSD 9.0

Cyberjock posted about some issues with hot swappable drives here: https://forums.freenas.org/index.php?threads/does-freenas-support-hotswap-drives.14546/ grant it that was 2013 and I don't know if his opinion has changed but I kind of share his mindset, it has not been something so rock solid in the past and so why take the risk unless crucial to one's setup?

 

[danb35]

Fair enough. It is nice to be able to swap in a replacement drive without taking down the server. For the average home user, though (or even the well-above-average home user), that's usually a matter of "minor convenience" rather than "operational necessity".

In my case, the "need" flows more from drive density than anything else. My 12-bay chassis is full; I have a 36-bay server that should be delivered today to replace it. Those, by and large, are going to be hot-swap chassis. I'm not going to be swapping drives willy-nilly, but if I have a drive failure, I'll most likely hot-swap it. I know the hardware supports it, I know the OS supports it, and I know the driver supports it.

 

[Wallybanger]

Well, ZFS is the only option.
That means RAIDZ1, RAIDZ2, RAIDZ3 and arbitrary numbers of mirrors.


RAIDZ2 is software RAID6, but better in every way.


Don't let it bother you. ZFS is more reliable than pretty much anything else out there. A spontaneous catastrophic loss of the pool is essentially unheard of - they seem to always be caused by things that would destroy a non-ZFS equivalent (failing drives, physical damage, ...).

Even if you end up with some corrupted data (for whatever reason), you're generally still able to save most of the pool's contents.


Unless you do something very stupid, you can run a lot of stuff in jails - assuming the software is available (web servers and torrent clients certainly are).
That said, I wouldn't run a web server on my NAS server, even jailed. The other two applications make sense, though.

All very good points. OK, it sounds like RAIDZ2 is probably pretty good then.

Yeah, I was thinking more about the webserver and it is probably best to have offsite web hosting. I don't need a public website constantly pointing to my home IP. How about an FTP though? That probably pretty safe, eh?

What about the Silverstone DS380B? also you will find many people on point out that hot swappable isn't really that much of a benefit in a FreeNAS box. Think about it, 98% of the time your drives are just going to sit in the box. When that 2% rolls around to replace a drive then some simple thumb screws usually do it. Most don't put it high on their priority list, especially if it comes with a higher price, because of the minimal advantage. In fact many veterans on here never recommend "hot swapping" on the fly. FreeNAS doesn't particularly like that and things could go wrong. If I am replacing anything, shut her down properly and do what you need to do. I don't run a production environment server that needs to be on 24/7 with mammal uptime. The couple of people that I give access to my box won't die if it is down for 30 minutes. ;) Your usage case may vary though.

OK, another good point. I probably don't NEED hot swap... it's just a want and it looks like that want is going to cost me an extra $100 for a case. I may be just fine without it. I was thinking that it might help tidy up wire routing.

I was looking at the DS380 but it looks like the extended Mini-ITX boards will only fit with modification to the case and the mods are less than ideal. Also, apparently the cooling is less than optimal since the fans just blow on the HDDs and not over them. That's where the PC-Q26 seems to be the better option.


Thanks for all of the replies so far guys, this is all good stuff!

 

[Robert Trevellyan]

How about an FTP though? That probably pretty safe, eh?

Why does anyone use FTP? Plain-text password transmission? No thanks.

If you feel compelled to have some kind of remote access, the preferred solution is a VPN, but SSH with public-key authentication, on a non-standard port, is also considered viable. The beauty of SSH is that it includes SFTP.

 

[Wallybanger]

Why does anyone use FTP? Plain-text password transmission? No thanks.

If you feel compelled to have some kind of remote access, the preferred solution is a VPN, but SSH with public-key authentication, on a non-standard port, is also considered viable. The beauty of SSH is that it includes SFTP.

Because they don't know any better ;)

I read about SFTP but had absolutely no idea how it worked. More research I'm settling in to read CyberJock's n00b powerpoint now.

 

[Robert Trevellyan]

I don't remember how I learned that SSH includes SFTP*. Seems less well-known than I would expect.

* According to Wikipedia, "It was designed by the Internet Engineering Task Force (IETF) as an extension of the Secure Shell protocol (SSH) version 2.0 to provide secure file transfer capabilities."

 

[Wallybanger]

Here's another question...

If disks are added to a VDev and VDevs are added to zpools and one of the VDevs dies it kills the whole zpool... why the hell would you ever add more than one VDev to a zpool? Can you have more than one zpool? If the VDevs are where the RAID functionality takes place it doesn't make sense to me why it would kill the whole pool if there are other VDevs within the pool.

Oh, and Cyberjock, your n00b ppt was great! Thanks :)

I don't remember how I learned that SSH includes SFTP*. Seems less well-known than I would expect.

* According to Wikipedia, "It was designed by the Internet Engineering Task Force (IETF) as an extension of the Secure Shell protocol (SSH) version 2.0 to provide secure file transfer capabilities."

Yeah, apparently OpenSSH comes with FreeBSD so I'm assuming there is some SSH functionality within FreeNAS but I haven't read the manual yet. Hopefully there is a way to easily and securely share files using SFTP.

 

[danb35]

why the hell would you ever add more than one VDev to a zpool?

Because having a single volume consisting of all your storage space, which you can carve up ad hoc into datasets as desired, is very useful in many cases.

it doesn't make sense to me why it would kill the whole pool if there are other VDevs within the pool.

Because data is striped across all the vdevs in the pool.

I'm assuming there is some SSH functionality within FreeNAS

Indeed there is. Simply turn on the switch on the services screen. I've not tried SFTP as such, but scp works quite well.

 

[Bidule0hm]

Maybe everything will be clear with this: https://forums.freenas.org/index.php?threads/comprehensive-diagram-of-the-zfs-structure.38865/ ;)

 

[Robert Trevellyan]

why the hell would you ever add more than one VDev to a zpool?

Another reason is if you need higher IOPS.

Hopefully there is a way to easily and securely share files using SFTP.

SFTP is designed for file access more than file sharing.

 

[Wallybanger]

Because having a single volume consisting of all your storage space, which you can carve up ad hoc into datasets as desired, is very useful in many cases.

Because data is striped across all the vdevs in the pool.

Indeed there is. Simply turn on the switch on the services screen. I've not tried SFTP as such, but scp works quite well.

It sounds like it's just creating more opportunities for data loss. I'm planning on running 8x 4tb drives as a RAIDZ2 array for about 24TB of usable space. Wouldn't that be preferable to having say 2X(4x 4TB) as 2 VDev's in one pool?

Ok, excellent news about SFTP.

Maybe everything will be clear with this: https://forums.freenas.org/index.php?threads/comprehensive-diagram-of-the-zfs-structure.38865/ ;)

That actually just created more confusion. Now I need to read about datasets and stuff.

Another reason is if you need higher IOPS.

SFTP is designed for file access more than file sharing.

OK, more IOPS makes sense.... but having 1 VDev kill the entire pool still sounds like a design flaw. Mind you, I'm no computer scientist.

File access is fine. As long as I can give someone an address and a password and let them securely upload and download stuff to a folder I'm happy.