Greetings from NE Ohio / Western PA

[Tom2112]

Hello All,

I've had an OpenIndiana / napp-it NAS server running at home for about seven or eight years. It was 8x2TB drives at RAIDZ2. I pretty much built it, shared out the drive space, and walked away. I've been happy as a clam with it the whole time. The 2TB drives in it are starting to go, so I figured it was time to build a new one.

When I looked at building another OpenIndiana/napp-it server, the whole thing had changed radically. Back seven or eight years ago, FreeNAS wasn't as robust a solution. It lacked a lot of the features available in napp-it's solution. Now FreeNAS seems to have caught up with a vengeance.

So... long story short, I'm looking into FreeNAS and specc'ing out a new server.

Tom

 

[gpsguy]

Welcome to the forums!

 

[Tom2112]

Thank you!

 

[Chris Moore]

Hello All,

I've had an OpenIndiana / napp-it NAS server running at home for about seven or eight years. It was 8x2TB drives at RAIDZ2. I pretty much built it, shared out the drive space, and walked away. I've been happy as a clam with it the whole time. The 2TB drives in it are starting to go, so I figured it was time to build a new one.

When I looked at building another OpenIndiana/napp-it server, the whole thing had changed radically. Back seven or eight years ago, FreeNAS wasn't as robust a solution. It lacked a lot of the features available in napp-it's solution. Now FreeNAS seems to have caught up with a vengeance.

So... long story short, I'm looking into FreeNAS and specc'ing out a new server.

Tom

That is the way of things. If you have some idea what you want to be able to do, I will be happy to make some suggestions. Also, are you interested in using retired enterprise gear (with new drives) or do you want to purchase all new?

 

[gpsguy]

You're welcome.

BTW, take a look at our resources for recommended hardware.

 

[Tom2112]

Thank you, Chris!

I'm still trying to feel out what the hardware costs will be. I plan to go with new hardware. I briefly considered recycling hardware, but my old system doesn't support ECC RAM. And I don't plan on making a second server to backup the array. So the server data needs to be as reliable as possible - and that means ECC ram. That makes a lot of things more expensive.

The FreeNAS server's purpose would simply be network storage via SMB. I have a ton of movies, music, TV shows, etc. I do run Plex and a web server from my main gaming PC, and those both reference files on the NAS, but all the processing going on the gaming PC. The beauty of that is that it is a beefy PC and can handle the load easily. I only ever use Plex and the webserver when I'm away from the PC, so it works out well. The NAS just needs to serve files. That said, I haven't looked into what plugins are available for FreeNAS or what else I can do with it.

I'm thinking:
CPU: Intel E3-1200 series, not sure which one yet
RAM: 16GB of ECC ram (do I need 32GB if I'm not deduping or encrypting?)
NAS Drives: Four 10TB HGST NAS drives
OS Drives: Two (mirrored) 128GB SSD
Cache Drive: One 128GB SSD? (Not sure about this yet, I need to do some more reading on how the caching works.)
Motherboard: probably SuperMicro (I like the IPMI feature), but don't know which one... needs 7 or 8 SATA ports (I don't want to buy a SAS card)

Your thoughts on ram, mobo choice, cpu, and caching would be appreciated.

 

[Chris Moore]

I can already say that a cache drive is not a thing for you unless you are doing virtualization.
It would also be to your advantage to use 6 drives in RAIDz2 instead of only 4 drives, especially if you want to use such large drives. Last time I looked, the value option was 8TB drives.
I can give you more information once I get home.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Tom2112]

Oh, I thought it would cache file transfers - kinda like a big buffer. I noticed in the past, when copying large amounts of data to/from my old NAS that it would go full speed (saturate the LAN) for a while, then hit some sort of limit, and would almost pause and gradually pick up speed again until saturation, then bog down again. That would repeat until the huge transfer was done. I wouldn't hit that limit with 20 or 30 gigs of data, but 100 gigs or more would.

 

[Chris Moore]

ARC and ZIL are cached in RAM so the general rule is to avoid disk caches until the RAM is maxed out.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Redcoat]

I'm still trying to feel out what the hardware costs will be. I plan to go with new hardware. I briefly considered recycling hardware, but my old system doesn't support ECC RAM. And I don't plan on making a second server to backup the array. So the server data needs to be as reliable as possible - and that means ECC ram. That makes a lot of things more expensive.

Hmmm.. - what @Chris Moore meant was considering a used server system, not "partial reuse" of what you have. There are good deals to be had (and he seems to have a nose for finding them ...). Based on my own experience (with my backup/test server) I recommend that you at least take a look. If I were to want another server, I'd go the same route without hesitation.

 

[Tom2112]

That's a good point. I hadn't considered buying a used server. Though, I have to admit, I'm hesitant because of the simple logic of age and use. Folks don't just leave servers sitting around turned off. So whatever you can find has been running non-stop for a couple of years. Age adds up. Drives aside, power supplies and fans are the first to go, then capacitors, and who knows what else. I'm not sure I see the value in that. Help me understand this idea. Are the cost savings that significant that the risk is worth it?

 

[Chris Moore]

I'm not sure I see the value in that. Help me understand this idea. Are the cost savings that significant that the risk is worth it?

The good thing about servers, as opposed to a regular PC, is that a server is built with top quality components because the manufacturer does not want the thing to fail. If a server fails, the company using it is less likely to want to buy that kind of server again. We have a decent sized data center where I work and (sadly I am not the ultimate decision maker) they have installed systems with hot swappable components in such a way that to even access the hot swap components (such as a power supply) the server must be shutdown and removed from the rack. This is an idiotic choice made by a person that doesn't even work there any more and it continues to be done that way because that is the, 'way it has always been done'. They don't even order cable management arms and when they do get them they throw them away. Sorry, I got on a rant for a moment. The systems I am in control of are handled differently. The point being that the system (software) admins get all pissy about having to shutdown a server to change a power supply and make it their mission to never buy that kind of hardware again. It is that kind of thinking that makes hardware vendors charge a premium for their servers up-front because they build them with the best components and they never know if you will buy from them again, but it is probably a NO if there is a failure.
All that said, a server that would have cost between $10000 and $15000 (or more) new can be had second hand for $1000 ish and the hardware is still good for another 6 years or more. I bought all my home servers this way except for the chassis that I recently bought as an upgrade and I got it on close-out where it would have been around $3000 or more new, I got it for $350 and it was still new in the box with plastic wrap and had never been used.
It depends a lot on what you want, but you can get a lot more bang for the buck buying second hand enterprise servers. I don't suggest second hand hard drives though, they usually have a higher than normal failure rate due to high hours and rough handling. Also, you don't want to go too old because it will not have the capabilities you may want and it will use (sometimes significantly) more power.

 

[Redcoat]

I've only done it once so far, about a year ago. A Dell C2100 FS12TY, 32 GB Memory, Dual Xeon E5630's, dual GB LAN + BMC, dual redundant power supplies, 12 drive bays (no drives) for $350+shipping ($80 as I recall). Put 6 new drives in it, fired it up and it's only been off for an hour or two since (once to install a pair of used laptop SATA HDD's as boot drives in place of USB sticks and once to replace those two with a $20 eBay used SSD).
@Chris Moore will be able to tell it better ...

 

[Tom2112]

I also work in a datacenter. I've seen my share of failures, but I have to admit, they have dropped significantly in more recent years. Our Cisco UCS is a sight to behold. (At least, I like to think so, LOL)

So where/how do you find these deals? Ebay? I'm not sure I'd even know what to search for...

 

[Chris Moore]

So where/how do you find these deals? Ebay? I'm not sure I'd even know what to search for...

Yes, if you know what you are looking for, you can find some really good gear on eBay. There is also a bunch of gear that would be good, if it were not FreeNAS / ZFS (compatibility) then there is the gear that is just plain junk. One of the things I do (for work) and for fun is shop for gear. I was just reviewing a quote from HPE (Hewlett Packard) today where they were offering to sell us 10TB hard drives for the bargain price of $1700 each. The system was around $150k and I think we can find a better deal.
One thing you can do with FreeNAS is run the Plex server in a Jail so all the transcoding is handled inside the server, but that uses some RAM and CPU cycles because it is all CPU based transcode. If you think you want to give that a try, you will want to go with a little stronger build, but if you are strictly interested in file serving, it does not take as much.
Do you have an idea how much (not counting the drives) you are willing to invest in the server?
Also, how much fan noise are you willing to tolerate?

 

[Tom2112]

I'm not too concerned about fan noise. The server would be kept in the basement. But some of those servers are pretty darn loud...

As for price... If I were building one myself from new parts:
CPU: $250
Mobo: $150
RAM: $200
I already have a good power supply and chassis. As with most storage, the main cost is in the drives. Call it $600 without the drives.

 

[Chris Moore]

I already have a good power supply and chassis

Well, I wasn't suggesting just getting some components, I was suggesting getting a whole server that would do the thing but there were a couple questions that matter that you didn't answer.
Did you want to try and run Plex on the server or just keep that on your gaming PC?
Did you want to take my suggestion about going to 6 drives at 8TB each in RAIDz2 or stick with the 10TB drives you had mentioned?

 

[Tom2112]

Oh, sorry. I'll just leave Plex on the gaming PC. I would prefer the option of having up to 6 storage drives. I can always decide the size/layout of 4 tens or 6 eights later.

 

[Tom2112]

Just poking around on eBay...
What are your thoughts on this:
https://www.ebay.com/itm/Dell-Power...537270?hash=item27e2b99376:g:~FcAAOSwKQ9Z-2Ca

 

[Chris Moore]

Just poking around on eBay...
What are your thoughts on this:
https://www.ebay.com/itm/Dell-Power...537270?hash=item27e2b99376:g:~FcAAOSwKQ9Z-2Ca

The one big problem I see, but it is correctable, is this: RAID Controller PERC 6i RAID Controller with BBU and Cables
That hardware RAID controller would need to be replaced with a plain SAS HBA and we have had several people use this model (R710) and be happy with it. Here is a SAS HBA that other people (including me) have used sucessfully:
https://www.ebay.com/itm/Dell-H310-...ZFS-FreeNAS-unRAID-High-Air-Flow/162834671120
Other than that, it might pull more power than a more recent generation system, but it should do all the things you are looking to do.