Best practices upgrade path

[312elements]

I’m in the final building of my server build. I’m a photographer and this machine will store my file library. Single user. I’ll be writing 20gb at a time. Machine will be connected to my editing rig via 10gbe twinax.

Specs
E3 1220 v6
Super micro x11ssm-f
16gb ecc ram
8x8tb wd red

I’ve got about $500 left to put into the system if you think it will lead to increased performance. Any suggestions on how to best spend it given my specific use and needs. I have no VM plans nor do I anticipate needing to accommodate additional users. Thanks in advance.

 

[kdragon75]

Buy more RAM.

 

[kdragon75]

Or if you don't already have an onside backup server, build one. It may be slow but if it keeps you working while the primary is getting fixed, its doing its job.

 

[312elements]

Or if you don't already have an onside backup server, build one. It may be slow but if it keeps you working while the primary is getting fixed, its doing its job.

Thank you for your reply. The current sever is going to be repurposed as backup. Just to be clear, your suggestion is to max out the ram before adding ssd’s and such. Budget would push the ram up to 48gb.

 

[kdragon75]

Thank you for your reply. The current sever is going to be repurposed as backup. Just to be clear, your suggestion is to max out the ram before adding ssd’s and such. Budget would push the ram up to 48gb.

I would imagine there are better was to spend $500 but in the context of the server and maximizing performance, yes. SSDs as cache drives should generally only be considered if you can not physically support the amount of RAM needed for caching. Obviously on the enterprise server side of things where the system will support 1TB+ of RAM its not always financially reasonable to max out your ram first but that's a different ballpark in another state.

 

[kdragon75]

Go Packers...

 

[312elements]

Go Packers...

This conversation went south quickly. Should be a good show with NE this weekend. I’ve got Brady and White on the roster in my fantasy league so I’m hoping for a high scoring game.

 

[Chris Moore]

16gb ecc ram

Like @kdragon75 said, with that amount of storage, you really should have more memory, because FreeNAS uses RAM for cache of read and write to the pool. I would say that you need a minimum of 32 GB, but I suppose you could get away with 16 GB. It is really down to performance and as you said, you will be the only user, and if you are only accessing one file at a time, caching may not be a significant issue.

With regard to having a backup, because one copy of the file is not a backup, I suggest that a second server is a good idea. I have been running a pair of servers since around 2012 and it has saved me from myself on more than one occasion. Also, snapshots are a good safety net to protect against accidental deletion.

If you are interested in a low cost build as a backup system, you might look at this hardware. I pulled these links a few weeks ago, so the auctions are surely done by now, so be cautious that eBay does not redirect you to different hardware:

CASE: Fractal-Design-Define-R5-FD-CA-DEF-R5-BK-Black-Silent-ATX-Midtower-Computer-Case - US $104.99
https://www.ebay.com/itm/253026336681

POWER: NIB Corsair CS-M Series CS550M 550W 80 Plus Modular Power Supply - US $69.00
https://www.ebay.com/itm/163279426704

System Board: Super Micro X9SCM-F Motherboard w/ Heatsink/Fan & I/O Shield - US 79.99
https://www.ebay.com/itm/192561781616

CPU: Intel Xeon E3-1230V2 3.30GHz Quad-Core CPU Processor SR0P4 LGA1155 - C737 - US $85.00
https://www.ebay.com/itm/283158542659

Memory: Samsung 8GB M391B1G73QH0-YK0 DDR3 1600MHz 240p UDIMM PC3-12800E ECC 2Rx8 - - US $62.88 * 2 = $125.76
https://www.ebay.com/itm/282764333255

Drive Controller: LSI-SAS-9211-8i-8-port-6Gb-s-PCI-E-Internal-HBA-Both-Brackets-IT-MODE - US $59.99
https://www.ebay.com/itm/152937435505

Drive Cables: Mini SAS to 4-SATA SFF-8087 Multi-Lane Forward Breakout Internal Cable - - US $12.99
https://www.ebay.com/itm/371681252206

Thermal Compound: Noctua NT-H1 Thermal Paste Grease Conductive Compound for CPU/GPU - US $6.95
https://www.ebay.com/itm/302624513215

I may have missed some accessories, and I didn't include drives, but this should get you all the key components.

Total: 544.67

 

[Chris Moore]

Fractal-Design-Define-R5

The Define R5 has eight hard drive bays and it is quiet. A very good choice for most small builds.

 

[312elements]

Like @kdragon75 said, with that amount of storage, you really should have more memory, because FreeNAS uses RAM for cache of read and write to the pool. I would say that you need a minimum of 32 GB, but I suppose you could get away with 16 GB. It is really down to performance and as you said, you will be the only user, and if you are only accessing one file at a time, caching may not be a significant issue.

With regard to having a backup, because one copy of the file is not a backup, I suggest that a second server is a good idea. I have been running a pair of servers since around 2012 and it has saved me from myself on more than one occasion. Also, snapshots are a good safety net to protect against accidental deletion.

If you are interested in a low cost build as a backup system, you might look at this hardware. I pulled these links a few weeks ago, so the auctions are surely done by now, so be cautious that eBay does not redirect you to different hardware:

CASE: Fractal-Design-Define-R5-FD-CA-DEF-R5-BK-Black-Silent-ATX-Midtower-Computer-Case - US $104.99
https://www.ebay.com/itm/253026336681

POWER: NIB Corsair CS-M Series CS550M 550W 80 Plus Modular Power Supply - US $69.00
https://www.ebay.com/itm/163279426704

System Board: Super Micro X9SCM-F Motherboard w/ Heatsink/Fan & I/O Shield - US 79.99
https://www.ebay.com/itm/192561781616

CPU: Intel Xeon E3-1230V2 3.30GHz Quad-Core CPU Processor SR0P4 LGA1155 - C737 - US $85.00
https://www.ebay.com/itm/283158542659

Memory: Samsung 8GB M391B1G73QH0-YK0 DDR3 1600MHz 240p UDIMM PC3-12800E ECC 2Rx8 - - US $62.88 * 2 = $125.76
https://www.ebay.com/itm/282764333255

Drive Controller: LSI-SAS-9211-8i-8-port-6Gb-s-PCI-E-Internal-HBA-Both-Brackets-IT-MODE - US $59.99
https://www.ebay.com/itm/152937435505

Drive Cables: Mini SAS to 4-SATA SFF-8087 Multi-Lane Forward Breakout Internal Cable - - US $12.99
https://www.ebay.com/itm/371681252206

Thermal Compound: Noctua NT-H1 Thermal Paste Grease Conductive Compound for CPU/GPU - US $6.95
https://www.ebay.com/itm/302624513215

I may have missed some accessories, and I didn't include drives, but this should get you all the key components.

Total: 544.67

Thanks for that. I’m currently running a synology setup, so that’s going to be the backup once the new machine is up and running, but that looks like a very capable build at that price point. How do you like that LSI card?

 

[Chris Moore]

How do you like that LSI card?

I have been using LSI SAS controllers to run my SATA drives for several years now and I find them to be much less trouble than the SATA controllers I had used before. I always suggest using them for all the storage drives. I still use SATA for the boot drives because that is just easier.

 

[Chris Moore]

E3 1220 v6
Super micro x11ssm-f

If you have not already bought this, you might want to compare that CPU with the one in my build suggestion.
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E3-1220+v6+@+3.00GHz&id=3131
CPU Mark score of 7976

https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E3-1230+V2+@+3.30GHz&id=1189
CPU Mark score of 8857

The only reason to go with newer hardware is that it sometimes uses less electricity. FreeNAS does not need the latest and greatest hardware and sometimes the latest hardware isn't even much faster. For the money, I would go with the old gear instead of new and put the money saved in a savings account to prepare for the next upgrade.

 

[pro lamer]

https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E3-1230+V2+@+3.30GHz&id=1189
CPU Mark score of 8857

The only reason to go with newer hardware is that it sometimes uses less electricity

Just for sanity: the CPU above is limited to 32GB RAM. Having said that...

I would say that you need a minimum of 32 GB, but I suppose you could get away with 16 GB

... having read that, keep in mind I am just a noob.

Please treat my post only as a sanity chime in...

Sent from my mobile phone

 

[Chris Moore]

having read that, keep in mind I am just a noob.

I am not trying to bust you for being a noob, but I want to provide some clarification on the memory situation for you and for the OP.
FreeNAS uses RAM to cache read and write operations to the storage pool and for memory resident executable code for the operating system and for the memory resident portion of any jails or VMs that are running. If you are not running any jails or other VMs, the amount of memory needed is greatly reduced and the primary function that having additional memory will serve is to make the NAS more responsive to requests for data.
FreeNAS employs something called an ARC (adaptive replacement cache) that holds recently accessed or frequently accessed data in memory so that it can be supplied more quickly when it is requested again. The more memory you have, the more data can be in cache. If you have only one person accessing the NAS and you rarely access the same file repeatedly, the benefits provided by cache are reduced. FreeNAS can run perfectly well with 8GB of memory and I ran my backup NAS for several years with only 16GB, while my primary FreeNAS had 32GB to allow it to have extra memory for the jails to run. There is a lot of flexibility even in the often mentioned rule of having one gigabyte of RAM for every terabyte of storage.
Also, the performance you want must be balanced against the cost of that performance. If you save a couple of seconds on accessing a file, how much is that worth to you? If you are a large company with many employees accessing NAS, that time is magnified by the number of employees and the amount of wait is also magnified by the number of simultaneous requests for data. A NAS built for such an environment would need to be more robust and the added cost might be worth it as it would be spread across more users, but when you have a single user, it is difficult to rationalize spending hundreds of extra dollars to save seconds. Let your own budget be your guide.

There is a lot of material on the forum to help you with your decision:
https://forums.freenas.org/index.php?resources/

Hardware Recommendations Guide Rev 1e) 2017-05-06
https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/

FreeNAS® Quick Hardware Guide
https://forums.freenas.org/index.php?resources/freenas®-quick-hardware-guide.7/

Why not to use RAID-5 or RAIDz1
https://www.zdnet.com/article/why-raid-5-stops-working-in-2009/

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

 

[Dice]

In line with @kdragon75 - the additional RAM bump from 16GB to <more> is the typical thing to suggest (by me, others, most).
Yet, as @Chris Moore points out - the flexibility of the RAM:TB ratio is flexible - depending on use-case. I'll try to through some examples.

There is a huge difference between an "off site" receiever of backups, where network speeds ..aren't too impressive... and drives typically recieves streams of files in a slow pace (not taxing the ZIL on the pool/taxing the drives to slow things to a crawl) in the first case. Contrast that to a small server with 20 users, using the office-files located on the NAS. similar files are requested multiple times. This is where things like a lot of RAM and a L2ARC SSD cache really makes ZFS shine. It is a tremendous difference in usage patterns and thus - hardware requirements.

Now, so for your case @312elements , the charachteristics as you point out are single user, pretty larger files. But - are you working with them directly off your intended FreeNAS machine, or is the FreeNAS intended as a "storage area" from which you fetch the files for work, to a local workstation? This difference in workflow will cause freenas to experience your workload closer to either the "off site" or the "20 people server" scenario. Well actually, the small office situation could be allieviated with L2ARC and RAM, whilst your's - would rather require additional VDevs to swallow the IO requirements editing on FreeNAS would require. The latter being faaaar more taxing on your setup.

I've directly relevant experience myself. When I built my server, I initially went with 16GB RAM for 7x6TB. The migration processs was a pain. It was best described as "grab whatever you can find to hold data on the side" I found my self looking at performance that was outmoust garbage, during the brutal rearranging of files/reorganization following a migration Transfers slowing down to crawl. I grabbed another 32GB RAM and went up to 48GB. My problems would decline, but not dissappear. It was a noticable difference, but I could still tax the system to a crawl. Luckily, once my organization was completed - I never saw any sort of taxing problem regarding RAM. So much so, that I went on to trying lower amounts of RAM. It would do just fine with 16GB. Due to Jails and virtualization requirements, I currently give the freenas VM 32GB. I believe that is overkill for a mostly idle server....

Now, onto what would be best spent for your remaining $500?
- Get an UPS.

Cheers, Dice

 

[Ender117]

Since lots of people are talking about a secondary NAS for onsite backup, I would like to seek your opinion for the following question:

If you have a fast internet connection (say 1Gbit fiber), can a good online backup solution replace onsite backup? With a robust FreeNAS build and proper snapshot, it appears that backup are mostly only useful for disaster recovery scenarios. OTOH a gigabit internet means restore/retrieve can be completed in reasonable time frame. Having a big primary pool also means a secondary NAS would cost you a bit fortune to build and run. What's your thought on this?

Edit: Maybe I should rephrase my question to "online backup vs secondary FN box for onsite backup", because there are a lot of way of onsite backup that does not need a dedicated box.

 

[Chris Moore]

Since lots of people are talking about a secondary NAS for onsite backup, I would like to seek your opinion for the following question:

I have two FreeNAS systems running side-by-side at home with a 1-way push update from primary to backup every half hour. The advantage for me is that I can take my primary down for any reason at any time and the other users can simply access the data on the backup. It reduces my stress because I don't have to rush to get the primary back online. It is not a "HA" configuration because there is nothing automatic about it, but it is all about making my life easier. Sure, it costs more in hardware and electricity, but it reduces my stress. If you don't care about the down time while you repair what ever it was that took the NAS down to begin with and then restore your data from backup, an off-site backup is fine. That keeps your data safe and it is a great idea to protect from fire or flood or other calamity that would physically destroy the server. Do have an off-site backup. I am not saying don't. The reason for an on-premises backup is not for disaster recovery. It serves a different purpose. If you are cutting corners, you just have to decide what you want to cut. For me personally, I decided that temporary outage of a local server was more likely than my house being flooded or burned down. I have a temporary outage three or four times a year for one reason or another. Usually because I am tinkering with the hardware. I have lived in this house almost 40 years and it has not burned down or been flooded yet. I looked at the frequency with which I need a particular thing. Having an off-site backup is great, but how often will I actually need it? Having an on-site backup is something I use frequently.

 

[Chris Moore]

Having a big primary pool also means a secondary NAS would cost you a bit fortune to build and run. What's your thought on this?

I found great deals on my hardware, deals that are hard to find, and I don't expect anyone else to be able to duplicate that. but some people could get lucky and find even better deals. For example, I bought my Chenbro 48 bay chassis for only $300, plus shipping, and my Supermicro 24 bay chassis for $150.
Still, it is certainly possible to build a very good server (not including drives) for around $650 and a backup NAS could be a bit less powerful and I can see building one for around $530, all based on the last set of prices I looked up a month or so ago. The cost of drives very much depends on your storage needs. I spent around $2400 on drives alone when I did my drive refresh around the beginning of 2018. I bought n entire case of drives to replace all the old drives I had been using, many of which were over five years in power on hours:



There is no single answer that fits everyone. You have to decide what is workable for you.

 

[Dice]

Since lots of people are talking about a secondary NAS for onsite backup, I would like to seek your opinion for the following question:

If you have a fast internet connection (say 1Gbit fiber), can a good online backup solution replace onsite backup? With a robust FreeNAS build and proper snapshot, it appears that backup are mostly only useful for disaster recovery scenarios. OTOH a gigabit internet means restore/retrieve can be completed in reasonable time frame. Having a big primary pool also means a secondary NAS would cost you a bit fortune to build and run. What's your thought on this?

I recon a 2nd server locally is the way to go if: a) you don't trust anyone else with your data, and/or b) have a users which requires access to data at any time.
If none of those premises are required, off site is the way to go.
At that point you're in for either paying for a service or DIY/ putting the box in a coolio or equivalent. At that point, I believe the cheaper option in the longer run, is to buy the service. Plus, you wouldnt want anything else to do with that service, other than making sure it actually keeps backuping as intended. Additionally, once the first DIY-backupserver has "paid for itself" and has become obsolete, needing new hardware gives a new hefty hardware investment on your part. (I dont take into account that price per TB is getting cheaper by the year, since ideally, to some extent that would also trickle down to paid services as well as customer grade hardware off the shelf. yet the miles may vary.)

Cheers

 

[Ender117]

For me personally, I decided that temporary outage of a local server was more likely than my house being flooded or burned down. I have a temporary outage three or four times a year for one reason or another. Usually because I am tinkering with the hardware. I have lived in this house almost 40 years and it has not burned down or been flooded yet. I looked at the frequency with which I need a particular thing. Having an off-site backup is great, but how often will I actually need it? Having an on-site backup is something I use frequently.

I believe that with a properly designed and built FreeNAS box, you should rarely have unplanned down time, so usually you can plan ahead. Additionally, a good online backup solution should allow you retrieve individual files easily. I believe that it's a good assumption that out of all the data in one's primary NAS, only a (relatively) small portion is accessed frequently so that with a gigabit internet retrieving data from the cloud should only means minimum interruption to life.

Of course, things can happen. Your primary may be down unexpectedly for an extended time but still not an disastrous event. Or maybe your internet is down etc. I figure that a good compromise here is some USB drives or an independent VM that host a copy of your most frequently accessed data. I feel you would need a fairly large "hot" data set to justify a dedicated secondary FN box.

I certainly can appreciate the benefit of having a second onsite backup. I just feel a dedicated server for it is kind of overkill

I found great deals on my hardware, deals that are hard to find, and I don't expect anyone else to be able to duplicate that. but some people could get lucky and find even better deals. For example, I bought my Chenbro 48 bay chassis for only $300, plus shipping, and my Supermicro 24 bay chassis for $150.
Still, it is certainly possible to build a very good server (not including drives) for around $650 and a backup NAS could be a bit less powerful and I can see building one for around $530, all based on the last set of prices I looked up a month or so ago. The cost of drives very much depends on your storage needs. I spent around $2400 on drives alone when I did my drive refresh around the beginning of 2018. I bought n entire case of drives to replace all the old drives I had been using, many of which were over five years in power on hours:

View attachment 26593

There is no single answer that fits everyone. You have to decide what is workable for you.

I know, used server market is full of surprises. I got my 24bay DAS for $200, though I had to spend another 50 on a special cable to make it work. It's entirely possible to spend another $400 to build a decent head unit with 64G RAM and 10Gb NIC. It's the drives and electricity that would cost you another 1-2k in the next 5 years.