My TrueNAS plan, 2 years on from learning - comments welcome

[Bikerchris]

Hello All,

I would be keen to have your thoughts and comments, does this server arrangement make sense for me?

Show : My background

I've been using FreeNAS and then TrueNAS in a disposable/learning situation for a few years now, and am closer to making it production. It is a brilliant and rock solid system with the right hardware. All hardware is most definitely burned in, all hard drives are thoroughly tested. I've really been grateful for the help from this community, thank you all.

Show : Transition away from Windows

I've had one machine that's been my primary storage, a server of sorts. Today I bought a 14TB drive to store all files (instead of the 4 mixed drives I'm currently using), backed up with Backblaze during the transition and to allow me to re-use the machine as a TrueNAS snapshot Server (Referenced as "Server2" below)

Show : How I'll be using the system(s)

Purpose is a mixture of Personal (weekly use) and Business (daily use). SyncThing will replace historic use of Dropbox and G Drive for hourly changed files (CAD Drawings, photos and videos). I then have personal photos/videos as well as Plex (am warming to Emby, Jellyfin install didn't go well) for entertainment via a Nvidia Shield attached to TV as well as PC's. I'll also be using the server for PC backups (3 Nos.)

Show : Site context

I did rent an office, but I was given 6 months to build my new office in the back garden and that's where my workstation is now. I've put Primary and Secondary servers in the house (loft), with the intention to protect them from roof failure at a later date. I'm keeping an eye on temperatures as well and plan to put them in an insulated box with good ventilation. My new concern is energy conservation, with KWh prices of around $1 being possible in my country (UK) soon.

Show : The motive

This isn’t related to a data privacy concern, but data integrity. I’ve spent decades storing data on hard drives using Windows machines, a few times files have become corrupted. I’d quite like to pass on my data to the next generation if possible…if they even care about it!
Thru L1Tech, Lawrence Systems and Craft Computing I learned of TrueNAS, it seemed to be the logical solution, even if it’s not as turnkey as the alternatives (Synology, etc.) and very disk consuming. Again, this community was and continues to be immensely helpful.

Also, Google Drive doesn’t seem to like files exceeding 5GB (throws up sync errors) and Dropbox seems to be less immediate than it has in the past, potentially because it’s a free account that I’ve been using for many years. I’m constantly teetering on the edge of using up that dropbox 20GB storage limit, it’s really annoying having to play musical data every week or so!

Show : The data

It's around 12TB, at least half is films & TV Shows (a controllable amount, as I have all original media disks), the rest is personal with around 1-2TB of important work files. I've come to a realisation that what I really need is a larger working dataset than I have now, which is Google Drive (1TB) and Dropbox (22GB).
I would say that 1GB of work files is added every 2 weeks, so 26GB per year. Probably about the same amount of personal videos/photos are also added. The entertainment media is something I can control, but may be around 1TB per year not allowing for deletion.

Show : The latest plan (revision 3)

Server 1 (See signature)
3TB (Pool 1 - 2 x 4TB MIRROR) daily use, machine backups. Hourly snapshots locally, 2 hourly replication to Server2 (during office hours)
17TB (Pool 2 - 8 x 4TB RAIDZ2) media/other. Daily snapshots locally, 1 daily replication to Server2.
Server2 (See signature)
~3TB (Pool 1 to match the above, by either Mirror or RAIDZ). Destination for replications.
~17TB (Pool 2 as above). Destination for replications.
Server3 (off-site, well, in my office, hardware TBC)
Pool 1 to be of equal or greater size as the above Pools 1 & 2 combined. Destination for replication of Server2.

I would really like to buy or build the cheapest and lowest performance ECC machine possible.

Show : Into the future

I will likely, slowly replace the drives with larger capacity as a precaution towards increased needs, though I will keep an eye on usage for a while more before committing. Some family members may take up something data intensive, such as video related, so I'll just have to make good observations. I can't remember whether I've trial'd pool expansion, but I'm fairly sure it's a case of replacing all pool drives with larger drives and pressing the correct buttons.

Show : How I nearly did it, Revision 1

9 x 4TB RAIDZ2, but then I realised I'm making data available that's rarely accessed, spinning up that many drives for only 5% of that data being accessed, seemed like a waste of electricity and drive longevity.

Show : How I nearly did it, Revision 2

9TB Pool 1 - 5 x 4TB RAIDZ2 - Active
9TB Pool 2 - 5 x 4TB RAIDZ2 - Inactive
But then I figured that my video/film storage may outgrow a 9TB pool and then I'd be back to musical data, shifting things around.

Show : About the network and UPS

I've invested in SFP+ and will eventually have a 10G connection between house and outbuilding/office with a redundant CAT6 cable. So far as power is concerned, if it should have a UPS, it does have one.

Thank you for taking the time to read, a lot of my knowledge has come from this forum. I look forward to receiving any comments.

 

[sretalla]

It all looks/sounds pretty good/reasonable.

Nothing obviously wrong with it and you seem to have done the right thinking and research.

 

[Bikerchris]

It all looks/sounds pretty good/reasonable.

Nothing obviously wrong with it and you seem to have done the right thinking and research.

Wow, I honestly wasn't expecting that. Thank you very much for taking the time to read through it all.

I've just created the 8 drive pool. So impressive that I was able to transfer (via replication) around 4.5TB of files within ~6 hours.

 

[NugentS]

Interesting motherboard - that doesn't mean bad - but it is interesting

CPU has 40 PCIe lanes available and the chipset will have a few probably slower ones (I didn't check). Be careful when adding cards.

SATA Express - which I think is a dead technology, overtaken, surpassed and replaced by NVMe but I guess they will work as SATA ports - so thats good.

 

[Etorix]

Fair reuse of a workstation motherboard, but 2 out of the 10 SATA ports come from an ASMedia controller rather than from the Intel PCH: Check whether this controller is actually well-supported under TrueNAS…

 

[jgreco]

but 2 out of the 10 SATA ports come from an ASMedia controller rather than from the Intel PCH:

Are you certain? The X99 PCH does support 10 SATA. I can't pull up the frickin' manual at ASUS because of broken Javascript or something that the UTM doesn't like. But even if that fails, it looks like the ASUS PIKE II card is supported (basically an HBA). It certainly isn't the worst option for NAS repurposing.

Nothing obviously wrong with it and you seem to have done the right thinking and research.

I know, right? How damn rude. If people don't leave easy low hanging fruit to snipe at, then it becomes so much harder because the poster is actually asking us to *think* about it.... grumble ... annoying .. grumble ... more work ...

 

[sretalla]

Are you certain? The X99 PCH does support 10 SATA

I have 2 of those... they do. (although on my boards I use HBAs anyway just for fun)

 

[jgreco]

They do... what? Have an ASMedia controller? Support 10 PCH SATA? Dance a jig?

 

[Etorix]

Are you certain? The X99 PCH does support 10 SATA. I can't pull up the frickin' manual at ASUS because of broken Javascript or something that the UTM doesn't like. But even if that fails, it looks like the ASUS PIKE II card is supported (basically an HBA). It certainly isn't the worst option for NAS repurposing.

I'm certain I'm not certain of anything, but my favourite search engine brought me here:

X99-E WS | Servers & Workstations | ASUS Global

X99-E WS, a new Compact Electronics Bay (CEB) workstation motherboard based on the Intel® X99 chipset and with support for four-way multi-GPU setups at full PCI Express 3.0 x16 speed for extreme graphics power. X99-E WS is engineered with premium components for premium efficiency, including an...

www.asus.com

and under Specifications>Storage:

Intel® X99 chipset : *3
1 x SATA Express port, gray, , Compatible with 2 x SATA 6.0 Gb/s ports
1 x M.2 x4 Socket 3, gray, , with M Key, type 2260/2280 storage devices support (PCIE mode)
8 x SATA 6Gb/s port(s), gray, *4
Intel® Rapid Storage Technology supports*5
ASMedia® SATA Express controller : *6
1 x SATA Express port, gray, , Compatible with 2 x SATA 6.0 Gb/s ports
ASMedia® SATA 6Gb/s controller :
2 x eSATA 6Gb/s port(s), Compatible with 2 x SATA 6.0 Gb/s ports

*4 8 x SATA 6.0 Gb/s ports* (4 x gray from controller 1, 4 x black from controller 2)
*5 These functions will work depending on the CPU installed
*6 These SATA ports are for data hard drivers only. ATAPI devices are not supported

The PCH does support 10 SATA, but 2 of them are packaged as SATA Express; 2 of the 10 on-board genuine SATA ports come from an unspecified ASM chip.
The motherboard is certainly very capable as a NAS, but some caution and/or a little more research appears to be in order here.

 

[Bikerchris]

Interesting motherboard - that doesn't mean bad - but it is interesting

CPU has 40 PCIe lanes available and the chipset will have a few probably slower ones (I didn't check). Be careful when adding cards.

SATA Express - which I think is a dead technology, overtaken, surpassed and replaced by NVMe but I guess they will work as SATA ports - so thats good.

It certainly is, I originally used it as a workstation (CAD drafting, 3D Visuals, video editing), but something wasn't quite right about it, so I built another and left this one aside for a few years. Then learnt of FreeNAS (back then) and thought it might be perfect.

Thank you for the hint about the PCI lanes, I will remember that.

Yes, I've kept away from SATA Express ports as a precaution. Thanks!

 

[Bikerchris]

Fair reuse of a workstation motherboard, but 2 out of the 10 SATA ports come from an ASMedia controller rather than from the Intel PCH: Check whether this controller is actually well-supported under TrueNAS…

Well spotted. I've used those ASMedia ports and so far there's no issue, but I do keep an eye on which drives are connected to them. Thanks!

 

[Bikerchris]

Are you certain? The X99 PCH does support 10 SATA. I can't pull up the frickin' manual at ASUS because of broken Javascript or something that the UTM doesn't like. But even if that fails, it looks like the ASUS PIKE II card is supported (basically an HBA). It certainly isn't the worst option for NAS repurposing.

Hi jgreco - hope all is well with you and yours!

The manual will be with you shortly, if of interest.

I know, right? How damn rude. If people don't leave easy low hanging fruit to snipe at, then it becomes so much harder because the poster is actually asking us to *think* about it.... grumble ... annoying .. grumble ... more work ...

Haha, I do listen to those with significantly greater knowledge than I...especially the older I get

 

[Bikerchris]

Dance a jig?

Yes indeed, it is quite a powerful motherboard, significant overkill for my uses - but it's what I had already paid for, so re-used it is!

Here's all the doc's I have, if of interest @Etorix @sretalla @NugentS

 

[Bikerchris]

Well, I've been having a further re-think.

I may end up not having a 3rd server as an off-site backup, though I may review this in the future.

I'm turning to backblaze again, because if I isolate the data that is critical from the total amount of data, it would be quite affordable to use Backblaze.

Although I have around 12TB in total, I feel that it isn't justified to backup my 5-6TB of media/movies, so that gets it down to 6TB of actual important data, and that's a very affordable £330'ish. There is a transaction fee for, but at the moment it will only be a few pounds per year so it's negligible to be honest.

I have spent quite a bit of time testing Backblaze, it's actually very clever and I emulated the scenario where the server is just dead and all data is lost. I was easily able to take the details and restore the files to another machine.

If I had to download all data (6TB) back down to a replacement server, it would cost around £9-10 per TB. This is very affordable in the event of a failure, of course I would mostly lean on the snapshot machine first in the event of a problem.

 

[NugentS]

£330 pcm over 12 months is approx £4K.
Thats "wasted" money
Spend half of that on a small TN box (and the other half on electricity) and you will be quids in (in theory)

I do use B2 - but I store < 10Gb of documents / details - so it costs me nothing. I did get the config wrong at one point and have an outstanding bill of (from memory) $0.02 for the past 6 months or so

 

[Bikerchris]

£330 pcm over 12 months is approx £4K.
Thats "wasted" money
Spend half of that on a small TN box (and the other half on electricity) and you will be quids in (in theory)

I do use B2 - but I store < 10Gb of documents / details - so it costs me nothing. I did get the config wrong at one point and have an outstanding bill of (from memory) $0.02 for the past 6 months or so

It's £330 per year, hopefully that was made clear above. If that were per month, I would completely agree with you.

 

[NugentS]

Oops - must read it all first

 

[Bikerchris]

Oops - must read it all first

That's alright, I've been guilty of similar.

 

[Bikerchris]

Just wish to update, in case anyone is interested.

I'm in the process of changing things around a little, mainly due to vastly increased energy prices (UK).

The new plan involves:

A Daily Server consisting G4560 on Supermicro board (bought both now) with 2 x 4TB mirrored HDD's. Just waiting on 2x8GB ECC RAM in post.

A snapshot/cold storage server consisting i3-9100 on Supermicro board. 2 x 4TB mirrored that receives snapshots of the above, then 6-8 drives that will hold cold storage. (Server2 in sig)

A third server, re-using a power hungry CPU/board (Server1 in sig) that just consists of any old drives laying around and weekly snapshots the above server.

If funds permit, I'll have backblaze running on Server2, but it'll only backup critical files (around 3-5TB).