New to TrueNas - Some questions before I get started.

[panzerscope]

Hello all,

New to this community and I imagine I will be posting more questions as time goes on.

Long story short, I am transitioning from a QNAP NAS to a TrueNas system. My workstation for this upcoming build is as follows:

HP Z840 Workstation
CPU: 2x Xeon E5-2696-v4
RAM: 128GB DDR4 ECC
HDD's: 4x 16TB Toshiba MG08ACA16TE Drives
SSD's: 2x 1TB Firecuda Drives

I will be using this TrueNas system to:

1.Act as my NAS
2.Host Plex
3.Virtualize Windows environments
4.Backup NAS data to a remote source (Dropbox)

Before I start I have a few questions:

1. TrueNas Core and TrueNas Scale, Which one ? I did some reading but still have not come to a conclusion as to which best suits my needs or if one would give me any advantages over the other.
2. If I were to swap out my network card or GPU is this a headache or will TrueNas simple detect the new hardware and run it with no particular issues (Assuming its supported)
3. I am wanting to run a RAID10 like system with Caching via the SSD's. I know TrueNas does not use typical RAID configurations, so it is new to me. How would this configuration look like in TrueNas. Is there a decent guide on it ?

Thanks in advance.
P

 

[NugentS]

1. Core for Stability, Scale for probably better virtualisation (eventually)
2. Should just work
3. Striped Mirrors = RAID10. Set up a 2 drive vdev as a pool, then add another 2 drive vdev. Easy peasy

 

[panzerscope]

1. Core for Stability, Scale for probably better virtualisation (eventually)
2. Should just work
3. Striped Mirrors = RAID10. Set up a 2 drive vdev as a pool, then add another 2 drive vdev. Easy peasy

Thanks, regarding point 3. Do you setup caching as part of the pool process, or is this something configured after the striped mirrors pool is created ?

 

[NugentS]

What do you mean by caching?
Sorry - just want to know if what you think of as caching is what ZFS thinks of as caching? There are two types (plus more subtypes), one is read caching, one isn't write caching, but is to do with speeding up some writes - but it isn't caching.

Either way - this can be added at any time. Do not do it initially. You need to test your initial performance and then make a decision.

BTW - what are you booting from?

 

[panzerscope]

What do you mean by caching?
Sorry - just want to know if what you think of as caching is what ZFS thinks of as caching? There are two types (plus more subtypes), one is read caching, one isn't write caching, but is to do with speeding up some writes - but it isn't caching.

Either way - this can be added at any time. Do not do it initially. You need to test your initial performance and then make a decision.

BTW - what are you booting from?

Correct, what I meant by caching was the increased performance in write/read speeds. Good point though on waiting to asses the standard HDD performance first.

Sorry, not sure why I

 

[panzerscope]

I cant edit my last post, for some reason it posted. My boot drive is an Intel NVMe 670P. I should also have mentioned my NIC will be an Intel X540-T2

 

[NugentS]

A "cache" will not improve your "read/write" speed. Its not like an unraid cache both types work a bit differently.
To "improve" reads you use an L2ARC and with the amount of memory you have - this is practical. It only works however IF you are "over-using" the ARC (RAM) in the first place (thats a very simplistic explanation). To improve writes you use a SLOG, but only if you are using sync writes, which is by default iSCSI and NFS. You have 128GB of RAM (ARC) which will hold a lot of data
I have no idea what a hosted VM on TN Core or Scale uses by default.

You are proposing Mirrors across 4 drives which will give you fast resilver times and is as many IOPS as you can get from HDD's (not that many). I would look at putting the VM's on the 1TB Firecuda Drives as a seperate pool and not worry about the mirrored HDD's

 

[panzerscope]

Hey guys,

So I have been having a good play. No need for L2ARC as my hit rate with ARC is 98%, so all good there so far as I can see. So I was looking at putting the SLOG of a mirrored pair of SSD's. I note that it is best to OP the drives to increase the lifetime. I have read up on how to do this but I wanted to be clear whether you create the SLOG VDev and then configure the OP, or if it is the other way round ?

Many thanks,
P

 

[NugentS]

In the GUI - you can't overprovision (unless someone corrects me) - TN GUI uses the entire disk, wiping any partition table and creating its own
Some SSD's have manufacturer tools that overprovision the disk - and just makes the entire disk look like a smaller model. This may work with the TN GUI (I haven't tried).
The other option is to create smaller partitions on the SSD's - and then create the pool at the commandline. Its how I use 3 * 20GB partitions on mirrored optane's 900p's as SLOGs on three different pools. This is a lot easier to do in Core than Scale (which was a ballache - although to be fair there were probably easier ways than I did it). Warning "Here be Dragons" this is very unsupported and may cause issues (it shouldn't.......). For instance, swapping one of the Optane's for another caused me no end of grief and a complete NAS rebuild when I screwed up. Also much easier to do in Core than Scale.

If you are using SSD's as data disks then its probably not worth overprovisioning - just live with the fact that you will use them up and have to change them from time to time and buy decent SSD's. [Don't buy and use Samsung QVO's for this]
Or buy enterprise drives which will last a lot longer (and cost a lot more)

Remember SLOG only effects Sync writes which is mostly NFS and some iSCSI (both of which should be sync writes anyway if dealing with VM's)

As for your proposed SLOG
At 10Gb - your SLOG needs to be about 6-7 GB (I use 20 for safety)
Also your Firecuda drives are not suitable as a SLOG - they don't (to my knowledge) have PLP which is absolutely nessesary for a SLOG to do its job. Technically my Optane's don't either - but I consider it close enough to take the risk (and I am not buying better - ouch)

 

[panzerscope]

Thanks for the info. I looked at the Optane options but not totally sure my system specs (In OP) will actually support that technology. Still searching for a clear answer on that. Are there Optane alternatives with high endurance and power backup ?

 

[NugentS]

Thanks for the info. I looked at the Optane options but not totally sure my system specs (In OP) will actually support that technology. Still searching for a clear answer on that. Are there Optane alternatives with high endurance and power backup ?

Yes.

I use 900p in U.2 format with a PCIe *8 Card and a bifurcated PCie3*8 slot. These almost have PLP & very high performance & endurance
Technically they don't have PLP - but optane memory doesn't lose data without power (can't remember the proper word) so I only lose data in transit across the Optane if the power suddenly goes out. My data, my risk.

The P4800x range does officially have PLP - but does cost a bit more (ok, a lot more than a bit)

If you are putting SLOG's on a Pool of HDD's then you could do a lot worse than an Intel DC S3710 (or 3700) which is a high endurance SATA drive (10DWPD) and PLP. Second hand off ebay - only need a small one.

BTW - as long as you have the PCIe lanes then you can use Optane. Don't pay any attention to Intel garbage that the system needs to support Optane - that is typical Intel bullcrap. But you do need PCIe lanes, and bifurcation if connecting more than one card to a single slot, or use a PEX/PLX card that doesn't need bifurcation but is more expensive.

Optane 900 comes in either AIC or U.2 - get the U.2 which makes more options available to you re multiple cards in one slot.
Optane 4800 comes in AIC, U.2 and I think in gumstick.
Note that the 900P has been discontinued - but surplus stock is still available.

Other options are the Radian Memory Systems RMS200/RMS300 - but they are difficult to get hold of
Older devices are ZeusRAM - but they are a bit long in the tooth now and are SATA / SAS based (I think) so only good for an HDD Pool

With your Z840 it will come down to how many PCIe slots and available lanes you have. Each CPU has 40 lanes so it should come down to physical size of the unit and wether you can physically fit stuff.

 

[panzerscope]

Yes.

I use 900p in U.2 format with a PCIe *8 Card and a bifurcated PCie3*8 slot. These almost have PLP & very high performance & endurance
Technically they don't have PLP - but optane memory doesn't lose data without power (can't remember the proper word) so I only lose data in transit across the Optane if the power suddenly goes out. My data, my risk.

The P4800x range does officially have PLP - but does cost a bit more (ok, a lot more than a bit)

If you are putting SLOG's on a Pool of HDD's then you could do a lot worse than an Intel DC S3710 (or 3700) which is a high endurance SATA drive (10DWPD) and PLP. Second hand off ebay - only need a small one.

BTW - as long as you have the PCIe lanes then you can use Optane. Don't pay any attention to Intel garbage that the system needs to support Optane - that is typical Intel bullcrap. But you do need PCIe lanes, and bifurcation if connecting more than one card to a single slot, or use a PEX/PLX card that doesn't need bifurcation but is more expensive.

Optane 900 comes in either AIC or U.2 - get the U.2 which makes more options available to you re multiple cards in one slot.
Optane 4800 comes in AIC, U.2 and I think in gumstick.
Note that the 900P has been discontinued - but surplus stock is still available.

Other options are the Radian Memory Systems RMS200/RMS300 - but they are difficult to get hold of
Older devices are ZeusRAM - but they are a bit long in the tooth now and are SATA / SAS based (I think) so only good for an HDD Pool

With your Z840 it will come down to how many PCIe slots and available lanes you have. Each CPU has 40 lanes so it should come down to physical size of the unit and wether you can physically fit stuff.

Thanks very much. Very helpful. You are right, the P4800x is somewhat more expensive. The 900P is much more affordable and likely still my best bet. Intel has been known to spout crap and I am not surprised we are seeing it again.

I am only using 3 add in cards in my system, so I should have enough PCIe lanes and slots.

Bifurcation is a possibility as it is supported in the BIOS but I assume I could simply have two separate 900P's in different slots. Main aim to have two in a mirror for redundancy.

 

[NugentS]

You can indeed have 1 each in different slots. Bifurcation just means you can stuff 2 into an x8 slot, with the right kit.
In your situation I would probably use a Intel DC S3710 (its a whole hell of a lot cheaper) as you are using a pool of HDD's, an Optane is probably overkill. Hell, even a little M10 16GB Optane (Not the H10) might help - although they are somewhat challenged technically

Also - you don't NEED a mirror. A SLOG is NOT pool critical. If it fails then the pool just goes back to using the ZIL on the pool (AIUI). Mind you - you might just notice the performance drop

 

[panzerscope]

You can indeed have 1 each in different slots. Bifurcation just means you can stuff 2 into an x8 slot, with the right kit.
In your situation I would probably use a Intel DC S3710 (its a whole hell of a lot cheaper) as you are using a pool of HDD's, an Optane is probably overkill. Hell, even a little M10 16GB Optane (Not the H10) might help - although they are somewhat challenged technically

Also - you don't NEED a mirror. A SLOG is NOT pool critical. If it fails then the pool just goes back to using the ZIL on the pool (AIUI). Mind you - you might just notice the performance drop

Thanks for the recommendation. So you think I would see a performance benefit from the S3710 just based mainly on the IOPS ? . I know the 900P is possibly overkill for a HDD array setup, it was simply attractive due to its Nvme nature being faster.

Thanks for all your help

 

[NugentS]

Consider what a SLOG actuially does.
Reads are not effected
Writes are written to the ZIL on the SLOG, and the ZIL in memory. Once they are written to the ZIL on the SLOG the transaction is complete and that is flagged back to the writing client. The NAS will at a later date flush the ZIL in memory to the permanent store as part of a larger transaction. The ZIL on the SLOG is never (under a steady state) read, its always written to.

A SATA HDD can do maybe 150MB/s(on a good day) with relatively high latency
A SATA SSD can do maybe 550MB/s almost all day with next to zero latency
You will notice a difference.

I recently bought 2 *200GB DC S3710 at £23.65 each on ebay. 100% lifetime left - so barely used. Mind you that was a remarkably good deal.
The DC S3700 would do just as well: https://www.ebay.co.uk/itm/295021924529?hash=item44b0ad50b1:g:d20AAOSwrghim7hA. I believe they are marginally slower than the 3710. Obviously the Optane 900P or up will be considerably faster & considerably better.

 

[panzerscope]

Consider what a SLOG actuially does.
Reads are not effected
Writes are written to the ZIL on the SLOG, and the ZIL in memory. Once they are written to the ZIL on the SLOG the transaction is complete and that is flagged back to the writing client. The NAS will at a later date flush the ZIL in memory to the permanent store as part of a larger transaction. The ZIL on the SLOG is never (under a steady state) read, its always written to.

A SATA HDD can do maybe 150MB/s(on a good day) with relatively high latency
A SATA SSD can do maybe 550MB/s almost all day with next to zero latency
You will notice a difference.

I recently bought 2 *200GB DC S3710 at £23.65 each on ebay. 100% lifetime left - so barely used. Mind you that was a remarkably good deal.
The DC S3700 would do just as well: https://www.ebay.co.uk/itm/295021924529?hash=item44b0ad50b1:g:d20AAOSwrghim7hA. I believe they are marginally slower than the 3710. Obviously the Optane 900P or up will be considerably faster & considerably better.

Thanks very much for the info. Very helpful. After looking around on ebay, I was able to find an Optane 900P (280GB) for £175, this will clearly manage all my needs. My TrueNas system runs on a UPS, so the power loss protection, or lack thereof on the 900P is not a deal breaker.

 

[NugentS]

Thats largely what I thought

 

[ThreeDee]

IF you go with Core .. do manual installs of Plex and whatever else you want to run instead of using plugins

..that's all I got, lol

 

[panzerscope]

So I received the Optane 900P today. Popped it in the system and all is good. However when creating the VDEV for Slog it gives me the warning that it is not in a mirror configuration, which is fine. It gives you the option to acknowledge that and to force the creation of the pool. However when I agree to the warning and continue, the "Create" button remains greyed out and I cannot continue. Something I am missing ?

Thanks guys.

 

[NugentS]

Shouldn't you be adding it to a pool, not creating its own pool