NVMe, SDD, HDD - how to bite this

[DamianS]

Hello,
I'm starting my adventure (actually more of a science) with NAS as a result I have a few questions, but before I start I'll introduce the HW (maybe it will be useful?):

MB: Asrock Z690 Extreme (2 NICs: 1G Intel I219V, 2.5G Realtek Dragon RTL8125BG)
CPU: Intel i5-13600K (14 cores - 20 threads, iGPU)
FAN: NH-D15
RAM: Corsair Vengeance LPX 64GB DDR4-3200 (I know - not ECC, but more on that later)
PSU: Seasonic Prime Ultra 750 W Titanium
CASE: Cooler Master HAF XB Evo Cube

and now a brief explanation of why only PC HW - once I know exactly what I need then I will buy the appropriate server-class components. I know that ZFS requires ECC RAM, but this is a "test environment" that will go to PC after my longer education.

On the software side, currently there is only TrueNAS, but eventually there is to be PROXMOX and under it TrueNAS, OPNsense and about 3 VMs. Also including (either in VMs or as plug-ins) PLEX (Audio only lossless Hi-Res: FLAC, DSF, DFF, etc. Video up to 4K 60FPS with DTS, DD, ATMOS audio), NextCloud (up to 6-8 devices), OpenVPN, AdGuard, SMB, NFS, etc...). Overall 1 "heavy" user and up to 4 "light" users.

And finally getting to the point: currently I can put 3x NVMe and 6-8x SATA into this MB, which I would like to move to the target NAS later. And here are my questions:
How to arrange these 3 NVMe most optimally?
Example:
1st m.2 (512GB?) for the system (currently under TrueNAS, but later PROXMOX):
- I have heard that both take up all the capacity and it is impossible to use the remaining space, right?
- how much should be the IOPS and TBW?
2nd m.2 (2TB?) for VM (TrueNAS, OPNsense and up to 2-3 Linux/Windows/others)
3rd m2 (2TB?) (???)
Here I would see Solidigm P44 Pro 512GB (TBW 0.5PB, good IOPS, DDR4 cache, good perf/power ratio, 256bit AES, 5 year warranty) and Seagate FireCuda 530 SSD (TBW 2.55PB and the rest like P44 Pro).

6x SATA - here I am considering 3 options:
A) 6x SSD like Crucial MX500 4TB or Samsung SSD 870 QVO 4TB.
B) 6x HDD of the likes of Toshiba Enterprise MG08ACA 12-16TB or Seagate Exos X - X18 12-16TB
C) 6x HDD 8TB + 2 SSD 4TB (???).

For A and B is it best to use RAID-Z2?

I know there is a big capacity spread (RAW: 24TB vs 72-96TB), but <> equal cost.
Based on current data, there would be up to 100GB of data to write in a peak per day - often much less (up to 30GB) with most of it to overwrite anyway. Generally much more data would be readable.
What if option A and the disks fill up?
What are your suggestions in both of these cases (NVMe and SATA) and why?

 

[Arwen]

I can't answer all of your questions, but here are a few answers;

ZFS does not require ECC memory, but if you care about your data, ECC memory is a good idea.

Virtualizing TrueNAS has some special requirements for stability and reliability. Here is a post on the subject. The main take away is always pass the disk controller to the TrueNAS VM.
https://www.truenas.com/community/t...completely-losing-your-data.12714/#post-59655

Yes, TrueNAS does not support sharing boot devices:

I have to waste an entire drive just for booting?

With the advent of TrueNAS Scale, we seem to be getting a large number of people from the Linux community who are not quite grasping what TrueNAS is. TrueNAS is not a "Linux distro" and it is not designed to try to meet your arbitrarily spec'd...

www.truenas.com

For your A-C storage options, it is both about what amount you need now, plus growth. And how much protection you need, (RAID-Z2 verses Mirrors).

Your option C does not specify why you want 2 x 4TB SSDs in in a 6 x 8TB HDD pool. The only sane reason for those big SSDs is ZFS Special Metadata vDev, but you list only 2. If you use RAID-Z2, then that suggests 3 x 4TB SSDs to have the same redundancy between the data vDevs and the Special Metadata vDev. Loss of either vDev is total pool loss, thus the desire to have the same level of redundancy with them.


Lots of new users to both TrueNAS & ZFS try to over-engineer their first TrueNAS server. Try using TrueNAS as a simple desktop VM with fake storage to get familiar with the GUI and command line.

 

[DamianS]

Thanks a lot for your response and sorry for my late reply.

Option C was such a loose option - I was more concerned with a possible suggestion from your experienced side....

I'm tentatively leaning towards option A - 6x SSD, as I don't need large capacities right now, and a scrub or possible array rebuild will always be faster on SSD than on HDD.
On 90% will be 1x vDev RAID-Z2 6x4TB, although I was also considering 2x vDev RAID-Z1 3x4T - performance-wise the 2nd option is better (2x IOPS) but the 1st is more secure (1x2 vs 2x1 Disk Failure).

Now I'm not 100% sure so please correct me if anything:
In case I need more capacity there are only 2 options:
1) replace the drives with larger 1 by 2 (and resilver or something like that after each)?
2) a second vDev but the same as 1 (6x4TB RAID-Z2)?
3) any other alternative?

And one last question:
I have a choice between SAMSUNG 870 QVO 4TB and CRUCIAL MC500 4TB (this one I prefer) - both similar on paper (2 is 30€ more expensive) - which according to you would be better and why.
I was also considering the TRANSCEND SSD230S 4TB, but it tends to fall off because it doesn't have HW AES, and I need to have some datasets encrypted.

 

[NugentS]

Do not use Samsung QVO - they are utter shite and belong in landfill.
To be fair - they are probably fine in a workstation - but not in a NAS. I haven't any personal experience of the others.

 

[DamianS]

	Kingston DC450R Data Center Series Read-Centric SSD 3.84TB	Crucial MX500 4TB
R/W:	560/520 MB/s (SLC-Cached)	560/510 MB/s
IOPS (R/W):	99k/26k	90k/90k
TBW:	2.82PB	1PB
MTBF:	2MH	1.8MH
HW:	256bit AES	256bit AES, TCG Opal 2.0

In addition to the aforementioned theory, I looked through a lot of different tests so as for me for the beginning of learning about NAS consumer ssd is sufficient.
As for Samsung - I have already encountered a similar opinion a few times regarding SATA SSD so something must be up....

 

[Arwen]

Thanks a lot for your response and sorry for my late reply.
...

You are welcome.

...
Now I'm not 100% sure so please correct me if anything:
In case I need more capacity there are only 2 options:
1) replace the drives with larger 1 by 2 (and resilver or something like that after each)?
2) a second vDev but the same as 1 (6x4TB RAID-Z2)?
3) any other alternative?
...

Yes, their are 2 ways to grow capacity:

• Replace all existing storage devices in a vDev with larger ones, one at time. A resilver will need to occur on each, before moving on to replace another.
• Add a second vDev. In general, one of same redundancy and number of storage devices is recommended. But, it is both possible and reasonable to change the vDev layout slightly.

I was also considering the TRANSCEND SSD230S 4TB, but it tends to fall off because it doesn't have HW AES, and I need to have some datasets encrypted.

Using HW encryption of a storage device, (aka SED = Self Encrypting Disk), has nothing to do with ZFS encryption. (Nor FreeBSD GELI encryption.) Thus, you comment does not make sense. If you want "some datasets encrypted", then it implies that others are not encrypted. So you would not be using SED, (or GELI), instead, you would use ZFS dataset encryption.

All encryption needs to be handled carefully. Loosing your password to TrueNAS is not fatal. With physical access to the computer, you can recover access and fix the password issue.

However, loosing the encryption password to ZFS dataset encryption is fatal. Their are no back doors. Nor known methods to "crack" the encryption easily. Loose the encryption password, and you have created a self inflicted ransomware with no ability to pay for recovery.

Loosing the encryption password to a single SED in a pool with 1 disk of redundancy is not fatal. But, you have to consider the device failed and replace it. (And replace it before another storage device failure.) Loosing all the encryption passwords to all SED in a pool IS FATAL.

Further, encryption does not prevent on-line access. Meaning whence you enter the encryption password and the storage becomes available, if a hacker gets in, all your data is available. At-rest encryption is meaningless to prevent on-line access.

Thus, unless you have specific needs, encryption is not recommended.

 

[DamianS]

THX again,

I thought that in order to have the entire drive hardware-encrypted it needed a version typically labeled SED. On the other hand, as for AES I thought that it only "accelerates" de/encryption of this "built-in" in ZFS... Holy sh*t...
For legal reasons, I need to have certain documents encrypted on the drive in case of theft - no matter if it's a dataset or an entire drive.
From what you wrote I conclude that it is "better" to have the whole disk encrypted.
For the sake of clarity - can you write me what are the differences between SED, HW AES and TCG OPAL if I need, for example, to have the whole disk encrypted? Can I do it from TrueNAS lvl or on a PC with an app from the manufacturer?

 

[Arwen]

Here is some info:

• Modern CPUs have AES / Encryption helper instructions to speed up de/encryption. This has nothing to do with storage or network. But, it CAN be used to encrypt storage data or network traffic.
• SED is for Self Encryption Disks. Before they can be used, you have to send them the password. Either from the BIOS, EFI or boot loader. Thus, if used before ZFS, less need for ZFS encryption. However, since the SED firmware is not available for security reviews, their is potential for bugs, data loss errors or out right back doors.
• ZFS supports top level dataset encryption, which can be inherited by all children datasets if desired.
• OPAL is just part of SED.

If you need everything encrypted, it is tricky. ZFS dataset encryption does not encrypt metadata, so some information "leaks". Most people would not have a problem with this, because it is necessary for ZFS to perform Scrubs, Re-silvers, or Sends.

None of this encryption would protect you if a hacker got in while the file system is mounted. Meaning SED & ZFS dataset encryption are "at rest". If you NAS was stolen, it almost certainly has its power cord pulled. Thus, on boot up, the thief would not have access to the encrypted SED or ZFS datasets. Unless they had the passwords too.

One last note, ZFS supports 2 methods to unlock datasets. If I remember correctly, the key based automatically un-locks the ZFS dataset on boot. This is good for disk disposal or returning a disk to the manufacturer for warranty replacement. Whence a disk is removed, any data encrypted remains encrypted. You MUST manually backup your ZFS encryption key, or risk getting locked out on boot drive failure.

The other ZFS method to unlock datasets is password based. On every reboot, you would have to enter the password before any encrypted datasets could be used. Of course, DON'T forget your password!


One last thing. Some people can get away with encrypting individual files. This allows you to pick and choose which files should be encrypted. It might even make sense to setup a separate directory / folder, or even a ZFS dataset, for that purpose. Thus, easier to review for security compliance.


I highly recommend you clearly understand what needs to be encrypted, how it is done, (including recovery / passwords), and what your options are with TrueNAS. We have had a few people setup encryption and "loose" the key or password. Thus, loosing the data. Some even lacked backups, so it was truly gone for good.