Maybe Over Engineered structure of datasets, users, groups

[Buddinski88]

Hello all,

I just reinstalled TrueNAS Scale for the second time, as I wanted to "play around" a bit with the first installation.
Currently I seem to be a bit stuck as I am no longer sure how to split/create the datasets.

Drives	Data-Pool	Dataset Level 1	Dataset Level 2	ZVOL	Groups	Users	Info
2x 1TB NVME mirror	boot-pool						Systempartition
	nvme-pool			vm-drives
3x 14TB HDD RAIDZ1	data-pool	backup	clients		services	backup-clients	Windows, MacOS
			systems		services	backup-services	Proxmox, Home Assistant, Homematic
			general		admin	buddy	old external drives
		homes	buddy		family	buddy	personal files
			katrin		family	katrin	personal files
			family		family		family related files, where all groupmembers should have access
		media	movies, tv-shows		services, family		maybe also based on plex or jellyfin
3x 1TB SSD RAIDZ1							Maybe L2ARC?

Probably I'm trying to complicate everything, because for example it was not possible to make the dataset "buddy" accessible via SMB.

I understood that it is possible to create a home directory per user, but this is then not in the dataset, but in the paths known from Linux (usr/buddy/...)?

Does my structure make sense at all? I would like to create different snapshot tasks per dataset.

I have watched quite a few videos (e.g. from Lawrence Systems or SpaceRex). Also I have read the official documentation, but I am - as I said - stuck.

Your help, corrections would be really appreciated!

Many greetings and many thanks,
Buddinski88

P.s. I'm not even sure if it's right in this section, since it's more about a configuration than an installation.

 

[neofusion]

Are the VM ZVOLs and boot-pool on the same NVMe mirror?
The OS will not see much benefit from being on media that fast and that your current configuration means that a reinstall of the OS, something that would otherwise be a trivial matter if you have a backup of your config, is suddenly a fair bit more involved.

My advice, knowing nothing about your other hardware, would be to install the OS on two of the SSDs instead (presumably you have the drives already, otherwise pick smaller drives). Dedicate the NVMe:s to VM:s only, and consider using the last SSD for L2ARC.

Edit: Clarified a sentence.

 

[ChrisRJ]

L2ARC only makes sense for very specific workloads. Even then it requires a certain amount of RAM.

As @neofusion mentioned, you missed to specify your hardware. Please check the forum rules (linked in red at top of the page). We need this information to give proper advice.

Also, the information that you intend to "run VMs" is not enough. What programs? What number of IOPS required? What performance do you expect?

 

[anodos]

I'd avoid nesting datasets within SMB shares.

 

[sfatula]

So, as an example of what @anodos is saying, look at media. You have dataset(s) under it. Well, media is media, just create a media dataset at what you call level 1 and the other stuff is simply subdirectories. Media should have 1k (edit: No, not 1k, that would be very very bad if you could, I mean 1M!) recordsize most likely.

I actually use a structure that most on the forums here would likely hate. But it works for me! I segregate top level datasest based on backup intent. So, I have a dataset for daily/important stuff to backup, a dataset for archival stuff which is backed up less often (but is pretty large too), and a nobackup dataset as I don't care if I lose anything on it. Makes backup configuration easier and while a single app might exist on all 3 of those, and it's therefore not as pretty as having say all stuff for jellyfin under jellyfin, it gives a lot of value to me. I have a lot of stuff in nobackup! That sounds bad but once you start looking at the stuff you have, a lot of apps have workspace, say transcoding space in jellyin, temp files of various kinds, I actually put Emby (similar to jellyfin) DVR recordings there as they are simply too big to be backed up as they change every day as I record more stuff, etc. etc. But I'm always different, lol.

 

[ChrisRJ]

Well, media is media, just create a media dataset at what you call level 1 and the other stuff is simply subdirectories.

I like the idea and it's great if it works for you that way. At least for me things are a little bit different. Under media I have separate datasets for photos (not possible to re-create) as well as my CDs (can be ripped again).

 

[Buddinski88]

Are the VM ZVOLs and boot-pool on the same NVMe mirror?
The OS will not see much benefit from being on media that fast and that your current configuration means that a reinstall of the OS, something that would otherwise be a trivial matter if you have a backup of your config, is suddenly a fair bit more involved.

My advice, knowing nothing about your other hardware, would be to install the OS on two of the SSDs instead (presumably you have the drives already, otherwise pick smaller drives). Dedicate the NVMe:s to VM:s only, and consider using the last SSD for L2ARC.

Edit: Clarified a sentence.

Exactly the Boots SSDs are mirrored and divided into two partitions (or 3 with SWAP).
Partition 1 = System
Partition 2 = Swap
Partition 3 = Thought for the ZVOLS

I want to virtualize e.g. a proxmox server, which acts as a third node in my cluster.

I have not thought about the reinstallation of TrueNAS. The question how often does this occur while running? With the current version of SCALE I should only have to make updates?

L2ARC only makes sense for very specific workloads. Even then it requires a certain amount of RAM.

As @neofusion mentioned, you missed to specify your hardware. Please check the forum rules (linked in red at top of the page). We need this information to give proper advice.

Also, the information that you intend to "run VMs" is not enough. What programs? What number of IOPS required? What performance do you expect?

Thanks for an answer. With the cache I'm not sure yet. As I said, the plates were available and I think you could "do" something else with them. Is there a recommendation for this?

Virtualize I would like, as I said for example Proxmox wherein then again other issues like a Docker VM or Homeassistant VM runs.

This is now not the current state of my system environment, but perhaps that also contributes:

In general, I made a lot of progress yesterday. The backups are now properly stored on the NAS and can also be restored.

Regarding the home directories I also managed to do it. There is now the structure shown in the first post. So I can also create a separate snapshot task for each dataset.

In my signature is now also the complete description of my NAS

I thank you all for the feedback and hope that you will accompany me further to make no major mistakes.

 

[neofusion]

So, as an example of what @anodos is saying, look at media. You have dataset(s) under it. Well, media is media, just create a media dataset at what you call level 1 and the other stuff is simply subdirectories. Media should have 1k recordsize most likely.

Did you mean record size 1MB?

Exactly the Boots SSDs are mirrored and divided into two partitions (or 3 with SWAP).
Partition 1 = System
Partition 2 = Swap
Partition 3 = Thought for the ZVOLS

I want to virtualize e.g. a proxmox server, which acts as a third node in my cluster.

I have not thought about the reinstallation of TrueNAS. The question how often does this occur while running? With the current version of SCALE I should only have to make updates?

Should, yes.
Sometimes things don't go as you hoped.

Even if you only need to make updates, no one is testing that things are A-OK for an update with that setup, it unlikely to be one of iXsystems test cases.

 

[sfatula]

Did you mean record size 1MB?


Should, yes.
Sometimes things don't go as you hoped.

Even if you only need to make updates, no one is testing that things are A-OK for an update with that setup, it unlikely to be one of iXsystems test cases.

Hah, yes, of course!

 

[Buddinski88]

Did you mean record size 1MB?


Should, yes.
Sometimes things don't go as you hoped.

Even if you only need to make updates, no one is testing that things are A-OK for an update with that setup, it unlikely to be one of iXsystems test cases.

That unsettles me a bit now. I think I have to set it up differently.

Probably two smaller NVME drives make more sense.

What would be your advice for the three remaining 1TB SSDs? As I said, I'm not completely through with the cache topic yet and honestly, the longer I read about it... it doesn't make sense with my setup.

I think if it does then it makes sense to increase the RAM in the future?

 

[sfatula]

More RAM almost always better than l2arc.

Truenas configures some swap space, no need to manually do so.

If you need the space, the 3 SSDs could be the VM space, and whatever is left, there is usually need for temporary files space, work files space, stuff you wouldn't need to back up. I ended up with almost 700GB of space just for work files, temporary files, etc. Not sure what VMs you will be running though. I have a Windows history zvol for versioned file history too.