After year i realized a huge mistake... 60TB of storage all in one pool - no room for VM or anything else

[Avion]

Hi,

TL:DR: Created a pool with all available disks and had no idea that I could not split it up or that it is not "dynamically associated". Need to create VMs now and more pools. What is the best possible way to do this?

For years I wanted to build a home-lab to better store my video production and game-dev files.
Especially RAW Video and Photo Footage has been a space killer in the past.
I bought three refirbished intel servers but the supplier betrayed us and send us damaged hardware and claimed it was the mail service. (Three deliveries, all same damage, sued them, won the case but still got the damage and time-loss)
Instead I used a workstation pc and populated it with 1TB SSD and 5 10GB Iron Wolf HDDs. (the 6th was not available at the time so I had to use the 5...)

I used to be at that time a windows only user and wanted to have better solution for my desperate storage requirements, a VM server solution to get more deeply into coding (code, break, reset, code again) and for some rendering and game-dev / hardware dev related threads.

I had no idea (still no idea, just more thoughts) about truenas and how to plan properly.
I just ran through a bunch of tutorials, setup the hardware, setup the drives and installed truenas.
I created one pool and selected all disks and just ran with it, believing that the storage is dynamically managed and associated to the pools.

Now 1.5 years later I finally get to start other curiculums and setups and wanted to create VMs for development and so on...
I realized the mistake, when I wanted to create a pool for the VMs. (do I need to create a pool for each VM?)
And I could not select anything, no drive, no capacity etc. Quick research resulted in the realisation that I possibly messed up the whole system.

So here is my outreach:

The plan for the system (for now)

1. I want to have two servers. One in my home-lab one in my work-lab. I got one, and would buy the 6th drive to complete it.
2. They should function as backup for each other.
3. I would prefer to have drives that I could mount on windows and access on linux split up in
   1. Footage (Video and Photography large volume of RAW storage and library of my own footage (up to 20TB but for now 15TB is fine)
   2. Projects (Game Dev, Hardware Dev etc. but is fine with 5TB max
   3. Documents
   4. Software and images etc.
   5. Datasets for A.I. training that I can access from the VMs
   6. VM pool?

I do have 1TB of SSD (could upgrade to two 1TB Sticks) and 60TB of HDD space that I would use in Raid 5 or 6.

So the big question for now:
How do I solve my predicament?

Should I backup everything and complete delete the pool and re-install it?​

Can I reduce the size of the pool and split it up?​

Would you create one pool with defined sizes for each of the contents proposed in the 3. .. chapter?​

​

​

Thank you very much for your help, guidance and answers. I am seriously trying to up my understanding of the matter and improve. So be blunt, I will not take an offence ;)​

 

[Patrick M. Hausen]

You can divide the pool as you see fit - just create as many datasets as you like. You might want to read up on fundamental terminology - like pools vs. datasets. Look here.

 

[HoneyBadger]

Hey @Avion

So let's frontload this one:

TL;DR - You're probably fine.

The default behavior, if you assigned all five HDDs to a single pool in TrueNAS, will likely have picked some manner of RAIDZ, so you at the very least have at least single-drive fault tolerance. The "pool" is a single, large chunk of storage - out of which you can carve datasets - and if you need them to have capacity limits, the dataset can have a quota applied to it. But assuming it made a RAIDZ2, you'll have roughly 30TB of usable space - and by default, there's no "hard stop" on which sub-dataset can consume any amount of it.

But let's dig in a little bit here. I'm assuming TrueNAS CORE, since that's the subforum you've posted in.

Can you provide either a screenshot of Storage -> Pools -> (Gear Icon) -> Status or the output of System -> Shell -> zpool status?

 

[Avion]

Hey, Thank you do. I'll read up on the datasets! That would be great ;D And I'll wrap my head around on where to use what.

After fixing it I would certainly start planning / building out the system during this week.

The output:

NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

root@truenas[~]# zpool status
pool: MainData
state: ONLINE
scan: scrub repaired 0B in 1 days 20:56:02 with 0 errors on Mon Nov 20 20:56:02 2023
config:

NAME STATE READ WRITE CKSUM
MainData ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/7734e7b5-6efe-11ed-85a1-049226c24d06 ONLINE 0 0 0
gptid/77413b54-6efe-11ed-85a1-049226c24d06 ONLINE 0 0 0
gptid/772980c5-6efe-11ed-85a1-049226c24d06 ONLINE 0 0 0
gptid/77181496-6efe-11ed-85a1-049226c24d06 ONLINE 0 0 0
gptid/774c5a5c-6efe-11ed-85a1-049226c24d06 ONLINE 0 0 0

errors: No known data errors

pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:00:05 with 0 errors on Tue Dec 5 03:45:05 2023
config: dsafldaf

 

[Avion]

Sorry, could not edit the previous post. Mod is reviewing it. After both posts get cleared I'll merge them. For now:

 

[Patrick M. Hausen]

Your pool is fine and exactly as it is supposed to be. You can subdivide it into datasets for different purposes as you see fit. Feel free to come back with any more questions.

While you are at the task of "housekeeping" - did you share the entire pool e.g. via SMB and put files into the top level dataset? You should definitely correct that in the long term. The documentation states that after creating your pool you should create datasets for sharing and not put files simply "on the pool".

No biggie, your files are safe, but there's a bit of homework ahead cleaning up and moving files around. A lot of forum regulars are there to help, just take your time. Also we have of course time zone issues and personal schedules - so a conversation with a particular individual might just come to a halt for a couple of hours or even days while they are busy with something else.

The part of the documentation I referred to can be found here.

A TrueNAS dataset is a file system that is created within a data storage pool. Datasets can contain files, directories (child datasets), and have individual permissions or flags. Datasets can also be encrypted, either using the encryption created with the pool or with a separate encryption configuration.

It is recommended to organize your pool with datasets before configuring data sharing, as this allows for more fine-tuning of access permissions and using different sharing protocols.

Kind regards,
Patrick

 

[Avion]

Hey Patrick, big fat thanks! Since the origional post I have been diving into terminology, documentation and many many videos on zvol / dataset. Man, how could i miss that crucial knowledge in the beginning ;D

Thank you for the help. I did actually exactly that and mounted pool in windows as network storage and created folders etc. etc.
I certainly will do the housekeeping.

 

[ChrisRJ]

Since the origional post I have been diving into terminology, documentation and many many videos on zvol / dataset.

I would really caution against watching arbitrary YouTube videos when it comes to TrueNAS. The vast majority of them is downright dangerous or at least misleading. Seriously, please validate advice from YouTube at this forum. Or even better, read the resources in the forum instead. YouTube content is mostly entertainment and not qualified advice. There are exceptions (e.g. Tom Lawrence in the case of TrueNAS), but most are rather crappy.

I have compiled a list of resources over time and put it into my signature ("Recommeded readings"). This would be a good starting point. At the end of the day TrueNAS (and ZFS) is made for mission-critical storage in the enterprise. With that come a number of limitations relative to SMB or home-user systems (like unRAID, Synology, etc.) that need to be understood.

Hope that helps.

 

[Avion]

I would really caution against watching arbitrary YouTube videos when it comes to TrueNAS. The vast majority of them is downright dangerous or at least misleading. Seriously, please validate advice from YouTube at this forum. Or even better, read the resources in the forum instead. YouTube content is mostly entertainment and not qualified advice. There are exceptions (e.g. Tom Lawrence in the case of TrueNAS), but most are rather crappy.

I have compiled a list of resources over time and put it into my signature ("Recommeded readings"). This would be a good starting point. At the end of the day TrueNAS (and ZFS) is made for mission-critical storage in the enterprise. With that come a number of limitations relative to SMB or home-user systems (like unRAID, Synology, etc.) that need to be understood.

Hope that helps.

Jepp! I can certify this ;D
I did read quite a bit (blogs) and watched a bunch of sys-admin videos on true-nas but here is the catch: I watched it from the "home-lab" perspective and after two days of reading only documentation (dry but necessary) and great posts here in this forum I can say this: The videos I watched did not downright lie but they leave out critical concepts that I should have known before I setup everything and risk critical data for my company and my own ressources. Interestingly I am normally one to fall for this but it was setup in a rush (a rush beeing prepare for days but a lot went into the hardware side and hirarchy and storage guestimates etc) and even finding a software stack to suit it all. After that time I had not enough capacity to dive into the bits and pieces deeper. #should have though ;D

Thanks guys for the help. I think in the mid-term I will reset everything and build up again from scratch (summer next year) but for now I would then create datasets within the "main-storage" pool to organize the different storage and simple windows access related folders and one zvol for the VM. (right?) ;)

 

[Avion]

Hey, now I actually do got more questions or rather clarification on a subject:

I wanted to create datasets and move all files and folders into each corresponding dataset. Also I wanted to create the VM using the main pool. (I'll read up more on VM later... focussed on storage for now).

But: In the side menue in the gui under Storage --> there should be an option Datasets right? But there is not. When I create a dataset for the pool through the settings sub-menue on the right I can create a dataset but it is indistinguishable from the folders that I created within windows SMB (network drive mounted in this case):

Is this correct? I know now that Datasets are filesystems but are they so to say "folders with sets of permissions, rights etc.?"
Or have I maybe setup the pool not right so that the option does not appear?

Thanks again for the guidance here.

 

[Redcoat]

under Storage --> there should be an option Datasets right?

Click the three dots at the RHS. to find it:

 

[Avion]

Click the three dots at the RHS. to find it: View attachment 73529

Sorry should have made that clarification more clear ;). ^^^that^^^^ I found of course. But if I would like to "manage" datasets i.e. I can only see and edit it under Storage-->Pools and then in the file-system itself. But there is not overview of datasets etc. right? In the documentation its referenced as to be in the left side gui menue under storage, topic "manage datasets".

 

[Redcoat]

In the documentation its referenced as to be in the left side gui menue under storage, topic "manage datasets".

I can't find this. Can you screen copy it and paste it here, please, so I can understand what you are seeking?

 

[Patrick M. Hausen]

In the documentation its referenced as to be in the left side gui menue under storage, topic "manage datasets".

That's in TrueNAS SCALE, not in TrueNAS CORE. Two different products.

 

[Redcoat]

That's in TrueNAS SCALE, not in TrueNAS CORE. Two different products.

Thanks, Patrick. That explains my inability to find it!

 

[Avion]

Yepp, that actually created a bunch of confusion as I somehow switched accidentially into the TrueNAS SCALE docu for a couple of topics and could not really place it in my setup. That hint fixed a whole series of question marks. Currently i am combating an issue that leaves my ubuntu desktop vm without a cursor. But that is another issue / thread most likely unrelated to the original question postet here.

Thanks for the help! It help me to jump-start this process by a lot!