Adding a TrueNAS server to my Homelab, First build!

[BlarHxD]

Hi all!

So I have been lurking for some time now and decided I want to use a TrueNAS SCALE server for my Homelab/home setup. I have been reading all the recommandations and debats about hardware and pool configurations and I believe I have good idea it now. Therefor I want to show this forum my setup and see if there maybe could be made some improvements or good advice for future tinkering :)

Here is my system:

• OS: TrueNAS-SCALE-22.12.3.3
• Case: Fractal Design Define r6
• Motherboard: SuperMicro X11SSM-F
• CPU: Intel Pentinum G4600 (2C/4T)
• RAM: 64GB ECC DDR4 Micron memory
• HBA: Inspur LSI 9300-8i (Modded with a 40mm noctura FLX fan & IT-Mode)
• NIC: 2 x Intel NT364 4 Ports Gigabit Adapter (Used for iSCSI)

drive configuration:

Boot

• 1 x 128GB Samsung SSD

ESXi Datastore (iSCSI)

• 3 x MIRROR | 2 wide | 6 x 960GB SM863a Samsung SSD

Vault (Samba)

• 1 x RAIDZ2 | 6 wide | 6 x 6TB Toshiba MG04ACA600E
• 1 x L2ARC | 250GB Micron SSD

Few pictures of my system:

Please Note: I did re-order the layout of the drives, for better ariflow and temps are good now!

 

[NugentS]

Looks good, you have avoided all the major pitfalls - even the overheating SAS HBA. Congratulations

Do you really require that many NICs? Wouldn't 10Gb be better?

First comment - iSCSI from ESXi uses sync writes by default means that your iSCSI connections will be very slow. You have three choices:

1. Live with it. Test it and see. You can always do steps 2 or 3 later on.
2. Set the dataset / zvol to sync = disabled. This is as fast as it can go. BUT you are risking virtual disk corruption if you have a sudden unexpected power outage - you could lose up to 5 seconds of transactions - a potential disaster in a database or virtual disk.
3. Use a SLOG (using appropriate hardware - which has very specific requirements) to run sync-enabled but accelerated. This won't be as fast as sync=disabled, but is faster than sync = enabled. Note that as your pool is SSD's the difference won't be as much as if your pool was HDD's

Suitable SLOG Hardware:

• High endurance, a SLOG is always written to and never read from in a steady state
• Low latency in comparison to the pool its attached to
• High Speed in comparison to the pool its attached to
• Some form of PLP (Power Loss Protection) so that if the power goes out / kernal panic you don't lose the data in flight

Ideally a SLOG should have the same redundancy as the primary vdevs, in your case that would be mirrored. I don't - I run a single Optane 900p as a SLOG and accept the failure risk. If a SLOG fails during normal use then it doesn't matter - things will just slow down. The bad failure mode is if the SLOG fails as the system unexpectedly reboots which I consider relatively unlikley

Second Comment - the L2ARC will probably do very little (if anything for you). Leave it out to start with and then after a few week / month of normal operation check the arc hit rate and if thats > 90% the the L2ARC is probably a waste of time. Having said that its unlikley to do you any harm as:

• Its suitably sized
• You do have enough ARC to use it.

Future tinkering - more ARC (memory)

 

[BlarHxD]

Looks good, you have avoided all the major pitfalls - even the overheating SAS HBA. Congratulations

Do you really require that many NICs? Wouldn't 10Gb be better?

First comment - iSCSI from ESXi uses sync writes by default means that your iSCSI connections will be very slow. You have three choices:

1. Live with it. Test it and see. You can always do steps 2 or 3 later on.
2. Set the dataset / zvol to sync = disabled. This is as fast as it can go. BUT you are risking virtual disk corruption if you have a sudden unexpected power outage - you could lose up to 5 seconds of transactions - a potential disaster in a database or virtual disk.
3. Use a SLOG (using appropriate hardware - which has very specific requirements) to run sync-enabled but accelerated. This won't be as fast as sync=disabled, but is faster than sync = enabled. Note that as your pool is SSD's the difference won't be as much as if your pool was HDD's

Suitable SLOG Hardware:

• High endurance, a SLOG is always written to and never read from in a steady state
• Low latency in comparison to the pool its attached to
• High Speed in comparison to the pool its attached to
• Some form of PLP (Power Loss Protection) so that if the power goes out / kernal panic you don't lose the data in flight

Ideally a SLOG should have the same redundancy as the primary vdevs, in your case that would be mirrored. I don't - I run a single Optane 900p as a SLOG and accept the failure risk. If a SLOG fails during normal use then it doesn't matter - things will just slow down. The bad failure mode is if the SLOG fails as the system unexpectedly reboots which I consider relatively unlikley

Second Comment - the L2ARC will probably do very little (if anything for you). Leave it out to start with and then after a few week / month of normal operation check the arc hit rate and if thats > 90% the the L2ARC is probably a waste of time. Having said that its unlikley to do you any harm as:

• Its suitably sized
• You do have enough ARC to use it.

Future tinkering - more ARC (memory)

Thanks for the kind words :) I would like to believe that I did alot of research, before starting building my TrueNAS.

regarding the 10Gb, I don't have any network gear that supports it as of yet. I also run roundrobin ISCSI and all of my 3 ESXi hosts are direct connected, why I use that many NICs. I have just gotten a 16 ports switch, where I would in the future, I will remove some of the NICs and let the switch handle it.

also regarding 10Gb, I just got a hold on this system, equipped as this link states it: https://www.supermicro.com/en/products/system/4U/6048/SSG-6048R-E1CR36N.cfm
I do believe I need to get more HBAs, as the controller is a raid controller and it can't be used with TrueNAS. but damn its a nice controller that can be used for all 36 bays.

I must admit, that I do not fully yet understand the full value of a SLOG device yet and what kind of hardware should be used for it. I will properly need to look into this.

As for iSCSI with that sync setting, I find the VMs running on it, very responsive. But I will try to disable the sync setting and see what kind of performance I will get afterwards.

and lastly the L2ARC, my understanding were that if I access the same files often, that it would benefit me to have one attached to my RAIDz2. Please correct me if I am wrong.

SuperMicro X11SSM-F can only hold 64gb of memory :)

 

[sfatula]

and lastly the L2ARC, my understanding were that if I access the same files often, that it would benefit me to have one attached to my RAIDz2. Please correct me if I am wrong.

Thats what the ARC does, your 64GB ram. Most people seem to believe that they need a l2arc, most people do not. As mentioned earlier, it is unlikely to help you. I run 3 VMs and 14 apps at home on Scale, and, the same memory as you. My mean arc hit ratio for the past week is 99.46%. So, leaves very little for an l2arc to do, waste of time and effort. It will also consume some of your RAM.

 

[BlarHxD]

Thats what the ARC does, your 64GB ram. Most people seem to believe that they need a l2arc, most people do not. As mentuioned earlier, it is unlikely to help you. I run 3 VMs and 14 apps at home on Scale, and, the same memory as you. My mean arc hit ratio for the past week is 99.46%. So, leaves very little for an l2arc to do, waste of time and effort. It will also consume some of your RAM.

Alright :D then I will remove it and use it for applications/docker stuff for fun I guess :D

 

[BlarHxD]

First comment - iSCSI from ESXi uses sync writes by default means that your iSCSI connections will be very slow. You have three choices:

1. Live with it. Test it and see. You can always do steps 2 or 3 later on.
2. Set the dataset / zvol to sync = disabled. This is as fast as it can go. BUT you are risking virtual disk corruption if you have a sudden unexpected power outage - you could lose up to 5 seconds of transactions - a potential disaster in a database or virtual disk.
3. Use a SLOG (using appropriate hardware - which has very specific requirements) to run sync-enabled but accelerated. This won't be as fast as sync=disabled, but is faster than sync = enabled. Note that as your pool is SSD's the difference won't be as much as if your pool was HDD's

I have disabled the sync and I can tell, there were some peformance increase. Its blazing fast. I will test with some more VM actions, just see how much. It was fast before, but there is some performance to gain by disabled the sync setting.

Thanks for letting me know! :)

 

[NugentS]

@BlarHxD
Obligatory Warning: Setting sync=disabled does put your data at risk. if you get a sudden power event / kernal panic than you can lose transactions if they are in ARC and not flushed to disk.

Now that you know that sync writes are slowing you down you should consider whether a SLOG is appropriate. A SLOG will make sync writes faster than without a SLOG, but not as fast as async. They will however be safe as they never get left in RAM. Note however that not all hardware / SSD's will do. A decent Optane is good (Not an M10, H10, H20) 900p, 905p, 1600x, P4801X are all good. They do not need to be big though.

 

[sfatula]

Speaking of power events, you should add a UPS to your system for many reasons! One that works with NUT (Network UPS Tools).

 

[BlarHxD]

900p, 905p, 1600x, P4801X are all good. They do not need to be big though.

Yo those drives are hella expensive then I take my chances and trust in my backup solution for my VMs, plus they are easy to deploy.

Rather invest more into my samba pool.

But I do understand the risks now. Thanks for the informative explaination.

Speaking of power events, you should add a UPS to your system for many reasons! One that works with NUT (Network UPS Tools).

All in due time :) it's on my shopping list for my homelab. I will look into NUT :)