Boot, SLOG and special vdev on the same mirror?

[Jamberry]

So after my previous sanity check post about my home lab, I have now decided to leave the VM storage on Proxmox and only use TrueNAS for files and stuff.

I got a new idea but I am not sure if it is dangerously stupid

These 1TB PLP drives from Kingston seem pretty solid. Why not use two of them for boot? Why not make another partition for SLOG? Heck, why not add a third partition for special vdev?

Something like:
Partition 1 / Boot / 40GB
Partition 2 / SLOG / 16GB
Partition 3 / special vdev / 700GB
and around 200GB for over provisioning. Not sure if this is really needed, because Partition 3 should not grew above 75% anyway as far as I understand it.


I saw in the forum, that boot + slog is not officially supported. So another option that would work for me is a 2,5" single SSD for Boot and two nvme drives for SLOG and vdev. Would that be supported without any hacks?

 

[Davvo]

Gosh, why?
SLOG without power protection is useless.
If you lose a special vdev, you lose your entire pool.

This is a very very bad idea.

I have to waste an entire drive just for booting?

With the advent of TrueNAS Scale, we seem to be getting a large number of people from the Linux community who are not quite grasping what TrueNAS is. TrueNAS is not a "Linux distro" and it is not designed to try to meet your arbitrarily spec'd...

www.truenas.com

Some insights into SLOG/ZIL with ZFS on FreeNAS

What is the ZIL? POSIX provides a facility for the system or an application to make sure that data requested to be written is actually committed to stable storage: a synchronous write request. Upon completion of a sync write request, the underlying filesystem is supposed to guarantee that a...

www.truenas.com

 

[joeschmuck]

Why not make another partition for SLOG? Heck, why not add a third partition for special vdev?

Because it's not supported by TrueNAS.

Could you do it? YES
Should you do it? NO

Myself and other have played around with that and while it will work, if you upgrade your system then the partitions may not be there when you are done. So we do not recommend it.

Why would someone want to use a 1TB SSD as a boot device? Use the least expensive drive you can find and make a backup of your configuration file anytime you make changes to your system.

 

[Etorix]

Dangerously stupid it is. But you've added a new layer to the plot by mixing SLOG and special vdev, that is two different critical components for pool integrity, making it even more dangerous that the "usual" boot+slog or boot+data propositions.

Using partitions rather whole drives is not supported by True NAS for ANY purpose, or any combination. Just accept it, and get suitable drives for each purpose. SLOG (assuming you need sync writes and have a use for it) requires PLP, low write latency and high endurance: No need to pay for 1 TB drives, unless you're using the capacity for extreme overprovisionning, and no need to mirror in home/lab use. Special vdev lives on regular SSD, no need for PLP. Boot will do with the cheapest small SSD you may find (any $9.99 Optane M10 16GB to be found?); no need to mirror either. The whole set may end up being cheaper than two big PLP SSDs.

 

[c77dk]

Remember to have at least the same level of redundancy on your special vdevs as on your data drives. As @Davvo wrote: dead sVDEV == dead pool.

And to follow the others: you can, but it's quite dangerous.

 

[Jamberry]

SLOG without power protection is useless.

Well, it does have power protection. That is why I wrote PLP

Because it's not supported by TrueNAS.

That is a valid point and also my main concern. But is SLOG and special vdev on the same disk also not supported?

There is some reasoning behind why I would love to combine these two. The PLP drives are not that much more expensive than normal drives. They use m.2 and because of that are very fast. And they have good endurance numbers. Also most inexpensive supermicro motherboards offer only two m.2.
If I follow the best practice guide I need, 1 SATA for boot, 2 SATA for special vdev and one m.2 for SLOG. That is a lot more than just two m.2

But I guess

Using partitions rather whole drives is not supported by True NAS for ANY purpose, or any combination.

will seal the deal for me to not do it.

To be honest I don't need that performance really. Sure offloading some stuff to iSCSI would be nice but honestly I just need an excuse to experiment with SLOG and special vdevs

 

[Davvo]

Well, it does have power protection. That is why I wrote PLP

Missed this, my bad.

But is SLOG and special vdev on the same disk also not supported?

Multiple partitions, of any kind, are not supported.
If you have a special vdev composed of single disk and it dies, your pool dies no matter the redundancy of the other vdevs. End of the story.

 

[Jamberry]

If you have a special vdev composed of single disk and it does, your pool dies. End of the story.

They would be mirrored.

But when you say:

Partitioning drives, of any kind, is not supported.

I can trash this idea.

 

[Jamberry]

(any $9.99 Optane M10 16GB to be found?

Holy cow, I did not realize these are that cheap on ebay! Cheapest new one I could find was a 250GB for 300$ in Europe.

 

[Etorix]

Holy cow, I did not realize these are that cheap on ebay! Cheapest new one I could find was a 250GB for 300$ in Europe.

I've found some for 9.99E locally, and hoarded half a dozen. Have a look on ricardo.ch…
Note that M10 are "cheap" consumer Optane intended to serve a fast cache to HDDs; they are strangled to a x2 PCIe link and have somewhat limited throughput and endurance, especially in the lowest 16 GB capacity. Perfect boot drives at that price, would not make a good SLOG (the 64 GB variant maybe, but best go for 900p or DC P4800X for SLOG use).

Speaking of SLOG, do you really need one? Do you have a workload with mandatory sync writes? (Reminder: SLOG is not used for async. Sync writes are an order of magnitude slower than async; a dedicated SLOG helps, but does NOT close the gap.)

 

[joeschmuck]

But is SLOG and special vdev on the same disk also not supported?

Nope. When you add an SLOG TrueNAS does not give you an option to partition the drive for any other use. But again, while you could manually make this happen, it's not recommended. When I tested this out myself many moons ago, I made it work but any upgrades could possibly be installed half-ass and then you have a mess on your hands. It's really not worth the risk if you are putting important data on that extra vdev. But if you wanted to fool around with it, I'm not going to tell you not to. It can be a good learning experience to be honest with you but you must understand the risk and that very few people here will help you troubleshoot an issue if you get into a jam.

 

[Jamberry]

Speaking of SLOG, do you really need one?

Not really need one, mostly only because I wanna play around with iSCSI.
My gaming PC has a SSD and a 2TB old crappy HDD. That would be fun to migrate to iSCSI.
Or my Proxmox host could move unimportant VMs (like mumble) to TrueNAS NFS storage.

When I tested this out myself many moons ago, I made it work but any upgrades could possibly be installed half-ass and then you have a mess on your hands.

Well that sounds like a no go to me. Thank you for your sanity check.

if you get into a jam

Brilliant wordplay on my username or just a happy coincidence?

 

[HoneyBadger]

Not really need one, mostly only because I wanna play around with iSCSI.
My gaming PC has a SSD and a 2TB old crappy HDD. That would be fun to migrate to iSCSI.
Or my Proxmox host could move unimportant VMs (like mumble) to TrueNAS NFS storage.

An iSCSI "game drive" won't need sync writes or an SLOG - after all, your desktop definitely doesn't use them now.

The ProxMox host could potentially be an option for needing sync writes - but only if you truly care about the VMs in an "every single bit written matters" sense, and by your own admission you are calling them "Unimportant VMs." If you're willing to rebuild them or restore from backup, you could omit the sync writes with the understanding that any data between "last backup" and "now" could potentially be lost.

Side note: Generally, running both iSCSI and NFS over the same physical interface(s) can lead to competition. Would you consider iSCSI for your ProxMox storage, or do you have enough separate interfaces to split iSCSI and NFS?

Well [manually partitioning] sounds like a no go to me. Thank you for your sanity check.

It's certainly possible, but has both feet soundly in the "Here Be Dragons" category.

 

[Patrick M. Hausen]

For the dedicated boot device I am repeating myself but Transcend still sells new M.2 SATA or 2.5" SATA SSDs in sizes 32, 64, 128 ... G for a reasonable price.
Also in case you are using Supermicro boards: some come with an insane number of SATA ports and all I came across have at least one "SuperDOM", i.e. powered SATA DOM port. Add another SATA DOM with a dedicated power connector (also at least one on probably all server boards) and you get a mirror for boot. "Wastes" two SATA ports but you don't need a position in your case to mount 2.5" drives.

Example:

A2SDi-4C-HLN4F mainboard
SC721 TQ-350B case

1 4x SATA via miniSAS HD --> HDD cage, 4x spinning drives
1x SuperDOM SATA --> DOM for boot
1x regular SATA plus DOM power connector --> second DOM for boot
2x regular SATA --> 2x 2.5" SSDs for metadata vdev
1x M.2 on mainboard --> PCIe SLOG

And if you get the 8 core A2SDi-8C-HLN4F instead, there are 2 miniSAS connectors for a total of 8 SATA disks via miniSAS ... that baby features 12 SATA ports in total!

HTH,
Patrick

 

[Jamberry]

@HoneyBadger you're right, that's actually actually a great idea! I could just take the "risk" and set it to async.

Side note: Generally, running both iSCSI and NFS over the same physical interface(s) can lead to competition. Would you consider iSCSI for your ProxMox storage, or do you have enough separate interfaces to split iSCSI and NFS?

I honestly never thought about that. Thanks for pointing that out I will take a look into it.

A2SDi-4C-HLN4F mainboard

Looks like an awesome motherboard! It is just a shame that it is not available to purchase in Switzerland, but I will look around. Maybe I can find something. Are these CPUs powerful enough for 10Gbit networking? Maybe I will upgrade to 10Gbit later on. Thank you for pointing out these system on chip motherboards. I honestly never really looked into them!

Unfortunately, it's the same is true for sata doms in Switzerland, they are way too expensive. Cheapest ones I could find are around $160.

The case you mentioned looks very similar to my TrueNAS mini system, but I probably wanna go with eight drives. To get the system noise down from my current system, I will try some fractal design R7 XL with sound dampening. Hot swap I currently have is nice, but I didn't use it that often and are fine with powering down my system for maintenance.

Awesome help I got from you guys. Thanks a lot again! What a great community!

 

[Patrick M. Hausen]

This particular Atom based series comes with 4x 1G networking. Given the awesome number of SATA ports I guess they are intended for NAS use and would not expect them to push a full 10G. I have a couple of them and at 1G they are awesome.

I changed my main home NAS for something with a little more oomph, because I run a couple of VMs and single core peak performance of these Atoms leaves a bit to be desired.

I am also really satisfied with the 5028D-TN4T I got for that system. Xeon D are expensive compared to standard boards with Ryzen or similar, but again: ECC memory, lots of ports for storage, rather quiet from the get go (and you can still exchange the case fan for a Noctua), IPMI (I have four servers at home and no monitor or keyboard for them), etc.

While Switzerland is not part of the EU, they are somewhat associated to the common market. Can't you order from Germany, e.g. jacob.de?

 

[Jamberry]

I also looked into the X10SDV-TLN4F. I could get that one for 950CHF. The thing that worries me is power consumption and age. This a 8 year old CPU. I know that age is not everything, and I am happy to buy a 2y old CPU, but 8y seems very old for me. According to servethehome.com the motherboard consumes 90w under load, but I was unable to find idle consumption numbers.

rather quiet from the get go

I am not concerned about fan noise, biggest problem I have is HDD noise, because my Supermicro HDD drive tray caddies use screws and not some kind of rubber screws.

Can't you order from Germany, e.g. jacob.de?

In theory I can. The problem is that most online shops are too lazy to return sales taxes. So I pay the German 20% plus my 8%. Then I also pay customs duty 50Fr. for checking the package. And then I pay a very high shipping fees. If I ever run into any problems and need to return it, the same cycle starts all over again. Some vendors like TrueNAS will cover everything for you in one single fee and do all the legal duty stuff for you. They currently want 331.19$ dollar to ship to Switzerland! Thank god the black friday sale also discounted the shipping :)

 

[Patrick M. Hausen]

No friend living across the border?

 

[Jamberry]

I have one, but driving 1h for both ways makes no economical nor environmental sense to me.

 

[Jamberry]

The new X12SDV series from Supermicro looks very interesting. 25Gbit Networking and 10 SATA Ports. Hope they will be on sale soon.