NVDIMMS and TrueNAS 12

Joined
Apr 14, 2020
Messages
8
So I saw on the feature list that NVDIMMs willl be support in TrueNas 12 and I was wondering if anyone had played around with them yet. I have a Gigabyte MH71-HB0 and it supports 4 Optane DC Memory DIMMs and I am really interested in experimenting with these, especially now that the price for them has become easily obtainable. In fact the price for a 256GB module ($300) is almost competitive with SSD drives. I was wondering if it would we possible to use these as our ZIL device. Instead of buying a an expensive PCIe Optane Drive, we just slap a couple of these 256GB modules in there in AP mode and boom 512GB ZIL device. Would this work in TrueNAS? On a side note, is TrueNAS coded to take advantage of the speed of the modules? I have read a lot about them lately and with the falling prices I would really like to pick a couple up. Thank you for the help guys.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
I was wondering if it would we possible to use these as our ZIL device. Instead of buying a an expensive PCIe Optane Drive, we just slap a couple of these 256GB modules in there in AP mode and boom 512GB ZIL device

You would want to be fairly sure you're battery backed for those as the amount of data written in just a few seconds could be a lot to lose.

Would this work in TrueNAS?
Maybe...
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
You would want to be fairly sure you're battery backed for those
It's Optane DC - 3D XPoint on a DIMM interface. No battery backing is needed there.

In fact the price for a 256GB module ($300) is almost competitive with SSD drives. I was wondering if it would we possible to use these as our ZIL device. Instead of buying a an expensive PCIe Optane Drive, we just slap a couple of these 256GB modules in there in AP mode and boom 512GB ZIL device.

Definitely cheaper than the PCIe devices but I believe it requires special drivers for the Optane DC modules; I know Intel was working towards implementing support for persistent memory in FreeBSD but I don't know if it was completed yet.
 
Joined
Apr 14, 2020
Messages
8
Well I noticed on the feature list that it specifically stated support for nvdimms, now where or not the includes the opt and modules is anyone’s guess. Could someone on the dev team weigh in here?
 
Joined
Apr 14, 2020
Messages
8
And yes just to reiterate in ap mode the Optane modules are persistent just like their ssd cousins, however in memory mode they are not persistent, hence the rebranding.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,456
we just slap a couple of these 256GB modules in there in AP mode and boom 512GB ZIL device.
...roughly 480GB of which would be wasted. SLOG isn't a write cache and doesn't work like one.
 
Joined
Apr 14, 2020
Messages
8
...roughly 480GB of which would be wasted. SLOG isn't a write cache and doesn't work like one.
This statement confuses me. Aren’t optane drives pretty much the default standard for a zil/slog device? If we had access to the same technology but on the memory bus, wouldn’t that make it faster? I know I currently have a 905 optane as my zil and while I don’t have that large of a pool at the moment I plan to have upwards of 100 tb eventually.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,456
This statement confuses me.
...because you don't understand how a SLOG device is used. At any given time, it will have no more than two transaction groups' (each of which would be five seconds) worth of data. See:
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
I think the question is more along the lines of "what's the smallest suitable SSD?" - say the P3700, for instance, that thing starts at 240 GB if I'm not mistaken. If you're really looking for speed, way oversized SSD capacities are probably inevitable.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
...because you don't understand how a SLOG device is used. At any given time, it will have no more than two transaction groups' (each of which would be five seconds) worth of data. See:
https://www.ixsystems.com/community/threads/proper-sizing-of-slog-and-transaction-group.35470/
https://www.ixsystems.com/blog/why-zil-size-matters-or-doesnt/

Unfortunately both of those links are 5 years old and refer to the "legacy write throttle" - the short version now is that with the default tunables, you'll never use more than 4G of a given SLOG device. Like most things in ZFS this can be adjusted, but you'll consume an equal amount of RAM to whatever you allow your "maximum SLOG usage" to be, which robs you of some ARC space. Obviously it's up to you to find the balance depending on your read/write usage, the "burstiness" of the workload, and how much RAM is installed.

I think the question is more along the lines of "what's the smallest suitable SSD?" - say the P3700, for instance, that thing starts at 240 GB if I'm not mistaken. If you're really looking for speed, way oversized SSD capacities are probably inevitable.

400G for that one specifically, but it's kind of moot as pointed out - to get the kind of speed and responsiveness you want out of an SLOG, you either end up with a grossly oversized SSD or a specialised (and usually extremely expensive) device. Oversized devices do help with wear-levelling to increase endurance though.
 

ehsab

Dabbler
Joined
Aug 2, 2020
Messages
45
Wouldnt two mirrored 128GB DCPMM modules in DA mode partitioned for SLOG and L2ARC be a good solution to nvme disks?
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Conceivably, but it's still cutting edge stuff. Any volunteers? Be sure to carry first aid kits.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
ñvdimm only enterprise.version
Only officially. NVDIMM implementation is challenging, but on proper hardware they simply appear as additional nvd devices. There is not software lockout that prevents them from working on TrueNAS CORE.
 

Rand

Guru
Joined
Dec 30, 2013
Messages
906
RMS is no NVDimm tough.

I upgraded (accidentally) to 12U1 yesterday and it seems my NVDimm (-N, not -P as per OP) continues to work fine albeit with another device identifier, but pool was migrated fine.
Not sure I still need the Loader variable I had in place any more, but have not tested further
 

ehsab

Dabbler
Joined
Aug 2, 2020
Messages
45
RMS is no NVDimm tough.

I upgraded (accidentally) to 12U1 yesterday and it seems my NVDimm (-N, not -P as per OP) continues to work fine albeit with another device identifier, but pool was migrated fine.
Not sure I still need the Loader variable I had in place any more, but have not tested further
What NVDimm are you using?
 

TrumanHW

Contributor
Joined
Apr 17, 2018
Messages
197
How would an NVDIMM compare to an Optane Persistent RAM performance be..?

In fact, what's the performance (comparatively) between Optane Persistent RAM vs say a P4800X or 905p ..?
 

TrumanHW

Contributor
Joined
Apr 17, 2018
Messages
197
Optane Persistent Memory 100 vs 200 performance & compatibility

Compatibility requires very expensive CPU
(I'd hoped they'd work with E5-2600)

Intel-Optane-PMem-100-vs-200.png


Optane Persistent DIMM 100 Optane Persistent DIMM 200
Maximum DIMM Speed / Performance 2,666 MT/s 3,200 MT/s
SP CPU Generation REQUIRED Xeon Scalable Processors (2nd Gen.) Xeon Scalable Processors (3rd Gen.)
CPU Compatibility






































































Xeon Platinum 8280L
Xeon Platinum 8280
Xeon Platinum 8276L
Xeon Platinum 8276
Xeon Platinum 8270
Xeon Platinum 8268
Xeon Platinum 8260Y
Xeon Platinum 8260L
Xeon Platinum 8260
Xeon Platinum 8256
Xeon Platinum 8253

Xeon Gold 6262V
Xeon Gold 6258R
Xeon Gold 6256
Xeon Gold 6254
Xeon Gold 6252N
Xeon Gold 6252
Xeon Gold 6250L
Xeon Gold 6250
Xeon Gold 6248R
Xeon Gold 6248
Xeon Gold 6246R
Xeon Gold 6246
Xeon Gold 6244
Xeon Gold 6242R
Xeon Gold 6242
Xeon Gold 6240Y
Xeon Gold 6240R
Xeon Gold 6240L
Xeon Gold 6240
Xeon Gold 6238T
Xeon Gold 6238R
Xeon Gold 6238L
Xeon Gold 6238
Xeon Gold 6234
Xeon Gold 6230T
Xeon Gold 6230R
Xeon Gold 6230N
Xeon Gold 6230
Xeon Gold 6226R
Xeon Gold 6226
Xeon Gold 6222V
Xeon Gold 6212U
Xeon Gold 6210U
Xeon Gold 6209U
Xeon Gold 6208U
Xeon Gold 5222
Xeon Gold 5220T
Xeon Gold 5220S
Xeon Gold 5220R
Xeon Gold 5220
Xeon Gold 5218T
Xeon Gold 5218R
Xeon Gold 5218N
Xeon Gold 5218B
Xeon Gold 5218
Xeon Gold 5217
Xeon Gold 5215L
Xeon Gold 5215

Xeon Silver 4215R
Xeon Silver 4215








4 and 8 Socket SP
Platinum 8380HL
Platinum 8380H
Platinum 8376HL
Platinum 8376H
Platinum 8360HL
Platinum 8360H
Platinum 8356H
Platinum 8354H
Platinum 8353H
Gold 6348H
Gold 6330H
Gold 6328HL
Gold 6328H
Gold 5320H
Gold 5318H

Highest Per-Core SP
Platinum 8380
Platinum 8368
Platinum 8362
Platinum 8360Y
Platinum 8358
Gold 6348
Gold 6342
Gold 6354
Gold 6346
Gold 6334
Gold 6326
Gold 5317
Gold 5315Y


Scalable Processors
Platinum 8352Y
Gold 6338
Gold 6336Y
Gold 6330
Gold 5320
Gold 5318Y
Silver 4314
Max SGX Enclave
Platinum 8380
Platinum 8368Q
Platinum 8368
Platinum 8352S
Gold 5318S

Cloud Optimized for VM
Platinum 8358P
Platinum 8352V

Liquid Cooled
Platinum 8368Q

Network / NFV Optimized
Platinum 8351N
Gold 6338N
Gold 6330N
Gold 5318N


Mobile Media Optimized
Platinum 8352M

Long-Life & NEBS-Thermal Friendly
Gold 6338T
Gold 5320T

Single-Socket Optimized
Platinum 8351N
Gold 6314U
 
Top