Recommended Intel Optane P4800X SLOG Configuration

[Chris Tobey]

Hi guys,

I am thinking of adding an Intel P4800X (375GB) as a SLOG for my FreeNAS fileserver. I am looking for recommended configurations and settings for obtaining the best results from such a setup. Things like overprovisioning, block size, etc.

All access is done via NFS mounts, hence the desire to speed up the sync-writes. Currently 50+ servers access it concurrently.

My other system specifications are:
Chassis: Supermicro CSE-846BE16-R920B
Motherboard: Supermicro X9DRD-7LN4F
CPUs: 2 x Xeon E5-2603v2
RAM: 12 x 16GB ECC DDR3-1866
SLOG: 2 x Intel S3500, to be replaced by Intel P4800X.
Interface: Currently 1x10GbE, but expandable to 4x10GbE should I be able to take advantage of it.

 

[Stux]

When I tested my p3700 I ser it to 4K sectors and 20% provisioning using the Intel SSD tools.

I expect the same settings would be optimal in your scenario too.

I go through the steps here:
https://forums.freenas.org/index.ph...n4f-esxi-freenas-aio.57116/page-4#post-403374

Ignore the partitioning stuff. You probably don’t want to place extra load on the p4800x.

 

[Chris Tobey]

Thanks, I plan to do this upgrade tomorrow. One more piece of this puzzle is that my system is running FreeNAS-9.3-STABLE-201602031011.

I was looking into how to test all of the different performance benchmarks and everyone seems to be using 11.1+. The GUI provides me the option of updating to the 11.2 STABLE builds (I have automatic updates disabled). Is this recommended?

 

[HoneyBadger]

11.2 is a beta, despite the "STABLE" name; update to the 11.1-U6 build at the latest.

20% will still give you 75GB which is hugely overkill for an SLOG; the current OpenZFS write throttle maxes out at 4GB of dirty data by default (although that's tunable) - how about 5% for an 18.75GB device, or 10% for a 37.5GB one (which is still overkill, but might give you the option of throwing swap on there if you desired.)

Also, going from mirrored S3500s to a P4800X is going to feel like engaging Warp 9 on the Enterprise.

Check out this graph of the expected MB/s throughput at a given recordsize. You might want to consider getting that second 10Gbps link ready for activity. ;)

 

[Chris Tobey]

Interesting.

Options are 11-STABLE and 11.2-STABLE, I guess I should go 11-STABLE?

My plan was 20% overprovision and switch to 4K sectors, not much else on that front.

At the moment I only have access to 1GE servers to read/write to the FreeNAS system that has a 10GE link. I plan to get a 10GE server and do some tests before and after the upgrade/install. Anyone have recommendations on what kind tests / tools to use?

FreeNAS is running baremetal with no jails. I have access to the console of the server itself (via SSH) and Windows 7/10 and CentOS 6.x servers to run any benchmarks.

 

[HoneyBadger]

11-STABLE is the correct train to be on, and I'm pretty sure that your lack of jails means you can just do a direct GUI upgrade - assuming you have an approved maintenance window, of course. What's your boot device?

As far as tests, I'll throw in the obvious and shameless plug for you to run the simple diskinfo -wS benchmark from the link in my signature once you have it set up with 4K sectors. That will give you the most accurate idea of what kind of sustained write performance to expect from datasets that go through the SLOG.

Beyond that, you could use a tool like iozone to simulate things like:

Database access (randomly read and write 8K)
VMware ESXi running Windows (randomly read and write 4K)
Big streaming copies (128K or larger writes)

Or use the Windows tools like Atto/AS-SSD/CrystalDiskMark.

Although you're going to find yourself in the amusing situation of a single 10Gbps link being a limiting factor in certain tests.

 

[Chris Tobey]

I actually made a mistake, I have two Intel S3500 120GBs as my boot device in a mirror.

I have two Samsung SSD 850 PRO 128GBs as my SLOG, also in a mirror.

 

[HoneyBadger]

Ah. Well in that case it's even better that you're upgrading your SLOG, because I don't believe the 850 PRO has power loss protection.

A mirror of SSDs for boot means that your upgrade installation/reboot time should also be quite spiffy.

 

[Chris Tobey]

I was able to get a Windows 10 PC running with the Intel P4800X installed. I then used Intel SSD Data Center Tool (isdct) to play with it.

First I was on firmware E2010324 which was also the latest listed in the FirmwareUpdateTool_v3_0_3. That being said, isdct said there was a firmware update available so I applied that, rebooted, and now I am on firmware E2010435:

Code:

PS C:\isdct> .\isdct.exe show -intelssd

- Intel Optane(TM) SSD DC P4800X Series PHKS7xxxxxxxxxxAGN -

Bootloader : EB3B0416
DevicePath : \\\\.\\PHYSICALDRIVE2
DeviceStatus : Healthy
Firmware : E2010435
FirmwareUpdateAvailable : The selected Intel SSD contains current firmware as of this tool release.
Index : 0
ModelNumber : INTEL SSDPED1K375GA
ProductFamily : Intel Optane(TM) SSD DC P4800X Series
SerialNumber : PHKS7xxxxxxxxxxAGN

Next I followed the instructions from Stux/manual to set the sector size to 4096:

Code:

PS C:\isdct> .\isdct.exe start -intelssd 0 -NVMeFormat LBAformat=3 SecureEraseSetting=1
WARNING! You have selected to format the drive!
Proceed with the format? (Y|N): y
Formatting...(This can take several minutes to complete)

- Intel Optane(TM) SSD DC P4800X Series PHKS7xxxxxxxxxxAGN -

Status : NVMeFormat successful.

Now I can see the 4096 sector size when I show all properties:

Code:

PS C:\isdct> .\isdct.exe show -a -intelssd

- Intel Optane(TM) SSD DC P4800X Series PHKS7xxxxxxxxxAGN -
...
SectorDataSize : 4096
SectorSize : 4096
...

 

[Chris Tobey]

Some benchmarks:
Samsung 850 EVO SSD 1TB (left) vs Intel P4800X 375GB (right)

 

[Chris Tobey]

I am, however, still having issues with the overprovisioning. It does not seems to want to let me do it at all. It is not REALLY necessary with the P4800X, and I am not using the drive for anything else, but I also do not need a 375GB SLOG.

Code:

PS C:\isdct> .\isdct.exe set -intelssd 0 MaximumLBA=20%
Error: Invalid sector size for max address.
PS C:\isdct> .\isdct.exe set -intelssd 0 MaximumLBA=21%
Error: Invalid sector size for max address.
PS C:\isdct> .\isdct.exe set -intelssd 0 MaximumLBA=22%
Error: Invalid sector size for max address.
PS C:\isdct> .\isdct.exe set -intelssd 0 MaximumLBA=23%
Error: Invalid sector size for max address.
PS C:\isdct> .\isdct.exe set -intelssd 0 MaximumLBA=24%
Error: Invalid sector size for max address.
PS C:\isdct> .\isdct.exe set -intelssd 0 MaximumLBA=25%
Error: Invalid sector size for max address.
PS C:\isdct> .\isdct.exe set -intelssd 0 MaximumLBA=26%
Error: Invalid sector size for max address.
PS C:\isdct> .\isdct.exe set -intelssd 0 MaximumLBA=20GB
Error: Invalid sector size for max address.
PS C:\isdct> .\isdct.exe set -intelssd 0 MaximumLBA=100%
Error: Invalid sector size for max address.

Anyone have experience with this or any ideas to try?

 

[Chris Tobey]

I had read the whitepaper from Intel last night on Over-Provisioning as well, they had this to say:

Whitepaper: https://www.intel.com/content/dam/w...nd-based-ssds-better-endurance-whitepaper.pdf

 

[HoneyBadger]

Interesting. I wonder if it's disabled at the firmware level for some reason. It's not like it will hurt the drive by being under-provisioned.

I would manually create a smaller aligned partition, giving the full 375GB drive would give you very large metaslabs.

 

[Chris Tobey]

Setup is Windows 10 machine with Mellanox 10GE card <-> 10GE Router <-> FreeNAS Server with Chelio 10GE card.

FreeNAS 9.3 with mirrored Samsung 850 Pro SSDs as SLOG. sync=always:

FreeNAS 11.1 with mirrored Samsung 850 Pro SSDs as SLOG. sync=always:

FreeNAS 11.1 with Intel P4800X 375GB as SLOG. sync=always:

 

[SMnasMAN]

thanks for the benchmarks / info, you have a setup EXACTLY like mine (except my optane is on the way currently).

What are the HDDs in your pool? (ie size / count / layout)? ( i didnt see it anywhere in this post) thanks!

 

[Chris Tobey]

I have one pool with two vdevs, each with six drives in Z-2. The drives are 4TB Seagate SAS drives. There is a second pool with one vdev of six drives in Z-2. The drives there are 8TB Seagate SAS drives.