Looking for a suitable configuration for max performance

pk167

Cadet
Joined
Jul 18, 2020
Messages
7
Hi Everyone, since I brought a server for building a HIGH PERFORMANCE and RELIABLE file server, so I brought serial SSDs and HDDs to archive this goal. Follow are all specification so far:

  • Motherboard make and model: Come from DELL R720xd
  • CPU make and model: Intel Xeon CPU E5-2660 *2
  • RAM quantity: 192GB
  • Hard drives, quantity, model numbers, and RAID configuration, including boot drives: WDC WD80EZAZ * 12 (2 striped 6x raidz2)
  • Hard disk controllers: H710p in IT mode
  • Network cards: Dell Intel X520-DA2
  • OS: TrueNAS core (from beta to RC1)

I've tried to some different combinations of SSDs and HDDs, but I still could not get writing/reading speed of 10Gbps. I've searched at community and even on internet, there is no similar best practice for reference. Does anyone could recommend me a right set of combination to maximum the performance? Thank you in advance!

#1
Cache: N/A
Slog: WDC SN720 512GB *2 (mirrored)
Metadata: N/A

#2
Cache: WDC SN720 512GB *2 (striped)
Slog: Intel S3710 200GB *2 (mirrored)
Metadata: N/A

#3
Cache: WDC SN720 512GB *2 (striped)
Slog: Intel Optane 16GB *2 (mirrored)
Metadata: N/A

#4
Cache: WDC SN720 512GB *2 (striped)
Slog: Intel P4801x 100GB *2 (mirrored)
Metadata: N/A

#5
Cache: N/A
Slog: Intel P4801x 100GB *2 (mirrored)
Metadata: WDC SN720 512GB *2 (mirrored)

#6
Cache: N/A
Slog: Intel S3710 200GB *2 (mirrored)
Metadata: WDC SN720 512GB *2 (mirrored) + Intel Optane 16GB *2 (mirrored)

#7
Cache: N/A
Slog: Intel S3710 200GB *2 (mirrored)
Metadata: WDC SN720 512GB *2 (mirrored) + Intel Optane P4801x 100GB *2 (mirrored)

#8
Cache: N/A
Slog: N/A
Metadata: WDC SN720 512GB *2 (mirrored) + Intel Optane P4801x 100GB *2 (mirrored)

#9
Cache: N/A
Slog: Intel S3710 200GB *2 (mirrored)
Metadata: WDC SN720 512GB *2 (mirrored) + Intel Optane P4801x 100GB *2 (mirrored)

Besides, is there any way to remove special vdev & dup vdev from existing pool? The only way I know is rebuild a pool then restore all data.
 
Last edited:

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
The issue is often - how you test. You need to find a test methodology and set-up where the NAS is the bottleneck... not the clients or the network or the testing software. In particular, there needs to be enough IO-depth or queue depth given the network latencies. When testing internally, we typically use a dozen or more testing clients.

Once you have added vdevs to a pool... they cannot be removed. The devices can be replaced.
 

pk167

Cadet
Joined
Jul 18, 2020
Messages
7
The issue is often - how you test. You need to find a test methodology and set-up where the NAS is the bottleneck... not the clients or the network or the testing software. In particular, there needs to be enough IO-depth or queue depth given the network latencies. When testing internally, we typically use a dozen or more testing clients.

Once you have added vdevs to a pool... they cannot be removed. The devices can be replaced.

Thanks for your reply. Would you please give me some substantive recommendations? As a rookie, I only know to use some benchmark tools, the results were not satisfied. I could not get 10Gb/s from the first time I tested.

I hope to get some suggestion that how to utilize all hardware I had to maximum the performance.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
Configuring for performance requires a total over view of what you want to do: Workload, clients, network, protocols, business needs.

With 12 drives, max performance is achieved through 5 or 6 mirror pairs... depending on how quickly a failed drive can be replaced.

With 2 RAIDZ2 VDEVs.. the performance is very sensitive to IO size. Are you just reading and writing large files?

Typical business would test with N clients... each running fio or vdBench with specified workload over specified protocol. There are lots of parameters to configure (iosize, iodepth, r/w mix, dataset size).. each of which can have a huge impact.

10GbE networks don't really provide 10Gb/s throughput.... there are protocol overheads and queueing inefficiencies. 8Gb/s is the most anyone should expect... its often lower.

The metadata cache doesn't really help unless there are lots of files and Metadata doesn't fit in RAM... you have lots of RAM.
The SLOG helps with synchronous writes...more so with small writes... especially for NFS
In most cases, the drives will be the performance limiter.... Mirrors are faster than RAID-Z2s.

So general guidance is don't focus on max performance, focus on what you need and configure for that goal.
 

pk167

Cadet
Joined
Jul 18, 2020
Messages
7
Configuring for performance requires a total over view of what you want to do: Workload, clients, network, protocols, business needs.

With 12 drives, max performance is achieved through 5 or 6 mirror pairs... depending on how quickly a failed drive can be replaced.

With 2 RAIDZ2 VDEVs.. the performance is very sensitive to IO size. Are you just reading and writing large files?

Typical business would test with N clients... each running fio or vdBench with specified workload over specified protocol. There are lots of parameters to configure (iosize, iodepth, r/w mix, dataset size).. each of which can have a huge impact.

10GbE networks don't really provide 10Gb/s throughput.... there are protocol overheads and queueing inefficiencies. 8Gb/s is the most anyone should expect... its often lower.

The metadata cache doesn't really help unless there are lots of files and Metadata doesn't fit in RAM... you have lots of RAM.
The SLOG helps with synchronous writes...more so with small writes... especially for NFS
In most cases, the drives will be the performance limiter.... Mirrors are faster than RAID-Z2s.

So general guidance is don't focus on max performance, focus on what you need and configure for that goal.

Thank you, MorganL. In my scenario, my TrueNAS works as central family space that storing photos, games, documents and video streaming. Not just reading and writing large files, most of scenario are small (all types of upon I mentioned except video streaming) I think. According to your suggestion, so I need to rebuild a pool with 6 pairs of mirrors, and running on 192GB RAM without any cache or metadata. Am I correct?
 
Last edited:

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
For the workloads you describe, your performance needs are not particularly high. Are they not being met with this configuration?

Mirrors are used for virtualization and database workloads that need high performance and high IOPS.

Test with video streaming and you'll have your decision.
 

pk167

Cadet
Joined
Jul 18, 2020
Messages
7
For the workloads you describe, your performance needs are not particularly high. Are they not being met with this configuration?

Mirrors are used for virtualization and database workloads that need high performance and high IOPS.

Test with video streaming and you'll have your decision.

Yes, gaming is also a high workload for me. Higher IOPS is able to get game loading faster & smoother, I am thinking do I still need to setup a cache if the pool has been build with fully mirrored. In other words, is it necessary to use cache if memory is large enough?

Besides, can these Optanes (16G/100G) be used for performance improvement? like Cache, or Slog?
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
With 192GB RAM, I doubt cache will help.
SLOG isn't needed unless you have iSCSI or NFS with reliability issues.
Mirrors might help, but I doubt you need that performance boost..... what IOPS are needed?
 

pk167

Cadet
Joined
Jul 18, 2020
Messages
7
With 192GB RAM, I doubt cache will help.
SLOG isn't needed unless you have iSCSI or NFS with reliability issues.
Mirrors might help, but I doubt you need that performance boost..... what IOPS are needed?

I have no concept of IOPS, but as much as possible for sure. I got some weird results after taking tests by benchmark tools (please refer attached pics). I find the reading performance is way more lower than my expected. It doesn't fulfill the performance based on 6 stripped mirrors I think. Can I adjust any parameters to improve it? Here are my current specification & configuration:

TureNAS:
  • Hard drives, quantity, model numbers, and RAID configuration, including boot drives: WDC WD80EZAZ 8TB * 12 (6 striped mirrors)
  • Cache: N/A
  • Slog: Intel P4801x 100GB
  • Metadata: WDC SN720 512GB *2 (mirrored)
[Client] Windows 10 Pro:
  • C drive (NVME: disabled write cache)
  • I drive (iSCSI: enabled write cache & buffer flushing)

C: (Intel SSD)
nvme_intel_aja.PNG

I: (iSCSI by TrueNAS)
iscsi_nas_aja.PNG


nvme_intel.PNG

71.33MB/s in Read??!
iscsi_nas.PNG


nvme_intel_cm.PNG

By the same condition, seems be normal on CrystalDiskMark
iscsi_nas_cm.PNG
 
Last edited:

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
The SSD benchmark tested is probably tuned to test an SSD with low latency... it doesn't have enough I/O in parallel (low queue depth) to load the NAS. So, its really testing latency. Note that the SSD latency is 17x lower... and so bandwidth is almost 17x higher. Network latency to HDDs is always higher than local SSD latency.

The testing is also on 1GB file size.... this means that caching can have a big impact and its unrealistic. Real tests should be 100G - 1TB.

CristalDiskMark has Queue depth and threads to test for parallelism. QD = 8, Threads = 1, Sequential numbers are good. But they may be overly good because of the 1GB test size and the caching effects.
 

pk167

Cadet
Joined
Jul 18, 2020
Messages
7
The SSD benchmark tested is probably tuned to test an SSD with low latency... it doesn't have enough I/O in parallel (low queue depth) to load the NAS. So, its really testing latency. Note that the SSD latency is 17x lower... and so bandwidth is almost 17x higher. Network latency to HDDs is always higher than local SSD latency.

The testing is also on 1GB file size.... this means that caching can have a big impact and its unrealistic. Real tests should be 100G - 1TB.

CristalDiskMark has Queue depth and threads to test for parallelism. QD = 8, Threads = 1, Sequential numbers are good. But they may be overly good because of the 1GB test size and the caching effects.

So, will increase the record size helps to reduce the latency? Currently it is 128K, do you suggest increase to 512K or 1M?
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
It all depends on the actual workload. The latency you have for access HDDs over 10Gbe is very reasonable. If you need lower, then Mirrors or SSDs are better.

However, I'd guess that the CrystalDiskMark results are closer t realistic... especially if you increase the size of the test to larger than your RAM size.
 
Top