Hello from Montreal, I would like to have your opinion on my config.

[911AS]

I was looking for a way to use my old PC that still works well. And I saw a video on YT from Craft Computing. I knew at that moment that this is what I wanted to do with my old PC!

Simply put, it's using a pool in TrueNAS which is shared via the iSCSI service and used as a local drive on a Windows station on which a Steam library is installed.
The guy clearly explains how to do it but assuming that we already have TrueNAS installed and an existing pool.

I installed TrueNAS on my old PC and figured out how to create a new pool. It worked but it was slow and some games crashed. I redid the pool configuration a second time and it was faster and the games no longer crashed. But I still didn't really know what I was doing...

I would first like to say thank you to the members of this forum ChrisRJ (for his recommended readings) and Cyberjock (for his powerpoint intended for beginners). It helped me a lot!

I would like to configure my old PC as best as possible with the components I already have on hand:
- Motherboard: EVGA X58 SLI3 (PCIe 2.0)
- CPU: i7-920@2.66Ghz
- RAM: 24GB DDR3-1600
-LAN: 1Gb/s
-Drives:
1x 6TB HDD SATA3
2x 1TB HDD SATA
2x 2TB NVme SSD PCie 3.0 (each installed with a PCie X16 adapter card)
1x 60GB SATA SSD
1x 32GB SATA SSD

I created a pool as follows:
1 mirrored vdev with 2x 1TB HDD SATA.
1 mirrored vdev with 2x 2TB NVme SSD.
1 L2ARC/ARC cache using 1x 60GB SATA SSD.

I chose to use L2ARC/ARC cache because I will share a zvol with iSCSI and I estimated its size based on the Cyberjock presentation:
- The size of the ARC must not exceed 7/8th the size of the memory.
- The size of L2ARC no more than 5 times the size of ARC.
- A 60GB SSD should not use more than 12GB of memory and that leaves another 12GB of memory which is larger than the bare minimum requirement of 8GB that must be available for TrueNAS.

I used the other 2 disks (6TB HDD and 32GB SSD) on another pool for backups (replication task).

Do you have any recommendations? I haven't installed the games yet. I can easily redo my pool base based on your expert advice.

Do you think a jumbo frame (mtu config change) could improve network performance to make it worth it?

Thank you in advance,

Annick

 

[Pitfrr]

Hello,

I'm not very familiar with iSCSI but from my understanding, in that case, the more IOPs, the better. Hence your mirror with the NVMe drives would be a good way to start (and I would keep the HDD mirror for some other stuff).
Also an L2ARC will not help you much (again not sure for iSCSI but I guess here too), you might want to try without you might experience better results. But better wait for a confirmation from more experienced forum members in that matter.

But what I wanted to point out is your motherboard. It was not easy to find but it apparently (page 45) uses a Realtek chipset for the network. And that is not a good idea with TrueNAS since the drivers are not reliable. So you might experience low performances due to this. So it might be good to test with an intel chipset and see if you get better performance (data rate and such).

 

[911AS]

Hello,

I'm not very familiar with iSCSI but from my understanding, in that case, the more IOPs, the better. Hence your mirror with the NVMe drives would be a good way to start (and I would keep the HDD mirror for some other stuff).

Is it better in terms of performance to have only one mirrored vdev with disks of the same size and speed rather than 2 vdevs with disks different in size and speed?

Thinking about it, I could also plug the 2 HDD drives each into the 2 SATA3 (6GB/s) connectors. The 2 NVme SSDs are on PCIe 2.0, technically their speed is limited to 2GB/s. If the question above is yes, would I be better off using HDDs?

Also an L2ARC will not help you much (again not sure for iSCSI but I guess here too), you might want to try without you might experience better results. But better wait for a confirmation from more experienced forum members in that matter.

The only reason tbh, is that in the documentation I read, it mentions that using vdev log(ZIL) and cache(l2-arc) may be appropriate if using iSCSI. But, I also read that you need to have at least 64GB of memory for it to be worth it.

The good news is that I can easily add/remove the vdev log and vdev cache and test if I see a difference in performance.

But what I wanted to point out is your motherboard. It was not easy to find but it apparently (page 45) uses a Realtek chipset for the network. And that is not a good idea with TrueNAS since the drivers are not reliable. So you might experience low performances due to this. So it might be good to test with an intel chipset and see if you get better performance (data rate and such).

Thank you for taking the time to do the research, it is appreciated. If I understand correctly, I could buy an Intel NIC card to improve network performance.

 

[HoneyBadger]

Your 1Gbps network card is likely to be the speed bottleneck - a local SATA SSD is capable of around 550MB/s of reads, and your network will likely limit you to ~110MB/s of reads, or about one-fifth of the load times of a SATA SSD. A local NVMe SSD will load in the order of "gigabytes per second" - if you're mostly loading smaller titles, the difference may not be felt, but larger titles or ones that rely heavily on asset streaming may have challenges with this setup.

 

[911AS]

In summary, it doesn't matter which pool config I use, it's my network connection that takes precedence.

 

[Pitfrr]

I'd say those speeds are not too bad given your system.

 

[911AS]

I'd say those speeds are not too bad given your system.

You mean it could be even worse? J/k.

Why is writing faster than reading? Shouldn't it be the opposite?

 

[Pitfrr]

You mean it could be even worse?

Yup... with a Realtek chipset this is not bad.

Why is writing faster than reading?

With TrueNAS, the RAM is used for cache (read and write).
When writing, you write to the RAM first before it gets written to the disks, so this is very fast. In the test you've done, you probably didn't fill up all the cache so the write operation is fast.
Reading is slower because you also read from the RAM but the first time, the data is not in the RAM but on the disk, so it takes longer to access.

 

[911AS]

I tried with a 64GiB file size in place of 64MiB. The speed is almost the same. It just much longer to execute. I thought once the memory will filled (see pictures below) that the speed will be affected but it seems is not the case.

The IOPS are also nearly identical.

The ZFS cache as expected for the big file use pretty much all the memory. But why with the smaller file, the server don't free up memory but allocate it to the services?

 

[Pitfrr]

At that point, I'm not familiar enough with TrueNAS's memory management to explain this behavior...

For the 64GB test, are you writing on your NVMe pool or the HDD pool?
If I understand correctly, your pool uses the SSD and the HDD drives, right, so no surprise you don't see differences because in that case the bottleneck is not the drives (but then something else).

 

[HoneyBadger]

The lower-than-gigabit reads are probably a result of the Realtek network card; if I recall, they seem to be notably poorer on the "sending" side than "receiving" which makes sense given their target market of consumer gaming gear.

I happen to have a Nehalem-era Xeon system kicking around - peak read speeds are a fair bit better, writes a little less so, but it's backed by only a couple of "well-aged" SATA-2 HDDs.

 

[911AS]

For the 64GB test, are you writing on your NVMe pool or the HDD pool?

All the tests were with the NVme only. Nothing else.

 

[911AS]

The lower-than-gigabit reads are probably a result of the Realtek network card; if I recall, they seem to be notably poorer on the "sending" side than "receiving" which makes sense given their target market of consumer gaming gear.

Ok, the read/write speed should be more or less equal which is clearly not the case for me.

I ordered a 2.5G NIC with the Intel I-225-V controller. From what a read in TrueNAS forum, it should be recognized by TrueNas Core since the version 12. I will try has is to see if I get a better read test results.

My other PC already have an integrated 2.5G NIC Intel I-226-V. I also ordered an unmanaged 2.5G switch box. I know, I know, using anything between 1G and 10G is illegal and punishable by TrueNAS forum law.

I have no usage for such a thing as a 10G setup. Even less with the local ISP's that cannot get more than 3Gb/s. If a can reach 250Mb/s, I will be more than happy.

 

[911AS]

One thing, I found is that the only pool config were my games never crashed, is using the NVme as a L2-ARC cache vdev and the HDD's as data vdev. Probably the best setup in this case for a Steam Library. Writes operation when installing the games and mostly reads operation after.

 

[911AS]

I installed a new 2.5G NIC(Intel i225-v rev-3) on my TrueNAS server, and my PC already had an integrated 2.5G NIC on the MB(Intel i226-v w/ last driver). I also bought an unmanaged 2.5G switch box.

I did some tests and the performance is very bad. I don't know why. I added below a picture of the test result and a picture of my network setup.

The setup is simple. The black cable (cat8, is the shortest i have) connect the switch box to the router (100/1000G). The left yellow cable connect my NAS server and the right yellow my PC (both cables are cat. 6). The switch box indicated the yellow cable has 2.5G speed and the black one 1G speed has expected.

What am I doing wrong?

Edit: 2024-02-03 - Removed picture because a changed my setup.

 

[911AS]

I recently changed my network configuration to finally obtain the performance expected from a 2.5Gb network.

To do this, I changed my ethernet switch with a YuanLey 4x2.5Gb ports and 2x10Gb SFP+ ports, I replaced the 2.5Gb Intel I225-V network card with an Intel X520-DA2 10Gb and I used a cable 6 foot multimode OM3 fiber optic cable with transceivers for Intel.

These changes fixed the performance issues. With Iperf3, I get a speed of 282MB/s in both directions. And with the CrystalDiskMark utility (the drive is a iSCSI share), a notable gain:

Even if is a somewhat expensive learning curve, I am happy with the final result.

 

[ChrisRJ]

Given your CPU and the overall age, this is really not bad.