Upgrade to 10Gb, worth it with my config?

[Lucas Rey]

Hello community, I kindly ask your opinion regarding a possible upgrade of my homelab server from 1Gb to 10Gb (or 2,5Gb), cause my provider just upgraded my home fiber to such speed. My goal is speedup transfer between my clients (10Gb too) to/from TrueNAS.
I have a Proxmox server where I virtualized TrueNAS, these are the TrueNAS specs:

• 16 Core, 24 GiB
• 4x4 TB WD Red (RAID-Z2) ==> This will become in the next weeks: 4x12 TB Seagate IronWolf Pro (RAID-Z2)
• Disks are passed through directly to TrueNAS
• Only one Dataset (NO compression) - Occupied Space 73%

While this is the main hardware (PROXMOX):
RAM: 128GB DDR4
CPU: Intel i9 9900K
MB: Q370M D3H GSM PLUS
Hard Disks connected to SATA3 onboard ports

Well, following is the speed test I did on TrueNAS CLI. It seems I can't get a good speed, did I do something wrong with the test? Is it worth upgrading to 10Gb at this point?

Code:

# dd of=/dev/null if=test.dat bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 77.050954 secs (272177293 bytes/sec)

# dd of=/dev/null if=test.dat bs=2048k count=10000
^C4196+0 records in
4196+0 records out
8799649792 bytes transferred in 12.853717 secs (684599596 bytes/sec)

Thank you
Lucas

 

[chuck32]

RAIDZ2 is not very fast, you are pretty much limited to the speed of a single drive.
With striped mirrors you can increase that speed. Or you set up a SSD pool.

In your case I would say if you do not need to transfer larger files often and do not need your Internet bandwidth (really 10G?) on your machine it probably won't make a difference to you. After the memory cache is used you will drop to your HDD speeds quickly.

Only other scenario I would say you could benefit is, if you have multiple clients accessing the machine simultaneously and you also got a 10G switch. This will not even saturate two 1G machines completely.

I just have a direct 10G connection to my PC and I would not want to miss it. But my pool is faster than yours and I have more memory.

Stay away from 2.5G, that is not working well. General consensus is stay at 1G or go 10G.

 

[probain]

Moar bigger is moar betterer!

Jokeas aside. 10G is nice. But unless you really feel and know that you're limited by 1G, in a practical sense. Not just a theoretical one. Then having 10G is mostly a (really) nice-to-have feature. Are transfer speeds a problem for you? Are you doing most stuff wireless? How often do you copy large files at the same time you're in a real hurry?

I like my 10G. I absolutely do not need it. And the costs do add up. Switches, SFP+ modules, cabling, NICs.. And all of these components run alot hotter than 1G variants.

Homelabbing is about learning and having fun IT-projects. So if you can afford it (disposable income). Go for it

 

[Lucas Rey]

Thanks both for replies. I often transfer, from my PC to TrueNAS, larger files (e.g. 4k mkv from 60 to 80GB), others are normal operations like transfer every 4 hours PC backup from client to nas or use samba to share the media.
Currently, at 1 Gb, I only have 80/90MB x sec transfer rate (PC to/from NAS).
Because I ordered 4x12TB seagate disks, I can move from RAID-Z2 to striped mirror to increase write speed, but I will sacrifice the data security. Will see.

However, the 10Gb will not be limited only to TrueNAS, cause as I wrote, my provider will upgrade my current 2,5Gb fiber to 10Gb. So the 10Gb switch, network cards, and cables will be used also other then transfer file to nas.

 

[ChrisRJ]

It seems I can't get a good speed, did I do something wrong with the test?

Yes. You basically tested your CPU (because of dev/null) with a single-threaded transfer.

Check out fio for such tests.

For being able to leverage 10 Gbps on the NAS side you either need a lot of HDD mirrors or simply a pair of SSDs.

It would also help to know more about your hardware.

As to the core question, for me it falls into the category "If you have to ask, you don't need it".

 

[Lucas Rey]

Yes. You basically tested your CPU (because of dev/null) with a single-threaded transfer.

Check out fio for such tests.

fio gots worst result:

# fio --ramp_time=5 --gtod_reduce=1 --numjobs=1 --bs=1M --size=200G --runtime=60s --readwrite=write --name=testfile
testfile: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1
fio-3.28
Starting 1 process
testfile: Laying out IO file (1 file / 204800MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=115MiB/s][w=114 IOPS][eta 00m:00s]
testfile: (groupid=0, jobs=1): err= 0: pid=5182: Sat Feb 17 10:11:20 2024
write: IOPS=135, BW=136MiB/s (142MB/s)(8218MiB/60500msec); 0 zone resets
bw ( KiB/s): min=96256, max=251577, per=100.00%, avg=139211.31, stdev=21017.43, samples=118
iops : min= 94, max= 245, avg=135.47, stdev=20.53, samples=118
cpu : usr=0.68%, sys=7.85%, ctx=65748, majf=0, minf=1
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,8218,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: bw=136MiB/s (142MB/s), 136MiB/s-136MiB/s (142MB/s-142MB/s), io=8218MiB (8617MB), run=60500-60500msec

 

[ChrisRJ]

What is the speed you need?

You told is very little about your hardware. Please do so as per the forum rules.

 

[Lucas Rey]

Added some hardware info to the main post. This is homelab, so no great specs.

What is the speed you need?

I don't need specific trasnfer speed, I'm looking to increase as much as possible the current speed. As I wrote I can trasfer file at about 80/90 MB/sec with current 1GB connection.

 

[Etorix]

Disks are passed through directly to TrueNAS

Do you mean that this is a virtualised instance AND you're passing virtual disks rather than passing the controller?

Resource - "Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

[---- 2024/01/16: Still relevant. Virtualization really doesn't change much. Updates made as appropriate. ----] [---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some...

www.truenas.com

 

[Davvo]

Imho, no.

 

[Lucas Rey]

Do you mean that this is a virtualised instance AND you're passing virtual disks rather than passing the controller?

Yes, this is my config. I know it's dangerous, but in the last 4 years I got no issue at all.

Anyway, what could be a different config? I have no space for a HBA card, and my motherboard has 6 sata ports. 4 used for TrueNAS VM, and other 2 for proxmox hypervisor.
So I believe a cannot pass the whole controller to TrueNAS

 

[ChrisRJ]

Yes, this is my config. I know it's dangerous, but in the last 4 years I got no issue at all.

Assuming that your data are valuable to you, this is a really bad argument. It is the same as saying "I didn't need my seat belt since I started driving cars". Would you really not use it, just because so far everything has been fine?

Of course, if you are willing to accept an increased risk to loose data, that approach is fine and you should ignore my statement.

 

[Davvo]

Anyway, what could be a different config?

Changing the server. You will lose data this way.

Resource - "Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

[---- 2024/01/16: Still relevant. Virtualization really doesn't change much. Updates made as appropriate. ----] [---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some...

www.truenas.com

 

[danb35]

Anyway, what could be a different config?

As @Davvo says, use more suitable hardware. What you do or don't have space or budget for doesn't in the least mitigate the problems with the design you've chosen.

 

[Etorix]

Anyway, what could be a different config?

Put Proxmox on NVMe so you can pass the SATA controller to TrueNAS.
Run TrueNAS bare metal and the rest as VMs/jails/containers.
Or get more suitable hardware.

Anyway with your setting any performance issue could involve TrueNAS, Proxmox or a mix of both. It may be hard to diagnose and we cannot help you because none of the experts would run that kind of config.

 

[Lucas Rey]

Thank you for your advices, I'll evaluate them. However this is a homelab not a farm, and even if my data is precious I cannot spend so many money to change my server.
I know that 4 years without issues doesn't mean I'm safe, but currently I cannot do in any other way.

 

[Etorix]

and even if my data is precious I cannot spend so many money to change my server.

At least make sure you have a backup…

 

[Lucas Rey]

Are the snapshots stored on the same disks safe?
I mean a good way to restore data?

 

[Patrick M. Hausen]

Are the snapshots stored on the same disks safe?

Yes, if you intend snapshots to cover the frequent event of accidental deletion, misconfiguration and similar mishaps. Definitely no, if you ask about your unsuitable setup corrupting on disk data structures.

You would need a backup by replication of these snapshots to a second system.

If money for hardware is an issue but time tinkering is not so much, consider eliminating Proxmox from the equation and install TrueNAS SCALE natively. You get a "well it sorta works" container implementation and a solid VM implementation as well as all the NAS features with SCALE. The "apps" ecosystem is probably going to improve.

 

[Lucas Rey]

Is this the right controller to pass to TrueNAS?