Build questions, performance small files upgrade 11.2u7/13.u1.1

[modtta4455]

Hello,

I recently had to upgrade my old box from 2017 due to space contraints.

Setup as follows:
G3930
Asus p10s-ws c236
32gb ecc udimm
Bequiet Gold psu 400w (system draws around 120w from the wall during write via 10gb, so i guess 400w is plenty. I do have seasonic gold 650/750 here too but i guess thats overpowered/less efficient)
Fractal define 5
Boot pool mirrored 16gb optane nvme m2 ssds
Coming from 6x8tb raidz2 to 8x8tb raidz2. should be the interleave setup till 20tb come down in price. 8x20 --> 115 netto, almost exactly the capacity of my tape library :)

I know generally all here prefer enterprise gear. I do have another box in rack intel r2308gz4gc e5-2650v2//64gb ecc reg but that is a bit loud for home use and refusef nvme boot or bios hacking for that. Do have mirrores 850evo boot pool though. Keep that in case i go 8x20tb.

Use case: simple file storage for home data hoarder, 1 user. Simple backup to tar lto6 library msl4048. general 1gb cabling. 10gb just for backup via icx6450. Generally all is served to windows share. All my pcs are windows only servers run unix/bsd. Tape backup via windows type share via debian since i didnt get nfs share to work...

During installation i encountered the good old panic error with the e5. Reseated rams and took out 2nd 10gb controller solved it. No idea what that was.

During restore to the g3930 box from tape i had consistent 160mb/s (capping from uncompressed lto6 tape) with large files but horrible 0-25mb with small stuff. Is that normal? Dd write test shows around 480 write and 600 non cached read. Tape attached to 2nd box connected to 10gb with dual 2695v2//128gb so no bottleneck there.

Questions as follows:

Can my 32 gb handle my 8x8 build? Never ran into any issues with the 6x8 even at 93% full

Any ideas about the small files slow down? Does seem a bit too brutal. Maybe i can improve that with separate zil/slog? I do have 16gb optane nvme on hand if that helps. If it craps out early with writes dont care. Use the same model for boot pool. Snached a hand full new ones for 5 eur each... do use them with passive gembird pcie adapters so i do not block the sata ports with the 2 onboard m2.
Power loss is no issue. Server is behind ups with shutdown feature. Built that because of the tape drives that really hate blackouts...

Will the g3930 be enough? Doesnt seem to cap out. Has about 40% load with write. Dont use vms.

The tape thing was already discussed. Does truenas support qle2500 fc? I do have all fibrechannel tape and could power down one box during backup.

Any further things i left out or should consider?

Best wishes from germany/hungary

 

[modtta4455]

Huhu, any ideas, anyone?

 

[Etorix]

During restore to the g3930 box from tape i had consistent 160mb/s (capping from uncompressed lto6 tape) with large files but horrible 0-25mb with small stuff. Is that normal?

I'd suspect it is a SMB issue with small files, not ZFS.

Can my 32 gb handle my 8x8 build?

Probably. With enough RAM, the "1 GB RAM per data TB" rule of thumb relaxes, and there are no VMs/apps here.

Any ideas about the small files slow down? Does seem a bit too brutal. Maybe i can improve that with separate zil/slog?

All pools have a ZIL (reserved on the disks by ZFS) for sync writes. Dedicated SLOG devices help only with sync writes—which is likely not the case. So forget it.
(If you did need a SLOG, Optane is indeed the recommendation, but the 16 GB modules somewhat lack performance and endurance for this workload and larger drives would be better. Keep these 16 GB M10 Optane as boot drives; at 5 E a piece, they are perfect for this use.)

Will the g3930 be enough? Doesnt seem to cap out. Has about 40% load with write.

Then it's enough. SMB is single-thread and benefits from high frequency, not high cores.

 

[modtta4455]

I'd suspect it is a SMB issue with small files, not ZFS.

Do you have any ideas in which direction i should start searching? Do I need special parameters or manual tweaking / customizing?

Probably. With enough RAM, the "1 GB RAM per data TB" rule of thumb relaxes, and there are no VMs/apps here.

top. As of now, i do not feel any Problems but wanted to make sure.

All pools have a ZIL (reserved on the disks by ZFS) for sync writes. Dedicated SLOG devices help only with sync writes—which is likely not the case. So forget it.
(If you did need a SLOG, Optane is indeed the recommendation, but the 16 GB modules somewhat lack performance and endurance for this workload and larger drives would be better. Keep these 16 GB M10 Optane as boot drives; at 5 E a piece, they are perfect for this use.)

Thanks. I also thought SMB = Asynch, but wanted to make sure. I have 4 more in stock in case i need em.

Then it's enough. SMB is single-thread and benefits from high frequency, not high cores.

Top.

Any further things i left out or should consider?