Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

Stymied with Microserver Gen8 Performance.

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE

Alex_L

Newbie
Joined
Apr 21, 2021
Messages
2
Hey,

I've been troubleshooting by googling and researching here for a couple of weeks, trying not to be 'that guy'. I've made a little progress, but need a hand!

I've had a 4-bay Drobo for a while now that it's time to replace. It has recently given me some heard stopping moments ("All your drives are broken!" red lights, to which the solution was to physically disassemble the thing, optimistically blow out some dust, put it back together again and it miraculously works - not the most confidence-inspiring thing).

So, I was after a system that I can leave on, and automate back up to - Time Machine & Full-disk backups from Macs, photos, videos, etc etc. Ultimately I'd also like to get similar set-ups for the rest of my family so we all have off-site backup with encrypted replication between sites so that the data is safe, but there's no nosing in each others' stuff because one needs boundaries. But that's for later! I want to get my setup working first...

Full of optimism, I picked up:
  • an HP Microserver Gen8 on eBay that had 16GB of ECC ram (it had a Celeron, I replaced it with a Xeon E3-1220L).
  • 4x shiny new 4TB Ironwolf drives (balancing ambition against being in trouble with my other half!).
  • 1x 512GB Samsung SATA SSD, and some kit to install it in the optical bay ("I'm not 100% sure it will be useful, but I can use it in something else if it isn't".
  • a usb -> mSATA adapter so I can boot from a 120GB Kingston mSATA drive I have, with the internal **USB 2.0** port.
It's a cracking little setup for what I want it for (AFAICT). Great. I updated the iLO & BIOS, which were both dated c. 2013.

First 'attempt' - 13-25 MB/s speeds
I installed TrueNAS on the mSATA drive on the internal USB 2 port, before putting any other drives in. This seems of note as it asked me if I wanted to create a swap partition to which my fresh eyes said 'of course, that sounds like a good idea!'. I then installed the rest of the drives, created a raidz1 pool with the IronWolf drives and left the SSD unconfigured.

It worked - I could copy, but only at 25MB/s. I poked about, disabling compression, atime and encryption, trying autotune - all with no effect.
Trying iperf gave me ~950 Mb/s (so theoretically ~115 MB/s) and 'dd' ing a 30GB file generated from /dev/random between the SSD and the main pool gave me 250 MB/s each way with 'zpool iostat'. It spiked to 400 going to the main pool, but I guess that's parity or some such.
At this point I discovered that two of the bays are SATA3, the other two and the ODD port are SATA2. Whilst not outstanding, it's ~SATA2 speed and perfectly in-line with what I hope the link-aggregated 1GbE connections will be, so all good there.

I tried SMB, I tried SCP, I tried AFP - all gave me 25MB/s. Hurrumph.

There was a little grasping at straws here and noticed that the USB mSATA drive was HOT. I saw a comment about swap. 25MB/s seemed suspiciously near half the throughput of a USB 2.0 link (25MB/s write + 25MB/s read ~= 480Mb/s (/ 8 ~=60MB/s). Was it copying everything to swap on the slow boot drive and then copying it to the pool? [I realise this is probably not how ZFS works and it is maybe a coincidence, but...]

[NB at this point reinstalling was faster than learning about managing swap on FreeBSD. Hey, my wife is expecting me to lay the new floor - there's only so much time in the world!]

I reinstall (with the Ironwolf drives already in), *it doesn't ask me about swap this time* and lo and behold...

Second 'attempt' - 45 MB/s speeds
Squinting a bit, that's double - and half a gigabit link. Check me out, troubleshooting all on my own. Still a way short, though. Clearly TrueNAS or ZFS don't like swap on a slow SSD! Or something.

Since then I've been tinkering to get up up towards the 100 MB/s mark. It hasn't budged. I've tried using NFS, WebDav, AFP, an SCP copy, SMB. I've tried autotune. The highest my CPU seems to go is about 15% utilisation. My ram usage varies between 3.6 and 8GB (of the 16 total) for services, but not more. The 'iperf' and 'dd' / 'pool iostat' results are the same.

Maybe that's okay and I just live with that. No need for link aggregation. It just seems a bit... weak. I'm sure it'll give me all the reliability and simple replication stuff that I need. I just don't have that inner glow of 'Cool, that's sorted!'.

Any ideas?

(NB I've also tried the Samsung SSD that (I think) I configured as an SLOG drive - and that made no difference, so I removed it and it's sitting unused. As I don't seem to be able to boot from that SATA, I'll probably remove it & use it for something else, unless you guys have any ideas!)
 

Basil Hendroff

Neophyte Sage
Joined
Jan 4, 2014
Messages
1,390
That does seem unusually low. I'm not at home atm, but from recollection, I get upwards of 85 MB/s from my Gen 8's.
 

KrisBee

Neophyte Sage
Joined
Mar 20, 2017
Messages
1,221
@Alex_L A HP microserver gen8 with your specs is definitely capable of saturating a 1Gbe link. My xfers from linux or win10 maxes at 117MB/s via SMB shares for single large files.

You did no say explicitly if all your clients are MacOS , but if so this is probably a case of tweaking the NAS share config. But before anything else, I would caution against using RAIDZ1 for reasons you can read about on this forum. A far safer pool topology for your four disks is RAIDZ2 or next best a stripe of two mirrors.

As to the swap thing, It's so long since I installed the system, as opposed to upgrading it, I've forgotten about that choice. In my case the system created a small 2G swap partition on each disk in my pool. In any case, doesn't "REPORTING/MEMORY" in the webui show swap utilisation as does "DISPLAY SYSTEM PROCESSES"? The later is the equiv of using the top command in a shell.
 
Last edited:

Alex_L

Newbie
Joined
Apr 21, 2021
Messages
2
Thanks for the responses, guys! I really appreciate it. I am mainly running macOS, yes. I've a linux box too and that was showing similar speeds. I didn't have a windows machine, but installed it on something with GbE. Strangely it started with a file copy dialogue that looked like the elevation profile from a bicycle race - 25MB/s - 60MB/s, but then just plateauing at 97 at the end of the 3GB file. Putting the same one across again gave me a fairly consistent 95.

I guess I have some tweaking to do!

Thanks for the pointers @KrisBee. Reading a little more on topology, I'll probably do the Z2 then - I really don't have *that* much to store so safety is likely better. I hadn't poked far enough to look at REPORTING/MEMORY - but yes, it does - zero utilisation of swap.

Okay - you've given me some hope with tinkering. I'll try different configurations and perhaps NFS instead or even AFP even though it isn't long for this world - and even if that doesn't work, I'm sure the speed will do.

Thanks again both!
 

Basil Hendroff

Neophyte Sage
Joined
Jan 4, 2014
Messages
1,390
That does seem unusually low. I'm not at home atm, but from recollection, I get upwards of 85 MB/s from my Gen 8's.
Just confirming I am averaging around 110 MB/s (both read and write) for large file transfers between a Win 10 client and RAID-Z1 pool on a Gen 8 microserver running TrueNAS 12.0-U3. Note that I'm also using slightly slower WD Red+ HDDs @5400 RPM over the faster Seagate IronWolf @7200 RPM. I have some IronWolf in the mix somewhere, but I prefer the Reds for the compact microserver form factor as they run cooler and quieter. While RAID-Z1 is no longer 'flavour of the month', I mitigate the risk by replicating datasets to backup TrueNAS microservers. If required, the pool can be reconstructed by reassembling these datasets. Storage-wise, this approach allows me to make the best use of the microserver's four drive bays while still managing my data protection risks.

copy.jpg
 
Last edited:
Top