Slow local disk reads, but writing decent

[geraldtheadmin]

I've recently upgraded to TrueNas 12 and reads from RAIDZ1 are down to about 30MB/s from the pool, about 10MB/s per disk. (performance was never blistering on this machine, but seems to have got worse).

The hardware is a HP Gen 8 Microserver 12Gb RAM, Celeron G1620T, booting TrueNas from USB pendrive. (I've read people have reservations about using USB pendrive boot and running on Celerons).

To investigate I removed all the disks and created a new pool on a test disk -- still reads are capped at 30MB/s.

doing various raw read and writes test to the single disk (various block sizes) still result in about 30MB/s

Code:

dd if=/dev/ada0 of=/dev/zero bs=1k count=400k

Testing the same disk in my Linux box I get 80MB/s reads . Writing to the disk in both Linux and TrueNas (direct and via a dataset in the ZFS pool) goes at about 79MB/s. During testing there was 6G or so of RAM free but processor load ave was 70%.

If feels like there's some sort of bottleneck reading from the onboard disk controller (B120i AHCI SATA mode).

Any ideas? I know it's all a bit underpowered, so I'm considering replacing the Celeron with a Xeon, replacing the pendrive with a SSD, adding a bit more RAM, or whether I'd be wasting my time? Any suggestions for more debugging of the hardware/drivers? Thanks.

 

[geraldtheadmin]

Found an spare RAID card and popped that in -- reads now seem to be limited only by my gigabit ethernet.

I also Discovered via freebsd forums tip on doing a read test with diskinfo, which might be useful for anyone needing to debug.

Code:

diskinfo -tv /dev/da0p1 

 

[HoneyBadger]

Found an spare RAID card

Danger.

Is this an actual "RAID card" complete with IR firmware and lousy drivers, or an LSI HBA that just happens to be masquerading as a RAID controller?

Either way - I would first look to something like drive cache being disabled in the BIOS of your Microserver. Writes won't be hit as heavily because ZFS does some "caching" of its own on writes.

 

[geraldtheadmin]

Is this an actual "RAID card" complete with IR firmware and lousy drivers, or an LSI HBA that just happens to be masquerading as a RAID controller?

It's a Dell PERC H200 -- being recognised as LSI SAS2008. These days I'm now slightly confused what passes as an actual RAID controller.

On board (and now disabled) it was B120i -- curiously this connected to the first 2 drives in the array at 6GBps and second 2 drives at 3GBps -- looking it up this is documented.

Either way - I would first look to something like drive cache being disabled in the BIOS of your Microserver. Writes won't be hit as heavily because ZFS does some "caching" of its own on writes.

Thanks, will check and do the experiments.

 

[HoneyBadger]

It's a Dell PERC H200 -- being recognised as LSI SAS2008. These days I'm now slightly confused what passes as an actual RAID controller.

Ah, that's good. My concern was that something was used that would have put another layer of abstraction (eg: each disk is its own RAID0, all disks in a RAID0 together)

On board (and now disabled) it was B120i -- curiously this connected to the first 2 drives in the array at 6GBps and second 2 drives at 3GBps -- looking it up this is documented.

That's pretty common for onboard SATA. For spinning disks a 3GBps limit is less of an impact but you'll feel the pinch on SSDs.

Thanks, will check and do the experiments.

That's my guess. Also ensure that the onboard SATA ports are set to "AHCI" instead of "RAID" if that's an option. I'm not certain either if you need to use the separate "Smart Storage Array" setup to set caching behavior.

 

[geraldtheadmin]

That's my guess. Also ensure that the onboard SATA ports are set to "AHCI" instead of "RAID" if that's an option. I'm not certain either if you need to use the separate "Smart Storage Array" setup to set caching behavior.

It was definitely set to "AHCI", not got round to finding the caching option yet.

Having just moved to TrueNAS 12, I've started playing with ZFS encryption options -- turning this on seems to reduce performance to similar poor levels of 30MB/s and CPU load ave is reported as 90%+. The Celeron has no aes encryption, limited threads and clock speed so I'm now wondering if that's part of the mix. Have seen enough potential in the existing server to justify ordering a Xeon to try. Will report back when it arrives.

 

[HoneyBadger]

Encryption will definitely have a significant impact on anything without hardware AES-NI instructions, and as you've already identified the Celeron as having fairly low horsepower to begin with. You could also boot a liveCD of Linux on the Microserver and just do some dd reads to see if the problem follows the hardware or the OS.

Which CPU did you purchase? Bear in mind that the Microservers were designed for low-TDP processors, so keep an eye on your temperatures if you put a conventional wattage Xeon inside.

 

[geraldtheadmin]

You could also boot a liveCD of Linux on the Microserver and just do some dd reads to see if the problem follows the hardware or the OS.

Subsequently I'm suspicious the original fake RAID hardware isn't particularly broken, just the processor didn't have enough grunt to service it, and previously it was on then edge, but going to TrueNAS 12 was subtle different of more demanding. As well as trying liveCD, might try booting Freenas 11 again and benchmarking while loading up the CPU. Wouldn't go back, though, love the replication/encryption so far in TrueNAS 12!

Which CPU did you purchase? Bear in mind that the Microservers were designed for low-TDP processors, so keep an eye on your temperatures if you put a conventional wattage Xeon inside.

I thought I'd bought a E3-1220Lv2 -- turns out google had auto 'corrected' it to a E3-1220v2, which has TDP of about 70W. I subsequently cancelled the order as I was concerned it wouldn't work. For the TDP the E3-1220Lv2 looks amazing, sadly, other people obviously think so, so struggling to get hold of one.
New gen 10 plus microserver seems easy option, but feels like there's still life in the old one. If only I hadn't gone for the 'bargain' celeron at the time!

 

[QonoS]

The E3-1220v2 is not as bad as you maybe think. 70W TDP is reached with Turbo activated. With Turbo deactivated those CPUs are quite okay from a power consumtion view point, especially for the price. You loose 400MHz with deactivating it, which means max. Freq. then is 3.1GHz with 4C4T. That is really sufficient. Another pluspoint is thermals which are much better and in case of PWM fan it stays quieter.

 

[geraldtheadmin]

The E3-1220v2 is not as bad as you maybe think. 70W TDP is reached with Turbo activated. With Turbo deactivated those CPUs are quite okay from a power consumtion view point, especially for the price. You loose 400MHz with deactivating it, which means max. Freq. then is 3.1GHz with 4C4T. That is really sufficient. Another pluspoint is thermals which are much better and in case of PWM fan it stays quieter.

Thanks wish I would have thought/known that before I cancelled -- have now ordered an i5 (though I suspect it doesn't have ECC :( ). Might look at the E3-1220v2 again and will report back when things arrive.

 

[ChrisRJ]

Unless you are running the CPU under full load all the time, TDP is irrelevant as an indicator for power consumption. At least for any somewhat recent CPU. Yes, TDP used to be more or less equivalent to power consumption, but that was more than ten years ago.

 

[Constantin]

For simple file SOHO servers using SMB, a CPU with fewer cores but higher clock speeds is likely preferable to a “wider but slower” CPU that has more cores but slower clock speeds.

That’s because SMB is still a single-thread process, so clock speed matters, having many cores does not ( TrueNAS SCALE may be changing some aspects of this in the future)

Having modern AE crypto support built in is a great to have, as are CPUs that can throttle their power needs commensurate with the processor load. Pretty much anything built in the last 10 years should feature both. The days of pre-Nehalem CPU room heaters are pretty much gone.