Small transfer speeds showing up for HDDs in syslog

[dmt0]

Hi All!

My FreeNAS performance is really bad.
Perhaps a big part of it is due to incorrect tuning.
However I've gone through a lot of values of many commonly tuned loaders and still didn't get good results.
I thought that the following issue might have a say in the performance too, or perhaps it might turn out to be a part of a bigger issue...


My Spec:
FreeNAS 8.0.3 RC3 x86 (32bit).
P4 2.8 GHz
4GB ram
4 identical WD HDDs - WD20EARS - SATA 3 Gb/s
Mobo: MSI 945GCM5 with Intel ICH7 chipset with on-board SATA controller that is supposed to allow 300MB/s transfer speeds.

I've looked through all the possible BIOS settings and didn't find anything relevant (could I have missed something?)


Here's my the relevant part of syslog:

Jan 9 23:58:36 freenas kernel: ada0 at ata0 bus 0 scbus0 target 1 lun 0
Jan 9 23:58:36 freenas kernel: ada0: <CF Card Ver5.04> ATA-5 device
Jan 9 23:58:36 freenas kernel: ada0: 100.000MB/s transfers (UDMA5, PIO 512bytes)
Jan 9 23:58:36 freenas kernel: ada0: 7279MB (14909328 512 byte sectors: 16H 63S/T 14791C)
Jan 9 23:58:36 freenas kernel: ada1 at ata2 bus 0 scbus1 target 0 lun 0
Jan 9 23:58:36 freenas kernel: ada1: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
Jan 9 23:58:36 freenas kernel: ada1: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
Jan 9 23:58:36 freenas kernel: ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Jan 9 23:58:36 freenas kernel: ada2 at ata2 bus 0 scbus1 target 1 lun 0
Jan 9 23:58:36 freenas kernel: ada2: <WDC WD20EARS-22MVWB0 51.0AB51> ATA-8 SATA 2.x device
Jan 9 23:58:36 freenas kernel: ada2: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
Jan 9 23:58:36 freenas kernel: ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Jan 9 23:58:36 freenas kernel: ada3 at ata3 bus 0 scbus2 target 0 lun 0
Jan 9 23:58:36 freenas kernel: ada3: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
Jan 9 23:58:36 freenas kernel: ada3: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
Jan 9 23:58:36 freenas kernel: ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Jan 9 23:58:36 freenas kernel: ada4 at ata3 bus 0 scbus2 target 1 lun 0
Jan 9 23:58:36 freenas kernel: ada4: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
Jan 9 23:58:36 freenas kernel: ada4: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
Jan 9 23:58:36 freenas kernel: ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)


Does anyone have an idea of why the drives do not work at 300MB/s as they are supposed to?

Cheers!

 

[b1ghen]

I can't really tell you why they show up as SATA 1.5Gb/s devices, but I doubt this will affect performance in any way. What speeds are you getting out of your pool?

 

[William Grzybowski]

Yeah, those disks definitely cannot saturate a SATA1 link

 

[dmt0]

I can't really tell you why they show up as SATA 1.5Gb/s devices, but I doubt this will affect performance in any way. What speeds are you getting out of your pool?

Well, I get a write speed of 12MBpS. Maximum I ever got out of my system is around 70MBpS - that was on 8.0.0. I don't know if I had that issue back than.

Basically what I'm thinking is that this could be a symptom of a bigger issue that someone has dealt with before.

 

[William Grzybowski]

Are you using ZFS? Did you check the 4k sector size box while creating it?

 

[dmt0]

Are you using ZFS? Did you check the 4k sector size box while creating it?

Yes, it's ZFS.
I'd need to check this evening if I use 4k sector...

 

[dmt0]

Are you using ZFS? Did you check the 4k sector size box while creating it?

Time didn't make this problem go away... :)

I just checked that I actually have 512K blocks:

# zpool set cachefile=/data/zfs/zpool.cache MainOne
# zdb -U /data/zfs/zpool.cache | grep ashift
ashift=9

Is there a way to move them to a 4k sector without destroying the data? I can't find the answer to this anywhere...

Are jumpers on those EARS drives a way to go?

 

[Joshua Parker Ruehlig]

i didn't check 4k either, I'm 90% sure you can't change once you've created a vdev. I waiting for more drives before I correclty create my raidz with 4k blocks. Live and learn

 

[dmt0]

So, just went through the process of moving all the data to the temporary storage, and reformatted my ZFS, forcing 4K sectors (in the meantime, also replaced one failing drive...).
Checked through the CLI, now it's indeed 4K:
ashift=12

Looking at console at WebGUI I see this:

Mar 17 18:03:15 freenas kernel: ada4 at ata3 bus 0 scbus2 target 1 lun 0
Mar 17 18:03:15 freenas kernel: ada4: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
Mar 17 18:03:15 freenas kernel: ada4: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
Mar 17 18:03:15 freenas kernel: ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)

for all drives.

Which number should I believe?

The transfer speed has not changed by the way.
It is now at about 100K per second - completely unusably slow.

When I try to create files using dd locally - I get 59 Megabyte/second speed. So it seems only transfers over NFS are problematic.
Disabling ZIL didn't do anything to performance.

Also tested my hypothesis that performance went down because I went from having 3 drives to 4. Created ZFS with 3 drives - no change in performance.

What to do?

 

[dmt0]

Looking back into my journal, it seems that the speed dropped dramatically (to 100K/s) after I changed a certain tunable (kern.maxvnodes 250000 + vfs.zfs.txg.write_limit_override 256M). But after I changed the tunable back to the original value - the speed didn't come back. Which got me thinking that some kind of corruption happened in the database or underlying configuration. So I decided to abandon the configuration and installed from scratch - using 8.2.0 Beta 2 (wanted to try the autotuner as well).
No change - still got he same speed.
Switch to a different network interface and there it is!!! Got a speed of 1.3 M/s without proper tunables.
What the hell has happened is beyond me. At some point the network interfaces decided to stop working properly - it's an Intel PRO/1000 GT.
Now I'm back to my onboard Broadcom thing that got new driver at around 8.02 I believe and should be working with jumbo frames now.
Having adjusted all the tunables (that used to give me 40MB/s in the past), I now get 3.8MB/s.

 

[lrusak]

Jan 9 23:58:36 freenas kernel: ada1 at ata2 bus 0 scbus1 target 0 lun 0
Jan 9 23:58:36 freenas kernel: ada1: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
Jan 9 23:58:36 freenas kernel: ada1: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
Jan 9 23:58:36 freenas kernel: ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)

Does anyone have an idea of why the drives do not work at 300MB/s as they are supposed to?

Cheers!

I had this happen to me as well. Changing the drives from AHCI to IDE in the BIOS corrected this

 

[dmt0]

Unfortunately, there's no such option in my BIOS :(

Restored the system to an old configuration that is known to work. Same speed issue. I guess it makes it a hardware problem that just happened.