SAS 15k rpm drive performance question

[Pitfrr]

Hi,


I'm playing around with 15k RPM SAS drives and I'd like to understand an unexpected (for me that is ;-) ) behaviour.
It's not directly related to FreeNAS so I'm not sure to post in the right category...

For the hardware:
I'm using a HP DL380G6 with dual xeon L5640 and 208GB of RAM. It's running ESXi 6.5 and FreeNAS 11 in a VM (with 64GB dedicated RAM and 4 vCPU). An LSI flashed HBA is passed through to the FreeNAS VM.
There are 5 SAS 146GB @15k RPM drives connected.

I thought first to wipe the drives and used the FreeNAS GUI. It was very slow, like two hours.
So I used dd to check it (and I found out later that the FreeNAS GUI does the same).
With dd if=/dev/zero | pv | dd of=/dev/da1 I played a bit (with the block sizes) and the best I could get out of it was 26MB/s.

I was surprised, not quite what I would have expected for a 15k RPM drive... :-O (I never used 15k RPM drives before).

I thought "well ok, maybe there is a hardware incompatibility somewhere.... too bad".

And I went on and created a RAIDz2 volume (I tried with RAIDz and RAIDz3 as well) and a shared (SMB) dataset and started to copy data on it (through a 1 gigabit network).
To my surprise I got a 110MB/s throughput (the maximum for a gigabit network). So here I thought the RAM is doing the caching but even when the RAM was filled up I still had a sustained 100-110MB/s (I filled up the dataset completely).
And during a SCRUB I had speeds of approx. 690MB/s.

So I checked the zero generator (I first used the /dev/urandom generator): dd if=/dev/zero | pv | dd of=/dev/null and had about 100MB/s (and with the /dev/urandom generator only 37MB/s that's why I didn't use it).


I would like to understand why the dd command is so slow when zeroing a drive, what am I missing? I tried it with SATA drives (@5900rpm) and I got a speed of 130MB/s by comparison.
And why is the urandom generator so slow (it seems normal since on my test system I get like 28MB/s but out of curiosity I'd like to understand)?
I'm also surprised that the zero generator seems a bit slow compared to my test system where I get 408MB/s?!? (would that be because of Vmware?)

Otherwise the system performs quite well (at least for my usage on a gigabit network).
From a 15k drive I would expect more iOPs (but in a single user environment I wouldn't feel the difference I suppose) and a bit more throughput on the drives (but with a gigabit network, no big difference neither) compared to what I’m used to.

Thanks for your feedback!

 

[kdragon75]

Sounds like you have a grasp on what your doing. I didn't see you mention what block sizes you used. i would try dd if=/dev/zero of=/dev/daX bs=200k. You should get around 200MB/s.

EDIT: 200k = 2048000 Bytes... Stupid math.

 

[Pitfrr]

I tried several different block sizes like:
8B; 16B; 32B; 64B; 128B; 256B; 512B;
1k; 2k; 4k; 8k; 16k; 32k; 64k; 128k; 256k; 512k;
1M; 2M; 4M; 8M: 16M
(even tried 256M :-P)

I got the best speeds with 256 (either B, k or M) therefore I used bs=256k

I also used the option conv=notrunc but that shouldn't have any effect on the speed.

 

[MrToddsFriends]

And why is the urandom generator so slow (it seems normal since on my test system I get like 28MB/s but out of curiosity I'd like to understand)?

The PRNG in FreeBSD implements the Fortuna algorithm. It's no big surprise for me that using it in the way you do results in a CPU bound benchmark.

https://en.wikipedia.org/wiki//dev/random#FreeBSD
https://forums.freenas.org/index.ph...e-benchmarks-and-cache.981/page-8#post-457992