OK, tried again this morning with a couple of 128GB 6Gb/s SSDs:
Code:
=== START OF INFORMATION SECTION ===
Model Family: Marvell based SanDisk SSDs
Device Model: SanDisk SD6SB1M128G1022I
Serial Number: 143874401733
LU WWN Device Id: 5 001b44 c8b2a21c5
Firmware Version: X231600
User Capacity: 128,035,676,160 bytes [128 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: Unknown (0x000a)
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 6
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Apr 14 06:24:20 2015 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Code:
[root@plexnas] ~# iostat -C -w 2 -d -t da /dev/da0
da0 da1 da2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id
127.96 466 58.19 55.75 0 0.02 11.44 12 0.13 1 0 0 0 99
128.00 3322 415.21 0.00 0 0.00 0.00 0 0.00 0 0 0 0 99
128.00 3315 414.36 0.00 0 0.00 0.00 0 0.00 0 0 0 0 99
128.00 3318 414.73 0.00 0 0.00 0.00 0 0.00 0 0 0 0 99
128.00 3319 414.92 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 3322 415.23 0.00 0 0.00 0.00 0 0.00 0 0 0 0 99
128.00 3328 415.98 0.00 0 0.00 0.00 0 0.00 0 0 0 0 99
128.00 3320 415.04 0.00 0 0.00 0.00 0 0.00 0 0 0 0 99
128.00 3329 416.10 0.00 0 0.00 0.00 0 0.00 0 0 0 0 99
128.00 3330 416.29 0.00 0 0.00 0.00 0 0.00 0 0 0 0 99
128.00 3329 416.17 0.00 0 0.00 0.00 0 0.00 0 0 0 0 99
128.00 3324 415.48 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
Started the second DD:
128.00 3320 414.98 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 3327 415.85 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 3324 415.48 0.00 0 0.00 0.00 0 0.00 0 0 0 0 99
128.00 3323 415.35 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 3316 414.48 0.00 0 0.00 0.00 0 0.00 0 0 0 0 99
128.00 3316 414.48 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 3292 411.55 0.00 0 0.00 0.00 0 0.00 0 0 0 0 99
128.00 3327 415.85 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 3329 416.10 0.00 0 0.00 0.00 0 0.00 0 0 0 0 99
128.00 3329 416.17 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 3333 416.60 0.00 0 0.00 0.00 0 0.00 0 0 0 0 99
128.00 3327 415.85 0.00 0 0.00 0.00 0 0.00 0 0 0 0 99
This time there was almost no change at all in the performance of the drive once I started the second dd command.
Here is the graph. This first long read was a single dd command, the second one was two dd commands running at the same time (dd if=/dev/da0 of=/dev/null bs=8M).
The slight gap in the first one was me stopping the process by accident.
So now I am beginning to wonder. The SSDs are 6 Gb/s, the LSI3008 is a 6 Gb/s HBA, the SAS expander is 6 Gb/s and my HGST drives are SATA3 6 Gb/s but obviously something is not right. So I decided to pull all my drives and go with just two drives plugged into the SAS expander (Like the SSDs) and try again, maybe it has something to do with the number of drives plugged into the expander (I know, I know, I'm reaching here....but worth a test):
Here is the smartctl output for the for the drives being tested:
Code:
=== START OF INFORMATION SECTION ===
Model Family: HGST Deskstar NAS
Device Model: HGST HDN724040ALE640
Serial Number: PK2338P4H9LXSC
LU WWN Device Id: 5 000cca 249d275e5
Firmware Version: MJAOA5E0
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Apr 14 07:01:07 2015 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
The only thing interesting that I note is the sector size (512 logical but 4096 physical). Since I am seeing issues with the drives while addressing them directly, I don't think there should be any issues with forcing 4k sector sizes (which I think is the default in freenas now when you create a new pool).
and...FAIL....
Code:
[root@plexnas] ~# iostat -C -w 2 -d -t da /dev/da0
da0 da1 da2 cpu
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id
128.00 1313 164.15 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 1325 165.60 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 1278 159.73 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 1304 162.98 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 1296 162.04 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 1303 162.92 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 1316 164.48 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 1323 165.42 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 988 123.50 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
Second DD started:
128.00 436 54.47 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 351 43.85 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 313 39.17 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 321 40.17 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 435 54.35 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 358 44.73 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 321 40.17 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 316 39.48 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
128.00 251 31.42 0.00 0 0.00 0.00 0 0.00 0 0 0 0 100
So I grabbed a Seagate SATA3 6 Gb/s drive we had laying around still wrapped and tried again:
Code:
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST250DM000-1BD141
Serial Number: Z3TS6YMX
LU WWN Device Id: 5 000c50 06508eab2
Firmware Version: KC48
User Capacity: 250,059,350,016 bytes [250 GB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Apr 14 07:45:41 2015 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Code:
[root@plexnas] ~# iostat -C -w 2 -d -t da /dev/da0
da0 da1 cpu
KB/t tps MB/s KB/t tps MB/s us ni sy in id
127.70 35 4.33 11.56 17 0.19 1 0 0 0 99
128.00 1092 136.55 0.00 0 0.00 0 0 0 0 100
128.00 1091 136.37 0.00 0 0.00 0 0 0 0 100
128.00 1096 137.06 0.00 0 0.00 0 0 0 0 100
128.00 1092 136.56 0.00 0 0.00 0 0 0 0 100
128.00 1096 137.06 0.00 0 0.00 0 0 0 0 100
128.00 1091 136.43 0.00 0 0.00 0 0 0 0 100
128.00 1097 137.12 0.00 0 0.00 0 0 0 0 100
Second DD Command started:
128.00 242 30.30 0.00 0 0.00 0 0 0 0 100
128.00 209 26.17 0.00 0 0.00 0 0 0 0 100
128.00 229 28.67 0.00 0 0.00 0 0 0 0 100
128.00 225 28.17 0.00 0 0.00 0 0 0 0 100
128.00 188 23.55 0.00 0 0.00 0 0 0 0 100
128.00 223 27.92 0.00 0 0.00 0 0 0 0 100
And here is the graph:
So I now know it is not specific to the HGST drives, although it appears (as of right now) to be specific to spinning media.
Tomorrow I will try a couple of 15K RPM SAS drives that I have left over from another project and see what happens with those drives.