I've got this build up with the specs as per original post other than (a) now using E5-1650 v4 as the CPU (b) P4800x as the slog; (c) no need for a separate HBA since this mobo has a built in 3008 (which I flashed to IT, of course). I'm going to document my testing of this as I get it ready in the event that's helpful to someone else and in the hopes I'll get some helpful comments from the knowledgeable folks here.
I ran diskinfo -wS on all the array SSDs to make sure none of them was out of whack. They all looked identical, here's a sample:
Code:
root@freenas[~]# diskinfo -wS /dev/da7
/dev/da7
512 # sectorsize
1920383410176 # mediasize in bytes (1.7T)
3750748848 # mediasize in sectors
4096 # stripesize
0 # stripeoffset
233473 # Cylinders according to firmware.
255 # Heads according to firmware.
63 # Sectors according to firmware.
ATA SAMSUNG MZ7LH1T9 # Disk descr.
S455NY0M210671 # Disk ident.
id1,enc@n500304801f1e047d/type@0/slot@8/elmdesc@Slot07 # Physical path
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM
Not_Zoned # Zone Mode
Synchronous random writes:
0.5 kbytes: 125.2 usec/IO = 3.9 Mbytes/s
1 kbytes: 124.1 usec/IO = 7.9 Mbytes/s
2 kbytes: 126.5 usec/IO = 15.4 Mbytes/s
4 kbytes: 130.9 usec/IO = 29.8 Mbytes/s
8 kbytes: 138.9 usec/IO = 56.2 Mbytes/s
16 kbytes: 153.3 usec/IO = 101.9 Mbytes/s
32 kbytes: 196.6 usec/IO = 158.9 Mbytes/s
64 kbytes: 275.3 usec/IO = 227.0 Mbytes/s
128 kbytes: 422.3 usec/IO = 296.0 Mbytes/s
256 kbytes: 722.0 usec/IO = 346.2 Mbytes/s
512 kbytes: 1342.7 usec/IO = 372.4 Mbytes/s
1024 kbytes: 2543.1 usec/IO = 393.2 Mbytes/s
2048 kbytes: 4896.4 usec/IO = 408.5 Mbytes/s
4096 kbytes: 9583.3 usec/IO = 417.4 Mbytes/s
8192 kbytes: 19007.7 usec/IO = 420.9 Mbytes/s
And here's the same for the P4800X I'm using as a SLOG (I x-posted these results to the SLOG thread)
Code:
root@freenas[~]# diskinfo -wS /dev/nvd0
/dev/nvd0
512 # sectorsize
375083606016 # mediasize in bytes (349G)
732585168 # mediasize in sectors
0 # stripesize
0 # stripeoffset
INTEL SSDPED1K375GA # Disk descr.
PHKS7481007J375AGN # Disk ident.
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM
Synchronous random writes:
0.5 kbytes: 14.6 usec/IO = 33.4 Mbytes/s
1 kbytes: 14.6 usec/IO = 66.8 Mbytes/s
2 kbytes: 14.9 usec/IO = 130.8 Mbytes/s
4 kbytes: 12.0 usec/IO = 326.1 Mbytes/s
8 kbytes: 13.6 usec/IO = 573.8 Mbytes/s
16 kbytes: 18.1 usec/IO = 864.2 Mbytes/s
32 kbytes: 24.7 usec/IO = 1264.5 Mbytes/s
64 kbytes: 40.7 usec/IO = 1535.0 Mbytes/s
128 kbytes: 74.5 usec/IO = 1678.4 Mbytes/s
256 kbytes: 132.2 usec/IO = 1891.3 Mbytes/s
512 kbytes: 233.7 usec/IO = 2139.8 Mbytes/s
1024 kbytes: 436.1 usec/IO = 2292.8 Mbytes/s
2048 kbytes: 851.0 usec/IO = 2350.3 Mbytes/s
4096 kbytes: 1673.7 usec/IO = 2389.9 Mbytes/s
8192 kbytes: 3304.5 usec/IO = 2420.9 Mbytes/s
I'm not going to "burn in" any of the drives as this is an all flash setup. Any disagreement there?
My next step is to set-up the pool (I have 8 drives which I'm going to use as 4 mirrored zdevs for max performance in my VM SAN use case) and then run some dd tests with compression off. Any other local tests to try before I move into testing out the 10G ethernet?