SSD pool slow performance with iSCSI sync=always

[kobazik]

Hi,

I got a pool of 4 Transcend SSD370 256 GB TS256GSSD370 in raidz setup and maximum I get over iscsi MPIO is 120BM/s with sync=always . With sync=disabled I can get 200MB/s which statures two 1gbit iscsi path.

I understand that sync is expensive operation but I expected to be not a problem for SSD pool.

Does my pool still requires a dedicated let say intel zil/slog device even for SSD pool?

Any comments appreciated.

Thanks

Dom

 

[depasseg]

Does my pool still requires a dedicated let say intel zil/slog device even for SSD pool?

Only if the speed of the SLOG device can write faster than the speed of your pool. Run a quick test to find the speed of your zpool. If it's not 360MB/s, then an Intel S3700 will likely help. Use this thread to test the speed of your zpool: https://forums.freenas.org/index.php?threads/nfs-performance-numbers-i-need-more-speed.30499/

Here's the speed of my Intel S3700:

Code:

[root@freenas1] ~# dd if=/dev/zero of=/dev/ada1 bs=64M count=5000
^C724+0 records in
723+0 records out
48519708672 bytes transferred in 131.212239 secs (369780358 bytes/sec)
[root@freenas1] ~# 

Code:

[root@freenas1] ~# iostat ada1 -t 1
       tty            ada1             cpu
tin  tout  KB/t tps  MB/s  us ni sy in id
   0     6 3820.81   2  8.13   0  0  0  0 99
   0   134 128.00 2884 360.56   0  0  2  0 98
   0    47 128.00 2856 357.01   0  0  1  0 98
   0    47 128.00 2861 357.64   0  0  1  0 99
   0    47 128.00 2877 359.64   0  0  1  0 98
   0    47 128.00 2866 358.27   0  0  1  0 98
   0    47 128.00 2861 357.64   0  0  1  0 98
   0    47 128.00 2884 360.51   0  0  1  0 99

 

[kobazik]

I have another pool of 6 WD 5400rpm drives with Intel S3500 300GB (max 315MB/s sustained write) and this pool does >200MB/s over iSCSI MPIO using 2x 1gbit and sync=always.

Do I still need zil/slog for SSD pool?

I thought it is only needed for magnetic pools.

 

[depasseg]

The only purpose of having a SLOG is to write data faster than it's possible to write it to the pool. If your SLOG is faster than your pool (without a SLOG), then a SLOG will help. Think of it this way, when you are doing Sync writes, the ZIL must be written to disk. If you don't have a SLOG, then it will write it to the Pool.

What is the speed of your 4 disk RAIDZ SSD pool?

 

[diehard]

Even though your pool is SSD's, any intensive sync writes will still be fairly slow if writing ZIL to the zpool.

Either determine that you can accept the risk of un-synced writes or obtain an SLOG with low write latency and high single queue depth throughput. .. Or accept your current performance.

 

[kobazik]

Thanks I understand how zil works. Just a bit surprised that 4 SSDs in RAIDZ struggle so much with sync=always. Each SSD on each own can do seq. read/write 470MB/s and 75k random IOPS. I would expect it be able to saturate 2Gbits even with penalty of having sync=always.

Correct me if I'm wrong but in this case would need at least Intel S3500 240GB (sustained write of 260MB/s) for SLOG to be able to saturate 2Gbits iscsi traffic :(

 

[diehard]

Ignore those optimal throughput ratings and random IO numbers.

The drive's write latency and QD=1 throughput are what matters.

You might not even get 260MB/sec with a S3500.

 

[kobazik]

Ignore those optimal throughput ratings and random IO numbers.

The drive's write latency and QD=1 throughput are what matters.

You might not even get 260MB/sec with a S3500.

I"m currently getting 220MB/s on 6x WD 5400rpm "raid-10" mirror/stripped pool over 2Gbits iscsi MPIO with Intel S3500 300GB (sustained 315 MB/s write) as SLOG

 

[kobazik]

Interesting so if I had pool of lots fast SSDs over 10Gbit ethernet, I would then need at least a pair of Intel S3700 800GB (sustained 450MB/s write) to be able to get close to 1GB/s write over iscsi.

I think I'm getting closer to moving away of iSCSI and towards Ceph and Rados :D

 

[diehard]

You're worried about not getting over 1GB/sec write and then thinking about Ceph?

Ugh

 

[depasseg]

A pair of anything doesn't improve performance for a SLOG since they are mirrored not striped. At that point you should be looking at something like a PCIe SSD or Zeus.

And yea, Ceph is not the answer to this problem.

 

[cyberjock]

That dd test is not a good test to determine if a disk makes a good slog. Writes will *never* be 64MB (the size of the block chosen), in fact, they won't even exceed 128KB, and can be as small as 4KB. iozone is a better test, and somewhere a year or two ago I did a thread and provided the exact command for testing for slog performance.

 

[HoneyBadger]

@cyberjock I think it's the -o flag that should force synchronous writes to the test file, but don't quote me on that. Along with -s to set the blocksize.

 

[depasseg]

Yeah, I was testing something else with that dd command and just used it for a quick and dirty "what can I quickly push to this S3700".

 

[depasseg]

Is this the thread you were thinking of: https://forums.freenas.org/index.php?threads/donations.19945/#post-112745

 

[depasseg]

So I ran

Code:

iozone -r 4k -s 256G -i 0 -i 1 -i 2 -f -o /dev/ada1 > /mnt/tank/media/ada-iozone-results.txt

and it filled up my boot volume (trying it with -s 32GB as a test now). My guess is that the iozone logging data is temporarily held on the boot ram drive. So be careful.

 

[marbus90]

http://www.anandtech.com/show/8792/transcend-ssd370-128gb-256gb-512gb-review/2

This SSD model basically grinds to a halt very fast. Also try striped mirrors instead of raidz - should give you 2-4 times the troughput instead...

as for sync write performance, that'll always be low even with a dedicated SLOG device. 100MB/s on a S3700, 250MB/s on a P3700...

but: 100MB/s of totally random troughput is damn fast. that's 25k IOPS which are written to the pool safely (if you use SSDs with supercaps).