bad or normal performance ?

[Branislav Kirilov]

I made a freenas server that occasionally serves windows machines that boot from iscsi zvols on that server. The pool has 4 intel ssd's in raidz1. There is separate storage pool from 6 mechanical disks also in raidz1.
My issue is that the performance I get from these ssd's is something that i should be pulling from single one or at least should be better.

Here are the tests first is SSD pool

Code:

dd if=/dev/zero of=/mnt/ssdstorage/test/temp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 315.186943 secs (340668244 bytes/sec)
dd if=/mnt/ssdstorage/test/temp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 248.083857 secs (432814064 bytes/sec)

Second pool from mechanical disks

Code:

dd if=/dev/zero of=/mnt/storage/test/temp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 472.189081 secs (227396581 bytes/sec)
dd if=/mnt/storage/test/temp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 264.581848 secs (405825960 bytes/sec)

Pool, is heavily fragmented, but with the type of usage that we have it is inevitable.

Code:

NAME           SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
ssdstorage    1.73T  1.29T   459G         -    46%    74%  1.00x  ONLINE  /mnt
storage       16.2T  5.45T  10.8T         -    16%    33%  1.00x  ONLINE  /mnt

Other info

Code:

last pid: 15020;  load averages:  0.25,  0.36,  0.29        up 76+01:01:43  16:02:32
70 processes:  1 running, 69 sleeping
CPU:  1.0% user,  0.0% nice,  3.5% system,  0.1% interrupt, 95.4% idle
Mem: 120M Active, 1118M Inact, 44G Wired, 8980K Cache, 1724M Free
ARC: 36G Total, 8318M MFU, 24G MRU, 11M Anon, 593M Header, 3551M Other
Swap: 20G Total, 80K Used, 20G Free

So when i put heavy write load to the server this is all i get. See the screenshots from gstat and iwstat, which seems very low to me.

What is worse that, everything that is connected to that pool, servers via iscsi and VM-s with qcow drives connected via NFS, they all get extremely sluggish and unresponsive.
So I was wondering can the bottleneck be somewhere else except the drives?
As i see it it's the drives.



Screen 3 is the usual disk usage that i see under heavy write, but disks, should be able to do around 200. At least 200 is what i saw when i did the dd tests.

What I'm missing or it's all normal?

 

[Mirfster]

The pool has 4 intel ssd's in raidz1.

Any particular reason you went with RaidZ1 instead of Mirrors?

What Make/Model SSDs?

ssdstorage 1.73T 1.29T 459G - 46% 74% 1.00x ONLINE /mnt

Your Pool is over 50% full and fragmented; I believe that this is where things start to slow down with iSCSI. Would wait for the more knowledgeable contributors to chime in though.

 

[Mirfster]

Tossing in suggested read written by the man, the myth, the legend @jgreco "Some differences between RAIDZ and mirrors, and why we use mirrors for block storage"

 

[Branislav Kirilov]

Yes I've read his post which explains a lot and I also found this http://constantin.glez.de/blog/2010/06/closer-look-zfs-vdevs-and-performance#raidz which kind of confirms it. I've chose raidz because of disk space, which wouldn't be enough with mirror. I knew mirror has better performance and even so I was expecting better performance from raidz.
Tomorrow we will build another server with 4 ssd's and i will do some tests to see if something better will come out from mirror.

@Mirfster Drives are

Intel SSD 540s Series (480GB, 2.5in SATA 6Gb/s, 16nm, TLC)
Sequential Read 560 MB/s;
Sequential Write 480 MB/s;
Random Read (8GB Span) 78000 IOPS;
Random Write (8GB Span) 85000 IOPS;

And we are not even scratching the speed of 1 disk and that's why i wrote here. Don't know if I can check something else for a problem, or it's normal given the circumstances. I don't have any prior experience with ZFS.

 

[jgreco]

Perhaps you could more fully document your hardware and networking setup.

 

[Branislav Kirilov]

Networking is trunk of 2x4 port intel cards.
MB is Asus P6T7 with Xeon L5639
Memory is 48GB non ECC obviously. I'll test tomorrow on 32gb ecc MB i have plenty of them.
SSD's are connected on HBA

mps0: <Avago Technologies (LSI) SAS2004> port 0xa000-0xa0ff mem 0xfbafc000-0xfbafffff,0xfba80000-0xfbabffff irq 30 at device 0.0 on pci9
mps0: Firmware: 20.00.04.00, Driver: 20.00.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>

which is IBM SERVERAID H1110 flashed with IT firmware of LSI 9211

 

[jgreco]

Trunking probably isn't helping.

The ancient Xeon with a slow clock CPU platform definitely isn't helping.

The HBA may be less-than-ideal for SSD connectivity (~450MB/sec vs ~500MB+ on mainboard ports).

74% capacity at that fragmentation suggests the system is struggling to allocate new space.

RAIDZ is rarely good for block storage. It'd be really interesting to see how mirrors compare.

 

[maglin]

I wanted ISCSI but performance of it with FreeNAS requires a lot of RAM and free disk space. Not to mention IOPS. Instead of a full gstat use gstat -p and only attached geoms will be displayed.

Maybe use 8 SSDs to build a stripped mirror array. And I believe the expensive SSDs are not required for the pool.


Sent from my iPhone using Tapatalk

 

[Branislav Kirilov]

maglin, I'm not sure what you mean by striped mirror. For 8 drives i make 4x2 mirrors, then i join them in stripe (not sure if i can do that in freenas)?
Or I select 4x2 and select stripe when creating the pool.

Here are some tests I did on another MB with 4 180GB intel ssd's connected on sata2 ports.
32GB ecc ram

single drive

Code:

107374182400 bytes transferred in 472.796067 secs (227104644 bytes/sec)
107374182400 bytes transferred in 400.472840 secs (268118513 bytes/sec)

2 drives stripe

Code:

107374182400 bytes transferred in 245.977435 secs (436520457 bytes/sec)
107374182400 bytes transferred in 207.326221 secs (517899675 bytes/sec)

2 drives mirror

Code:

107374182400 bytes transferred in 481.734910 secs (222890598 bytes/sec)
107374182400 bytes transferred in 206.421118 secs (520170530 bytes/sec)

stripe 4 disks

Code:

107374182400 bytes transferred in 137.301966 secs (782029461 bytes/sec)
107374182400 bytes transferred in 122.194672 secs (878714109 bytes/sec)

raidz1 4 disks

Code:

107374182400 bytes transferred in 165.938397 secs (647072554 bytes/sec)
107374182400 bytes transferred in 124.265403 secs (864071415 bytes/sec)

mirror 2x2

Code:

107374182400 bytes transferred in 266.218392 secs (403331196 bytes/sec)
107374182400 bytes transferred in 122.972793 secs (873153971 bytes/sec)

P.S what is difference in selecting 4x1 or 1x4 when creating stripe for example. Does it matter if it's vertical or horizontal?

 

[jgreco]

Some of us have long considered the volume manager to be a bit of a design disaster.

I don't know which way means which, you kinda gotta look and figure it out yourself. One of those means four drives in a four way mirror, one vdev of that. Extreme redundancy. The other means one drive in a one-way mirror (i.e. not actually mirrored), four vdevs worth of that. No redundancy.

 

[Bidule0hm]

1 line = 1 vdev.
1 column = 1 drive in a vdev.