I think my FreeNAS 9.10.2-U6 is no longer using my L2ARC

Itamar Croitoru · Nov 19, 2018

HoneyBadger said:
It's possible there's a second controller that sas2ircu doesn't recognize, so it isn't displaying it.

pciconf -lv will generate a LOT of info but hopefully one of the lines will give us a clue on if there's an additional SAS card in play.

The entire output:

Code:

[root@freenas] ~# sas2ircu list
LSI Corporation SAS2 IR Configuration Utility.
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved.


		 Adapter	  Vendor  Device					   SubSys  SubSys
 Index	Type		  ID	  ID	Pci Address		  Ven ID  Dev ID
 -----  ------------  ------  ------  -----------------	------  ------
   0	 SAS2308_2	 1000h	87h   00h:04h:00h:00h	  1000h   3000h
SAS2IRCU: Utility Completed Successfully.
[root@freenas] ~# pciconf -lv
hostb0@pci0:0:0:0:	  class=0x060000 card=0x02a31028 chip=0xd1308086 rev=0x11 hdr=0x00
	vendor	 = 'Intel Corporation'
	device	 = 'Core Processor DMI'
	class	  = bridge
	subclass   = HOST-PCI
pcib1@pci0:0:3:0:	   class=0x060400 card=0x02a31028 chip=0xd1388086 rev=0x11 hdr=0x01
	vendor	 = 'Intel Corporation'
	device	 = 'Core Processor PCI Express Root Port 1'
	class	  = bridge
	subclass   = PCI-PCI
pcib2@pci0:0:5:0:	   class=0x060400 card=0x02a31028 chip=0xd13a8086 rev=0x11 hdr=0x01
	vendor	 = 'Intel Corporation'
	device	 = 'Core Processor PCI Express Root Port 3'
	class	  = bridge
	subclass   = PCI-PCI
none0@pci0:0:8:0:	   class=0x088000 card=0x00000000 chip=0xd1558086 rev=0x11 hdr=0x00
	vendor	 = 'Intel Corporation'
	device	 = 'Core Processor System Management Registers'
	class	  = base peripheral
none1@pci0:0:8:1:	   class=0x088000 card=0x00000000 chip=0xd1568086 rev=0x11 hdr=0x00
	vendor	 = 'Intel Corporation'
	device	 = 'Core Processor Semaphore and Scratchpad Registers'
	class	  = base peripheral
none2@pci0:0:8:2:	   class=0x088000 card=0x00000000 chip=0xd1578086 rev=0x11 hdr=0x00
	vendor	 = 'Intel Corporation'
	device	 = 'Core Processor System Control and Status Registers'
	class	  = base peripheral
none3@pci0:0:8:3:	   class=0x088000 card=0x00000000 chip=0xd1588086 rev=0x11 hdr=0x00
	vendor	 = 'Intel Corporation'
	device	 = 'Core Processor Miscellaneous Registers'
	class	  = base peripheral
none4@pci0:0:16:0:	  class=0x088000 card=0x00000000 chip=0xd1508086 rev=0x11 hdr=0x00
	vendor	 = 'Intel Corporation'
	device	 = 'Core Processor QPI Link'
	class	  = base peripheral
none5@pci0:0:16:1:	  class=0x088000 card=0x00000000 chip=0xd1518086 rev=0x11 hdr=0x00
	vendor	 = 'Intel Corporation'
	device	 = 'Core Processor QPI Routing and Protocol Registers'
	class	  = base peripheral
ehci0@pci0:0:26:0:	  class=0x0c0320 card=0x02a31028 chip=0x3b3c8086 rev=0x05 hdr=0x00
	vendor	 = 'Intel Corporation'
	device	 = '5 Series/3400 Series Chipset USB2 Enhanced Host Controller'
	class	  = serial bus
	subclass   = USB
pcib3@pci0:0:28:0:	  class=0x060400 card=0x02a31028 chip=0x3b428086 rev=0x05 hdr=0x01
	vendor	 = 'Intel Corporation'
	device	 = '5 Series/3400 Series Chipset PCI Express Root Port 1'
	class	  = bridge
	subclass   = PCI-PCI
pcib4@pci0:0:28:4:	  class=0x060400 card=0x02a31028 chip=0x3b4a8086 rev=0x05 hdr=0x01
	vendor	 = 'Intel Corporation'
	device	 = '5 Series/3400 Series Chipset PCI Express Root Port 5'
	class	  = bridge
	subclass   = PCI-PCI
ehci1@pci0:0:29:0:	  class=0x0c0320 card=0x02a31028 chip=0x3b348086 rev=0x05 hdr=0x00
	vendor	 = 'Intel Corporation'
	device	 = '5 Series/3400 Series Chipset USB2 Enhanced Host Controller'
	class	  = serial bus
	subclass   = USB
pcib5@pci0:0:30:0:	  class=0x060401 card=0x02a31028 chip=0x244e8086 rev=0xa5 hdr=0x01
	vendor	 = 'Intel Corporation'
	device	 = '82801 PCI Bridge'
	class	  = bridge
	subclass   = PCI-PCI
isab0@pci0:0:31:0:	  class=0x060100 card=0x02a31028 chip=0x3b148086 rev=0x05 hdr=0x00
	vendor	 = 'Intel Corporation'
	device	 = '3420 Chipset LPC Interface Controller'
	class	  = bridge
	subclass   = PCI-ISA
atapci0@pci0:0:31:2:	class=0x01018f card=0x02a31028 chip=0x3b208086 rev=0x05 hdr=0x00
	vendor	 = 'Intel Corporation'
	device	 = '5 Series/3400 Series Chipset 4 port SATA IDE Controller'
	class	  = mass storage
	subclass   = ATA
atapci1@pci0:0:31:5:	class=0x010185 card=0x02a31028 chip=0x3b268086 rev=0x05 hdr=0x00
	vendor	 = 'Intel Corporation'
	device	 = '5 Series/3400 Series Chipset 2 port SATA IDE Controller'
	class	  = mass storage
	subclass   = ATA
mps0@pci0:4:0:0:		class=0x010700 card=0x30001000 chip=0x00871000 rev=0x01 hdr=0x00
	vendor	 = 'LSI Logic / Symbios Logic'
	device	 = 'SAS2308 PCI-Express Fusion-MPT SAS-2'
	class	  = mass storage
	subclass   = SAS
em0@pci0:5:0:0: class=0x020000 card=0x135e8086 chip=0x105e8086 rev=0x06 hdr=0x00
	vendor	 = 'Intel Corporation'
	device	 = '82571EB Gigabit Ethernet Controller'
	class	  = network
	subclass   = ethernet
em1@pci0:5:0:1: class=0x020000 card=0x135e8086 chip=0x105e8086 rev=0x06 hdr=0x00
	vendor	 = 'Intel Corporation'
	device	 = '82571EB Gigabit Ethernet Controller'
	class	  = network
	subclass   = ethernet
mpt0@pci0:3:0:0:		class=0x010000 card=0x1f0f1028 chip=0x00581000 rev=0x08 hdr=0x00
	vendor	 = 'LSI Logic / Symbios Logic'
	device	 = 'SAS1068E PCI-Express Fusion-MPT SAS'
	class	  = mass storage
	subclass   = SCSI
bce0@pci0:2:0:0:		class=0x020000 card=0x02a31028 chip=0x163b14e4 rev=0x20 hdr=0x00
	vendor	 = 'Broadcom Corporation'
	device	 = 'NetXtreme II BCM5716 Gigabit Ethernet'
	class	  = network
	subclass   = ethernet
bce1@pci0:2:0:1:		class=0x020000 card=0x02a31028 chip=0x163b14e4 rev=0x20 hdr=0x00
	vendor	 = 'Broadcom Corporation'
	device	 = 'NetXtreme II BCM5716 Gigabit Ethernet'
	class	  = network
	subclass   = ethernet
vgapci0@pci0:1:3:0:	 class=0x030000 card=0x02a31028 chip=0x0532102b rev=0x0a hdr=0x00
	vendor	 = 'Matrox Electronics Systems Ltd.'
	device	 = 'MGA G200eW WPCM450'
	class	  = display
	subclass   = VGA
[root@freenas] ~#

We already saw the SAS 2308 in the list, so I'm guessing that's the card I bought and flashed into IT mode:

Code:

mps0@pci0:4:0:0:		class=0x010700 card=0x30001000 chip=0x00871000 rev=0x01 hdr=0x00
	vendor	 = 'LSI Logic / Symbios Logic'
	device	 = 'SAS2308 PCI-Express Fusion-MPT SAS-2'
	class	  = mass storage
	subclass   = SAS

I also notice this, which looks like the onboard SAS controller:

Code:

mpt0@pci0:3:0:0:		class=0x010000 card=0x1f0f1028 chip=0x00581000 rev=0x08 hdr=0x00
	vendor	 = 'LSI Logic / Symbios Logic'
	device	 = 'SAS1068E PCI-Express Fusion-MPT SAS'
	class	  = mass storage
	subclass   = SCSI

How do I tell which one is controlling the drive?

Itamar Croitoru · Nov 19, 2018

I'm guessing that the

Code:

scbus1

is my indicator. So scbus0 would be the 2308 and scbus1 would be the onboard, including my SLOG drive, L2ARC drive, boot drive and backplane.

HoneyBadger · Nov 19, 2018

Pretty safe assumption there, and at least the onboard controller appears to be close enough to the old SAS1068E chip to use its driver.

It's a very old controller though, not even SAS2 so you might actually see improved performance if you ran the R310 drive bays to the onboard SATA ports. It's an older chipset driving that as well though. Hard to say.

In any case, you appear to have cache and log devices in your pool. The former will fill on its own over time and the latter can be pressed into service by setting zfs set sync=always poolname/dataset as desired. Your performance will take a hit though - possibly a significant one.

Itamar Croitoru · Nov 19, 2018

I've enabled both and already set the sync to all pools.

The balance between using both seems ok. At least I now should be able to handle any further sudden power loss in stride with this, hopefully.

Thanks for all the help.

Ericloewe · Nov 20, 2018

Do you have a ton of HP-branded disks in there? If not, I think you have a serious case of hardware RAID.

Itamar Croitoru · Nov 20, 2018

Ericloewe said:
Do you have a ton of HP-branded disks in there? If not, I think you have a serious case of hardware RAID.

I have a number of Dell, HP and IBM drives.

Ericloewe · Nov 20, 2018

Okay, just checking. Note that your SAS1 HBA will only support disks up to 2 TB.

Itamar Croitoru · Nov 20, 2018

Ericloewe said:
Okay, just checking. Note that your SAS1 HBA will only support disks up to 2 TB.

To be honest, I'm running out of IOPS before GB.

Itamar Croitoru · Nov 23, 2018

I'm actually seeing fairly worse write performance since enabling this. What is a good way to improve it?

Does moving the SLOG SSD from the internal connected bay to an array connected bay help? In that case, it would be written through the same HBA card as the array I'm trying to protect with it.
Does switching to a PCIe card type SSD help?
Would just a newer server with faster specs help?
Is there some point where I've added too many drives to maintain good performance with the SLOG? How do I calculate this?

Ericloewe · Nov 23, 2018

You mean sync=always? That's to be expected, since you weren't doing any sync writes before.

Itamar Croitoru said:
Does moving the SLOG SSD from the internal connected bay to an array connected bay help? In that case, it would be written through the same HBA card as the array I'm trying to protect with it.

Maybe, probably not.

Itamar Croitoru said:
Does switching to a PCIe card type SSD help?

An NVMe SLOG would help and would also cost an arm, a leg and three kidneys. (Thanks, Intel!)

Itamar Croitoru said:
Would just a newer server with faster specs help?

That always helps, but it's a bit of a brute-force approach with limited ability to be applied where it matters most.

Itamar Croitoru said:
s there some point where I've added too many drives to maintain good performance with the SLOG?

No, and it's not about the SLOG, it's about the sync writes being made.

HoneyBadger · Nov 23, 2018

Itamar Croitoru said:
I'm actually seeing fairly worse write performance since enabling this. What is a good way to improve it?

Since you've never used sync writes before, this is to be expected - you're basically comparing against unsafe writes. A faster SLOG device is the answer.

Itamar Croitoru said:
Does moving the SLOG SSD from the internal connected bay to an array connected bay help? In that case, it would be written through the same HBA card as the array I'm trying to protect with it.

The improvement will likely come from getting it off of the SAS1 HBA - although my suggested connection would be via the internal SATA ports in order to avoid congestion with the other write load hitting there, and skip the added latency of crossing the SAS HBA.

Itamar Croitoru said:
Does switching to a PCIe card type SSD help?

Absolutely. See the link in my signature for benchmarks of suggested ones. While @Ericloewe is correct in that they cost a decent chunk more money than a SATA/SAS SLOG device, I wouldn't consider the roughly CAD$350 cost of an Optane 900p (Amazon Canada) to be that expensive given its performance and endurance. If you can get a second-hand Intel P3700 that is also a viable option.

Itamar Croitoru said:
Would just a newer server with faster specs help?

The bottleneck right now is the SLOG; if that's removed, you may find another one, but right now address the pressing issue.

Itamar Croitoru said:
Is there some point where I've added too many drives to maintain good performance with the SLOG? How do I calculate this?

ZFS rarely (if ever?) gets slower with more drives/vdevs in a pool; it generally does the opposite - more vdevs means more simultaneous writes.

Itamar Croitoru · Nov 23, 2018

I've run a fresh zilstat with the MB and pool specific and txg switches and counted, looks like 5 seconds per commit, so it's triggering because of the default 5 second timeout and not the RAM filling.

Here are the results, if I'm understanding them correctly, my txg isn't pushing so much that 46 x 146 GB 15k RPM SAS drives can't handle in mirrored pairs

Code:

zilstat -M -p iS2-R10-01 txg
waiting for txg commit...
	   txg	   N-MB	 N-MB/s N-Max-Rate	   B-MB	 B-MB/s B-Max-Rate	ops  <=4kB 4-32kB >=32kB
   3428984		  6		  6		  0		 18		 18		  1	141	  0	  0	141
   3428985		 16		  3		  8		 48		  9		 16	509	  0	 23	486
   3428986		  6		  1		  2		 35		  7		 11	328	  0	  7	321
   3428987		 10		  2		  3		 57		 11		 13	587	  0	  6	581
   3428988		  8		  1		  2		 50		 10		 13	672	  0	185	487
   3428989		  6		  1		  3		 36		  7		 17	445	  0	 32	413
   3428990		  5		  1		  1		 38		  7		  9	375	  0	 34	341
   3428991		  5		  1		  1		 38		  7		  9	442	  0	 44	398
   3428992		  5		  1		  2		 36		  7		 12	401	  0	 21	380
   3428993		  7		  1		  2		 41		  8		 12	585	  0	 16	569
   3428994		  5		  1		  1		 36		  7		 10	341	  0	  2	339
   3428995		 17		  3		 11		 50		 10		 21	507	  0	 17	490

Thoughts on these results?

Important Announcement for the TrueNAS Community.

I think my FreeNAS 9.10.2-U6 is no longer using my L2ARC

Itamar Croitoru

Dabbler

Itamar Croitoru

Dabbler

HoneyBadger

actually does care

Itamar Croitoru

Dabbler

Ericloewe

Server Wrangler

Itamar Croitoru

Dabbler

Ericloewe

Server Wrangler

Itamar Croitoru

Dabbler

Itamar Croitoru

Dabbler

Ericloewe

Server Wrangler

HoneyBadger

actually does care

Itamar Croitoru

Dabbler

Similar threads