I think my FreeNAS 9.10.2-U6 is no longer using my L2ARC

[Itamar Croitoru]

Setup:
I've been running this particular FreeNAS 9.10 for a few years now as a production iSCSI for my ESXi cluster. It currently runs on a Dell PowerEdge R310 with two Compellent storage arrays, filled with a total of 48 x 146 GB 15k RPM SAS drives in RAID 10 configuration.

I setup a SLOG and L2ARC and filled up the RAM to the max the R310 can hold of ECC RAM, 48 GB.

I've been keeping it updated and now it is at 9.10.2-U6 and will stay there now until I buy new hardware to install 11.2 on as this is my choice for an upgrade path.

Issue:
I noticed recently a performance issue on my VMs in ESX and after some checking, I noticed that my MTU size was wrong on the NICs on my FreeNAS. I corrected the size and rebooted the FreeNAS to make sure everything reloaded well. Since the reboot, I'm seeing the ARC hit ratio around 53% but noticed that my L2ARC is not being used at all. All stats for it are zeros.

Could it be that my L2ARC is not recognized somehow after the last reboot?

All drives show up with in camcontrol and are found during bootup according to my dmesg log.

I've googled around but I can't seem to find the process to confirm that the L2ARC and SLOG are actually being used still.

 

[Ericloewe]

What's the output of zpool status? [CODE][/CODE] tags please.

 

[HoneyBadger]

L2ARC is not persistent on reboot in any OpenZFS implementation yet; has the system been up long enough to warm the cache?

For SLOG, run zilstat and see if it's recording any commits.

 

[Itamar Croitoru]

What's the output of zpool status? [CODE][/CODE] tags please.

Code:

[root@freenas] ~# zpool status
  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Sat Nov 10 03:45:37 2018
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  da96p2	ONLINE	   0	 0	 0

errors: No known data errors

  pool: iS2-R10-01
 state: ONLINE
  scan: scrub repaired 0 in 1h15m with 0 errors on Sun Nov  4 01:15:25 2018
config:

		NAME											STATE	 READ WRITE CKSUM
		iS2-R10-01									  ONLINE	   0	 0	 0
		  mirror-0									  ONLINE	   0	 0	 0
			gptid/cdf95e24-5bf9-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/cfb93d7f-5bf9-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-1									  ONLINE	   0	 0	 0
			gptid/0c8207ac-5bfa-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/0e4459e4-5bfa-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-2									  ONLINE	   0	 0	 0
			gptid/40c59530-5bfa-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/42846bc3-5bfa-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-3									  ONLINE	   0	 0	 0
			gptid/7aeadbbd-5bfa-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/7cb4a7c0-5bfa-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-4									  ONLINE	   0	 0	 0
			gptid/b480b746-5bfa-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/b648ab85-5bfa-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-5									  ONLINE	   0	 0	 0
			gptid/fdbe0f66-5bfa-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/ff6fea09-5bfa-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-6									  ONLINE	   0	 0	 0
			gptid/351980da-5bfb-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/36d10dd2-5bfb-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-7									  ONLINE	   0	 0	 0
			gptid/6ce8f2eb-5bfb-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/6e9818ec-5bfb-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-8									  ONLINE	   0	 0	 0
			gptid/afb209e8-5bfb-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/b164e521-5bfb-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-9									  ONLINE	   0	 0	 0
			gptid/faf995c4-5bfb-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/fcaa6d06-5bfb-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-10									 ONLINE	   0	 0	 0
			gptid/ade838bf-6e44-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/3f1e030b-5bfc-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-11									 ONLINE	   0	 0	 0
			gptid/80aaa0a1-5bfc-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/82865353-5bfc-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-12									 ONLINE	   0	 0	 0
			gptid/bb4d6081-5bfc-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/bcfb9d07-5bfc-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-13									 ONLINE	   0	 0	 0
			gptid/f9839c40-5bfc-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/fb31e338-5bfc-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-14									 ONLINE	   0	 0	 0
			gptid/3b023f1b-5bfd-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/3b94501c-5bfd-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-15									 ONLINE	   0	 0	 0
			gptid/8e4b3084-5bfd-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/902161c6-5bfd-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-16									 ONLINE	   0	 0	 0
			gptid/d214949c-5bfd-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/d3cc15f4-5bfd-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-17									 ONLINE	   0	 0	 0
			gptid/81a3170d-5bff-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/83832864-5bff-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-18									 ONLINE	   0	 0	 0
			gptid/bcba48fd-5bff-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/be8b2bc7-5bff-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-19									 ONLINE	   0	 0	 0
			gptid/f9387756-5bff-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/f9d8957d-5bff-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-20									 ONLINE	   0	 0	 0
			gptid/31c68bb0-5c00-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/325c7adc-5c00-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-21									 ONLINE	   0	 0	 0
			gptid/730704c8-5c00-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/74d5f034-5c00-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
		  mirror-22									 ONLINE	   0	 0	 0
			gptid/b85ce99b-5c00-11e8-b48d-00151770088c  ONLINE	   0	 0	 0
			gptid/ba2f5426-5c00-11e8-b48d-00151770088c  ONLINE	   0	 0	 0

errors: No known data errors
[root@freenas] ~#

 

[Itamar Croitoru]

L2ARC is not persistent on reboot in any OpenZFS implementation yet; has the system been up long enough to warm the cache?

For SLOG, run zilstat and see if it's recording any commits.

Uptime is currently just over 5 days. How long does it take to warm up?

zilstat appears to run for a long time. Is it like tail where I wait and kill it or should I just let it run the entire output?

 

[Ericloewe]

Uptime is currently just over 5 days. How long does it take to warm up?

Considering your pool doesn't have an L2ARC, I'd say an infinite amount. But it doesn't magically disappear from the pool, so it was either never there or was removed.

 

[HoneyBadger]

Uptime is currently just over 5 days. How long does it take to warm up?

zilstat appears to run for a long time. Is it like tail where I wait and kill it or should I just let it run the entire output?

As mentioned by @Ericloewe your output shows that your pool has neither an L2ARC (cache) or SLOG (log) vdev.

Also based on vdev count, you're short two drives (2x23=46) of the 48 you stated.

zilstat reports real-time information - just kill it with Ctrl-C when you've seen enough. Is is showing all zeroes or were there significant non-zero lines at any time?

 

[Itamar Croitoru]

Considering your pool doesn't have an L2ARC, I'd say an infinite amount. But it doesn't magically disappear from the pool, so it was either never there or was removed.

It certainly had those when I added them. Maybe a patch removed them or something like that did but, it certainly wasn't me.

 

[Itamar Croitoru]

As mentioned by @Ericloewe your output shows that your pool has neither an L2ARC (cache) or SLOG (log) vdev.

Also based on vdev count, you're short two drives (2x23=46) of the 48 you stated.

zilstat reports real-time information - just kill it with Ctrl-C when you've seen enough. Is is showing all zeroes or were there significant non-zero lines at any time?

It ran for about 20-25 lines and only one had non-zero numbers

 

[Itamar Croitoru]

Would anything break if I just specified the SLOG and L2ARC to the same partitions again?

 

[Itamar Croitoru]

I'm going to just add them back and monitor for now. If/when they disappear again, I'll post again.

 

[HoneyBadger]

It certainly had those when I added them. Maybe a patch removed them or something like that did but, it certainly wasn't me.

Upgrades to the FreeNAS UI should never change pool configuration.
Post the output of zpool history iS2-R10-01 please.

It ran for about 20-25 lines and only one had non-zero numbers

I'm betting sync writes are not being used, and your SLOG will be doing nothing. The one non-zero line was likely just some metadata housekeeping.
Post the output of zfs get sync please.

Would anything break if I just specified the SLOG and L2ARC to the same partitions again?

It shouldn't, but as noted above you may not actually be using that SLOG. What devices are you using for cache/log and how are they connected?

 

[Itamar Croitoru]

Upgrades to the FreeNAS UI should never change pool configuration.
Post the output of zpool history iS2-R10-01 please.



I'm betting sync writes are not being used, and your SLOG will be doing nothing. The one non-zero line was likely just some metadata housekeeping.
Post the output of zfs get sync please.



It shouldn't, but as noted above you may not actually be using that SLOG. What devices are you using for cache/log and how are they connected?

zpool history iS2-R10-01

Code:

[root@freenas] ~# zpool history iS2-R10-01
History for 'iS2-R10-01':
2018-05-20.02:48:33 zpool create -o cachefile=/data/zfs/zpool.cache -o failmode=continue -o autoexpand=on -O compression=lz4 -O aclmode=passthrough -O aclinherit=passthrough -f -m /iS2-R10-01 -o altroot=/mnt iS2-R10-01 mirror /dev/gptid/cdf95e24-5bf9-11e8-b48d-00151770088c /dev/gptid/cfb93d7f-5bf9-11e8-b48d-00151770088c
2018-05-20.02:48:38 zfs inherit mountpoint iS2-R10-01
2018-05-20.02:48:38 zpool set cachefile=/data/zfs/zpool.cache iS2-R10-01
2018-05-20.02:50:23 zpool add -f iS2-R10-01 mirror /dev/gptid/0c8207ac-5bfa-11e8-b48d-00151770088c /dev/gptid/0e4459e4-5bfa-11e8-b48d-00151770088c
2018-05-20.02:51:50 zpool add -f iS2-R10-01 mirror /dev/gptid/40c59530-5bfa-11e8-b48d-00151770088c /dev/gptid/42846bc3-5bfa-11e8-b48d-00151770088c
2018-05-20.02:53:28 zpool add -f iS2-R10-01 mirror /dev/gptid/7aeadbbd-5bfa-11e8-b48d-00151770088c /dev/gptid/7cb4a7c0-5bfa-11e8-b48d-00151770088c
2018-05-20.02:55:05 zpool add -f iS2-R10-01 mirror /dev/gptid/b480b746-5bfa-11e8-b48d-00151770088c /dev/gptid/b648ab85-5bfa-11e8-b48d-00151770088c
2018-05-20.02:57:07 zpool add -f iS2-R10-01 mirror /dev/gptid/fdbe0f66-5bfa-11e8-b48d-00151770088c /dev/gptid/ff6fea09-5bfa-11e8-b48d-00151770088c
2018-05-20.02:58:40 zpool add -f iS2-R10-01 mirror /dev/gptid/351980da-5bfb-11e8-b48d-00151770088c /dev/gptid/36d10dd2-5bfb-11e8-b48d-00151770088c
2018-05-20.03:00:14 zpool add -f iS2-R10-01 mirror /dev/gptid/6ce8f2eb-5bfb-11e8-b48d-00151770088c /dev/gptid/6e9818ec-5bfb-11e8-b48d-00151770088c
2018-05-20.03:02:06 zpool add -f iS2-R10-01 mirror /dev/gptid/afb209e8-5bfb-11e8-b48d-00151770088c /dev/gptid/b164e521-5bfb-11e8-b48d-00151770088c
2018-05-20.03:04:12 zpool add -f iS2-R10-01 mirror /dev/gptid/faf995c4-5bfb-11e8-b48d-00151770088c /dev/gptid/fcaa6d06-5bfb-11e8-b48d-00151770088c
2018-05-20.03:06:04 zpool add -f iS2-R10-01 mirror /dev/gptid/3d46e4f5-5bfc-11e8-b48d-00151770088c /dev/gptid/3f1e030b-5bfc-11e8-b48d-00151770088c
2018-05-20.03:07:57 zpool add -f iS2-R10-01 mirror /dev/gptid/80aaa0a1-5bfc-11e8-b48d-00151770088c /dev/gptid/82865353-5bfc-11e8-b48d-00151770088c
2018-05-20.03:09:35 zpool add -f iS2-R10-01 mirror /dev/gptid/bb4d6081-5bfc-11e8-b48d-00151770088c /dev/gptid/bcfb9d07-5bfc-11e8-b48d-00151770088c
2018-05-20.03:11:19 zpool add -f iS2-R10-01 mirror /dev/gptid/f9839c40-5bfc-11e8-b48d-00151770088c /dev/gptid/fb31e338-5bfc-11e8-b48d-00151770088c
2018-05-20.03:13:07 zpool add -f iS2-R10-01 mirror /dev/gptid/3b023f1b-5bfd-11e8-b48d-00151770088c /dev/gptid/3b94501c-5bfd-11e8-b48d-00151770088c
2018-05-20.03:15:30 zpool add -f iS2-R10-01 mirror /dev/gptid/8e4b3084-5bfd-11e8-b48d-00151770088c /dev/gptid/902161c6-5bfd-11e8-b48d-00151770088c
2018-05-20.03:17:23 zpool add -f iS2-R10-01 mirror /dev/gptid/d214949c-5bfd-11e8-b48d-00151770088c /dev/gptid/d3cc15f4-5bfd-11e8-b48d-00151770088c
2018-05-20.03:29:27 zpool add -f iS2-R10-01 mirror /dev/gptid/81a3170d-5bff-11e8-b48d-00151770088c /dev/gptid/83832864-5bff-11e8-b48d-00151770088c
2018-05-20.03:31:06 zpool add -f iS2-R10-01 mirror /dev/gptid/bcba48fd-5bff-11e8-b48d-00151770088c /dev/gptid/be8b2bc7-5bff-11e8-b48d-00151770088c
2018-05-20.03:32:46 zpool add -f iS2-R10-01 mirror /dev/gptid/f9387756-5bff-11e8-b48d-00151770088c /dev/gptid/f9d8957d-5bff-11e8-b48d-00151770088c
2018-05-20.03:34:21 zpool add -f iS2-R10-01 mirror /dev/gptid/31c68bb0-5c00-11e8-b48d-00151770088c /dev/gptid/325c7adc-5c00-11e8-b48d-00151770088c
2018-05-20.03:36:12 zpool add -f iS2-R10-01 mirror /dev/gptid/730704c8-5c00-11e8-b48d-00151770088c /dev/gptid/74d5f034-5c00-11e8-b48d-00151770088c
2018-05-20.03:38:08 zpool add -f iS2-R10-01 mirror /dev/gptid/b85ce99b-5c00-11e8-b48d-00151770088c /dev/gptid/ba2f5426-5c00-11e8-b48d-00151770088c
2018-05-20.03:43:19 zfs create -o volblocksize=16K -V 2.75T iS2-R10-01/iS2-R10-01
2018-05-20.03:43:24 zfs set org.freenas:description= iS2-R10-01/iS2-R10-01
2018-05-22.15:43:34 zfs create -o volblocksize=16K -V 100M iS2-R10-01/Zabbix
2018-05-22.15:43:39 zfs set org.freenas:description= iS2-R10-01/Zabbix
2018-06-12.09:30:02 zpool replace iS2-R10-01 12182350450284575462 gptid/ade838bf-6e44-11e8-b48d-00151770088c
2018-06-12.09:30:16 zpool detach iS2-R10-01 12182350450284575462
2018-07-01.00:00:12 zpool scrub iS2-R10-01
2018-08-12.00:00:12 zpool scrub iS2-R10-01
2018-09-16.12:52:17 zpool import -c /data/zfs/zpool.cache.saved -o cachefile=none -R /mnt -f 775940954750883654
2018-09-16.12:52:17 zpool set cachefile=/data/zfs/zpool.cache iS2-R10-01
2018-09-23.00:00:11 zpool scrub iS2-R10-01
2018-11-04.00:00:11 zpool scrub iS2-R10-01
2018-11-14.15:51:39 zpool import -c /data/zfs/zpool.cache.saved -o cachefile=none -R /mnt -f 775940954750883654
2018-11-14.15:51:39 zpool set cachefile=/data/zfs/zpool.cache iS2-R10-01
2018-11-19.10:37:43 zfs create iS2-R10-01/jails
2018-11-19.10:40:43 zfs create -o mountpoint=/iS2-R10-01/jails/.warden-template-standard -p iS2-R10-01/jails/.warden-template-standard
2018-11-19.10:46:39 zfs snapshot iS2-R10-01/jails/.warden-template-standard@clean
2018-11-19.10:47:13 zfs clone iS2-R10-01/jails/.warden-template-standard@clean iS2-R10-01/jails/Webmin
2018-11-19.22:40:17 zpool add -f iS2-R10-01 log /dev/gptid/eb532412-ec75-11e8-8337-00151770088c
2018-11-19.22:46:30 zpool add -f iS2-R10-01 cache /dev/gptid/d84534d8-ec76-11e8-8337-00151770088c

[root@freenas] ~#

zfs get sync

Code:

[root@freenas] ~# zfs get sync
NAME														   PROPERTY  VALUE	 SOURCE
freenas-boot												   sync	  standard  default
freenas-boot/.system										   sync	  standard  default
freenas-boot/.system/configs-eab18b758b91471d95803a91d80bfcda  sync	  standard  default
freenas-boot/.system/cores									 sync	  standard  default
freenas-boot/.system/rrd-eab18b758b91471d95803a91d80bfcda	  sync	  standard  default
freenas-boot/.system/samba4									sync	  standard  default
freenas-boot/.system/syslog-eab18b758b91471d95803a91d80bfcda   sync	  standard  default
freenas-boot/ROOT											  sync	  standard  default
freenas-boot/ROOT/9.10.2-U1									sync	  standard  default
freenas-boot/ROOT/9.10.2-U2									sync	  standard  default
freenas-boot/ROOT/9.10.2-U2@2016-12-28-11:46:40				sync	  -		 -
freenas-boot/ROOT/9.10.2-U2@2017-01-20-13:48:09				sync	  -		 -
freenas-boot/ROOT/9.10.2-U2@2017-03-26-12:25:22				sync	  -		 -
freenas-boot/ROOT/9.10.2-U4									sync	  standard  default
freenas-boot/ROOT/9.10.2-U5									sync	  standard  default
freenas-boot/ROOT/9.10.2-U6									sync	  standard  default
freenas-boot/ROOT/9.10.2-U6@2017-05-29-17:34:41				sync	  -		 -
freenas-boot/ROOT/9.10.2-U6@2017-07-04-02:00:41				sync	  -		 -
freenas-boot/ROOT/9.10.2-U6@2017-10-01-01:09:54				sync	  -		 -
freenas-boot/ROOT/Initial-Install							  sync	  standard  default
freenas-boot/ROOT/default									  sync	  standard  default
freenas-boot/ROOT/default-20170508-233326					  sync	  standard  default
freenas-boot/grub											  sync	  standard  default
iS2-R10-01													 sync	  standard  default
iS2-R10-01/Zabbix											  sync	  standard  default
iS2-R10-01/iS2-R10-01										  sync	  standard  default
iS2-R10-01/jails											   sync	  standard  default
iS2-R10-01/jails/.warden-template-standard					 sync	  standard  default
iS2-R10-01/jails/.warden-template-standard@clean			   sync	  -		 -
iS2-R10-01/jails/Webmin										sync	  standard  default
[root@freenas] ~#

They are two SSDs directly installed in the Dell R310. For the ZIL, I have a Intel SC 7310 400GB and a 128GB Micron SSD for L2ARC.

 

[HoneyBadger]

I'm afraid I'm going to be the bearer of a trifecta of bad news here.

zpool history iS2-R10-01

Code:

[root@freenas] ~# zpool history iS2-R10-01
History for 'iS2-R10-01':
2018-11-19.22:40:17 zpool add -f iS2-R10-01 log /dev/gptid/eb532412-ec75-11e8-8337-00151770088c
2018-11-19.22:46:30 zpool add -f iS2-R10-01 cache /dev/gptid/d84534d8-ec76-11e8-8337-00151770088c

1. The zpool history log tells me that you have never had a cache or log device in your pool until just now.
2. It has also only ever been composed of 46 of those 48 drives.

zfs get sync

Code:

[root@freenas] ~# zfs get sync
NAME														   PROPERTY  VALUE	 SOURCE
iS2-R10-01													 sync	  standard  default
iS2-R10-01/Zabbix											  sync	  standard  default
iS2-R10-01/iS2-R10-01										  sync	  standard  default
iS2-R10-01/jails											   sync	  standard  default
iS2-R10-01/jails/.warden-template-standard					 sync	  standard  default
iS2-R10-01/jails/.warden-template-standard@clean			   sync	  -		 -
iS2-R10-01/jails/Webmin										sync	  standard  default
[root@freenas] ~#

3. You also have sync=standard set, and with iSCSI that means you've never been doing synchronous writes.

They are two SSDs directly installed in the Dell R310. For the ZIL, I have a Intel SC 7310 DC S3710 400GB and a 128GB Micron SSD for L2ARC.

Assuming you typo'd and intended to say the DC S3710, that's a good choice for a SATA SLOG. Are they connected via onboard SATA connectors, or via an HBA? If so, what model?

(Please don't say "through the integrated RAID controller.")

 

[Itamar Croitoru]

I'm afraid I'm going to be the bearer of a trifecta of bad news here.



1. The zpool history log tells me that you have never had a cache or log device in your pool until just now.
2. It has also only ever been composed of 46 of those 48 drives.



3. You also have sync=standard set, and with iSCSI that means you've never been doing synchronous writes.



Assuming you typo'd and intended to say the DC S3710, that's a good choice for a SATA SLOG. Are they connected via onboard SATA connectors, or via an HBA? If so, what model?

(Please don't say "through the integrated RAID controller.")

As for using 46 out of 48 drives, that's because I couldn't get the last two to show up when adding drives. Only one of them does and I required mirrored pairs. So somewhere in my array is one bad drive and I never bothered to track down which and deal with it. I guess I need to get around to that.

Yes, I typo'd and it is a DC S3710 for the SLOG, I'm fairly tired at this point.

I can't open the server to check which way I have it cabled in. Is there a command to check which controller a drive is connected through?

 

[HoneyBadger]

I can't open the server to check which way I have it cabled in. Is there a command to check which controller a drive is connected through?

camcontrol devlist should give us some information;

sas2ircu list should show a list of LSI SAS2 cards in your system. I don't believe you have access to storcli show all on that version but that might produce a list of cards.

dmesg | grep -i sas might also provide information about the controller itself.

 

[Itamar Croitoru]

camcontrol devlist should give us some information.

Code:

[root@freenas] ~# camcontrol devlist
<HP EH0146FARUB HPD8>			  at scbus0 target 11 lun 0 (pass0,da0)
<HP EH0146FARUB HPD8>			  at scbus0 target 12 lun 0 (pass1,da1)
<HP EH0146FARWD HPD9>			  at scbus0 target 13 lun 0 (pass2,da2)
<HP EH0146FARWD HPD9>			  at scbus0 target 14 lun 0 (pass3,da3)
<HP EH0146FARWD HPD9>			  at scbus0 target 15 lun 0 (pass4,da4)
<HP EH0146FARWD HPD9>			  at scbus0 target 16 lun 0 (pass5,da5)
<SEAGATE ST9146852SS HT64>		 at scbus0 target 17 lun 0 (pass6,da6)
<SEAGATE ST9146852SS HT64>		 at scbus0 target 18 lun 0 (pass7,da7)
<SEAGATE ST9146852SS HT64>		 at scbus0 target 19 lun 0 (pass8,da8)
<SEAGATE ST9146852SS HT64>		 at scbus0 target 20 lun 0 (pass9,da9)
<HITACHI HUC151414CSS600 K516>	 at scbus0 target 21 lun 0 (pass10,da10)
<HITACHI HUC151414CSS600 K516>	 at scbus0 target 22 lun 0 (pass11,da11)
<HITACHI HUC151414CSS600 K516>	 at scbus0 target 23 lun 0 (pass12,da12)
<HITACHI HUC151414CSS600 K516>	 at scbus0 target 24 lun 0 (pass13,da13)
<HITACHI HUC151414CSS600 K516>	 at scbus0 target 25 lun 0 (pass14,da14)
<HITACHI HUC151414CSS600 K516>	 at scbus0 target 26 lun 0 (pass15,da15)
<XYRATEX EB-2425-E6EBD 2005>	   at scbus0 target 27 lun 0 (pass16,ses0)
<HITACHI HUC151414CSS600 K516>	 at scbus0 target 28 lun 0 (pass17,da16)
<HITACHI HUC151414CSS600 K516>	 at scbus0 target 29 lun 0 (pass18,da17)
<SEAGATE ST9146852SS HT64>		 at scbus0 target 30 lun 0 (pass19,da18)
<SEAGATE ST9146852SS HT64>		 at scbus0 target 31 lun 0 (pass20,da19)
<SEAGATE ST9146852SS HT64>		 at scbus0 target 32 lun 0 (pass21,da20)
<HP EH0146FARWD HPD9>			  at scbus0 target 33 lun 0 (pass22,da21)
<SEAGATE ST9146852SS HT64>		 at scbus0 target 34 lun 0 (pass23,da22)
<HITACHI HUC151414CSS600 K516>	 at scbus0 target 35 lun 0 (pass24,da23)
<HITACHI HUC151414CSS600 K516>	 at scbus0 target 36 lun 0 (pass25,da24)
<HITACHI HUC151414CSS600 K516>	 at scbus0 target 37 lun 0 (pass26,da25)
<HP EH0146FARWD HPD9>			  at scbus0 target 38 lun 0 (pass27,da26)
<HITACHI HUC151414CSS600 K516>	 at scbus0 target 39 lun 0 (pass28,da27)
<HP EH0146FARUB HPD8>			  at scbus0 target 40 lun 0 (pass29,da28)
<HP EH0146FARWD HPD9>			  at scbus0 target 41 lun 0 (pass30,da29)
<HP EH0146FARWD HPD9>			  at scbus0 target 42 lun 0 (pass31,da30)
<HP EH0146FARUB HPD8>			  at scbus0 target 43 lun 0 (pass32,da31)
<XYRATEX EB-2425-E6EBD 2005>	   at scbus0 target 47 lun 0 (pass33,ses1)
<HP EH0146FARWD HPDD>			  at scbus0 target 54 lun 0 (pass34,da32)
<HP EH0146FARWD HPDD>			  at scbus0 target 55 lun 0 (pass35,da33)
<HP EH0146FARWD HPD4>			  at scbus0 target 56 lun 0 (pass36,da34)
<HP EH0146FARWD HPD4>			  at scbus0 target 57 lun 0 (pass37,da35)
<HP EH0146FARWD HPD4>			  at scbus0 target 58 lun 0 (pass38,da36)
<HP EH0146FARWD HPD4>			  at scbus0 target 59 lun 0 (pass39,da37)
<HP EH0146FARWD HPDD>			  at scbus0 target 60 lun 0 (pass40,da38)
<HP EH0146FARWD HPDD>			  at scbus0 target 61 lun 0 (pass41,da39)
<HP EH0146FARWD HPDD>			  at scbus0 target 62 lun 0 (pass42,da40)
<HP EH0146FARWD HPDD>			  at scbus0 target 63 lun 0 (pass43,da41)
<HP EH0146FARWD HPDD>			  at scbus0 target 64 lun 0 (pass44,da42)
<HP EH0146FARWD HPDD>			  at scbus0 target 65 lun 0 (pass45,da43)
<HP EH0146FARWD HPDD>			  at scbus0 target 66 lun 0 (pass46,da44)
<HP EH0146FARWD HPDD>			  at scbus0 target 67 lun 0 (pass47,da45)
<HP EH0146FARWD HPDD>			  at scbus0 target 68 lun 0 (pass48,da46)
<HP EH0146FARWD HPDD>			  at scbus0 target 69 lun 0 (pass49,da47)
<FUJITSU MBE2147RC D905>		   at scbus0 target 70 lun 0 (pass50,da48)
<FUJITSU MBE2147RC D905>		   at scbus0 target 71 lun 0 (pass51,da49)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 72 lun 0 (pass52,da50)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 73 lun 0 (pass53,da51)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 74 lun 0 (pass54,da52)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 75 lun 0 (pass55,da53)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 76 lun 0 (pass56,da54)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 77 lun 0 (pass57,da55)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 78 lun 0 (pass58,da56)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 79 lun 0 (pass59,da57)
<FUJITSU MBE2147RC D905>		   at scbus0 target 81 lun 0 (pass60,da58)
<FUJITSU MBE2147RC D905>		   at scbus0 target 82 lun 0 (pass61,da59)
<FUJITSU MBE2147RC D905>		   at scbus0 target 83 lun 0 (pass62,da60)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 84 lun 0 (pass63,da61)
<IBM-ESXS ST9146752SS   E E621>	at scbus0 target 85 lun 0 (pass64,da62)
<FUJITSU MBE2147RC D906>		   at scbus0 target 86 lun 0 (pass65,da63)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 87 lun 0 (pass66,da64)
<FUJITSU MBE2147RC D905>		   at scbus0 target 88 lun 0 (pass67,da65)
<FUJITSU MBE2147RC D905>		   at scbus0 target 89 lun 0 (pass68,da66)
<FUJITSU MBE2147RC D905>		   at scbus0 target 90 lun 0 (pass69,da67)
<FUJITSU MBE2147RC D905>		   at scbus0 target 91 lun 0 (pass70,da68)
<FUJITSU MBE2147RC D905>		   at scbus0 target 92 lun 0 (pass71,da69)
<FUJITSU MBE2147RC D905>		   at scbus0 target 93 lun 0 (pass72,da70)
<XYRATEX EB-2425-E6EBD 2005>	   at scbus0 target 94 lun 0 (pass73,ses2)
<FUJITSU MBE2147RC D905>		   at scbus0 target 95 lun 0 (pass74,da71)
<FUJITSU MBE2147RC D905>		   at scbus0 target 96 lun 0 (pass75,da72)
<FUJITSU MBE2147RC D905>		   at scbus0 target 97 lun 0 (pass76,da73)
<FUJITSU MBE2147RC D905>		   at scbus0 target 98 lun 0 (pass77,da74)
<FUJITSU MBE2147RC D905>		   at scbus0 target 99 lun 0 (pass78,da75)
<FUJITSU MBE2147RC D905>		   at scbus0 target 100 lun 0 (pass79,da76)
<FUJITSU MBE2147RC D905>		   at scbus0 target 101 lun 0 (pass80,da77)
<FUJITSU MBE2147RC D905>		   at scbus0 target 102 lun 0 (pass81,da78)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 103 lun 0 (pass82,da79)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 104 lun 0 (pass83,da80)
<IBM-ESXS ST9146752SS   E E621>	at scbus0 target 105 lun 0 (pass84,da81)
<FUJITSU MBE2147RC D906>		   at scbus0 target 106 lun 0 (pass85,da82)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 108 lun 0 (pass86,da83)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 109 lun 0 (pass87,da84)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 110 lun 0 (pass88,da85)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 111 lun 0 (pass89,da86)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 112 lun 0 (pass90,da87)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 113 lun 0 (pass91,da88)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 114 lun 0 (pass92,da89)
<SEAGATE ST9146853SS YS09>		 at scbus0 target 115 lun 0 (pass93,da90)
<FUJITSU MBE2147RC D905>		   at scbus0 target 116 lun 0 (pass94,da91)
<FUJITSU MBE2147RC D905>		   at scbus0 target 117 lun 0 (pass95,da92)
<FUJITSU MBE2147RC D905>		   at scbus0 target 118 lun 0 (pass96,da93)
<XYRATEX EB-2425-E6EBD 2005>	   at scbus0 target 119 lun 0 (pass97,ses3)
<FUJITSU MBE2147RC D905>		   at scbus0 target 120 lun 0 (pass98,da94)
<FUJITSU MBE2147RC D905>		   at scbus0 target 121 lun 0 (pass99,da95)
<ATA OCZ-VERTEX 1.5>			   at scbus1 target 0 lun 0 (pass100,da96)
<ATA INTEL SSDSC2BA40 0110>		at scbus1 target 1 lun 0 (pass101,da97)
<ATA MTFDDAK128MAM-1J 040H>		at scbus1 target 2 lun 0 (pass102,da98)
<DP BACKPLANE 1.07>				at scbus1 target 8 lun 0 (pass103,ses4)
<PLDS DVD-ROM DS-8D3SH HD51>	   at scbus5 target 0 lun 0 (pass104,cd0)
<iDRAC LCDRIVE 0323>			   at scbus8 target 0 lun 0 (pass105,da99)
<iDRAC Virtual CD 0323>			at scbus9 target 0 lun 0 (pass106,cd1)
[root@freenas] ~#

The devices are these:
<ATA INTEL SSDSC2BA40 0110> at scbus1 target 1 lun 0 (pass101,da97)
<ATA MTFDDAK128MAM-1J 040H> at scbus1 target 2 lun 0 (pass102,da98)

 

[HoneyBadger]

Those two SSDs are in the SAS topology based on their device names of da97 and da98 but they're showing up with their actual identifiers so I'm hopeful they're not behind a PERC RAID card.

sas2ircu list?

 

[Itamar Croitoru]

Those two SSDs are in the SAS topology based on their device names of da97 and da98 but they're showing up with their actual identifiers so I'm hopeful they're not behind a PERC RAID card.

sas2ircu list?

I believe I only have the single LUN array. Here's my output:

Code:

[root@freenas] ~# sas2ircu list
LSI Corporation SAS2 IR Configuration Utility.
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved.


		 Adapter	  Vendor  Device					   SubSys  SubSys
 Index	Type		  ID	  ID	Pci Address		  Ven ID  Dev ID
 -----  ------------  ------  ------  -----------------	------  ------
   0	 SAS2308_2	 1000h	87h   00h:04h:00h:00h	  1000h   3000h
SAS2IRCU: Utility Completed Successfully.
[root@freenas] ~#

 

[HoneyBadger]

It's possible there's a second controller that sas2ircu doesn't recognize, so it isn't displaying it.

pciconf -lv will generate a LOT of info but hopefully one of the lines will give us a clue on if there's an additional SAS card in play.