ZFS Pool Unavailable after upgrade 9.3 --> 11

[FNASFan]

Had a very healthy 4 disk, 1 ZFS volume FreeNAS running 9.3 with no problems for years... until a few hours ago when I decided to upgrade to 11.1. Now I get a CRITICAL alert in the GUI "The volume ZFS1 state is UNKNOWN" and the volume is indeed unavailable.

zpool import shows 3 of the 4 disks in the 4 disk volume seem to not be able to be opened but I have no clue as to why - they were fine before the upgrade. Has anyone seen anything like this after an upgrade?

Any suggestions gratefully received!

Code:

# zpool import
   pool: ZFS1
	 id: 11950453963679700988
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
	devices and try again.
   see: http://illumos.org/msg/ZFS-8000-3C
 config:

	ZFS1											UNAVAIL  insufficient replicas
	  raidz1-0									  UNAVAIL  insufficient replicas
		gptid/7f05f148-556a-11e6-9f20-3c4a9277a9d8  ONLINE
		10413270059427606477						UNAVAIL  cannot open
		16224969683152897032						UNAVAIL  cannot open
		3977721074817620361						 UNAVAIL  cannot open

 

[HoneyBadger]

Full system specs please including CPU/motherboard/HBA/drive type.

Are the "missing" drives visible in the BIOS? Can you use smartctl to query them?

 

[FNASFan]

Hi @HoneyBadger . Thanks for replying.

• All 4 disks show in the GUI (ada0 through ada3). 3 are WD Caviar Green, one is a WD Blue - I'm not sure how to tie the serial numbers to the identifiers in zpool import though.
• smartctl --scan shows the following:
  
  Code:

  /dev/ada0 -d atacam # /dev/ada0, ATA device
  /dev/ada1 -d atacam # /dev/ada1, ATA device
  /dev/ada2 -d atacam # /dev/ada2, ATA device
  /dev/ada3 -d atacam # /dev/ada3, ATA device
  

  
  
• ada0
  
  Code:

  === START OF INFORMATION SECTION ===
  Model Family:	 Western Digital Caviar Green (AF)
  Device Model:	 WDC WD20EARS-00MVWB0
  Serial Number:	WD-WCAZA3616153
  LU WWN Device Id: 5 0014ee 25ac59913
  Firmware Version: 51.0AB51
  User Capacity:	2,000,398,934,016 bytes [2.00 TB]
  Sector Size:	  512 bytes logical/physical
  Device is:		In smartctl database [for details use: -P show]
  ATA Version is:   ATA8-ACS (minor revision not indicated)
  SATA Version is:  SATA 2.6, 3.0 Gb/s
  Local Time is:	Thu Nov 15 20:07:13 2018 GMT
  SMART support is: Available - device has SMART capability.
  SMART support is: Enabled
  
  === START OF READ SMART DATA SECTION ===
  SMART overall-health self-assessment test result: PASSED
  

  
  
• ada1
  
  Code:

  === START OF INFORMATION SECTION ===
  Model Family:	 Western Digital Caviar Green (AF)
  Device Model:	 WDC WD20EARS-00MVWB0
  Serial Number:	WD-WCAZA3585475
  LU WWN Device Id: 5 0014ee 2b01b60a3
  Firmware Version: 51.0AB51
  User Capacity:	2,000,398,934,016 bytes [2.00 TB]
  Sector Size:	  512 bytes logical/physical
  Device is:		In smartctl database [for details use: -P show]
  ATA Version is:   ATA8-ACS (minor revision not indicated)
  SATA Version is:  SATA 2.6, 3.0 Gb/s
  Local Time is:	Thu Nov 15 20:12:12 2018 GMT
  SMART support is: Available - device has SMART capability.
  SMART support is: Enabled
  
  === START OF READ SMART DATA SECTION ===
  SMART overall-health self-assessment test result: PASSED
  

  
  
• ada2
  
  Code:

  === START OF INFORMATION SECTION ===
  Model Family:	 Western Digital Blue
  Device Model:	 WDC WD20EZRZ-00Z5HB0
  Serial Number:	WD-WCC4N2KK0LU5
  LU WWN Device Id: 5 0014ee 2b7d724ff
  Firmware Version: 80.00A80
  User Capacity:	2,000,398,934,016 bytes [2.00 TB]
  Sector Sizes:	 512 bytes logical, 4096 bytes physical
  Rotation Rate:	5400 rpm
  Device is:		In smartctl database [for details use: -P show]
  ATA Version is:   ACS-2 (minor revision not indicated)
  SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
  Local Time is:	Thu Nov 15 20:12:41 2018 GMT
  SMART support is: Available - device has SMART capability.
  SMART support is: Enabled
  
  === START OF READ SMART DATA SECTION ===
  SMART overall-health self-assessment test result: PASSED
  

  
  
• ada3
  
  Code:

  === START OF INFORMATION SECTION ===
  Model Family:	 Western Digital Caviar Green (AF)
  Device Model:	 WDC WD20EARS-00MVWB0
  Serial Number:	WD-WCAZA3594528
  LU WWN Device Id: 5 0014ee 205707cad
  Firmware Version: 51.0AB51
  User Capacity:	2,000,398,934,016 bytes [2.00 TB]
  Sector Size:	  512 bytes logical/physical
  Device is:		In smartctl database [for details use: -P show]
  ATA Version is:   ATA8-ACS (minor revision not indicated)
  SATA Version is:  SATA 2.6, 3.0 Gb/s
  Local Time is:	Thu Nov 15 20:13:08 2018 GMT
  SMART support is: Available - device has SMART capability.
  SMART support is: Enabled
  
  === START OF READ SMART DATA SECTION ===
  SMART overall-health self-assessment test result: PASSED
  

  
  
• Trying to get the other info now...

 

[FNASFan]

• CPU: AMD Athlon(tm) II Neo N36L Dual-Core Processor (1297.87-MHz K8-class CPU)
  
• Motherboard/HBA: ... er... it's a HP ProLiant MicroServer G7?

 

[FNASFan]

Getting more curious: if I gpart list, only the WD Blue is showing up - this seems to be the only drive working in the pool now:

Code:

Geom name: ada2
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada2p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   rawuuid: 7eee6189-556a-11e6-9f20-3c4a9277a9d8
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada2p2
   Mediasize: 1998251364352 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawuuid: 7f05f148-556a-11e6-9f20-3c4a9277a9d8
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251364352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029127
   start: 4194432
Consumers:
1. Name: ada2
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2

Why wouldn't the WD Greens show up?

 

[FNASFan]

Output of gpart show:

Code:

=>	  34  60751805  da0  GPT  (29G)
		34	  1024	1  bios-boot  (512K)
	  1058		 6	   - free -  (3.0K)
	  1064  60750768	2  freebsd-zfs  (29G)
  60751832		 7	   - free -  (3.5K)

=>		63  3907029105  ada0  MBR  (1.8T)
		  63  3907029105		- free -  (1.8T)

=>		63  3907029105  ada1  MBR  (1.8T)
		  63  3907029105		- free -  (1.8T)

=>		34  3907029101  ada2  GPT  (1.8T)
		  34		  94		- free -  (47K)
		 128	 4194304	 1  freebsd-swap  (2.0G)
	 4194432  3902834696	 2  freebsd-zfs  (1.8T)
  3907029128		   7		- free -  (3.5K)

=>		63  3907029105  ada3  MBR  (1.8T)
		  63  3907029105		- free -  (1.8T)

Has upgrading FreeNAS blown away my partitions or are they just not reading for some reason?

 

[HoneyBadger]

Output of gpart show:

Code:

=>	  34  60751805  da0  GPT  (29G)
		34	  1024	1  bios-boot  (512K)
	  1058		 6	   - free -  (3.0K)
	  1064  60750768	2  freebsd-zfs  (29G)
  60751832		 7	   - free -  (3.5K)

=>		63  3907029105  ada0  MBR  (1.8T)
		  63  3907029105		- free -  (1.8T)

=>		63  3907029105  ada1  MBR  (1.8T)
		  63  3907029105		- free -  (1.8T)

=>		34  3907029101  ada2  GPT  (1.8T)
		  34		  94		- free -  (47K)
		 128	 4194304	 1  freebsd-swap  (2.0G)
	 4194432  3902834696	 2  freebsd-zfs  (1.8T)
  3907029128		   7		- free -  (3.5K)

=>		63  3907029105  ada3  MBR  (1.8T)
		  63  3907029105		- free -  (1.8T)

Has upgrading FreeNAS blown away my partitions or are they just not reading for some reason?

Yikes. That's a little terrifying. Did you do the upgrade from the GUI, or manually via CD? Upgrading shouldn't have touched the partitions on those disks at all.

 

[FNASFan]

@HoneyBadger just a simple GUI upgrade.

There are differences between the 3 WDGreen and the (seemingly working) WDBlue in dmesg too. Not sure how to do a side-by-side in the forum, so I'll [...] the differences:

• Working (seemingly) ada2:
  
  Code:

  ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
  ada2: <WDC WD20EZRZ-00Z5HB0 80.00A80> ACS-2 ATA SATA 3.x device
  ada2: Serial Number WD-WCC4N2KK0LU5
  ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
  ada2: Command Queueing enabled
  ada2: 1907729MB (3907029168 512 byte sectors)
  GEOM_ELI: Device ada2p1.eli created.							 ]
  GEOM_ELI: Device ada2p1.eli destroyed.						   ] Different
  GEOM_ELI: Detached ada2p1.eli on last close.					 ]
  

  
  
• Not working (seemingly) ada3 (but ada0 and ada1 are identical)
  
  Code:

  ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
  ada3: <WDC WD20EARS-00MVWB0 51.0AB51> ATA8-ACS SATA 2.x device
  ada3: Serial Number WD-WCAZA3594528
  ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
  ada3: Command Queueing enabled
  ada3: 1907729MB (3907029168 512 byte sectors)
  ada3: quirks=0x1<4K>											 ] Different
  

What does "quirks" do?

 

[FNASFan]

Seems from here that this quirk is used to set the physical sector size in cases where it isn't reported by the hardware. There appears to be a performance impact of mixing disks with 4k and 512b physical sectors in a ZFS volume, and a risk of data loss if a disk with 4k physical sectors is being used as if it had 512b sectors. I wonder if WD Caviar Greens fall into the latter category so would have been used by default as a 512 (this would be consistent with what's being reported in smartctl). If the quirk to address this was set in a later kernel version than the one in 9.3, this might then lead to the possibility that those 3 drives were being used as 512b physical sector disks in 9.3 and now in 11.1 trying to be used as 4k. I would imagine no data on the drive would be readable if this was the case. Thoughts?

 

[HoneyBadger]

The quirk does what you assume it does - telling FreeBSD/ZFS that the underlying physical sector is 4K, not 512b.

There's a (big!) performance impact of using ashift=9 on 4K-physical disks that lie about their sector size, which is why the quirk exists.

Data loss shouldn't happen in any case; you can box yourself into a corner if you use ashift=9 and have to replace a failed 512b drive with a 4K one. Ergo, defaults are now ashift=12 but I'm uncertain when that would have changed. 9.3 might not have done that.

What do you get if you do zdb -U /data/zfs/zpool.cache | grep ashift? You might need to do zdb -E -U /data/zfs/zpool.cache | grep ashift

Although I suppose it's possible that if previously the OS was "counting by 512" and now it's "counting by 4096" then obviously "sector #64" won't be where it used to be.

 

[FNASFan]

I've just rolled back to 9.3 and lo! The volume is back! Some more info from the now working system:

• zpool status - note the warning given here about block size mismatches previously not present in the broken 11.1 - any ideas if this is the issue?
  
  Code:

    pool: ZFS1
   state: ONLINE
  status: One or more devices are configured to use a non-native block size.
  	Expect reduced performance.
  action: Replace affected devices with devices that support the
  	configured block size, or migrate data to a properly configured
  	pool.
    scan: scrub repaired 0 in 1h27m with 0 errors on Sun Nov 26 01:28:03 2017
  config:
  
  	NAME											STATE	 READ WRITE CKSUM
  	ZFS1											ONLINE	   0	 0	 0
  	  raidz1-0									  ONLINE	   0	 0	 0
  		gptid/7f05f148-556a-11e6-9f20-3c4a9277a9d8  ONLINE	   0	 0	 0  block size: 512B configured, 4096B native
  		gpt/da2									 ONLINE	   0	 0	 0  block size: 512B configured, 4096B native
  		gpt/da3									 ONLINE	   0	 0	 0  block size: 512B configured, 4096B native
  		gpt/da4									 ONLINE	   0	 0	 0  block size: 512B configured, 4096B native
  
  errors: No known data errors
  

  
  
• All smartctl output remains consistant
• Previously missing gpart list data now present
  
  Code:

  Geom name: ada0
  modified: false
  state: OK
  fwheads: 16
  fwsectors: 63
  last: 3907029134
  first: 34
  entries: 128
  scheme: GPT
  Providers:
  1. Name: ada0p1
     Mediasize: 2147483648 (2.0G)
     Sectorsize: 512
     Stripesize: 4096
     Stripeoffset: 0
     Mode: r1w1e1
     rawuuid: 871eecba-7691-11e0-b017-000c29228e23
     rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
     label: swap-da4
     length: 2147483648
     offset: 65536
     type: freebsd-swap
     index: 1
     end: 4194431
     start: 128
  2. Name: ada0p2
     Mediasize: 1998251367936 (1.8T)
     Sectorsize: 512
     Stripesize: 4096
     Stripeoffset: 0
     Mode: r1w1e2
     rawuuid: 87397955-7691-11e0-b017-000c29228e23
     rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
     label: da4
     length: 1998251367936
     offset: 2147549184
     type: freebsd-zfs
     index: 2
     end: 3907029134
     start: 4194432
  Consumers:
  1. Name: ada0
     Mediasize: 2000398934016 (1.8T)
     Sectorsize: 512
     Stripesize: 4096
     Stripeoffset: 0
     Mode: r2w2e5
  
  Geom name: ada1
  modified: false
  state: OK
  fwheads: 16
  fwsectors: 63
  last: 3907029134
  first: 34
  entries: 128
  scheme: GPT
  Providers:
  1. Name: ada1p1
     Mediasize: 2147483648 (2.0G)
     Sectorsize: 512
     Stripesize: 4096
     Stripeoffset: 0
     Mode: r1w1e1
     rawuuid: 8564df73-7691-11e0-b017-000c29228e23
     rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
     label: swap-da2
     length: 2147483648
     offset: 65536
     type: freebsd-swap
     index: 1
     end: 4194431
     start: 128
  2. Name: ada1p2
     Mediasize: 1998251367936 (1.8T)
     Sectorsize: 512
     Stripesize: 4096
     Stripeoffset: 0
     Mode: r1w1e2
     rawuuid: 857e85b3-7691-11e0-b017-000c29228e23
     rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
     label: da2
     length: 1998251367936
     offset: 2147549184
     type: freebsd-zfs
     index: 2
     end: 3907029134
     start: 4194432
  Consumers:
  1. Name: ada1
     Mediasize: 2000398934016 (1.8T)
     Sectorsize: 512
     Stripesize: 4096
     Stripeoffset: 0
     Mode: r2w2e5
  
  Geom name: ada2
  modified: false
  state: OK
  fwheads: 16
  fwsectors: 63
  last: 3907029134
  first: 34
  entries: 128
  scheme: GPT
  Providers:
  1. Name: ada2p1
     Mediasize: 2147483648 (2.0G)
     Sectorsize: 512
     Stripesize: 4096
     Stripeoffset: 0
     Mode: r1w1e1
     rawuuid: 7eee6189-556a-11e6-9f20-3c4a9277a9d8
     rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
     label: 1
     length: 2147483648
     offset: 65536
     type: freebsd-swap
     index: 1
     end: 4194431
     start: 128
  2. Name: ada2p2
     Mediasize: 1998251364352 (1.8T)
     Sectorsize: 512
     Stripesize: 4096
     Stripeoffset: 0
     Mode: r1w1e2
     rawuuid: 7f05f148-556a-11e6-9f20-3c4a9277a9d8
     rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
     label: 1
     length: 1998251364352
     offset: 2147549184
     type: freebsd-zfs
     index: 2
     end: 3907029127
     start: 4194432
  Consumers:
  1. Name: ada2
     Mediasize: 2000398934016 (1.8T)
     Sectorsize: 512
     Stripesize: 4096
     Stripeoffset: 0
     Mode: r2w2e5
  
  Geom name: ada3
  modified: false
  state: OK
  fwheads: 16
  fwsectors: 63
  last: 3907029134
  first: 34
  entries: 128
  scheme: GPT
  Providers:
  1. Name: ada3p1
     Mediasize: 2147483648 (2.0G)
     Sectorsize: 512
     Stripesize: 4096
     Stripeoffset: 0
     Mode: r1w1e1
     rawuuid: 863c0c34-7691-11e0-b017-000c29228e23
     rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
     label: swap-da3
     length: 2147483648
     offset: 65536
     type: freebsd-swap
     index: 1
     end: 4194431
     start: 128
  2. Name: ada3p2
     Mediasize: 1998251367936 (1.8T)
     Sectorsize: 512
     Stripesize: 4096
     Stripeoffset: 0
     Mode: r1w1e2
     rawuuid: 8657db39-7691-11e0-b017-000c29228e23
     rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
     label: da3
     length: 1998251367936
     offset: 2147549184
     type: freebsd-zfs
     index: 2
     end: 3907029134
     start: 4194432
  Consumers:
  1. Name: ada3
     Mediasize: 2000398934016 (1.8T)
     Sectorsize: 512
     Stripesize: 4096
     Stripeoffset: 0
     Mode: r2w2e5
  

  
  
• gpart show now shows everything
  
  Code:

  =>		34  3907029101  ada0  GPT  (1.8T)
  		  34		  94		- free -  (47k)
  		 128	 4194304	 1  freebsd-swap  (2.0G)
  	 4194432  3902834703	 2  freebsd-zfs  (1.8T)
  
  =>		34  3907029101  ada1  GPT  (1.8T)
  		  34		  94		- free -  (47k)
  		 128	 4194304	 1  freebsd-swap  (2.0G)
  	 4194432  3902834703	 2  freebsd-zfs  (1.8T)
  
  =>		34  3907029101  ada2  GPT  (1.8T)
  		  34		  94		- free -  (47k)
  		 128	 4194304	 1  freebsd-swap  (2.0G)
  	 4194432  3902834696	 2  freebsd-zfs  (1.8T)
    3907029128		   7		- free -  (3.5k)
  
  =>		34  3907029101  ada3  GPT  (1.8T)
  		  34		  94		- free -  (47k)
  		 128	 4194304	 1  freebsd-swap  (2.0G)
  	 4194432  3902834703	 2  freebsd-zfs  (1.8T)
  

  
  
• dmesg | grep ada3 does show the quirk still, but now shows extra lines
  
  Code:

  ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
  ada3: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
  ada3: Serial Number WD-WCAZA3594528
  ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
  ada3: Command Queueing enabled
  ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
  ada3: quirks=0x1<4K>
  ada3: Previously was known as ad10				] New!
  GEOM_ELI: Device ada3p1.eli created.			  ] New!
  

 

[FNASFan]

What do you get if you do zdb -U /data/zfs/zpool.cache | grep ashift?

On the now working system I get

Code:

ashift: 9

I'll bounce it back to 11.1 now and try it again on that...

 

[FNASFan]

Back on 11.1 and the output is the same (and we're back to UNKNOWN state alert and no volume):

Code:

ashift: 9

 

[HoneyBadger]

On the now working system I get

Code:

ashift: 9

I'll bounce it back to 11.1 now and try it again on that...

This could definitely be a case where the new quirk is causing FreeBSD to "count sectors" in a manner that makes it unable to recognize the disk layout. It was looking for the partition starting at offset (128x512b=64K) but if it's "counting by 4K" in the new version due to the quirk changing something, it would look for the partition starting at offset (128x4K=512K) and not find a valid one.

If this is reproducible from a clean build, I would say this should be filed as a bug report; the upgrade process needs to check for the presence of an ashift=9 pool and abort the upgrade if it detects one. I don't know that I have any extra 512e disks I can try it with though.

Offshoot: Is it possible to back up all of that data and recreate the pool? You've got 4K disks and really should be on ashift=12

 

[FNASFan]

I've gone colder on the quirk being the cause as it seems to be in the dmesg output of those disks on the working 9.3 build:

dmesg | grep ada3 does show the quirk still, but now shows extra lines

Code:

ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
ada3: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
ada3: Serial Number WD-WCAZA3594528
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3: quirks=0x1<4K>
ada3: Previously was known as ad10				] New!
GEOM_ELI: Device ada3p1.eli created.			  ] New!

9.3 seemed to indicate that it knew the physical layout was 4k (I assume from the quirk setting it as the greens don't report that), but was using it as 512 anyway (hence the warning):

Code:

status: One or more devices are configured to use a non-native block size.
	 Expect reduced performance. action: Replace affected devices with devices that support the
	 configured block size, or migrate data to a properly configured
	 pool.

Code:

gpt/da2									 ONLINE	   0	 0	 0  block size: 512B configured, 4096B native

I wonder if 11.1 is treating them as 4k because of the quirk and ignoring the fact that they're actually being used as 512? (grasping at straws here)

I'll have another dig around this morning for other clues, but regarding moving to ashift=12 - if I were able to bring back the other disks, can that be done to individual disks in the pool one at a time, or is a back-up/recreate pool/restore the only option?

 

[Johnnie Black]

WDC WD20EARS-00MVWB0

These old WD Green drives have a jumper, pins 7 and 8 IIRC, to toggle between 512n and 4k(512e) mode, it might be worth a try changing the current setting and trying again with FreeNAS 11.

While doing this shouldn't cause any risks to your data, i.e., if it doesn't work just set it back as it was, it might be a good idea to make sure backups are up to date before trying it.

 

[FNASFan]

Cheers for the tip, @Johnnie Black . I want to try this, so I popped into my local computer shop and asked the nice young man if they had any HDD jumpers - he looked at me like I was a time traveller from the stone age (not too far from the truth). I'll instead head on over to eBay or a "History of Computing" museum to see if they stock any ;)

 

[FNASFan]

@HoneyBadger I managed to set the vfs.zfs.min_auto_ashift tunable to 9 (eventually - it's ignored if set through the GUI - I think because of this - so I had to hack it in in a not-pretty way) just in case it was affecting pre-existing pools and not just newly created ones somehow as this seems to have been added in 9.10, but that was a long shot and it did't work. I'm more convinced this is right down at the device level rather than ZFS level, but now I'm out of leads. Any other things you think it's worth checking?

 

[HoneyBadger]

These old WD Green drives have a jumper, pins 7 and 8 IIRC, to toggle between 512n and 4k(512e) mode, it might be worth a try changing the current setting and trying again with FreeNAS 11.

While doing this shouldn't cause any risks to your data, i.e., if it doesn't work just set it back as it was, it might be a good idea to make sure backups are up to date before trying it.

That jumper increases the sector numbers by 1 - old OSes (WinXP) used to start their partition on "physical sector 63" which would make you misaligned for 4K sectors (63x512=32256 byte offset) so you'd set the jumper that added +1 to the LBA address, making the old OS install on "physical sector 64" instead (64x512=32768 byte offset) and be aligned. Still didn't avoid the read-modify-write but it at least stopped the issue of everything being misaligned.

Setting this jumper on a populated drive probably won't damage your data but it will definitely make it unavailable.

 

[HoneyBadger]

I wonder if 11.1 is treating them as 4k because of the quirk and ignoring the fact that they're actually being used as 512? (grasping at straws here)

I wonder if 11.1 is handling the "quirk" differently, because it really does look like it's searching for a partition calculated with 512b sectors, but treating the disk as using 4K sectors.

@HoneyBadger I managed to set the vfs.zfs.min_auto_ashift tunable to 9 (eventually - it's ignored if set through the GUI - I think because of this - so I had to hack it in in a not-pretty way) just in case it was affecting pre-existing pools and not just newly created ones somehow as this seems to have been added in 9.10, but that was a long shot and it did't work. I'm more convinced this is right down at the device level rather than ZFS level, but now I'm out of leads. Any other things you think it's worth checking?

Don't do this - you'll just end up painting yourself further into a corner. While min_auto_ashift only affects new pools, you don't want your new pools to be anything below ashift=12 in order to avoid this cropping up again.

I don't think there's anything wrong with your device (other than the firmware being a liar) but rather there's a "quirk in the quirks" and it's being treated differently between the 9.3 and 11.1 branches.

Right now, I would advise you to stay on 9.3, where your pool works, and try to figure out a way to back up your data. Once that's done, install 11.1-U6 fresh, and create a brand new pool from scratch (which should be ashift=12) and then restore it.