SOLVED Multipathed Disk Replacement: Unable to GPT format the disk

[MatthewSteinhoff]

TL;DR: When trying to replace a multipath disk under FreeNAS-9.10.2-U5 I get the message...

Unable to GPT format the disk "multipath/disk14"

Now what?

Long story...

I've been seeing this error grouping in my logs for a few weeks...

Code:

(da24:mps0:0:77:0): READ(10). CDB: 28 00 16 9d 6a 30 00 00 a0 00
(da24:mps0:0:77:0): CAM status: SCSI Status Error
(da24:mps0:0:77:0): SCSI status: Check Condition
(da24:mps0:0:77:0): SCSI sense: RECOVERED ERROR asc:17,6 (Recovered data without ECC - data auto-reallocated)
(da24:mps0:0:77:0): Info: 0x169d6a49
(da24:mps0:0:77:0): Actual Retry Count: 12

Drive da24 is also known as da8 which is multipath/disk18. The drive multipath/disk18 is one of the seven drives that make up the RAIDZ2 volume group V-Live600.

Selecting multipath/disk18 and clicking 'Replace' allows me to choose multipath/disk14 as the replacement. When I click 'Replace Disk', I get this message...

Unable to GPT format the disk "multipath/disk14"

Here is the full log dump...

Code:

Jul 11 09:54:15 bulk GEOM_ELI: Device multipath/disk18p1.eli destroyed.
Jul 11 09:54:15 bulk GEOM_ELI: Detached multipath/disk18p1.eli on last close.
Jul 11 09:54:15 bulk notifier: geli: No such device: /dev/multipath/disk18p1.
Jul 11 09:54:15 bulk notifier: dd: /dev/multipath/disk14: Invalid argument
Jul 11 09:54:15 bulk notifier: 1+0 records in
Jul 11 09:54:15 bulk notifier: 0+0 records out
Jul 11 09:54:15 bulk notifier: 0 bytes transferred in 0.001323 secs (0 bytes/sec)
Jul 11 09:54:15 bulk notifier: dd: /dev/multipath/disk14: Invalid argument
Jul 11 09:54:15 bulk notifier: 1+0 records in
Jul 11 09:54:15 bulk notifier: 0+0 records out
Jul 11 09:54:15 bulk notifier: 0 bytes transferred in 0.001190 secs (0 bytes/sec)
Jul 11 09:54:16 bulk manage.py: [middleware.exceptions:37] [MiddlewareError: Unable to GPT format the disk "multipath/disk14"]
Jul 11 09:54:16 bulk manage.py: [rollbar:1265] Got unexpected status code from Rollbar api: 403
Response:
{
  "err": 1,
  "message": "project is not enabled"
}
Jul 11 09:54:16 bulk manage.py: [rollbar:1097] Exception while posting item ApiError(u'project is not enabled',)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/rollbar/__init__.py", line 1095, in _send_payload
	_post_api('item/', payload, access_token=access_token)
  File "/usr/local/lib/python2.7/site-packages/rollbar/__init__.py", line 1138, in _post_api
	return _parse_response(path, SETTINGS['access_token'], payload, resp)
  File "/usr/local/lib/python2.7/site-packages/rollbar/__init__.py", line 1274, in _parse_response
	raise ApiError(json_data.get('message') or 'Unknown error')
ApiError: project is not enabled

The drives and partitions do exist...

Code:

# ls -l /dev/multipath/disk18*
crw-r-----  1 root  operator  0x1f6 Jun 19 09:46 /dev/multipath/disk18
crw-r-----  1 root  operator  0x222 Jun 19 09:46 /dev/multipath/disk18p1
crw-r-----  1 root  operator  0x224 Jun 19 09:46 /dev/multipath/disk18p2

# ls -l /dev/multipath/disk14*
crw-r-----  1 root  operator  0x164 Jun 19 09:46 /dev/multipath/disk14[/CMD]

I see the GEOM being created at boot...

 GEOM_MULTIPATH: disk14 created
GEOM_MULTIPATH: da34 added to disk14
GEOM_MULTIPATH: da34 is now active path in disk14
GEOM_MULTIPATH: da35 added to disk14



I seem to remember something about having to fully wipe the GPT headers and some scary sysctl tweaks but, before I try a procedures years old and a version or two behind, I thought it best to ask for advice.

Suggestions?

Cheers,
Matt

 

[zoomzoom]

Try using gpart to clear the partition table, create an mbr one, clear that one, then try the replace via the GUI

1. gpart destroy -F daX
2. gpart create -s mbr daX ; gpart add -t ntfs -a 4k daX
3. gpart destroy -F daX

• Where daX is the replacement disk

Please utilize [code] brackets, not [cmd] brackets for code output, as [cmd] brackets are intended for one line commands​

 

[MatthewSteinhoff]

Where daX is the replacement disk

Should I use the multipath address or one of the two direct-to-disk paths? Or does it matter?

Cheers,
Matt

 

[zoomzoom]

It should be whatever device that disk shows up as under /dev. It's likely /dev/adaX, but I do know some motherboards use da versus ada for SATA disks.

• Do you by chance know why yours are listed under /dev/multipath (as I don't have that directory)? Did you by chance create your pool via cli versus the WebUI?

 

[MatthewSteinhoff]

Do you by chance know why yours are listed under /dev/multipath (as I don't have that directory)? Did you by chance create your pool via cli versus the WebUI?

I'm using SAS drives in an external enclosure that supports multipathing. The multipath is created automatically by, I imagine, BSD. Everything I do is through the WebGUI including pool creation.

There are three paths to every device: the two physical paths and the multipath device which includes both paths. In this case, that is...

$ ls -l /dev/da34 /dev/da35 /dev/multipath/disk14
crw-r----- 1 root operator 0x130 Jun 19 09:46 /dev/da34
crw-r----- 1 root operator 0x132 Jun 19 09:45 /dev/da35
crw-r----- 1 root operator 0x164 Jun 19 09:46 /dev/multipath/disk14

When building a volume group in FreeNAS, only the multipath device is seen. When I do 'View Disks' under the storage menu, I see nothing. When I do 'View Multipaths', I see the actionable device and the subdevices that create that device...



So, I'm not sure if it is best to act on the multipath - which is what FreeNAS uses - or the underlying device paths?

Cheers,
Matt

 

[zoomzoom]

@MatthewSteinhoff (Just an FYI: I have no experience with SAS drives) You should be able to determine the correct disk device to utilize in gpart by grabbing the S.M.A.R.T data of the possible devices and matching the serial to the correct drive: smartctl -a /dev/da34 | grep Serial

• The device path is what would be utilized, as the device itself needs to have it's partition table cleared (or at least this is how I solved a similar issue with one of my SATA disks in my pool). However, while I assume this should be the same for SAS drives (as SAS is simply an interface), I can't be one hundred percent sure since I have no experience with SAS drives.

 

[MatthewSteinhoff]

Nope. Did the gpart thing and I'm still getting the same error.

Code:

# smartctl -a /dev/da34 | grep Serial
Serial number:		PPWZEAHD

# smartctl -a /dev/da35 | grep Serial
Serial number:		PPWZEAHD

# smartctl -a /dev/multipath/disk14 | grep Serial

# gpart show /dev/da34	 
gpart: No such geom: /dev/da34.

# gpart show da34
gpart: No such geom: da34.

# gpart show /dev/da35
gpart: No such geom: /dev/da35.

# gpart show da35
gpart: No such geom: da35.

# gpart show multipath/disk14
=>		34  1125768536  multipath/disk14  GPT  (545G)
		  34  1125768536					- free -  (545G)

# gpart destroy -F /dev/da34
gpart: arg0 'da34': Invalid argument

# gpart destroy -F da34
gpart: arg0 'da34': Invalid argument

# gpart destroy -F /dev/da35
gpart: arg0 'da35': Invalid argument

# gpart destroy -F da35
gpart: arg0 'da35': Invalid argument

# gpart destroy -F multipath/disk14
multipath/disk14 destroyed

# gpart add -t ntfs -a 4k /dev/da34
gpart: No such geom: /dev/da34.

# gpart add -t ntfs -a 4k da34
gpart: No such geom: da34.

# gpart add -t ntfs -a 4k multipath/disk14
gpart: Invalid alignment param: Invalid argument

# gpart add -t ntfs  multipath/disk14
multipath/disk14s1 added

# gpart destroy -F multipath/disk14
multipath/disk14 destroyed

# gpart show multipath/disk14
gpart: No such geom: multipath/disk14.

It looks like gpart didn't like the 4k alignment. Other than that, I did exactly what you suggested.

After doing the above, I went into the WebGUI and tried to replace the questionable multipath device with the good disk14 multipath and it again reported "Unable to GPT format the disk "multipath/disk14"".

Cheers,
Matt

 

[zoomzoom]

That's because you skipped the create command, as you're trying to add a partition to a drive without a partition table: gpart destroy -F multipath/disk14 ; gpart create -s mbr multipath/disk14 ; gpart add -t ntfs multipath/disk14 ; gpart destroy -F multipath/disk14

• If that doesn't solve it, try using forum search (if you haven't already), search the FreeBSD forum, and try googling... at least until someone familiar with SAS pools replies.

 

[MatthewSteinhoff]

Sorry. I did that step but missed copying it into my reply...

Code:

# gpart create -s mbr multipath/disk14 
multipath/disk14 created

#  gpart add -t ntfs -a 4k multipath/disk14
gpart: Invalid alignment param: Invalid argument

#  gpart add -t ntfs  multipath/disk14
multipath/disk14s1 added

#  gpart destroy -F multipath/disk14
multipath/disk14 destroyed

I've done a fair amount of searching and haven't yet found the magic recipe. I'll follow-up here if I come across the solution.

I really appreciate your time. Thank you.

Cheers,
Matt

 

[wblock]

There is no need to create a fake partition. Do the destroy -F, then a create GPT, then another destroy -F.

If the multipath stuff is opening that device for write, the GEOM system could prevent other things from writing to the partition table. In that case, it might be necessary to use the dangerous "yes, I really mean to do this" sysctl.

 

[MatthewSteinhoff]

Upgraded to 11.0-U2 and still having this problem. Went ahead and created a bug report...

https://bugs.freenas.org/issues/25407

I don't see a lot of FreeNAS users using multipath. At least it doesn't come up often in the forums. I wonder if it is a feature heavily tested back at the mothership?

Cheers,
Matt

 

[MatthewSteinhoff]

FreeNAS developer Alexander Motin figured it out. For reasons unknown, the drive was formatted with 520-byte sectors...

Geom name: da34
Providers:
1. Name: da34
Mediasize: 585399674080 (545G)
Sectorsize: 520
Mode: r1w1e1
descr: HITACHI DKR5C-J600SS

Using sg_format...

sg_format --verbose --format --size=512 --six /dev/da34

...we were able to convert the drive's sector size. Once that change was made, FreeNAS could use the drive.

Cheers,
Matt

 

[zoomzoom]

Out of curiosity, why would one format a HDD today with anything lower than 4K sectors? (IIRC, it takes more than 512b to store 512b worth of data in a 512b sector)

 

[MatthewSteinhoff]

The FreeNAS developer said 512 so we did 512.

It looks like, even though the drives were described as 'new old stock', they may have been used or formatted for a commercial array. Netapp seems to like a 520 sector size.

Cheers,
Matt

 

[Stux]

Out of curiosity, why would one format a HDD today with anything lower than 4K sectors? (IIRC, it takes more than 512b to store 512b worth of data in a 512b sector)

One reason is to install FreeNAS
https://bugs.freenas.org/issues/25433