Truenas how to replace disk using command line

[denis4o]

Not having the gptid might be sign of a hardware (controller) issue.

I doubt that since the disk that was never replaced had gptid

Code:

gpart list
Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,9e51ede7-2030-11ea-b2ff-7085c25f11d1,0x80,0x400000)
   rawuuid: 9e51ede7-2030-11ea-b2ff-7085c25f11d1
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada0p2
   Mediasize: 1998251364352 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(2,GPT,a799872b-2030-11ea-b2ff-7085c25f11d1,0x400080,0xe8a08808)
   rawuuid: a799872b-2030-11ea-b2ff-7085c25f11d1
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251364352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029127
   start: 4194432
Consumers:
1. Name: ada0
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e4

Geom name: ada2
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada2p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,d29a2ca8-7acf-11eb-aa59-7085c25f11d1,0x80,0x400000)
   rawuuid: d29a2ca8-7acf-11eb-aa59-7085c25f11d1
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada2p2
   Mediasize: 1998251364352 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(2,GPT,eb78e373-7acf-11eb-aa59-7085c25f11d1,0x400080,0xe8a08808)
   rawuuid: eb78e373-7acf-11eb-aa59-7085c25f11d1
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251364352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029127
   start: 4194432
Consumers:
1. Name: ada2
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e4

Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 15628053127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,fb438f93-9265-11ed-be0b-7085c25f11d1,0x80,0x400000)
   rawuuid: fb438f93-9265-11ed-be0b-7085c25f11d1
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada1p2
   Mediasize: 7999415652352 (7.3T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   efimedia: HD(2,GPT,fb50dea3-9265-11ed-be0b-7085c25f11d1,0x400080,0x3a3412a08)
   rawuuid: fb50dea3-9265-11ed-be0b-7085c25f11d1
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 7999415652352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 15628053127
   start: 4194432
Consumers:
1. Name: ada1
   Mediasize: 8001563222016 (7.3T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5

Geom name: ada3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada3p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,14b75f55-b803-11eb-8966-7085c25f11d1,0x80,0x400000)
   rawuuid: 14b75f55-b803-11eb-8966-7085c25f11d1
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada3p2
   Mediasize: 1998251364352 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(2,GPT,1b8422cd-b803-11eb-8966-7085c25f11d1,0x400080,0xe8a08808)
   rawuuid: 1b8422cd-b803-11eb-8966-7085c25f11d1
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251364352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029127
   start: 4194432
Consumers:
1. Name: ada3
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e4

Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 30031215
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: da0p1
   Mediasize: 272629760 (260M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0
   efimedia: HD(1,GPT,d6b7c845-4a66-11e8-841c-30f9edbae477,0x28,0x82000)
   rawuuid: d6b7c845-4a66-11e8-841c-30f9edbae477
   rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
   label: (null)
   length: 272629760
   offset: 20480
   type: efi
   index: 1
   end: 532519
   start: 40
2. Name: da0p2
   Mediasize: 15103328256 (14G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 272650240
   Mode: r1w1e1
   efimedia: HD(2,GPT,d716d376-4a66-11e8-841c-30f9edbae477,0x82028,0x1c21d40)
   rawuuid: d716d376-4a66-11e8-841c-30f9edbae477
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 15103328256
   offset: 272650240
   type: freebsd-zfs
   index: 2
   end: 30031207
   start: 532520
Consumers:
1. Name: da0
   Mediasize: 15376000000 (14G)
   Sectorsize: 512
   Mode: r1w1e2

 

[danb35]

There is no error when I try to offline the disk via the GUI, but nothing happens.

That is odd. There may be a bug in TrueNAS, it may be (as you're suspecting) that it doesn't deal with the adanp2 designations, or it could be something else. I'd think it's worth a bug report (link's at the top of the page, or you can do it through the TrueNAS web UI), but if you want to address it as quickly as possible, the instructions I gave above should take care of it--and use gptids, so TrueNAS should be happier with it going forward.

 

[danb35]

Not having the gptid might be sign of a hardware (controller) issue.

No, as later reported, he followed a bad set of instructions that put devices into the pool without the gptid. Not a fatal situation, but also not ideal.

 

[denis4o]

but if you want to address it as quickly as possible, the instructions I gave above should take care of it

If I am reading your instructions correctly, it might not work for my case:

1. I started with a RAIDZ1 pool of 4 x 2TB HDDs (1 designated with gptid and 3 with adaXp2) and wanted to expand it. I have no more SATA ports available :(
2. I was able to offline and replace the gptid one with a 8 TB HDDs and I wanted to do the same with the other 3, but I was unable to
3. Now I have a mix of 8 TB and 3 x 2 TB, although the pool size has not changed and is still 6 TB usable (85% full)

Maybe I don't really understand the point of sparse file ... will it take up space (I guess not )? Also, is there a possibility of data loss?

 

[danb35]

I'm not suggesting you create a degraded pool, but that resource has the commands to partition the disk and determine its gptid. Offline an old disk, install a new one, partition it and determine the gptid of the second partition, then use zpool replace to replace ada3p2 with whatever the gptid is. Repeat with the remaining disks.

 

[Davvo]

Nuking the pool and recreating it with the drives you want is also an option if you have backup.
Do note that RAIDZ1 is not advisable for large (multiple TB) HDDs.

 

[denis4o]

Nuking the pool and recreating it with the drives you want is also an option if you have backup.

I'm in a process of backing up just in case and eventually I might nuke the pool and recreate it. I just don't want to deal with setting up my jails again.

Do note that RAIDZ1 is not advisable for large (multiple TB) HDDs.

Why is that? I use the NAS as a home file server, serving movies, downloading torrents and backing up photos and videos. Nothing too serious.

partition the disk and determine its gptid

I was under the impression that the disks are properly partitioned I have their gptids (rawuuid):

Code:

root@freenas:~ # gpart list
Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,9e51ede7-2030-11ea-b2ff-7085c25f11d1,0x80,0x400000)
   rawuuid: 9e51ede7-2030-11ea-b2ff-7085c25f11d1
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada0p2
   Mediasize: 1998251364352 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(2,GPT,a799872b-2030-11ea-b2ff-7085c25f11d1,0x400080,0xe8a08808)
   rawuuid: a799872b-2030-11ea-b2ff-7085c25f11d1
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251364352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029127
   start: 4194432
Consumers:
1. Name: ada0
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e4

Geom name: ada2
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada2p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,d29a2ca8-7acf-11eb-aa59-7085c25f11d1,0x80,0x400000)
   rawuuid: d29a2ca8-7acf-11eb-aa59-7085c25f11d1
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada2p2
   Mediasize: 1998251364352 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(2,GPT,eb78e373-7acf-11eb-aa59-7085c25f11d1,0x400080,0xe8a08808)
   rawuuid: eb78e373-7acf-11eb-aa59-7085c25f11d1
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251364352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029127
   start: 4194432
Consumers:
1. Name: ada2
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e4

Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 15628053127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,fb438f93-9265-11ed-be0b-7085c25f11d1,0x80,0x400000)
   rawuuid: fb438f93-9265-11ed-be0b-7085c25f11d1
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada1p2
   Mediasize: 7999415652352 (7.3T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   efimedia: HD(2,GPT,fb50dea3-9265-11ed-be0b-7085c25f11d1,0x400080,0x3a3412a08)
   rawuuid: fb50dea3-9265-11ed-be0b-7085c25f11d1
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 7999415652352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 15628053127
   start: 4194432
Consumers:
1. Name: ada1
   Mediasize: 8001563222016 (7.3T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5

Geom name: ada3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada3p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,14b75f55-b803-11eb-8966-7085c25f11d1,0x80,0x400000)
   rawuuid: 14b75f55-b803-11eb-8966-7085c25f11d1
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada3p2
   Mediasize: 1998251364352 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(2,GPT,1b8422cd-b803-11eb-8966-7085c25f11d1,0x400080,0xe8a08808)
   rawuuid: 1b8422cd-b803-11eb-8966-7085c25f11d1
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 1998251364352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029127
   start: 4194432
Consumers:
1. Name: ada3
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e4

Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 30031215
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: da0p1
   Mediasize: 272629760 (260M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0
   efimedia: HD(1,GPT,d6b7c845-4a66-11e8-841c-30f9edbae477,0x28,0x82000)
   rawuuid: d6b7c845-4a66-11e8-841c-30f9edbae477
   rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
   label: (null)
   length: 272629760
   offset: 20480
   type: efi
   index: 1
   end: 532519
   start: 40
2. Name: da0p2
   Mediasize: 15103328256 (14G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 272650240
   Mode: r1w1e1
   efimedia: HD(2,GPT,d716d376-4a66-11e8-841c-30f9edbae477,0x82028,0x1c21d40)
   rawuuid: d716d376-4a66-11e8-841c-30f9edbae477
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 15103328256
   offset: 272650240
   type: freebsd-zfs
   index: 2
   end: 30031207
   start: 532520
Consumers:
1. Name: da0
   Mediasize: 15376000000 (14G)
   Sectorsize: 512
   Mode: r1w1e2

Or I am wrong? I just need to find out how to get that listed under /dev maybe add them via glabel? I want to apologize if what I am saying is stupid. I just started digging in more deeply into TrueNAS, FreeBSD and ZFS. I set this server up 5-6 years ago and the only maintenance it had was the disk replacement (obviously not done properly). Has been working since then flawlessly.

 

[danb35]

It doesn't matter that the disks already in the pool are properly partitioned; you need to properly partition the replacements--and when you do the replacement, use the gptid rather than adanp2.

 

[denis4o]

So that will still put the pool in a degraded state since I will have to physically remove the HDD that I want to replace and install the new one and partition it - I have no spare SATA ports left. Is that even possible? In my previous experience if I detach ada2 (for example) from the server and I issue the command zpool replace home_server ada2p2 gptid/a799872b-2030-11ea-b2ff-7085c25f11d1 won't I get

Code:

cannot open 'ada2p2': no such device in /dev

or something similar

 

[danb35]

Yes, it will put the pool in a degraded state. And yes, it's still possible to replace devices. Do zpool status to see how the missing device is identified, then use that identifier in the zpool replace command.

 

[denis4o]

Thanks @danb35 for you time! I will give it a go tomorrow.

 

[Davvo]

Why is that? I use the NAS as a home file server, serving movies, downloading torrents and backing up photos and videos. Nothing too serious.

RAID5 is the hardware equivalent of RAIDZ1

Why RAID 5 stops working in 2009

The storage version of Y2k? No, it's a function of capacity growth and RAID 5's limitations.

www.zdnet.com

 

[Chris Moore]

This RAID level conversation relates to something I have been battling at work. We have a storage cluster at work that is made up of a large number of 6TB hard drives. Each RAID group within the cluster is RAID-6. This worked fine when the cluster was new... Three times in the last 9 months we have had to recover from a multi drive failure because when one drive in a RAID group failed and started trying to rebuild, another drive would be so stressed by the rebuild that it would fail also. Last weekend, that was enough to cause a third drive to fail and if you know your RAID levels, you know that recovering from three drives being down is no easy task.
TLDR, if you plan to run the array for a longish time, you probably want RAID-z3. It costs an extra drive, but it gives you a little more security.
On top of that, understand that no RAID is a backup. If the data matters to you, you should have it recorded in three places, one of them being off-site.
Additionally, make the plan at the start to replace your disks around the 5 year mark. Engineering told me this weekend that none of their reliability projections are valid beyond 5 years. That isn't to say the disks can't last longer, it is just a warning that beyond 5 years, you are on borrowed time. The disks we are running at work, they were manufactured in 2016 and the system went into production in 2017 after validation tests were completed.

 

[denis4o]

@danb35 I was wondering would it be an issue if the partitions are of a different size, because my old partitions are 1.8T and the new ones would be 7.3T. I know there is an automatic pool expand feature in TrueNAS but I don't know if it would work (I hope it does). I backed up my data just in case, but I am still cautions and haven't taken any actions yet.

 

[danb35]

I was wondering would it be an issue if the partitions are of a different size

No.

 

[denis4o]

Firstly, I'd like to thank @danb35 for the comments and for the guidance. I have replaced 3 out of the 4 HDDs already.

Secondly, I'd like to add something for the folks who might read the topic later and come across the same problem. If the HDDs have been partitioned, you'd want to delete the partition information by deleting the first and the last 512 bytes on the HDD and then re-pratition it with the steps that @danb35 described in his tutorial for creating a degraded pool. The commands that I used to delete the partition info on the HDDs are:

Code:

dd if=/dev/zero of=/dev/ada3 bs=1m count=1

dd if=/dev/zero of=/dev/ada3 bs=1m oseek=`diskinfo ada3 | awk '{print int($3 / (1024*1024)) - 4;}'`

P.S. replace ada3 with your drive designator