migrated to new usb boot devices and now swap won't mount

[colmconn]

Hey,

I've recently migrated to a new boot device.

I did a fresh install of FreeNAS on the new device and booted up from it and uploaded a copy of my config and rebooted. The problem I'm having now is that despite the swap partitions existing on my pool disk drives the swapon command fails to add these drives to the available swap on boot.

I've included some info from the kernel log, the output of gpart, geli list, swapinfo and the contents of /etc/fstab. If you need any more info please ask.

Can any one tell me please how I fix this so that the three swap partitions on my hard drives in my main pool will be successfully added to the system as swap on boot?

Thanks,

Colm.



I'm running:
FreeBSD starbug.local 9.3-RELEASE-p25 FreeBSD 9.3-RELEASE-p25 #0 r281084+d3a5bf7: Wed Sep 2 15:00:10 PDT 2015 root@build3.ixsystems.com:/tank/home/jkh/build/FN/objs/os-base/amd64/tank/home/jkh/build/FN/FreeBSD/src/sys/FREENAS.amd64 amd64

When I boot up the machine, I find the following messages in the log (I've trimmed hopefully irrelevant other log entries, but if you need more please ask):
Sep 5 14:57:10 starbug GEOM_ELI: Device ada3p1.eli created.
Sep 5 14:57:10 starbug GEOM_ELI: Encryption: AES-XTS 256
Sep 5 14:57:10 starbug GEOM_ELI: Crypto: hardware
Sep 5 16:22:59 starbug notifier: geli: Cannot access ada3p1 (error=1).
Sep 5 16:22:59 starbug notifier: swapon: /dev/ada2p1.eli: No such file or directory
Sep 5 16:22:59 starbug notifier: swapon: /dev/ada0p1.eli: No such file or directory
Sep 5 16:22:59 starbug notifier: swapon: /dev/ada1p1.eli: No such file or directory
Sep 5 16:22:59 starbug notifier: geli: Cannot access ada3p1 (error=1).
Sep 5 16:22:59 starbug notifier: swapon: /dev/ada2p1.eli: No such file or directory
Sep 5 16:22:59 starbug notifier: swapon: /dev/ada0p1.eli: No such file or directory
Sep 5 16:22:59 starbug notifier: swapon: /dev/ada1p1.eli: No such file or directory

In the following ada[0-2] are the drives in my main storage pool and ada3 is an SSD containing my system dataset and jails.

My /etc/fstab contains:
/dev/ada3p1.eli none swap sw 0 0
/dev/ada2p1.eli none swap sw 0 0
/dev/ada0p1.eli none swap sw 0 0
/dev/ada1p1.eli none swap sw 0 0

~# gpart show /dev/ada0
=> 34 5860533101 ada0 GPT (2.7T)
34 94 - free - (47k)
128 4194304 1 freebsd-swap (2.0G)
4194432 5856338696 2 freebsd-zfs (2.7T)
5860533128 7 - free - (3.5k)

~# gpart show /dev/ada1
=> 34 5860533101 ada1 GPT (2.7T)
34 94 - free - (47k)
128 4194304 1 freebsd-swap (2.0G)
4194432 5856338696 2 freebsd-zfs (2.7T)
5860533128 7 - free - (3.5k)

~# gpart show /dev/ada2
=> 34 5860533101 ada2 GPT (2.7T)
34 94 - free - (47k)
128 4194304 1 freebsd-swap (2.0G)
4194432 5856338696 2 freebsd-zfs (2.7T)
5860533128 7 - free - (3.5k)

~# gpart show /dev/ada3
=> 34 468862061 ada3 GPT (223G)
34 94 - free - (47k)
128 4194304 1 freebsd-swap (2.0G)
4194432 464667656 2 freebsd-zfs (221G)
468862088 7 - free - (3.5k)

And swap info shows:
Device 1K-blocks Used Avail Capacity
/dev/ada3p1.eli 2097152 0 2097152 0%

# geli list
Geom name: gptid/6f026f72-4ee2-11e5-8b2c-d0509978574f.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 128
Crypto: hardware
UsedKey: 0
Flags: NONE
KeysAllocated: 56
KeysTotal: 56
Providers:
1. Name: gptid/6f026f72-4ee2-11e5-8b2c-d0509978574f.eli
Mediasize: 237909835776 (221G)
Sectorsize: 4096
Mode: r1w1e1
Consumers:
1. Name: gptid/6f026f72-4ee2-11e5-8b2c-d0509978574f
Mediasize: 237909839872 (221G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 2147549184
Mode: r1w1e1

Geom name: ada3p1.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 256
Crypto: hardware
Flags: ONETIME, W-DETACH, W-OPEN
KeysAllocated: 1
KeysTotal: 1
Providers:
1. Name: ada3p1.eli
Mediasize: 2147483648 (2.0G)
Sectorsize: 4096
Mode: r1w1e0
Consumers:
1. Name: ada3p1
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 65536
Mode: r1w1e1

Geom name: gptid/5e6b9083-34dc-11e5-96d9-d0509978574f.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 128
Crypto: hardware
UsedKey: 0
Flags: NONE
KeysAllocated: 699
KeysTotal: 699
Providers:
1. Name: gptid/5e6b9083-34dc-11e5-96d9-d0509978574f.eli
Mediasize: 2998445408256 (2.7T)
Sectorsize: 4096
Mode: r1w1e1
Consumers:
1. Name: gptid/5e6b9083-34dc-11e5-96d9-d0509978574f
Mediasize: 2998445412352 (2.7T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1

Geom name: gptid/5f2a9fac-34dc-11e5-96d9-d0509978574f.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 128
Crypto: hardware
UsedKey: 0
Flags: NONE
KeysAllocated: 699
KeysTotal: 699
Providers:
1. Name: gptid/5f2a9fac-34dc-11e5-96d9-d0509978574f.eli
Mediasize: 2998445408256 (2.7T)
Sectorsize: 4096
Mode: r1w1e1
Consumers:
1. Name: gptid/5f2a9fac-34dc-11e5-96d9-d0509978574f
Mediasize: 2998445412352 (2.7T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1

Geom name: gptid/5db1f87e-34dc-11e5-96d9-d0509978574f.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 128
Crypto: hardware
UsedKey: 0
Flags: NONE
KeysAllocated: 699
KeysTotal: 699
Providers:
1. Name: gptid/5db1f87e-34dc-11e5-96d9-d0509978574f.eli
Mediasize: 2998445408256 (2.7T)
Sectorsize: 4096
Mode: r1w1e1
Consumers:
1. Name: gptid/5db1f87e-34dc-11e5-96d9-d0509978574f
Mediasize: 2998445412352 (2.7T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1

 

[joeschmuck]

Describe what you did and give good details on your old boot device, the new boot device, where they were/are connected and anything else you can think of. If you messed around with the SATA connections, re-seat them all on the MB and drive side. And what steps did you do starting with booting your system after you built up your boot device? Hopefully you imaged your boot device, booted, auto-import your pool, restore your config file. That is what I do when I'm testing out new versions of FreeNAS and it has always worked.

 

[colmconn]

Thanks for replying.

I still have the old boot USB device (a 16GB KINGSTON USB 2 key). It's not been reused for anything else

Note that the following is from memory as it's been over a week since I replaced the drive and have only recently noticed the 3 swap partitions on my hard drives are not being added to swap automatically.

1. Save config.
2. Save encryption keys (key 0 and key 1)
3. Shutdown freeness
4. Install FreeNAS onto the new USB boot key (a low profile SANDISK Cruzer "16GB" Fit; quotes because the thing isn't really 16GB but just under 16 unlike the KINGSTON). As with the KINGSTON, the installation was done on my Mac by booting the FreeNAS installation CD in VirtualBox and installing directly onto the SANDISK USB key.
5. Boot from newly created boot drive
6. Upload config and wait for reboots to complete
7. In the storage page in the GUI, I import my encrypted pool volume, by entering passphrase and uploading key. However, this only resulted in the pool being decrypted and available for that boot. The keys never survived reboot and I upload the key and enter passphrase on every reboot instead of just the pool passphrase as one would typically expect. I found a previous thread in the forum and bug report that was very similar, if not identical, to the situation I faced but can't find it now. I'll keep looking and post links if I find them again. I eventually got around this problem by re-keying the pool which made the problem go away, that is I only have to enter the passphrase to unlock the pool after a reboot. I've had no problems with the system since changing the boot USB key, other than this issue with swap.

In the past week or so I installed a spare SSD that I had. I used the volume manager to add the drive to the system and let it partition and import it SSD drive. I use it for my .system and since Saturday it also houses my jails. It was around when I installed the SSD that I noticed that the only swap partition being used was the one created on it.

 

[joeschmuck]

I need to stop right here because you are dealing with encryption. I have no personal experience with encryption and it can be tricky. I would never recommend someone use it either, never, not unless you have tested out how to replace a drive before putting any data on the system and took fantastic notes and had all your keys backed up.

Sorry I can't help you more but I don't want any data destroyed because I told you to do something and the encryption screwed it up. I saw the .eli extension and I was hoping those were just the swap partitions but maybe not. If you can read your data, I'd back it all up. Once that is done then pursue recovery. You may end up destroying the pool and recreating it. Again, if you don't need encryption, don't use it. You can encrypt specific files if you need to, True Crypt or something similar can offer a lot to a person.

 

[colmconn]

Not a problem. Thanks for taking the time to reply. I just want to say that my system is humming along quite nicely and I've no problems de-crypting and mounting my pool on each reboot. The only issue is related to the failure of the system to add the swap partitions on the 3x3TB hard drives in my pool. I figured that since swap is always encrypted even on otherwise unencrypted systems that someone may be able to help me fix it.