SOLVED How to setup FreeNAS on a (partitioned) single SSD with boot and jails.

seltz

Dabbler
Joined
Dec 15, 2015
Messages
28
I have a fairly low-use machine, and wanted to run FreeNAS as a partition on an SSD to allow space for jails.

Motivations:
- iocage (i.e. jails) cannot run on freenas-boot. I believe iocage's own recommendations says it won't run on freebsd-boot partitions, and the FreeNAS GUI won't let you either.
- I don't want to have another usb drive or ssd drive dedicated for freebsd-boot, nor do I want to use the full ssd drive for just freebsd-boot. I realize this is not the normal recommendation (freenas's installer only lets you install on the whole disk) and carries some risk of failure. I consider my needs fairly minimal, and only have so many SATA/USB ports for storage devices.

Prior Work I based this on:
- Jeremy Lea's 'Installing FreeNAS on a Partition': https://forums.servethehome.com/ind...ing-FreeNAS-on-a-partition.27798/#post-271656
- A prior failed attempt of trying to do zfs send / recv to copy & boot freenas-boot instead of mirroring: https://www.ixsystems.com/community...pool-onto-a-usb-drive-and-boot-from-it.85839/

What you need if starting from scratch, ignoring your storage hard drives:
- USB drive A - flashed with downloaded freenas iso.
- USB drive B - to install freenas on to start with. I used a fairly small one 8gb, since the mirror can be larger.
- SSD drive C - that will be the final place for freenas-boot, swap, jails, etc.

Steps:
1. Install FreeNAS onto USB drive B as normal.
2. Boot FreeNAS from USB drive B.
3. From here, run the following commands (I'm just running these one-by-one)

#ssd drive - wipe contents (SSD is ada0)
gpart destroy -F /dev/ada0

#ssd drive - setup gpt partitioning scheme
gpart create -s gpt ada0

#ssd drive - setup boot partition
# and embed bootstrap code in
# partition scheme metadata
# note: i follow gpart's manpage advice on making the size 472
# so the swap partition will be aligned correctly.

gpart add -b 40 -s 472 -t freebsd-boot ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0


#ssd drive - add swap partition at first available offset - will be ada0p2
gpart add -s 16g -t freebsd-swap -l swap0 ada0

#ssd drive - add boot partition - will be ada0p3
gpart add -s 16g -t freebsd-zfs -l root0 ada0

#ssd drive - add jail partition - will be ada0p4
gpart add -t freebsd-zfs -l jail0 ada0

#setup jail pool
zpool create jail /dev/ada0p4
umount /jail
zpool export jail


#setup boot as mirror from our usb drive (usb drive B is da0, da0p2 is the freenas-boot pool)
zpool attach freenas-boot /dev/da0p2 /dev/ada0p3

#wait for resilver! run 'zpool status' to check status
zpool offline freenas-boot /dev/da0p2
zpool detach freenas-boot /dev/da0p2


4. Reboot/remove usb drive B, and boot from SSD. You may need to import the 'jail' pool within the freenas GUI before setting up jails.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,741
;)
 

narcdor

Cadet
Joined
Sep 20, 2020
Messages
1
Long time lurker and very basic FreeNAS user, I had to register so I could say THANK YOU!!! Finally I can run the boot partition and the jails on one SSD. That waste of space was so irritating. Now I can reclaim a 3TB drive for time machine since I was one whole drive short for that.
 

mkabir

Cadet
Joined
Dec 16, 2020
Messages
1
Excellent article. Thank you for the step by step guide.

Had a similar scenario, with the addition of a second SSD. Was setting up mirrored boot with two 250 GB SSD and wanted to use unused portion for jail. Followed all steps from seltz upto step '#ssd drive - add jail partition - will be ada0p4' for both SSD. Then created a mirrored pool with two partitions for jail. This is for TrueNAS. Note that, needed to use boot-pool for turenas vs freenas-boot for freenas.

All steps are listed below. Copying seltz's (thanks seltz) steps for completeness.

Steps:
1. Install TrueNAS onto USB drive B as normal.
2. Boot TrueNAS from USB drive B.
3. From here, run the following commands (running these one-by-one)

#ssd drive - wipe contents (First SSD is ada0, Second SSD is ada1)
gpart destroy -F /dev/ada0
gpart destroy -F /dev/ada1

#ssd drive - setup gpt partitioning scheme
gpart create -s gpt ada0
gpart create -s gpt ada1

#ssd drive - setup boot partition
# and embed bootstrap code in
# partition scheme metadata
# note: i follow gpart's manpage advice on making the size 472
# so the swap partition will be aligned correctly.

gpart add -b 40 -s 472 -t freebsd-boot ada0
gpart add -b 40 -s 472 -t freebsd-boot ada1

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

#ssd drive - add swap partition at first available offset - will be ada0p2, ada1p2
gpart add -s 16g -t freebsd-swap -l swap0 ada0
gpart add -s 16g -t freebsd-swap -l swap0 ada1

#ssd drive - add boot partition - will be ada0p3, ada1p3 (I had allocated 32g, probably not needed)
gpart add -s 16g -t freebsd-zfs -l root0 ada0
gpart add -s 16g -t freebsd-zfs -l root0 ada1

#ssd drive - add jail partition - will be ada0p4, ada1p4
gpart add -t freebsd-zfs -l jail0 ada0
gpart add -t freebsd-zfs -l jail0 ada1

Steps below are bit different from post above (colored text):

#setup jail pool
zpool create jail mirror /dev/ada0p4 /dev/ada1p4
umount /jail
zpool export jail

#setup boot as mirror from our usb drive (usb drive B is da0, da0p2 is the boot-pool)
zpool attach boot-pool /dev/da0p2 /dev/ada0p3
zpool attach boot-pool /dev/da0p2 /dev/ada1p3


#wait for resilver! run 'zpool status' to check status

4. Reboot/remove usb drive B, and boot from SSD. You will need to import the 'jail' pool within the truenas GUI before setting up jails.

Thanks.
 
Joined
Jan 16, 2021
Messages
2
#setup boot as mirror from our usb drive (usb drive B is da0, da0p2 is the boot-pool)
zpool attach boot-pool /dev/da0p2 /dev/ada0p3
zpool attach boot-pool /dev/da0p2 /dev/ada1p3

Sadly this does not work for me, I get the error:
cannot attach ada0p3 to da0p2: no such device in pool

after booting from the usb drive, and performing all the previous commands.

Doing a gpart list gives the usb drive boot partition:

Code:
2. Name: da0p2
   Mediasize: 15720251392 (15G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 544768
   Mode: r1w1e2
   efimedia: HD(2,GPT,b3fe011a-5808-11eb-bd2e-5404a6934ea3,0x428,0x1d48000)
   rawuuid: b3fe011a-5808-11eb-bd2e-5404a6934ea3
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 15720251392
   offset: 544768
   type: freebsd-zfs
   index: 2
   end: 30704679
   start: 1064


I do have the same device names for the usb drive and the ssd as in your post (but use only 1 ssd), but it seems the boot-pool is not created with the da0p2 device, but with a gptid:

Code:
root@truenas[~]# zpool status
  pool: boot-pool
 state: ONLINE
config:

    NAME                                          STATE     READ WRITE CKSUM
    boot-pool                                     ONLINE       0     0     0
      gptid/b3fe011a-5808-11eb-bd2e-5404a6934ea3  ONLINE       0     0     0


and if I try to attach the gptid I get the error:

Code:
root@truenas[~]# zpool attach boot-pool gptid/b3fe011a-5808-11eb-bd2e-5404a6934ea3 ada0p3
cannot attach ada0p3 to gptid/b3fe011a-5808-11eb-bd2e-5404a6934ea3: can only attach to mirrors and top-level disks


So I don't know where to go from here.

Also I'm having serious trouble with usb drives developing errors, I can't imagine anyone running off a usb drive. I've exhausted my collection of usb drives with this exercise...
 
Joined
Jan 16, 2021
Messages
2
I found the issue, it's a buggy error message, but the issue is difference in block size between devices, as discussed here.
 

enricostn

Cadet
Joined
Jan 31, 2021
Messages
3
Hi,
many thanks for this guide!

I'm trying to follow all the steps but with a 32GB USB drive as "USB drive B".

The only commands I changed are the following:

#ssd drive - add swap partition at first available offset - will be ada0p2
gpart add -s 32g -t freebsd-swap -l swap0 ada0

#ssd drive - add boot partition - will be ada0p3
gpart add -s 32g -t freebsd-zfs -l root0 ada0

Also, since I was getting the same error as @Per Wahlstrom , I used the gptid instead of /dev/da0p2.

From what I understand I've been able to reach the last steps without any error, but when I remove the USB drive B and try to boot from the SSD "drive C" I get a black empty screen with a non-blinking cursor.

Any hint? Thanks!
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,741
@enricostn Did you copy the freebsd-boot or EFI partition?
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,741
This step:
Code:
#ssd drive - setup boot partition
# and embed bootstrap code in
# partition scheme metadata
# note: i follow gpart's manpage advice on making the size 472
# so the swap partition will be aligned correctly.
gpart add -b 40 -s 472 -t freebsd-boot ada0
gpart add -b 40 -s 472 -t freebsd-boot ada1

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1


Please post the output of gpart list <drive> for both your new SSD (ada0?) and your existing USB boot device ...
 

enricostn

Cadet
Joined
Jan 31, 2021
Messages
3
Hi @Patrick M. Hausen
yes I executed those commands.

In any case, I discovered that I missed your comment about doing the same from the UI...

I looked for the documentation and I found:

After that I just had to:

1. detach the "USB drive B" from the boot pool
2. reboot removing "USB drive B"
3. manually create the pool for the rest of the SSD disk from the shell and import it from the UI

Everything works as expected, thanks!
 

Fongaboo

Dabbler
Joined
Nov 12, 2013
Messages
30
Looks like it's mostly going straightforward.

When I run...

Code:
zpool attach freenas-boot /dev/da0p2 /dev/ada0p3


..it barks back with...

Code:
cannot open 'freebsd-root': no such pool


I ran
Code:
zpool list
and got...

PXL_20210301_044454686.MP~2.jpg


(As an aside, I named the remaining space on my SSD 'jails' plural, just cuz that's how i had my storage for iocage named on my previous install)

So I re-ran the command but substituted 'freenas-boot' with 'boot-pool' and it seemed to be successful. But just wanted to confirm... is it just that TrueNAS made a name change since you wrote the post or something?
 

Fongaboo

Dabbler
Joined
Nov 12, 2013
Messages
30
P.S. Can I skip the last two commands and keep my USB stick in place as a backup? Or actually will that really slow things down?
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,741
So I re-ran the command but substituted 'freenas-boot' with 'boot-pool' and it seemed to be successful. But just wanted to confirm... is it just that TrueNAS made a name change since you wrote the post or something?
Yes, obviously :wink:
 

shaxs

Cadet
Joined
Mar 7, 2021
Messages
2
Okay I am a total newbie to TrueNAS. I have a 512gb NVME drive (nvd0) I want to partition to run the OS off and a larger partition for other things like virtual machines. I was following this thread with the differences being my USB drive is 32gb and my boot pool is called "boot-pool". I was able to do everything until I got to:

#setup jail pool
zpool create jail /dev/nvd0p4

When I run that command, I get:

Code:
root@truenas[~]# zpool create jail /dev/nvd0p4
invalid vdev specification
use '-f' to override the following errors:
/dev/nvd0p4 is part of exported pool 'jail'


Here is my gpart list for nvd0:


Code:
root@truenas[~]# gpart list nvd0
Geom name: nvd0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 937703047
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: nvd0p1
   Mediasize: 241664 (236K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0
   efimedia: HD(1,GPT,e125ef1b-7f6f-11eb-b95a-a8a1592f6dde,0x28,0x1d8)
   rawuuid: e125ef1b-7f6f-11eb-b95a-a8a1592f6dde
   rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
   label: (null)
   length: 241664
   offset: 20480
   type: freebsd-boot
   index: 1
   end: 511
   start: 40
2. Name: nvd0p2
   Mediasize: 34359738368 (32G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 262144
   Mode: r0w0e0
   efimedia: HD(2,GPT,f84ad6de-7f6f-11eb-b95a-a8a1592f6dde,0x200,0x4000000)
   rawuuid: f84ad6de-7f6f-11eb-b95a-a8a1592f6dde
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: swap0
   length: 34359738368
   offset: 262144
   type: freebsd-swap
   index: 2
   end: 67109375
   start: 512
3. Name: nvd0p3
   Mediasize: 34359738368 (32G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 262144
   Mode: r0w0e0
   efimedia: HD(3,GPT,0328057f-7f70-11eb-b95a-a8a1592f6dde,0x4000200,0x4000000)
   rawuuid: 0328057f-7f70-11eb-b95a-a8a1592f6dde
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: root0
   length: 34359738368
   offset: 34360000512
   type: freebsd-zfs
   index: 3
   end: 134218239
   start: 67109376
4. Name: nvd0p4
   Mediasize: 411384221696 (383G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 262144
   Mode: r0w0e0
   efimedia: HD(4,GPT,0b02d149-7f70-11eb-b95a-a8a1592f6dde,0x8000200,0x2fe43488)
   rawuuid: 0b02d149-7f70-11eb-b95a-a8a1592f6dde
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: jail0
   length: 411384221696
   offset: 68719738880
   type: freebsd-zfs
   index: 4
   end: 937703047
   start: 134218240
Consumers:
1. Name: nvd0
   Mediasize: 480103981056 (447G)
   Sectorsize: 512
   Mode: r0w0e0


Not sure what to do at this point. I was considering just buying a smaller NVME SSD, but I really want to use what I have if possible.
 

shaxs

Cadet
Joined
Mar 7, 2021
Messages
2
Not sure what happened, but I re-did everything over again and it worked. I also go the jail imported and my first plugin installed. Thanks for the great walkthrough!
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,741
Not sure what happened, but I re-did everything over again and it worked. I also go the jail imported and my first plugin installed. Thanks for the great walkthrough!
Glad it worked out.
 

zierbeek

Explorer
Joined
Apr 4, 2021
Messages
54
Hi all, I followed the exact tutorial. How do I know if the partitioning was successful?
I'm using a 32g usb drive and changed the value of 16g to 32g. not for the swap but for the other one
and how do I import the jail ?
 
Last edited:

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,741
gpart show?
 

zierbeek

Explorer
Joined
Apr 4, 2021
Messages
54
I get this in part show
Code:
gpart show ada0
=>       40  468862048  ada0  GPT  (224G)
         40        472     1  freebsd-boot  (236K)
        512   33554432     2  freebsd-swap  (16G)
   33554944   62914560     3  freebsd-zfs  (30G)
   96469504  372392584     4  freebsd-zfs  (178G)
 
Last edited:
Top