Further explanation of Partitions/Swap on disks

[sfcredfox]

Overall Intent
Could anyone provide some additional guidance on the output below and explain if any action should be taken regarding partitions and SWAP? Perhaps even explain if I've done something stupid based on my explanation of how the system was created, and a recommendation to fix it if possible/needed?

My system has two pools:
Pool1 'Tank1' - 3 vdevs (first created with only 1 vdev, first pool created on the system). Each vdev is 5 disks. It is da1-15. I added the other vdevs at two later times and allowed the system to start filling them.

'Tank1' also has a SLOG on da0 (done after creating the pool). I was under the impression that you did not have to remove any original log partition (ZIL) from the original disks. Am I wrong?

Pool2 'Tank2' - 1 vdev of 5 disks. **Edit - This pool is da16-20.** I moved the system data set to this pool after it was created because I wanted it off the first one.

Questions
I am assuming the additional partitions are swap? Or the original ZIL for the pool? Both?

Why did it create partitions on some drives, but not others?

Was this a result (having partitions on some and not others) of the original system dataset being created on the first pool, but then not created on subsequent disks in pool 1 because I moved it to pool 2 before hand?

What are the gptid partitions for?

Partial output of gstat (reference for partitions)
dT: 1.002s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
0 0 0 0 0.0 0 0 0.0 0.0| cd0
0 166 0 0 0.0 166 1940 0.1 1.2| da0
0 0 0 0 0.0 0 0 0.0 0.0| da1
0 1 1 4 4.4 0 0 0.0 0.4| da2
0 1 1 4 6.5 0 0 0.0 0.6| da3
0 0 0 0 0.0 0 0 0.0 0.0| da4
0 166 0 0 0.0 166 1940 0.1 1.3| da0p1
0 0 0 0 0.0 0 0 0.0 0.0| da1p1
0 0 0 0 0.0 0 0 0.0 0.0| da1p2
0 0 0 0 0.0 0 0 0.0 0.0| da2p1
0 1 1 4 4.4 0 0 0.0 0.4| da2p2
0 0 0 0 0.0 0 0 0.0 0.0| da3p1
0 1 1 4 6.5 0 0 0.0 0.7| da3p2
0 0 0 0 0.0 0 0 0.0 0.0| da4p1
0 0 0 0 0.0 0 0 0.0 0.0| da4p2
0 1 1 4 6.6 0 0 0.0 0.7| da5
0 0 0 0 0.0 0 0 0.0 0.0| da6
0 1 1 4 4.7 0 0 0.0 0.5| da7
0 0 0 0 0.0 0 0 0.0 0.0| da8
0 0 0 0 0.0 0 0 0.0 0.0| da9
0 1 1 4 5.4 0 0 0.0 0.5| da10
0 0 0 0 0.0 0 0 0.0 0.0| da11
0 0 0 0 0.0 0 0 0.0 0.0| da12
0 0 0 0 0.0 0 0 0.0 0.0| da13
0 0 0 0 0.0 0 0 0.0 0.0| da14
0 0 0 0 0.0 0 0 0.0 0.0| da15
0 0 0 0 0.0 0 0 0.0 0.0| da16
0 0 0 0 0.0 0 0 0.0 0.0| da17
0 0 0 0 0.0 0 0 0.0 0.0| da18
0 0 0 0 0.0 0 0 0.0 0.0| da19
0 0 0 0 0.0 0 0 0.0 0.0| da20
0 166 0 0 0.0 166 1940 0.1 1.5| gptid/70ef691f-6b8f-11e4-a62f-001b7857b9ca
0 0 0 0 0.0 0 0 0.0 0.0| da2p1.eli
0 0 0 0 0.0 0 0 0.0 0.0| gptid/31dbce50-3af4-11e4-a5b2-001cc467f92a
0 0 0 0 0.0 0 0 0.0 0.0| da3p1.eli
0 1 1 4 4.4 0 0 0.0 0.4| gptid/327ae548-3af4-11e4-a5b2-001cc467f92a
0 0 0 0 0.0 0 0 0.0 0.0| da4p1.eli
0 1 1 4 6.5 0 0 0.0 0.7| gptid/3319098c-3af4-11e4-a5b2-001cc467f92a
0 0 0 0 0.0 0 0 0.0 0.0| da5p1.eli
0 0 0 0 0.0 0 0 0.0

Final Questions
The documentation says that disabling SWAP is strongly discouraged. I don't have any interest in doing so knowing this.

Why would someone want to disable SWAP on FreeNAS (They're stupid is not quite the answer I'm looking for, more of their logic or reasons why)?

All the posts I could find about SWAP were bug reports asking for the ability to turn it off.
Are people wanting to remove it from certain disks, but not others? (Assuming you want it left on because are you not supposed to remove it entirely)?

Are those additional partitions on my disks hurting anything even though tank1 has a separate log and is not holding the system data set?
Should I try to remove them or otherwise care about this at all?

Thanks

 

[dlavigne]

Huh? What additional partitions? Please post a screenshot of Volume Status.

 

[danb35]

When a disk is added to a ZFS pool in FreeNAS and swap hasn't been disabled, two partitions are created. The first partition is swap, by default it's 2 GB. The second partition, comprising the remaining space on the disk, is used for ZFS. Thus, each of your data disks (with the possible exception of L2ARC and SLOG devices; I don't know how those are handled) should have two partitions on it. Your boot device under 9.2.x and before has four partitions.

What I'm seeing in your post looks like two partitions on many disks, which is what's expected.

 

[sfcredfox]

What I'm seeing in your post looks like two partitions on many disks, which is what's expected.

I think I'm understanding this, just not why some of the other disks don't seem to have those addition partitions. At least maybe it's just not listed?

Also, can anyone address the additional questions I had about why people would want the ability to disable swap?

Can you have it be enabled on one pool for the system to function correctly, but turn it off on some drives?

 

[mjws00]

Someone may be able to go deeper, but the basics are this.

Swap is there so that if something goes terribly horribly wrong, FreeNAS can use it. Under normal circumstances it should never be used.

The complication with swap, and the way they have implemented it, is that they have spread swap out over masses of disks. In your case with ~20 disks you have 40GB of swap which is more swap than you have memory. Normally OS's wouldn't do that. However, it's not a big deal as a percentage of most pools. Still meh.

Where it gets a little dicey and where some folks might want to disable it is for hot-swapping drives. You SHOULD be able to pull a hot-swap drive, plug it back in and just online it. But the issue gets complicated, as you are now dumping a swap device... it gets locked open. I'm not sure what happens if there are open memory pages on it... hard crash? Dunno. You can turn off swap on that single device first and things proceed normally. Replacing via the gui with a fresh drive handles things gracefully. But lets say the left side of my chassis is running hot and I want to shuffle a few drives over... Normal zfs (even FreeNAS with no swap), I can just yank it, move it, online it. No fuss. Swap all over the data pool screws that up.

The other thing swap does is allows you to add a drive that may have +- an exact sector count as a replacement. You can decrease swap a little if you throw an error that the device is to0 small while replacing a drive.

The party line is shut your server down to move disks. That is safest. But the whole reason we have hot-swap chassis is so that we can leave them running. Dropping a disk with a full resilver works. But moving a disk or temporarily disconnecting it doesn't work well with the current swap model. If there is logic beyond what I've mentioned I'd love to hear it. But frankly, I'd MUCH rather set swap on one drive for "failsafe" and then leave my pool capable. Plus if I throw .system on there, my archive pool can spin down to a more cold storage mode instead of spinning a dozen disks that could sit for a week or a month.

Anyway. That's what I've seen. I think the mass micro-swap model solved a problem that doesn't exist for many. It seems more likely that it addressed the needs of some with very limited hardware a long time ago. As I said, I'm open to hearing more logic. I would be disappointed if TrueNAS didn't handle hot-swap more elegantly. I'm likely to build the next config with a swap/.system pool, and a data pool without swap to get in the way. To my mind that gives the best of both worlds.

 

[cyberjock]

Yes, a hard crash is the result of dumping a swap device.... if it is in use. But as you said the swap shouldn't be used, so this shouldn't ever be a problem unless there are other problems too. If your system is having that many problems I'd consider a crash as a good thing because you should be giving the server attention because things are VERY wrong.

TrueNAS itself doesn't have swapspace at all. We work with the customers very tightly and issues that someone has that would require swapspace are immediately resolved on our end via direct support via the developers. The hardware and software is tightly controlled with our support contracts, so alot of problems that might develop with "random freenas user's hardware" is significantly less likely with TrueNAS.

I wrote up a rather lengthy comment about the .system dataset, but I just deleted it. If you are silly enough to think that moving it elsewhere is a good idea, you'll find out sooner or later why the idea is crazy.

 

[mjws00]

Can't see why building .system on a different ssd backed pool would be silly, but thanks. Want me to name it 'tank' and build it first? How could that be any different than a standard install. No need to delete stuff, we're all interested in your knowledge.

I do appreciate the confirmation on TrueNAS swap, that was my guess, but I don't own one and hoped you'd pop by.

 

[sfcredfox]

I am interested in those comments as well.

mjws00, thanks for the additional explanation.

I moved my system data set to a smaller, less important pool via the GUI also, but I don't quite see how that relates to SWAP space unless the two coincide and I'm missing that fact. I also get the sense that the .system and the system data set are related? Or not?

I purely wanted any additional duties/system logging/etc off the pool that was used as an iSCSI target an onto a pool that wouldn't be competing for performance. Maybe that's not the same thing as Cyber is talking about?

Based on the risk that a system could go down as a result of a part of the swap being active on the drive that's bad or needing to be replaced, can a user turn swap off on one pool and leave it default behavior on another? In other words, can tell FreeNAS not to use one pool's disks for that function, but let it use the others? And without breaking anything, or loosing general support?

I suspect that the problems people are complaining about when they attempt to offline drives and they won't because it's using the SWAP and something has gone wrong, so the GUI isn't able to release that, at that point, you should know to use a command to release the swap space? I know there are already some posts about that, but I wonder how it's related.

 

[sfcredfox]

Also, on the GSTAT output, can anyone explain what these entries are referring to?

0 166 0 0 0.0 166 1940 0.1 1.5| gptid/70ef691f-6b8f-11e4-a62f-001b7857b9ca
0 0 0 0 0.0 0 0 0.0 0.0| da2p1.eli
0 0 0 0 0.0 0 0 0.0 0.0| gptid/31dbce50-3af4-11e4-a5b2-001cc467f92a
0 0 0 0 0.0 0 0 0.0 0.0| da3p1.eli
0 1 1 4 4.4 0 0 0.0 0.4| gptid/327ae548-3af4-11e4-a5b2-001cc467f92a
0 0 0 0 0.0 0 0 0.0 0.0| da4p1.eli
0 1 1 4 6.5 0 0 0.0 0.7| gptid/3319098c-3af4-11e4-a5b2-001cc467f92a
0 0 0 0 0.0 0 0 0.0 0.0| da5p1.eli

These were different from the others and I can't filter the search results for gptid to get useful stuff for FreeNAS. Also, why are their entires that have .eli at the end?
Does any of this stuff matter? I'm just curious when trying to figure out what's going on with pool performance and looking at how devices are being used and if they are causing bottle necks, issues, or showing I've misconfigured something.

 

[mjws00]

Cyber is used to people trying to jam everything on a USB stick or doing odd things to try and save two pennies of power. The reality is we need .system to be persistent and reliable for samba and logging. I am not interested in under resourcing, but didn't really specify. One thing to consider is that I am FAR more interested in TrueNAS like features and performance. High availability, proper hot-swap, and fault tolerance are far more useful to me than "my sickbeard won't update." In addition I have archive and cold-storage workloads to consider along with performance based loads. So I'm not pretending for a second that my needs are common. Only discussing system potentials and caveats.

Swap is not causing pools to stay alive in most cases. It just hangs out and helps the odd person that needs it. It is a complete non issue, except for the problems of device removal. It also is not impacting performance. It's just empty space available if FreeNAS needs it. All the swap related bugs I could find were with respect to interface issues and misbehaved plugins or software.

Some of these things are just the nature of an appliance, and the evolution of the product over time. With 9.3 all of a sudden the devs have a whole new system pool available to use. They aren't limited to 2GB. But they aren't forcing a transition to DOM, ssd, or hdd boot devices. Give them a known good, reliable (possibly mirrored), and large space and they could easily ditch these things. However, huge masses of people would cry that they liked booting off usb. No installs, and no need for a "system" drive. Catch 22.

Does any of this stuff matter? Probably not. For most just click OK, stick with the defaults, and enjoy. There are some of us crazies that like to dive deeper.

Someone else will have to get into gstat. ;)

 

[cyberjock]

If you want to disable swap, just add a script to your bootup sequence that runs swapoff -a.

Code:

[root@mini] ~# swapoff -a
swapoff: removing /dev/ada3p1.eli as swap device
swapoff: removing /dev/ada1p1.eli as swap device
swapoff: removing /dev/ada2p1.eli as swap device
swapoff: removing /dev/ada4p1.eli as swap device
swapoff: removing /dev/ada5p1.eli as swap device

Not that I recommend this practice as your swap shouldn't be used anyway. If it is being used you should be dealing with *that* issue long before thinking the answer is to disable swap. :p An unused swap doesn't cause problems, so there's no problem to solve by disabling it. But there are plenty of problems that *may* develop if you do disable it. So why do something for no gain that could potentially have downsides? If that's your logic can I 'borrow' $500? I promise to pay you back. That's as good of an example of any with no gain and only potential downsides. ;)

 

[embrais]

I am not trying to resurrect this thread in a bad way. But I have a unit I am currently burning in and testing scenarios before going live.

FreeNAS-9.10-STABLE-201606270534 (dd17351)
2xIntel(R) Xeon(R) CPU E5520 @ 2.27GHz
48GB RAM
2 x120gb SSD boot
2 x120gb SSD ZIL (short stroked/provisioned)
2 x 120gb SSD cache
18 x 1TB Seagate Constellation SAS drives (RAD10)
swap creation disabled on pools

After reading MANY articles about Freenas/SWAP I have tried the following as a best of both worlds... or perhaps a fools comprimise?

create swapfile

dd if=/dev/zero of=/data/.swapfile bs=1024k count=32000
chmod 0600 /data/.swapfile

create a post init startup entry
mdconfig -a -t vnode -f /data/.swapfile -u 0 && swapon /dev/md0

after reboot

Device 512-blocks Used Avail Capacity
/dev/md0 65536000 0 65536000 0%

Since swap is for worst case scenario and the boot volume is mirrored and the swap is a file not a partition, this to me allows for swap just in case without the headaches

Am I missing something?

 

[Stux]

post-init is AFTER mounting the data pools. You may need swap to mount the pools in a crash scenario. In my tests it works pre-init, which I believe is before mounting the pools.

Verify that you have swap using top

 

[Chris Moore]

I wrote up a rather lengthy comment about the .system dataset, but I just deleted it. If you are silly enough to think that moving it elsewhere is a good idea, you'll find out sooner or later why the idea is crazy.

Where would you put the system dataset?

 

[Ericloewe]

That would seem to be old advice, as the GUI supports this now.

I'll also go ahead and lock this thread.