SOLVED sockstat Not Working

[nello]

About three days ago CrashPlan on my server became inaccessible to all clients.

I tried to see verify that it is listening on the right ports, I and get an error regarding the sockstat command

Code:

crashPlan@crashplan_1:~ % sockstat -l | grep java
sockstat: struct xtcpcb size mismatch
crashPlan@crashplan_1:~ % 

I get the error struct xtcpcb size mismatch regardless of the parameters that I use, for example:

Code:

crashPlan@crashplan_1:~ % sockstat
sockstat: struct xtcpcb size mismatch
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS     
crashPlan sshd      12381 4  stream -> ??
root     sshd       12361 6  stream -> ??
root     cron       5046  4  dgram  -> /var/run/logpriv
root     syslogd    4913  4  dgram  /var/run/log
root     syslogd    4913  5  dgram  /var/run/logpriv
root     syslogd    4913  6  udp6   *:514                 *:*
root     syslogd    4913  7  udp4   *:514                 *:*

Can anyone give me a clue what the error struct xtcpcb size mismatch is trying to tell me?

Thank you.

- nello

 

[BigDave]

To the House Elf in Chicago:
Dear Mr. nello,
Listen here Old Boy, you've been a part of this forum for too long to go posting in the "New to FreeNAS" section eh? :p

 

[dlavigne]

That looks like an ABI error. Did you install crashplan using Plugins or Jails? What is the exact build version for FreeNAS (from System -> Information)?

 

[nello]

Did you install crashplan using Plugins or Jails?

I installed the plugin about 1-2 years ago and I've been fiddling with it ever since. In late May 2016, I upgraded CrashPlan to version 4.7 using these instructions:

https://forums.freenas.org/index.php?threads/crashplan-ver-4-7.43190/#post-283648

What is the exact build version for FreeNAS (from System -> Information)?

FreeNAS-9.10-STABLE-201606270534 (dd17351)


You didn't ask, but ...

When I upgraded from FreeNAS 9.3 to 9.10 everything seemed to work just fine; I don't recall any irregularities. The only difference I notice is that HandBreakCLI takes about twice as long to transcode .mpg to mp4.

Thank you for your reply.

- nello

 

[DrKK]

Sir,

If the jail is based on a FreeBSD 9 (i.e., FreeNAS 9.3 etc) template, and you have recently gone to a FreeBSD 10 (i.e., FreeNAS 9.10) kernel, then, you are running in a backwards compatibility mode, and the "sockstat" that is in your jail now is "wrong" vis-a-vis the 10-based network stack.

As a result, one common that happens (I know, because I had the same problem) is that when you use sockstat, many of your sockets don't show up.

Go ahead and copy /usr/bin/sockstat from the *HOST* into the /usr/bin/sockstat on the jail, and it will work fine now ;) i.e., you have to use the sockstat for FreeBSD 10 in your jail, since the network is still a 10 service, even if your jail is 9.

Tricky, isn't it, with running antique jails in new FreeBSD's. I discovered this the hard way, and with an assist from Allan Jude.

 

[DrKK]

It's funny, because I was going to make a post on this, and then I was like, "nah, no one will ever notice this." And then sure enough, here's a guy...

 

[nello]

If the jail is based on a FreeBSD 9 (i.e., FreeNAS 9.3 etc) template, and you have recently gone to a FreeBSD 10 (i.e., FreeNAS 9.10) kernel ....

Yes, this is the situation. It's a CrashPlan plugin Jail running on FreeNAS 9.10.

Thank you for the enlightenment.

- nello

 

[DrKK]

Yes, this is the situation. It's a CrashPlan plugin Jail running on FreeNAS 9.10.

Thank you for the enlightenment.

- nello

Drop to shell in the FreeNAS appliance and do this:

cp /usr/bin/sockstat /mnt/{name-of-jail-zpool}/{name-of-jail-dataset}/usr/bin

Then it will ask if you want to overwrite, say yes.

Then you should be able to use sockstat in your deprecated jail.

 

[nello]

cp /usr/bin/sockstat /mnt/{name-of-jail-zpool}/{name-of-jail-dataset}/usr/bin

Thank you. I was dreading trying to figure out the paths.

 

[DrKK]

Of course, for the record, the correct procedure is to kill the old jails and make new ones, which use the FreeBSD 10 templates, and then wipe out the old templates and jails. I slowly did that over the course of about a week. Just finished the last one on Sunday.

 

[nello]

Of course, for the record, the correct procedure is to kill the old jails and make new ones, which use the FreeBSD 10 templates, and then wipe out the old templates and jails.

I've read things to this effect. But, I don't recall explicit instructions on how to activate FreeBSD 10 templates. In other words, what is the command such that all future Jails are based on the FreeBSD 10 template(s)? And, does changing the template affect existing (FreeNAS 9.x) Jails?

 

[DrKK]

I've read things to this effect. But, I don't recall explicit instructions on how to activate FreeBSD 10 templates. In other words, what is the command such that all future Jails are based on the FreeBSD 10 template(s)? And, does changing the template affect existing (FreeNAS 9.x) Jails?

First of all: anyone that chooses not to update their jails will still find their jails..."work"....But generally speaking, you will be frozen in time (no upgrades to your jail's software), and, other sundry problems (or at least, annoyances) that we are talking about in this thread.


So basically how this works:

1. If you attempt to create a new jail after upgrading to 9.10, then any subsequent jails you create will be on the 10-template; the first jail you create will download the template. This is without regard to whether or not 9 jails are, or are not, present.
2. You don't have a choice in the matter; this is how it works.
   
3. Any existing jails that you have on the 9-Template will continue to operate as is, even though the kernel under which they are running is now 10
4. As was said above, this mismatch means that there will be certain wonkiness, particular with respect to the programs you might be used to in /usr/bin, which was the reason for this thread in the first place; I don't know about anyone else, but the 9- vs 10- mismatch state with the jails, even if it "works", still leaves me without the confidence that I've come to expect about my FreeNAS/FreeBSD stuff.
   
5. Those that do not upgrade their jails to the 10 template will find it is impossible/nearly impossible to upgrade the software their jails run through packages or ports, so that your jail is frozen in time, running on a respirator.

So, it leads to this state where you constantly *want* to be upgrading your jails, and with each day that goes by, your software is falling out of date in the jails. So it's just mounting pulse rate, and it's inevitable anyway, so you just say: Gah. Screw it.

So how I upgrade (I do not claim this is max efficiency or max elegance):

1. I do *NOT* delete the old jail, but I *do* stop it, and I do turn off "autostart" for it.
   
2. I create the new jail on the new template simply by "add jail" in the JAIL screen.
   
3. I open an ssh session to the APPLIANCE ITSELF, so I can navigate to any directories I need to, including all the jail directories; for now I leave that on the side.
   
4. I configure ssh in the new jail shell from the GUI, then log in to the new jail
5. I install, in the new jail, the software packages that are relevant.
   
6. If I need to reference, or copy over, configuration files, logs, and other material from the OLD jail, I do so. This is very important. Everyone has the software in their jails "just so" (if you're not using the plugins, anyway).
7. I thoroughly test the new jail, make sure nothing is screwed up
8. I LEAVE THE OLD JAIL AROUND, but turned off, for several days, JUST IN CASE I have missed something
9. If I had port forwarding or other IP address specific services I was running, then I turn off the new jail, change the IP address of the old jail to something much higher so that there will be no confusion, then change the IP address of the new jail to what the old jail was. Then I turn the new jail back on. This means any outside ingresses to the old jail are now going to the new jail, without having to reconfigure the router or anything else.
   
10. After about a week of everything presumably working correctly, and I no longer need the old jail after meditating on it for the whole time, then, I go into the GUI, and I blow away the old jail. By the way (and yes, I've filed the bug report) any periodic snapshots for the old jail will not be automatically removed, you'll have to do that yourself in the STORAGE->PERIODIC SNAPSHOT TASKS. If you have more old jails to upgrade, return to step #1.
    
11. Once this procedure has been done to every single one of the old jails, then, the OLD JAIL TEMPLATES ARE NO LONGER NEEDED, and they can take up some space (in my case, I had lost about 5GB in old jail templates that I will never use again). Those datasets for the old templates (so, all of the template datasets except the most recent one) can be blown away in the GUI->STORAGE screen. There are a variety of ways to see which templates go with which active jails, but, if you've moved all of your 9 jails to 10 jails in a FreeNAS, then, it should be a simple matter: the inexperienced user can simply navigate to his jail "root" directory (shown in the jails->configuration screen in the GUI), so, an "ls -al" and look at the dates on the template datasets. One of those will obviously be your new template. The other ones are probably old---a year or more. Those are the ones whose datasets you blow away.
12. BUT BEFORE YOU DO THAT you will want to remove PRECISELY the no-longer used templates from the "warden". To see the exact names of templates the warden knows about, do a "warden template list". Now that you've identified in step #11 which ones have to be smoked, smoke them (very carefully) with the "warden template delete" command.
13. Then, go back to the GUI, go to the STORAGE menu, find the datasets that went with the templates you just smoked, and now, delete the dataset(s) for them. This will finally reclaim the space that was wasted on the old jail templates.

And, profit. I hope that was helpful.