crashplan update failing due to old java version

[drinny]

I noticed today that my crashplan jail using inbound bandwidth (which struck me as odd as I wasn't doing any restores) so I went to investigate. I was unable to connect to my headless crashplan service on my client which has happened in the past when crashplan decides to update itself. Looking into it further I noticed quite a few new directories from today in "/usr/pbi/crashplan-amd64/share/crashplan/upgrade".

Over the span of several hours. Already crashplan is updating... but failing. Looking at logs in "/usr/pbi/crashplan-amd64/share/crashplan/log":

I see lots of files with the following:

Fri Mar 25 14:47:42 CDT 2016 : Sourcing ../../install.vars...
======================================================
Fri Mar 25 14:47:42 CDT 2016 : Current CrashPlan Backup Engine:
Fri Mar 25 14:47:42 CDT 2016 : Stopping using ../../bin/CrashPlanEngine...
Stopping CrashPlan Engine ... OK
Fri Mar 25 14:47:52 CDT 2016 : Ensuring the UpgradeUI is not running.
Fri Mar 25 14:47:52 CDT 2016 : UpgradeUI is shut down.
Fri Mar 25 14:47:52 CDT 2016 : JAVACOMMON is set: /usr/pbi/crashplan-amd64/linux-sun-jre1.7.0/bin/java
Fri Mar 25 14:47:52 CDT 2016 : Current Java Version: 1.7
Fri Mar 25 14:47:52 CDT 2016: The Current java is not compatible. Embedding a compatible version.
Fri Mar 25 14:47:52 CDT 2016 : Download JVM from http://download.code42.com/installs/proserver/jre/jre-7-linux-i586.tgz
Fri Mar 25 14:47:52 CDT 2016 : downloading the JRE using /usr/local/bin/wget
--2016-03-25 14:47:52-- http://download.code42.com/installs/proserver/jre/jre-7-linux-i586.tgz
Resolving download.code42.com (download.code42.com)... 216.17.8.19
Connecting to download.code42.com (download.code42.com)|216.17.8.19|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://download.code42.com/installs/proserver/jre/jre-7-linux-i586.tgz [following]
--2016-03-25 14:47:52-- https://download.code42.com/installs/proserver/jre/jre-7-linux-i586.tgz
Connecting to download.code42.com (download.code42.com)|216.17.8.19|:443... connected.
ERROR: cannot verify download.code42.com's certificate, issued by '/C=US/O=thawte, Inc./CN=thawte SSL CA - G2':
Unable to locally verify the issuer's authority.
To connect to download.code42.com insecurely, use `--no-check-certificate'.
Fri Mar 25 14:47:52 CDT 2016 : Unable to download JRE from http://download.code42.com/installs/proserver/jre/jre-7-linux-i586.tgz; please check network connection
Fri Mar 25 14:47:52 CDT 2016 : Starting using ../../bin/CrashPlanEngine...
Starting CrashPlan Engine ... Using standard startup
OK

So it won't continue because java is out of date. I looked for what script started this upgrade process in hopes to add "--no-check-certificate" to the wget command, but didn't have much luck. I tried downloading the .tgz myself, unpacking it at "/usr/pbi/crashplan-amd64", and then renaming the directory to match the old one. But doing so, I get:

/usr/pbi/crashplan-amd64/linux-sun-jre1.7.0/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory

When trying to startup crashplan with the new java package in place. libjli.so seems to be in the same path to where it was before, so I'm not sure why it says that there is "no such file or directory".

Does anyone have any clues on what to do next?

 

[drinny]

Nevermind. I got it working again. I had to just manually define the ld_library_path to include the location of libjli.so. I also modified the update script to ignore the certificate check on the download. After those two fixes, the upgrade was able to proceed fine and crashplan is now updated.

 

[jchong]

Hi drinny, I'm having the exact same problem. Could you provide more details on how you fixed this? I'm not sure where to define the ld_library_path


Thanks

 

[drinny]

You need to set the LD_LIBRARY_PATH environment variable to the new location of your java installation. Probably prior to doing that, you could add "--no-check-certificate" to your upgrade.sh script (after the automatic process fails and during the hour before it retries). After it downloads the proper java environment, you need to set your LD_LIBRARY_PATH environment variable to that particular installation (or an updated java installation - java 8 works as well). With those two things done, the plugin should likely work.

You'll need to be somewhat comfortable in the CLI to do this. If the above is totally greek to you, I probably wouldn't attempt it. As much of a change as this was to the default plugin environment and given that I upgraded to FreeNAS 9.10 a week or so prior, I just decided to build a new jail from scratch and manually install crashplan into that jail instance instead. That way, I could have a jail running off of a 9.10 (FreeBSD 10.X) binaries and not have to tip toe around the whole /usr/pbi plugin structure that I don't really know much about.

Maybe someone with more plugin knowledge than I could update the plugin to reflect the changes in Crashplan 4.6.0. The default plugin installs Crashplan version 3.6.3, it looks like which is pretty ancient at this point. The whole plugin could probably use a refresh.

 

[Eric Blau]

I'm in the same boat. I installed CrashPlan 4.5.0 which I had prior to the 9.3 -> 9.10 FreeNAS upgrade. I'm trying to get CrashPlan running again now that I blew away my jails.

I was following the instructions here:

https://forums.freenas.org/index.ph...home-computers-to-crashplan-on-freenas.39265/

I hit a couple snags:

The Linux kernel module was not being loaded by FreeNAS even after installing the CrashPlan plugin and rebooting so I was getting ELF binary errors trying to start the JRE bundled with CrashPlan. I fixed that by manually doing a "kldload linux". Java then runs ok and CrashPlan starts.

I then hit an issue with having only an IPv6 nameserver configured in /etc/resolv.conf. I fixed that by commenting out the IPv6 nameserver and adding an IPv4 address.

CrashPlan then started and updated itself to version 4.6.0. Now that it did that, it no longer starts, giving the same error that you mention:

/usr/pbi/crashplan-amd64/linux-sun-jre1.7.0/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory

If I set LD_LIBRARY_PATH to add /usr/pbi/crashplan-amd64/share/crashplan/jre/lib/i386/jli, java at least runs from the command line, but CrashPlan still does not start.

Is there anything else you had to do? Where did you set LD_LIBRARY_PATH so that it will be sourced by the CrashPlan start script? Is that what you set it to?

Thanks in advance for any help!

 

[Eric Blau]

I see you installed from a fresh jail instead of messing with the plugin. Any tips or pointers on how to do that? I'm comfortable with the command line and had CrashPlan 4.5.0 working with the previous FreeNAS 9.3 plugin template, but I'm having problems now that I've upgraded to 9.10. Simple things like doing a "pkg install" are broken.

 

[drinny]

Check your install.vars in /usr/pbi/crashplan-amd64/share/crashplan and see what JAVACOMMON is set to. Make sure that's set to the new installation of java.

If so and you have the LD_LIBRARY_PATH set, I think it should work? Especially if you're able to get java to run from the CLI without the error.

Any hints in /var/log/engine_error.log and/or /var/log/engine_output.log as to what might be getting hung up?

 

[Eric Blau]

Thanks. I got it working.

I double-checked all my steps and found that I forgot to rename jre to linux-sun-jre1.7.0 under /usr/pbi/crashplan-amd64/share/crashplan after the CrashPlan upgrade.

I then put:

export LD_LIBRARY_PATH=/usr/pbi/crashplan-amd64/share/crashplan/linux-sun-jre1.7.0/lib/i386/jli

right near the top of the CrashPlanEngine script. That allowed CrashPlan to start successfully.

It looks like everything is running fine now. Now I just need to clean things up, making sure the "linux" module is loaded on FreeNAS reboot and making sure the IPv4 nameserver settings persist.

Thanks for the pointers and help!

 

[drinny]

I see you installed from a fresh jail instead of messing with the plugin. Any tips or pointers on how to do that? I'm comfortable with the command line and had CrashPlan 4.5.0 working with the previous FreeNAS 9.3 plugin template, but I'm having problems now that I've upgraded to 9.10. Simple things like doing a "pkg install" are broken.

Once I knew all of the above about the LD path and what not being broken, setting up the jail wasn't too much harder. I should have taken better notes. :) But off the top of my head:

• I made a new 9.10 jail (there are a few things that may need to be adjusted here that are global to 9.10 jails at the moment - /usr/local/etc/pkg/repos/FreeBSD.conf still points to 9 instead of 10 by default and I've been having a bug where new jails that I create get assigned the same MAC address as existing jails so that needs to be manually fixed).
• Installed the port from /usr/ports/sysutils/linux-crashplan. The java binaries you'll have to download manually. I think it uses java 7 as its dependency, but you can install java 8 and point it to that location and it seems to work just fine.
• I had to change my .ui_info file to be immutable from the host OS because it kept getting overwritten every time crashplan would start and not let me connect to it headless.
• I also had to modify my.service.xml to allow connections from my LAN (I'm lazy and didn't want to have to ssh tunnel to my server every time I wanted to connect to my headless server).
• Added storage to the new jail in the same places where it was mounted in the plugin.
• Adopted the new computer to replace my old one once crashplan was running so I didn't have to start from scratch.

I think that's roughly about what it entailed? Again, I didn't take very good notes and just sort of fixed things as I went. But that's the jist of it.

Also, my new jail isn't creating engine_error.log nor engine_output.log on startup. I haven't bothered to look into that one since crashplan is working, though.

 

[Eric Blau]

Once I knew all of the above about the LD path and what not being broken, setting up the jail wasn't too much harder. I should have taken better notes. :) But off the top of my head:

• I made a new 9.10 jail (there are a few things that may need to be adjusted here that are global to 9.10 jails at the moment - /usr/local/etc/pkg/repos/FreeBSD.conf still points to 9 instead of 10 by default and I've been having a bug where new jails that I create get assigned the same MAC address as existing jails so that needs to be manually fixed).
• Installed the port from /usr/ports/sysutils/linux-crashplan. The java binaries you'll have to download manually. I think it uses java 7 as its dependency, but you can install java 8 and point it to that location and it seems to work just fine.
• I had to change my .ui_info file to be immutable from the host OS because it kept getting overwritten every time crashplan would start and not let me connect to it headless.
• I also had to modify my.service.xml to allow connections from my LAN (I'm lazy and didn't want to have to ssh tunnel to my server every time I wanted to connect to my headless server).
• Added storage to the new jail in the same places where it was mounted in the plugin.
• Adopted the new computer to replace my old one once crashplan was running so I didn't have to start from scratch.

I think that's roughly about what it entailed? Again, I didn't take very good notes and just sort of fixed things as I went. But that's the jist of it.

Also, my new jail isn't creating engine_error.log nor engine_output.log on startup. I haven't bothered to look into that one since crashplan is working, though.

Thanks, those are roughly the steps I followed as well, but I used the CrashPlan 3.6.3 plugin as a starting point with a new FreeNAS 9.10 / FreeBSD 10 jail by recreating all of my jails.

I figured out the repository issue by looking at the FreeNAS bug tracker but that caused me some trouble at first. I'm a FreeBSD newbie so I stumble when I hit FreeBSD-specific issues like this pkg repository problem.

I'll have to keep an eye on .ui_info changing. Are you talking about in the jail or on the server where you're running CrashPlanDesktop? I already protected .ui_info on the Linux machine where I'm running CrashPlanDesktop so I think I'm good. I noticed it changed in the jail once I adopted my old computer. I'm hoping that is a one-time thing.

I did the same thing you did with my.service.xml, setting serviceHost to 0.0.0.0 so that CrashPlanDesktop can connect directly to it.

The adoption seems to have worked fine. It is scanning all of my files now and synchronizing file and block information. I'm crossing my fingers. Thanks again!

 

[drinny]

Thanks, those are roughly the steps I followed as well, but I used the CrashPlan 3.6.3 plugin as a starting point with a new FreeNAS 9.10 / FreeBSD 10 jail by recreating all of my jails.

I figured out the repository issue by looking at the FreeNAS bug tracker but that caused me some trouble at first. I'm a FreeBSD newbie so I stumble when I hit FreeBSD-specific issues like this pkg repository problem.

I'll have to keep an eye on .ui_info changing. Are you talking about in the jail or on the server where you're running CrashPlanDesktop? I already protected .ui_info on the Linux machine where I'm running CrashPlanDesktop so I think I'm good. I noticed it changed in the jail once I adopted my old computer. I'm hoping that is a one-time thing.

I did the same thing you did with my.service.xml, setting serviceHost to 0.0.0.0 so that CrashPlanDesktop can connect directly to it.

The adoption seems to have worked fine. It is scanning all of my files now and synchronizing file and block information. I'm crossing my fingers. Thanks again!

Yeah, I've been riding that Crashplan 3.6.3 plugin for a few years now. But with the recent changes in 4.6.0 and the further that I felt like I was getting from the 3.6.3 plugin, I just chose to do a straight up jail from scratch. I felt I had a little more flexibility that way instead of trying to cram in my hacks into a plugin architecture that I wasn't as comfortable with as a generic jail. Future changes would allow me to take advantage of that flexibility as well.

The .ui_info file that kept changing on my was on the server side. It didn't seem to change in the 3.6.3 plugin jail after the initial upgrade to a new version of crashplan, but for whatever reason in the my version from scratch it seemed to change on every restart of the service. No matter. Once it was immutable, there was no way crashplan was changing it. :)

My adoption went smoothly as well. It took a while to synchronize everything, but after that it was smooth sailing without any need to re-upload anything. It was pretty slick.

 

[jchong]

drinny - thanks for the tip, I was able to fix this by adding the LD_LIBRARY_PATH to the startup script (after updating java)

 

[Eric Blau]

Yeah, I've been riding that Crashplan 3.6.3 plugin for a few years now. But with the recent changes in 4.6.0 and the further that I felt like I was getting from the 3.6.3 plugin, I just chose to do a straight up jail from scratch. I felt I had a little more flexibility that way instead of trying to cram in my hacks into a plugin architecture that I wasn't as comfortable with as a generic jail. Future changes would allow me to take advantage of that flexibility as well.

The .ui_info file that kept changing on my was on the server side. It didn't seem to change in the 3.6.3 plugin jail after the initial upgrade to a new version of crashplan, but for whatever reason in the my version from scratch it seemed to change on every restart of the service. No matter. Once it was immutable, there was no way crashplan was changing it. :)

My adoption went smoothly as well. It took a while to synchronize everything, but after that it was smooth sailing without any need to re-upload anything. It was pretty slick.

Did you create your own jail through the FreeNAS WebUI? I'm looking into doing the same (creating my own jail) but I ran into a couple issues with mounting the Linux proc filesystem under /compat/linux/proc and the bash file descriptor device in /dev/fd.

Jails created through FreeNAS don't seem to have permissions to mount anything.

 

[drinny]

Did you create your own jail through the FreeNAS WebUI? I'm looking into doing the same (creating my own jail) but I ran into a couple issues with mounting the Linux proc filesystem under /compat/linux/proc and the bash file descriptor device in /dev/fd.

Jails created through FreeNAS don't seem to have permissions to mount anything.

I did. I ran into the same problem, but just kept forging on. Crashplan seems to work OK without having those two mounted, so I never bothered to look further into getting those mounted.

There may be a sysctl that can bypass those restrictions? I know that there were several options under security.jails.*, I believe?

 

[Steve Theodorou]

Nevermind. I got it working again. I had to just manually define the ld_library_path to include the location of libjli.so. I also modified the update script to ignore the certificate check on the download. After those two fixes, the upgrade was able to proceed fine and crashplan is now updated.

Which file exactly did you have to modify to skip the certificate check?

 

[drinny]

Which file exactly did you have to modify to skip the certificate check?

I'm pretty sure I modified the "upgrade.sh" script itself inside the temporary upgrade directory. Or it might have been "update.sh"? One of those two...

 

[subzer011]

i found the upgrade.sh file but i don't see where to put `--no-check-certificate' or where exactly to place "export LD_LIBRARY_PATH=/usr/pbi/crashplan-amd64/share/crashplan/linux-sun-jre1.7.0/lib/i386/jli"

can someone pls provide step by step instructions how to fix? i'm getting same upgrade errors in the log as OP.

 

[Talz]

I'm stuck here too. Same errors as OP, lost at same place as above.

It seems it generates a new temporary upgrade directory each time, so my changes to the upgrade.sh don't get used.

 

[Talz]

Reapplying the manual update fix from https://forums.freenas.org/index.php?threads/crashplan-not-updating.40374/#post-254182
fixed it for me (again). I thought since I was already on 4.6.0 this magic was used up, but no.

 

[subzer011]

Reapplying the manual update fix from https://forums.freenas.org/index.php?threads/crashplan-not-updating.40374/#post-254182
fixed it for me (again). I thought since I was already on 4.6.0 this magic was used up, but no.

thank you!! this worked for me too.