cd'ing to storage in portjail with zsh + oh-my-zsh triggers system hang

Status
Not open for further replies.

mgalgs

Cadet
Joined
Feb 21, 2014
Messages
4
Setup:

  • A ZFS storage pool that was originally created on FreeNAS 8.x but was recently updated to the latest ZFS version after updating to FreeNAS 9.2.1
  • portjail with some storage from my ZFS pool
  • zsh with oh-my-zsh [1]
  • Hardware: ASUS mobo with AMD CPU, 8GB of Patriot DDR3, 2 1TB Seagate HDD. More details at [2].
For some reason, whenever I cd into the directory where my "storage" is mounted in my portjail things become unhappy. With the old version of ZFS the whole system would hang. Nothing in my kernel log, nothing on screen. The only recovery was a hard power cycle. After updating my ZFS version the system stays up but that shared storage directory gets borked. I can still ssh in as root to the main system and can ssh to my portjail, just as long as I stay out of that directory. So I am able to (somewhat) reboot the machine (I do have to help it along with a Ctl-alt-delete; it never finishes rebooting on its own).

Other things I've tested:

  • memtest86: ran all night, no failures
  • Using bash to cd into the "problem" area: no hang
  • Using zsh *without* oh-my-zsh enabled to cd into the "problem" area: no hang
  • On a different machine (with zsh+oh-my-zsh) cd to the "problem" area over NFS: no hang
What I haven't done yet:

  • Narrowed down exactly what customization oh-my-zsh has enabled that triggers the hang. That would be useful for debugging but this certainly isn't an oh-my-zsh bug (shell customizations should definitely not be able to take a system offline...)
Any ideas what's going on? Can anyone else reproduce this? Here are the rough repro steps:

  1. Create a portjail
  2. Add some storage from a ZFS data set to your portjail
  3. pkg install zsh
  4. Install oh-my-zsh [3]
  5. cd /media/yourstorage
[1] https://github.com/robbyrussell/oh-my-zsh
[2] http://mgalgs.github.io/2012/04/24/freenas-hardware-build.html
[3] https://github.com/robbyrussell/oh-my-zsh#the-automatic-installer-do-you-trust-me
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I'm not going to delete your thread or anything, but this forum is for FreeNAS support. That being said, you're more likely to get an answer at that programs support forums/IRC than here. We don't as a rule support the stuff inside the jail just as VMWare doesn't support the OSes that run in their VM framework. You *might* find the answer here. But I'm doubtful at the moment. Especially since there's literally something like 20000 packages in the pkg repository and I've never heard of that one. Even the FreeBSD forums *may* be better.
 

mgalgs

Cadet
Joined
Feb 21, 2014
Messages
4
Hmm, not sure how you concluded that this isn't a FreeNAS problem (or at least a ZFS problem). I haven't seen anything like this on any of the different platforms I use zsh+oh-my-zsh on, and I also didn't find any similar reports on the Internet. And it's not due to lack of usage of this software. zsh is downloaded about 6,000 times per month and the oh-my-zsh project has +14,000 stars on GitHub (somewhere around the top 20 most popular project on github), so it's not like these are some obscure pieces of software (even if "you haven't heard of them" ;) ).

Having said all that, I haven't found the specific use case that triggers this error, so unless someone has seen the same problem I realize it's going to be hard to debug with the information I've given. I'll try to narrow down exactly what is causing the hang and will report back.

In the meantime, if anyone feels like trying to reproduce it that would be helpful in confirming the existence of a bug somewhere.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Hmm, not sure how you concluded that this isn't a FreeNAS problem (or at least a ZFS problem)

I never concluded anything. All I was saying is that generally we don't really care about the stuff going on inside the jail. If you look at some of our bug tickets many of them are closed to 3rd party to resolve. FreeNAS provides the jail framework, and that's about it. Many programs don't work right with ZFS. Read up on df and du on ZFS. They aren't quite 100% because of how ZFS works. And anyone who uses FreeBSD or even Linux knows about those 2 commands.

Generally, if you can prove it the framework that FreeNAS provides, the ticket will be handled accordingly. But... many tickets put in either are proven to not be the jail framework or are at least not proven to be a problem for us.

Ex1: One user puts in a ticket because he gets weird errors from one application when he writes very large files. Turns out of he does a dd write inside his jail it stops writing at 4GB. Not our problem.. ticket closed to 3rd party to resolve. What was weird about this was that I created the same jail with the same version and I created a 10GB file with zero problems. We don't even know why he couldn't write 4GB but I could!

Ex2: User puts in a ticket because some program he tries to compile inside his jail is using some wacky calculation to determine freespace and his getting a total free space of something like 120KB. The compiler crashes with an insufficient disk space message. It's clear the program doesn't run with the way ZFS handles free space, but again, that's not *our* problem to solve. The problem was that his program wasn't using the proper method for determining free space and as a result the program wouldn't compile. Ticket was closed to 3rd part to resolve. It's our fault because we use ZFS. But it wasn't our problem to fix either. ;) (This was similar in why df/du don't quite work with ZFS).

I haven't seen anything like this on any of the different platforms I use zsh+oh-my-zsh on, and I also didn't find any similar reports on the Internet. And it's not due to lack of usage of this software. zsh is downloaded about 6,000 times per month and the oh-my-zsh project has +14,000 stars on GitHub (somewhere around the top 20 most popular project on github), so it's not like these are some obscure pieces of software (even if "you haven't heard of them" ;) ).

Eh, those numbers aren't "blow my hair back". I've seen small obscure programs have numbers higher than that. 6000 downloads a month is just 50000 a year. That's not very much in a world with more than 3 billion computers. But it really doesn't matter how obscure it is or isn't. That doesn't determine how well a program works or doesn't work.

Edit: One thing to think about is this.. if there's only 50000 downloads a year, what's the chances that one of those 50000 in a year are likely to install it in a FreeNAS jail. You're looking for a very specific situation, and so anything that isn't a very very large project likely has a very small subset of users meeting these conditions. It's the sad truth. So I'm not really surprised you aren't finding much information on your specific issue.

Having said all that, I haven't found the specific use case that triggers this error, so unless someone has seen the same problem I realize it's going to be hard to debug with the information I've given. I'll try to narrow down exactly what is causing the hang and will report back.
A case that triggers the error would be useful as it might point a finger at what is going wrong.
 

mgalgs

Cadet
Joined
Feb 21, 2014
Messages
4
Well I switched my shell to bash a week ago (no longer using zsh or oh-my-zsh) and have still been seeing these hangs... It seems to happen when I access a "Storage" dataset that I created following the instructions in the docs [1]. I still haven't figured out what the usage pattern is that triggers the hang... Is this feature ("Storage" dataset in a port jail) commonly used? Any known issues? As long as I stay out of the shared dataset everything is stable... I also updated to 9.2.1.2 but I still see the hangs.

[1] http://doc.freenas.org/index.php/Adding_Jails#Adding_Storage
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Most everyone adds storage. That's the ONLY way to get data from outside the jail to be available inside the jail. So there's tons of people using it without your problems(including myself), and you are the only one currently complaining about it. So either you're using the system in a way that is unique(but should be valid) and you're the only one that's found the problem, you have some kind of a hardware problem, you have user error(honestly.. this is what I think is most likely), or you are using the jails in a way that has been deemed "out of bounds" or whatever you want to call it for FreeNAS.

The reasons I say user error is most likely is because 95% of the time it is user error. ;)
 

mgalgs

Cadet
Joined
Feb 21, 2014
Messages
4
Fair enough I guess. I'll update this thread if I figure anything out.
 
Status
Not open for further replies.
Top