Another [Errno 28] No space left on device: '/tmp/*'

Status
Not open for further replies.

vitaprimo

Dabbler
Joined
Jun 28, 2018
Messages
27
Hi,

I'm new around here and FreeNAS newbie too; last year I gave it a quick test drive but I was learning so many things at the time I just didn't gave it the time or prior research: I wanted to virtualize it.

This time around I have a sound system that works completely off network storage and I realized I have several disk bays that are unused so I weighted my options I thought FreeNAS was the best idea. So last week I went ahead and set it up, virtualized, of course, on the VMware platform, it's got a mirrored [virtualized] set of boot drives, each on a different server, one thick provision, one thin provisioned (I don't know why I did this maybe I didn't notice, I don't remember) and on physical disk, it's an older disk as I am sort of putting it through its paces before going in deep, the next day I ordered new drives though, they should be here today or tomorrow; the storage bus, I was careful to have the least virtualization possible down the BIOS, I actually went into the out-of-band controller and changed the settings twice so they were no lingering unapplied settings and restarted the server so the changes took effect correctly. On vCenter, I gave the storage bus to the VM, like I mentioned earlier, the server works without its own storage, boot off a flash drive just as FreeNAS, although it could boot from the network as well but I haven't got around to that just yet.

It seems every time I'm set into a new topic a million more appear and pull my attention. ADHD aside, before giving a shot at posting I searched a little for cases like mine and it turns out there are tons, hence the title, but from the few I went over they don't exactly match.

From the message appearing on the web interface I'm running out of space, from the string it shows I know it's a disk location, not memory, and everything else works normal! I can't stress that enough. The WebDAV shares work, the SMB shares work, the NFS shares work, Netdata is working and not skipping a beat. I opened VMware Fusion on a Mac to connect to FreeNAS' virtualized console as I forgot to set up SSH and ran zpool list, a command I found mentioned on a similar case and the boot drive it's 1/8 of its capacity, and being virtualized, if it were full I can always expand it, I guess, the storage drive, a 3TB used guinea disk had about half a terabyte used from a three or four disposable VMs I threw in there for testing, so it's also nowhere near capacity.

This information matches the one presented by vCenter, since one of the boot drives is thin provisioned, I can actually see how much data there is in there, and the storage drive is mounted as an NFS share and again correctly shows me how much there is in total, available and occupied...along with a million other metrics.

I'm not a developer, nor know a lot about this but that would only leave the web engine out, don't it? Once more, everything else, well, as little as I've managed to set up; somehow I couldn't make the plugins work, Active Directory gave me a little fight too but once I requested a Kerberos ticket from the DC in the web interface's little terminal window it got down to business and not once have I had an authentication error after that, no matter from where I log in.

Except for this little mishap, I'm very impressed with FreeNAS overall. Sorry for repeating so much but I want to be clear; everything aside from the web interface--the few things I set up are working fine, I already took out the VMs from FreeNAS preparing to restart but I haven't restarted yet because I have a feeling the error will go away which might be good but I probably have a set of disks in a van coming my way already and I rather deal with it now before going nuts when I actually have something to lose.

I did my homework before installing, I went through all the warnings related to virtualization, my checklist: physical disks along all PCI storage bus is in complete control of the FreeNAS VM, it even took the built-in NICs of the server but they are 1gig card so they aren't assigned in favor of the 10gig NICs, the computer is en enterprise grade server, actually, it is a rack mount Cisco server, on the memory side it's got only the minimum recommended, but then again, at the moment not much of the storage is being used and I'm stressing it a little on purpose. I'm planning to doubling the memory or maybe tripling it, I think I'll take the 1gig (besides de initial 8) per terabyte approach mentioned in the docs.

And, here is my error:

Environment:

Software Version: FreeNAS-11.1-U5 (8e2a858a1)
Request Method: POST
Request URL: https://cinder7.*.*/account/login/


Traceback:
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
42. response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response
249. response = self._get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
187. response = self.process_exception_by_middleware(e, request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
185. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "./freenasUI/account/views.py" in login_wrapper
320. extra_context=extra_context,
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/views.py" in inner
47. return func(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/decorators/debug.py" in sensitive_post_parameters_wrapper
76. return view(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapped_view
149. response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
57. response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/views.py" in login
82. auth_login(request, form.get_user())
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/__init__.py" in login
109. request.session.cycle_key()
File "/usr/local/lib/python3.6/site-packages/django/contrib/sessions/backends/base.py" in cycle_key
311. self.create()
File "/usr/local/lib/python3.6/site-packages/django/contrib/sessions/backends/file.py" in create
113. self.save(must_create=True)
File "/usr/local/lib/python3.6/site-packages/django/contrib/sessions/backends/file.py" in save
134. fd = os.open(session_file_name, flags)

Exception Type: OSError at /account/login/
Exception Value: [Errno 28] No space left on device: '/tmp/sessionidj38pfzwypch1xh0sbyxn4iuxf8ipsug9'


I went back to the console just after the error presented in a browser to take screenshots and there was a line that caught my attention, on the fifth, it reads "ValidHTTPSConnection, req", a certificate problem perhaps? The CA is local and has a OCSP working I just checked.
freenas-console.png

Here's my mathematically wrong 1/8 occupied boot drive, I guess I'm forgetting how to count now. It's 1/20th, I used Spotlight to calculate this time.
freenas-zpoollist.png

Above, the difference from the 500GB in storage I mentioned earlier is because I had already taken out the VMs from FreeNAS so the reading it correct. Everything works fine and I the ZFS performance hit mentioned just about everywhere is barely noticeable, if at all. The other storage is clustered, so vs a single ZFS disk with three high I/O VMs is not a fair comparison anyway.

freenas-web.png

freenas-netdata.png

Sidenote: I didn't know about Netdata, such a cool utility I love it instantly before knowing what was going on, it's now on several systems, I haven't deployed the server part yet, though from the quick doc-skimming I made it seems it doesn't actually need one.

Any ideas? I'll be monitoring my inbox for advice and/or scolding if need be, gotta learn some way. For the time being I have to run I'm actually taking a FreeNAS webinar just now, hopefully it hasn't started. :D THANKS!
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080

vitaprimo

Dabbler
Joined
Jun 28, 2018
Messages
27
THANKYOU-THANKYOU-THANKYOU!

...and wow, that was extremely fast. I moved it to the boot disk(s) for now, just as I install new disks, I don't think it'd write 10GB+ of logs in a couple of days max.

This is what I get for multitasking the setup.

First I selected reboot from the console and at the very end it failed because the lack of space; then from vCenter I issued a graceful reboot and it went through fine. I'm even more impressed and excited for the new array. I discovered watching the console booting up that FreeNAS has control over the server's optical drive too. What would be that good for these days, I've never used it before.

You've already saved me today but if you're feeling generous, when it was booting back up I read in the console something about RAID5, I barely managed to take the screenshot because I was in a webinar in the same computer--did I misconfigure the BIOS? Or is it just an I'm-prepared-for-anything thing? It does have a ton of options for "hardware" RAID and software RAID, "AHCI", is the option selected. It also had a compatibility option but this one allows the OS to detect hot swapping of drives, so I thought this was the direct option.

freenas-raid5.png


You can just ignore me, before that, thanks again!

This editor rocks BTW, I need to find out more about it.
 
Status
Not open for further replies.
Top