HELP PLEASE! :) Cron <root@freenas> /bin/sh /usr/local/sbin/save_rrds.sh

[John Richardson]

Hey all,

I'm pretty new to all this so this - but I have had freenas running for some time and I'm trying to learn as much as I can on my own... but this one has me stumped (probably pretty easy for y'all!).

Anyways, I've been getting these output emails from Freenas - like one an hour or so....

Subject:

Cron <root@freenas> /bin/sh /usr/local/sbin/save_rrds.sh

Message:

rm: /data/rrd_dir.tar.bz2: Read-only file system
mv: rename rrd_dir.tar.bz2.58340 to /data/rrd_dir.tar.bz2: Read-only file system

Do I just need to go into the terminal and sudo on those actions that Freenas is trying to complete - or do I need to adjust some permissions somewhere so that it can do this stuff going forward?

Thanks in advance for your help!

John

 

[jgreco]

/data is not supposed to be read-only. Something may be wrong.

What version of FreeNAS are you using, and what hardware, especially the boot device?

 

[John Richardson]

Asrock C2550D4I, 4 WD 3Tb, and the boot device is just a little thumb drive plugged into the mobo.

Version of Freenas =
FreeNAS-9.3-STABLE-201503200528

 

[John Richardson]

I've used the thumb drive for a while (and I've rebuilt the machine a couple of times) - never with this issue though.... I have been playing with linux in a VM on freenas - I wonder if I did something when trying to mount the shares in linux?

 

[jgreco]

Well, waitaminit...

/data isn't used for storing RRD's on 9.3. That's supposed to be over in /var/db/system.

But the script still refers to the file being over in /data. Hm.

In the new ZFS model of things, the root partition (which holds /data) isn't read-only.

Does this persist if you reboot? And if so, reboot, then log in, type "mount", and show us the results for root. Like this:

Code:

# mount
freenas-boot/ROOT/FreeNAS-FreeNAS-9.3-CURRENT-201503130047 on / (zfs, local, noatime, nfsv4acls)

 

[John Richardson]

Ok, So I just completed a reboot - not sure if I'll get another email or not but here is what comes up from 'mount'...

[root@freenas ~]# mount
freenas-boot/ROOT/default on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs, local, multilabel)
tmpfs on /etc (tmpfs, local)
tmpfs on /mnt (tmpfs, local)
tmpfs on /var (tmpfs, local)
freenas-boot/grub on /boot/grub (zfs, local, noatime, nfsv4acls)
storage on /mnt/storage (zfs, local, nfsv4acls)
storage/homenas on /mnt/storage/homenas (zfs, local, nfsv4acls)
storage/jails on /mnt/storage/jails (zfs, local, nfsv4acls)
storage/jails/.warden-template-VirtualBox-4.3.12 on /mnt/storage/jails/.warden-template-VirtualBox-4.3.12 (zfs, local, nfsv4acls)
storage/jails/.warden-template-pluginjail on /mnt/storage/jails/.warden-template-pluginjail (zfs, local, nfsv4acls)
storage/jails/.warden-template-pluginjail-open-x86 on /mnt/storage/jails/.warden-template-pluginjail-open-x86 (zfs, local, nfsv4acls
)
storage/jails/VirtualBox on /mnt/storage/jails/VirtualBox (zfs, local, nfsv4acls)
storage/jails/plexmediaserver_1 on /mnt/storage/jails/plexmediaserver_1 (zfs, local, nfsv4acls)
storage/jails/transmission_1 on /mnt/storage/jails/transmission_1 (zfs, local, nfsv4acls)
storage/jails_2 on /mnt/storage/jails_2 (zfs, local, nfsv4acls)
storage/jails_2/.warden-template-VirtualBox-4.3.12 on /mnt/storage/jails_2/.warden-template-VirtualBox-4.3.12 (zfs, local, nfsv4acls
)
storage/jails_2/.warden-template-pluginjail on /mnt/storage/jails_2/.warden-template-pluginjail (zfs, local, nfsv4acls)
storage/jails_2/VMS on /mnt/storage/jails_2/VMS (zfs, local, nfsv4acls)
storage/jails_2/plexmediaserver_1 on /mnt/storage/jails_2/plexmediaserver_1 (zfs, local, nfsv4acls)
storage/jails_2/transmission_1 on /mnt/storage/jails_2/transmission_1 (zfs, local, nfsv4acls)
storage/.system on /var/db/system (zfs, local, nfsv4acls)
storage/.system/cores on /var/db/system/cores (zfs, local, nfsv4acls)
storage/.system/samba4 on /var/db/system/samba4 (zfs, local, nfsv4acls)
storage/.system/syslog-e8f18e18f78a4738b9cfa45b0c82db88 on /var/db/system/syslog-e8f18e18f78a4738b9cfa45b0c82db88 (zfs, local, nfsv4
acls)
storage/.system/rrd-e8f18e18f78a4738b9cfa45b0c82db88 on /var/db/system/rrd-e8f18e18f78a4738b9cfa45b0c82db88 (zfs, local, nfsv4acls)
devfs on /mnt/storage/jails_2/VMS/dev (devfs, local, multilabel)
procfs on /mnt/storage/jails_2/VMS/proc (procfs, local)
/mnt/storage/homenas/Torrents on /mnt/storage/jails_2/VMS/media (nullfs, local)
/mnt/storage/homenas/VirtualBox on /mnt/storage/jails_2/VMS/media (nullfs, local)
/mnt/storage/jails/VirtualBox on /mnt/storage/jails_2/VMS/media (nullfs, local)
/mnt/storage/homenas on /mnt/storage/jails_2/VMS/mnt (nullfs, local)
devfs on /mnt/storage/jails_2/plexmediaserver_1/dev (devfs, local, multilabel)
procfs on /mnt/storage/jails_2/plexmediaserver_1/proc (procfs, local)
/mnt/storage/homenas/Torrents on /mnt/storage/jails_2/plexmediaserver_1/media (nullfs, local)
devfs on /mnt/storage/jails_2/transmission_1/dev (devfs, local, multilabel)
procfs on /mnt/storage/jails_2/transmission_1/proc (procfs, local)
/mnt/storage/homenas/Torrents on /mnt/storage/jails_2/transmission_1/media (nullfs, local)
[root@freenas ~]#

 

[John Richardson]

Again - I'm fairly new to Freenas so I probably created some bastard monster haha...

Thanks again for helping!

 

[jgreco]

Whoa. Just needed the root. But that's fine, no worries!

What happens if you do "touch /data/blah; rm /data/blah" now as the root user? Because it looks like the ROOT filesystem is mounted read/write, and it looks like you do have the /var/db/system/rrd dataset too.

 

[John Richardson]

Sorry - noob... how do I do this just as the root user... i tried sudo mount and it gave me:

[root@freenas ~]# sudo mount
Sorry, user root is not allowed to execute '/sbin/mount' as root on freenas.local.

So I just typed mount... here's the last command you suggested...

[root@freenas ~]# touch /data/blah; rm /data/blah

[root@freenas ~]# sud touch /data/blah; rm /data/blah
bash: sud: command not found
rm: /data/blah: No such file or directory

[root@freenas ~]# sudo touch /data/blah; rm /data/blah
Sorry, user root is not allowed to execute '/usr/bin/touch /data/blah' as root on freenas.local.
rm: /data/blah: No such file or directory

[root@freenas ~]#

 

[John Richardson]

I fear I've created an unholy monster haha - just as a side note - if you're interested in seeing the build...

https://forum.teksyndicate.com/t/ultra-silent-server/77566

 

[jgreco]

You're already root; see, [root@freenas]

Omit the sudo. I apologize for the confusion.

 

[John Richardson]

[root@freenas ~]# touch /data/blah; rm /data/blah
[root@freenas ~]#

 

[jgreco]

Okay, great. Guessing: no more e-mails. Let us know if I'm wrong.

Would be interesting to know how things got in that bad state.

 

[John Richardson]

Awesome man!

Thanks so much for your help! God knows what I did - I'm sure it was my fault though.

 

[John Richardson]

DAMNIT

rm: /data/rrd_dir.tar.bz2: Read-only file system
mv: rename rrd_dir.tar.bz2.15299 to /data/rrd_dir.tar.bz2: Read-only file system

Do you think it could have something to do with my virtual machines? Should I try turning them off to see if that helps?

 

[jgreco]

I'm not clear on what it's supposed to be doing. Storing the rrd's on /data is an old behaviour and the intent isn't entirely obvious.

Can you check the following things? Trying to see what might be different here.

First off, redo the touch/rm test above.

Then:

Code:

# ls -al /var/db/collectd/rrd
lrwxr-xr-x  1 root  wheel  51 Mar 19 18:28 /var/db/collectd/rrd@ -> /var/db/system/rrd-790a6bc1b7664bef8761693b41f806b2

That's supposed to be a symlink on 9.3. In the old days it was a directory that held RRD data.

Code:

# /usr/local/bin/sqlite3 /data/freenas-v1.db 'SELECT sys_pool FROM system_systemdataset ORDER BY -id LIMIT 1'
storage1

That should return the name of your main pool. This is the pool on which the system dataset is stored. This is a new 9.3 thing.

Code:

# /usr/local/bin/sqlite3 /data/freenas-v1.db 'SELECT sys_rrd_usedataset FROM system_systemdataset ORDER BY -id LIMIT 1'
1

This indicates that the rrd's are to be stored in a dataset.

I'm probably the wrong person to be debugging this as I mostly know how this was supposed to work on versions prior to 9.3 with ZFS, but it doesn't hurt to see if we can figure it out anyways.

 

[John Richardson]

Ok - here we go (THANK YOU!)

[root@freenas ~]# touch /data/blah; rm /data/blah
touch: /data/blah: Read-only file system
rm: /data/blah: No such file or directory
[root@freenas ~]#

[root@freenas ~]# ls -al /var/db/collected/rrd
ls: /var/db/collected/rrd: No such file or directory
[root@freenas ~]#

[root@freenas ~]# /usr/local/bin/sqlite3 /data/freenas-v1.db 'SELECT sys_pool FR
OM system_systemdataset ORDER BY -id LIMIT 1'
storage

[root@freenas ~]# /usr/local/bin/sqlite3 /data/freenas-v1.db 'SELECT sys_rrd_use
dataset FROM system_systemdataset ORDER BY -id LIMIT 1'
0
[root@freenas ~]#

 

[danb35]

It looks like /data/ is read-only again, which it shouldn't be. It also looks like you have the system set to not save the rrd data on the system dataset, and if I'm following the logic of the save_rrds.sh script correctly that makes a difference in how it behaves. If you go into the web GUI to System -> System Dataset, are the boxes for syslog and reporting database checked? If not, try checking them both and saving, and see if that helps.

 

[John Richardson]

Thanks for the input... only syslog was checked. I checked off reporting database and saved... I'll let you know if I get another email.

John

 

[John Richardson]

Well - actually it won't let me check the box! It gives me an error -