Volume will only mount in read-only mode

[Joe Mays]

I'm not sure this belongs in storage, but I'm not sure where it actually DOES belong, so this is as good a place as any. If there is a better topic please let me know.

I have a backup machine that copies everything over from the primary via a dedicated network connection. It's been running fine for months. I began getting an alert that the replication was failing a day or two ago. I rebooted the machine, and it seemed to come up fine until the very last step. It displays menu, displays the info line just before the menu prompt appears, pauses for a period of time, then redisplays the menu, over and over. It never gets to a menu prompt.

Apparently this is before it starts the web interface, because that never comes up for me to connect to the GUI, but after the network interface has started up, because I can ping the system fine.

I can hit the alt-F2 to get another console screen, log in there, and things seems to be fine. I can go to the raid volume, look at the files, read the contents, so I don't think it's a problem with mounting the raid volume. It seems to be mounted fine.

I'm getting bootup emails from the system. The syslogs, however, do not seem to be updating.

Obviously, I can't give debug info from the WebGUI since I can't get into it, but --

Build FreeNAS-9.3-STABLE-201602031011
Platform Intel(R) Xeon(TM) CPU 2.80GHz
Memory 16357MB
System Time Fri Mar 11 14:34:29 EST 2016

 

[Joe Mays]

I created a user account so I could ssh into the machine, so I can get to a command prompt that way if anyone knows what I should be looking for.

 

[Joe Mays]

I could be wrong, but I think I found at least part of the problem. The entire volume seems to be mounted as read only.


[root@winsan02] /mnt/WinCustVol/WinCommonData/home/webservice/l/la/launchpad@win.net/WWW# mount
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201602031011 on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs, local, multilabel)
tmpfs on /etc (tmpfs, local)
tmpfs on /mnt (tmpfs, local)
tmpfs on /var (tmpfs, local)
freenas-boot/grub on /boot/grub (zfs, local, noatime, nfsv4acls)
WinCustVol on /mnt/WinCustVol (zfs, local, read-only, nfsv4acls)
WinCustVol/WinCommonData on /mnt/WinCustVol/WinCommonData (zfs, NFS exported, local, read-only, nfsv4acls)
WinCustVol/WinCustMailData on /mnt/WinCustVol/WinCustMailData (zfs, NFS exported, local, read-only, nfsv4acls)
WinCustVol/.system on /var/db/system (zfs, local, read-only, nfsv4acls)
WinCustVol/.system/cores on /var/db/system/cores (zfs, local, read-only, nfsv4acls)
WinCustVol/.system/samba4 on /var/db/system/samba4 (zfs, local, read-only, nfsv4acls)
WinCustVol/.system/syslog-8f93c363d0054e6cb350f5e5f5f1f086 on /var/db/system/syslog-8f93c363d0054e6cb350f5e5f5f1f086 (zfs, local, read-only, nfsv4acls)
WinCustVol/.system/rrd-8f93c363d0054e6cb350f5e5f5f1f086 on /var/db/system/rrd-8f93c363d0054e6cb350f5e5f5f1f086 (zfs, local, read-only, nfsv4acls)
WinCustVol/.system/configs-8f93c363d0054e6cb350f5e5f5f1f086 on /var/db/system/configs-8f93c363d0054e6cb350f5e5f5f1f086 (zfs, local, read-only, nfsv4acls)

 

[dlavigne]

Anything in /var/log/messages?

 

[nightshade00013]

If I was troubleshooting this I would disconnect all the HDD's and work the GUI aspect first and apart since there are a couple problems the way things are looking. After it's booted up see if the same thing is happening and a menu is still unavailable along with the GUI. If that is the case it could be a bad update, I had this happen once on a test system so I manually booted to a previous version in grub. Problem was solved and then I reran the update through the GUI and the problem was solved.

After that is fixed you can reattach the drives and see if they come up like they should, my guess is that if a bad update happened and borked the UI that the problem there will fix itself.

You may also try another fresh install of FreeNAS and import a backup of your settings or set things up again and see if that fixes things. But seriously if the GUI isn't working right then something in the install is incorrect and trying to work the read only aspect is working things the wrong way.

 

[Joe Mays]

No relevant log messages, because the last log updates are from just before the problem started. Evidently, syslogd is not writing new log messages, either.

I tried backing up through the updates, but the same problem was still occurring in previous versions. Going to try the "reinstall and recover settings" suggestion now.

 

[Joe Mays]

I tried booting the machine on my desktop and chose "Verbose Mode" for booting, which yields the following information.

There is a message that comes up a few times before the menu, then keeps coming up each time it reloads the menu.

...pause...
...menu reloads...
...pause...
"ichwd0: timer set to 63 ticks"
"ichwd0: timer reloaded"
...pause...
...menu reloads....
...pause...
"ichwd0: timer set to 63 ticks"
"ichwd0: timer reloaded"
...pause...
...menu reloads....
...pause...
"ichwd0: timer set to 63 ticks"
"ichwd0: timer reloaded"
...pause...
...menu reloads....

 

[nightshade00013]

Install on a new stick. I would disconnect any drives just in case you could possibly select one as the instal destination then see what happens. Looks like something in the original install is goofed.