SOLVED CIFS dead in the water

[ivorycruncher]

I'm currently on FreeNAS 9.2.1.2. After not touching my computer or NAS for a couple days, I tried to access a file share, and found that it was unresponsive. Checking the server, I'm seeing repeated instances of the following text in the server notifier logs:

Jun 19 16:41:30 server notifier: django.db.utils.DatabaseError: database disk image is malformed
Jun 19 16:41:30 server notifier: winbindd not running? (check /winbindd.pid).
Jun 19 16:41:30 server notifier: smbd not running? (check /smbd.pid).
Jun 19 16:41:30 server notifier: nmbd not running? (check /nmbd.pid).
Jun 19 16:41:30 server notifier: Performing sanity check on Samba configuration: FAILED​

I tried stopping the CIFS service and restarting it, but now it won't restart, not even after a reboot. Also, any time the nav column is refreshed, the following alert appears:

Jun 19 16:47:10 server manage.py: [freeadmin.navtree:235] Failed to generate navtree for app freenasUI.sharing: database disk image is malformed​

Looking under Sharing > Windows (CIFS) Shares, it shows None, even though I previously had several shares defined. Output from testparm is as follows:

Load smb config files from /usr/local/etc/smb4.conf
max_open_files: increasing sysctl_max (11095) to minimum Windows limit (16384)
rlimit_max: increasing rlimit_max (11095) to minimum Windows limit (16384)
params.c:OpenConfFile() - Unable to open configuration file "/usr/local/etc/smb4
.conf":
No such file or directory
Error loading services.​

I'm a technical guy, but honestly kind of a noob when it comes to FreeBSD. Any insight into what's going on and how to fix would be greatly appreciated, because all my data is currently being held hostage by my FreeNAS.

 

[anodos]

You might have a corrupted config or broken permissions on your system dataset. Enter shell and see if the file /usr/local/etc/smb4.conf exists. Post output of "getfacl /mnt/<pool>/.system" where you input your pool name instead of <pool>.

 

[ivorycruncher]

I confirmed the config file doesn't exist. I tried creating a blank file, but it was deleted the next time I tried to start the CIFS service.

I have two pools mounted. The getfacl command on .system for one of them came back with "no such file or directory". On the other one, I got the following:


# file: /mnt/<pool>/.system
# owner: root
# group: wheel
owner@:rwxp--aARWcCos:------:allow
group@:r-x---a-R-c--s:------:allow
everyone@:r-x---a-R-c--s:------:allow

 

[anodos]

post output of "zfs get aclmode <pool>/.system"

 

[ivorycruncher]

NAME PROPERTY VALUE SOURCE
<pool>/.system aclmode passthrough inherited from <pool>

 

[anodos]

Nevermind. Still thinking.

 

[ivorycruncher]

I did in the command I ran, just didn't necessarily want to advertise my actual pool name.

 

[anodos]

I did in the command I ran, just didn't necessarily want to advertise my actual pool name.

Got it. I figured that out about 10 seconds after posting. :)

I think that you should get a fresh USB stick, install FreeNAS on it, import your pool and apply your old config to it. This would be a good opportunity to upgrade to a newer version of FreeNAS.

 

[ivorycruncher]

FYI, I just tried manually creating a new share, and got this error:

Request Method: POST
Request URL: https://server/admin/sharing/cifs_share/add/
Software Version: FreeNAS-9.2.1.2-RELEASE-x64 (002022c)
Exception Type: DatabaseError
Exception Value:
database disk image is malformed
Exception Location: /usr/local/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py in execute, line 450
Server time: Fri, 19 Jun 2015 17:39:56 -0500


It would definitely seem that the database got corrupted somehow, though no idea how since the system was idle and did not lose power (have it on a UPS).

Unfortunately, it seems I do not have a recent backup of my config (either forgot to, or misplaced the last one), so my last known good config backup is missing a significant number of changes to my system, including the addition of a new volume with two new disks and a number of share mount point modifications. I'm kind of scared to restore a backup with that many changes reverted.

I did export a config backup of the current state. Is there any way to determine which parts of the database are corrupt and repair them, and then restore the repaired backup?

EDIT: I actually just figured out how to browse the sqlite3 database via the terminal, and I'm getting the "database disk image is malformed" error when running a select command on the sharing_cifs_share table. Is there anyway to reset just that one table in the database, so that I can just manually recreate the shares?

 

[ivorycruncher]

Well, that was fun. I'm back online now. After giving myself a crash course in sqlite3 (I'm a quick study ^_^), I perused the freenas-v1.db database, and as far as I could tell, only the sharing_cifs_share table was corrupt. After researching repair methods, I ended up performing the following steps:

Dump accessible contents of original database to new database:
cd /data
sqlite3 freenas-v1.db ".dump" | sqlite3 fixed.db

Verify integrity of new database (integrity check should return "ok"):
sqlite3 fixed.db
pragma integrity_check;
.quit

Replace database:
mv freenas-v1.db freenas-v1.db.bad
mv fixed.db freenas-v1.db

Basically this just created a new copy of the existing database with all the data that it was able to extract out of the original one. The table schema was intact, so it created a new, blank sharing_cifs_share table with all the necessary data fields in it, just no records, and as far as I can tell, all other tables were still working, so the only thing that should have been lost was the CIFS share records.

Anyway, with all that done, I refreshed my web interface, and did not see anymore system event errors. I then attempted to create a Windows CIFS share and start the CIFS service, and confirmed I was able to do so succussfully. I then proceeded to simply recreate all my file shares from scratch. I now also have a current FreeNAS config on hand from after this repair. If anything like this happens again in the near future, I will assume my USB drive is giving out and get a new one. Guess I should probably also upgrade to 9.3 soon. :)

 

[anodos]

Well, that was fun. I'm back online now. After giving myself a crash course in sqlite3 (I'm a quick study ^_^), I perused the freenas-v1.db database, and as far as I could tell, only the sharing_cifs_share table was corrupt. After researching repair methods, I ended up performing the following steps:

Dump accessible contents of original database to new database:
cd /data
sqlite3 freenas-v1.db ".dump" | sqlite3 fixed.db

Verify integrity of new database (integrity check should return "ok"):
sqlite3 fixed.db
pragma integrity_check;
.quit

Replace database:
mv freenas-v1.db freenas-v1.db.bad
mv fixed.db freenas-v1.db

Basically this just created a new copy of the existing database with all the data that it was able to extract out of the original one. The table schema was intact, so it created a new, blank sharing_cifs_share table with all the necessary data fields in it, just no records, and as far as I can tell, all other tables were still working, so the only thing that should have been lost was the CIFS share records.

Anyway, with all that done, I refreshed my web interface, and did not see anymore system event errors. I then attempted to create a Windows CIFS share and start the CIFS service, and confirmed I was able to do so succussfully. I then proceeded to simply recreate all my file shares from scratch. I now also have a current FreeNAS config on hand from after this repair. If anything like this happens again in the near future, I will assume my USB drive is giving out and get a new one. Guess I should probably also upgrade to 9.3 soon. :)

Well done! New USB is a good idea.

 

[cyberjock]

The problem with that approach is that if an entry is missing from the database, it will NOT be created. Say you go to check some checkbox and the setting doesn't exist in your database. You'll get an error (you may not even get an error that makes it obvious what the problem is) and then it won't be applied. You'll have a heck of a time figuring out what the problem is at that point. :P

For your own long-term sanity, the answer is a new database that isn't corrupt or potentially missing entries.

 

[ivorycruncher]

The problem with that approach is that if an entry is missing from the database, it will NOT be created. Say you go to check some checkbox and the setting doesn't exist in your database. You'll get an error (you may not even get an error that makes it obvious what the problem is) and then it won't be applied. You'll have a heck of a time figuring out what the problem is at that point. :p

For your own long-term sanity, the answer is a new database that isn't corrupt or potentially missing entries.

I appreciate the concern, though I work in IT for a living, and am aware of such possibilities. FWIW, I compared a copy of the bad database to older backups I made a while back, and the only table that was an issue was the sharing_cifs_share one. All other tables had the appropriate data fields and, where applicable, valid records in them, so for the moment, I feel reasonably confident that the database itself is no longer an issue. At the very least, the fix is adequate for the moment.

That said, not knowing what caused the corruption, the USB drive itself is in question. I've ordered a new Sandisk Cruzer Fit drive, which will arrive within a week or so. At that point, my plan is to install the latest release version of FreeNAS on it and upload a current config backup from my existing FreeNAS installation. That should take care of any further concerns.

 

[anodos]

I appreciate the concern, though I work in IT for a living, and am aware of such possibilities. FWIW, I compared a copy of the bad database to older backups I made a while back, and the only table that was an issue was the sharing_cifs_share one. All other tables had the appropriate data fields and, where applicable, valid records in them, so for the moment, I feel reasonably confident that the database itself is no longer an issue. At the very least, the fix is adequate for the moment.

That said, not knowing what caused the corruption, the USB drive itself is in question. I've ordered a new Sandisk Cruzer Fit drive, which will arrive within a week or so. At that point, my plan is to install the latest release version of FreeNAS on it and upload a current config backup from my existing FreeNAS installation. That should take care of any further concerns.

Make that two USB drives and mirror them. :)

 

[guglez]

The problem with that approach is that if an entry is missing from the database, it will NOT be created. Say you go to check some checkbox and the setting doesn't exist in your database. You'll get an error (you may not even get an error that makes it obvious what the problem is) and then it won't be applied. You'll have a heck of a time figuring out what the problem is at that point. :p

For your own long-term sanity, the answer is a new database that isn't corrupt or potentially missing entries.

What to do if some records are already missing in my DB? Can I recreate them somehow?

Right now a lot of menus/features are not working in web gui. For example:

Alert button is not working: {"error": true, "events": [], "message": "Error: no such column: system_alert.node"}
I can't add NFS share:
{"error": true, "events": [], "message": "Error: no such column: services_nfs.nfs_srv_v4_v3owner"}

and so on. Do I have any options?

[root@freenas] ~# sqlite3 /data/freenas-v1.db "pragma integrity_check;"
ok

 

[SweetAndLow]

What to do if some records are already missing in my DB? Can I recreate them somehow?

Right now a lot of menus/features are not working in web gui. For example:

Alert button is not working: {"error": true, "events": [], "message": "Error: no such column: system_alert.node"}
I can't add NFS share:
{"error": true, "events": [], "message": "Error: no such column: services_nfs.nfs_srv_v4_v3owner"}

and so on. Do I have any options?

[root@freenas] ~# sqlite3 /data/freenas-v1.db "pragma integrity_check;"
ok

This thread is over 2 years old. You should probably make a new thread to ask your question.

Sent from my Nexus 5X using Tapatalk