bad dir ino: mangled entry

[OldDude]

OK, I have 2 FreeNAS devices which have been running smoothly for a couple of months. Both are running 8.0.3 Release 64-bit

Systems are identical (Intel i5, 16G RAM, 4 2TB drives).

One is visible to the Manufacturing floor, the 2nd is a pseudo "Hot" backup which is RSync'd every 15 minutes. (If anything ever happens to the first, the Manufacturing shifts can be quickly returned to production using the 'clone').

Suddenly today, I found that I could no longer access one of the devices using the Web GUI. I was greeted with a rather blunt but obscure error message:

Uhandled Exception: An Unhandled exception was thrown by the application.

Fantastic!.... this is about as informative as saying "ERROR: There was an error detected"
(Honestly, who writes these messages?!?)

So, I go to the console and see multiple lines of errors (2 identical lines which directly correlate with how many times I tried to connect to the server via the WebGUI) each line says:

/: bad dir ino 78512 at offset 24: mangled entry

All other aspects of the system appear to be running properly. The Rsync jobs are still actively being handled and data can still be read or written to the system.

zpool status reports no known errors and shows all drives.

So.... is the "bad dir" reference referring to the USB drive? and implying that it has become corrupted?
Would that explain why my GUI has suddenly died, but why everything else still works? or should I be concerned and start preparing for an immanent drive failure?

All help is appreciated.

 

[ProtoSD]

So.... is the "bad dir" reference referring to the USB drive? and implying that it has become corrupted?
Would that explain why my GUI has suddenly died, but why everything else still works? or should I be concerned and start preparing for an immanent drive failure?

All help is appreciated.

I think it is a fair guess to assume it's your USB drive. Since FreeNAS runs off 3 RAM disks after it boots that would explain why (almost) everything is still working.

I would

Prepare another USB key
Copy the /data/freenas-v1.db to a temp directory on your pool
Shutdown and boot from the new USB drive
open a command prompt from the console
mount -uw /
Copy the freenas-v1.db from the temp directory to /data
reboot
See if the GUI is responding etc.
Then run a scrub on your pool

Do you run regular scrubs?

I really think it's just a flaky/failing flash drive. You could pop it into another FreeBSD system and run fsck on it, but they tend to react unpredictably when they start to fail. I had one that suddenly decided it was a read-only device, even after trying to wipe it.

 

[OldDude]

Thanks for the quick response protosd.

I'll pick up a replacement USB to be safe.

I had already rebooted the system once (so much for the RAM drives), so I'm guessing that the USB is the flaky part.

Should I expect a high USB failure rate?

This particular one is only 5 months old and I had another one fail completely on the other system within about 3 months.

It's a little disconcerting.... especially because I have units running overseas as well.

I love to travel, but trips like that where everybody is pacing the floors before you arrive are never fun.

 

[ProtoSD]

Should I expect a high USB failure rate?

This particular one is only 5 months old and I had another one fail completely on the other system within about 3 months.

It's a little disconcerting.... especially because I have units running overseas as well.

I love to travel, but trips like that where everybody is pacing the floors before you arrive are never fun.

That's a good question about the failure rate, the reason for the RAM disks is to keep writes to the flash drive to a minimum, but they seem to fail anyway. If I were in a production environment I'd probably look for a higher end industrial flash drive, but I can't recommend anything specific, maybe some others here can. Some people opt for compact flash with a SATA adapter, but there are also the DOM (disc on module) flash drives that plug directly into a SATA port. I really don't know if those will last any longer, but I'd expect them to be a little better.

As for the systems overseas, I'd look into having a spare ready and a cron script to backup the config database. I hear you about the floor pacing visits.

I've had two USB flash drives fail, but they're cheapo commercial models.

 

[OldDude]

Here's an absolute n00b question....

Is the freenas-v1.db file the same as the backup file I create from the GUI, or does it contain additional info?

Whenever any changes are made from the GUI, I immediately backup into a directory that is RSync'd to the main server.

I'm just wondering if it would be better to perform the backup using a cron job. (Though, most of the support crew who would handle an emergency in my absence are not very comfortable with command line interfaces)

 

[ProtoSD]

That's a good question and I don't know the answer. What you could try is go to Restore config and point it to a copy of the freenas-v1.db and see what happens when you Restore/Import that from the GUI. Of course restoring it to a system that already has those settings won't tell you anything, so it needs to be a new or "factory restored" system with empty settings.

 

[StephenFry]

Should I expect a high USB failure rate?

I'm a FreeNAS newbie, but have enough experience running (intensive) programs off USB flash drives to not even try it for FreeNAS.

Currently I'm using my safest method, which is a good quality CF card on a CF2IDE converter. If you really want to use USB, I prefer to "make" one myself, by using a USB SD-card reader and a quality SD-card.

It seems the commercial USB memory sticks often use inferior memory; I don't bother any more.