Routine Maintenance Checks

Status
Not open for further replies.

maclark7029

Dabbler
Joined
Apr 6, 2014
Messages
25
Hi,

I was wondering what others are doing wrt routine maintenance checks of a freenas server. I have regularly sched SMART tests and SCRUBS running, but are there command-line type checks that are useful? For example, I usually run a "zpool status" once a week. Are there others that are useful?

Thanks,
Mike
 
L

L

Guest
There are some logs worth a look in /var/log. I like to look at messages occasionally. Beware though, these are really "unixy" messages and most of them are informational rather than errors.
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
Mike:

The regimen I use is fairly simple. It is mostly the "standard Cyberjock" method, plus a few extra overkills:
  • Odd weeks of the month: Perform a SMART "long" test on each drive. (Absolutely essential)
  • Even weeks of the month: Perform a ZFS scrub. (Absolutely essential)
  • Approximately twice a week, perform a SMART "short" test on each drive. (Probably useless, but it's pretty much free, so why not)
  • After each scrub, run a zpool status, make sure everything looks good
  • Once a day, run a smartctl -x on each drive, and save the result in a text file for later research purposes if something goes wrong.
  • Check the drive temperatures once a day -- this usually just means viewing the smartctl output from the previous step.
  • Once a day, save the FreeNAS config db file
  • On the first of each month, get out a can of compressed air and clear out all fans including CPU fan/heatsink
  • Cyberjock has a script that checks for database integrity---you can find that in the forum somewhere
  • Always keep one brand new cold spare hard drive on hand, and immediately replace any drive that shows any problem
  • Approximately once a day, or whenever I think of it, look at the "reporting" in the GUI, make sure everything looks right, no unusual use of swap space, CPU spikes, memory leaks, etc.
  • Once a night, "pull" my FreeNAS data onto my SyncBackFree backup solution on my Windows box. Just to be sure. (I think this step is a DrKK'ism, most people do other forms of backing up).
The nice thing about FreeNAS is that a properly conceived pool, correctly put together in ZFS, with the above fairly routine tasks, pretty much lowers your chance of any data loss to zero.

I have never lost a single bit of data. Ever.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
LOL @ the Cyberjock standard comment.
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630

panz

Guru
Joined
May 24, 2013
Messages
556
I shouldn't use "canned air" because sometimes it sprinkles some water, caused by humidity condensing during the spraying. I use a "dry" Abac compressor (it is a non-fluid lubricated compressor) which can produce dried air (Abac is one of the best compressor maker and, I'm proud, it is an Italian brand).
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Canned air is bad to use on a running system for many reasons including that the expanding gas results in cold air, and introducing a sudden temperature differential is a very bad idea. I have seen power supplies blow out when blown with a sudden blast of chilly canned air.
 

panz

Guru
Joined
May 24, 2013
Messages
556
Moreover, an often underestimated procedure is to "lock" all the fans you're going to clear with compressed air because, by spinning, they can act as a generator and induce an electrical potential difference capable of damaging the motherboard.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
That's also a good point. You'll laugh when I tell you that the standard tool around here for this is a 4" narrow zip-tie.... because it's electrically safe and there's always plenty around.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
That's also a good point. You'll laugh when I tell you that the standard tool around here for this is a 4" narrow zip-tie.... because it's electrically safe and there's always plenty around.
I still have nightmares when I encounter zip-ties, ever since I cut a fan wire trying to remove one. I guess that's a good use for the ones that every case and PSU manufacturer (Seasonic excluded) likes to add as an "accessory". It does sounds like a genuinely good method of holding the fans.
I've given up on compressed air to clean fans, though, and just wipe mine. Better results and less dust floating around that'll probably be sucked right in once the computer is running again.

These days, I only use compressed air to clean PSUs, PCBs with exposed solder points and connectors/slots.

LOL @ the Cyberjock standard comment.
Sounds like a drink. Any amateur (or professional) bartenders here want to invent said drink?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Since most of my work involves racked gear, usually not being removed from the rack, usually a vac or compressed air is as good as it gets.

How'd you manage to cut a fan wire? Usually that's a sign of using the wrong tool for the job... try something like an Xcelite 170M and you should have no problem with precision cuts.
 

panz

Guru
Joined
May 24, 2013
Messages
556
I don't blow compressed air without putting a vacuum cleaner connected with a big funnel-shaped bag (plastic-like made, ESD certified) near the operation area. It will collect most of the dust. You should always be careful not to induce electrostatic charge in the air, so always use an humidifier and an ESD air ionizer. I also use removable fans: they're A LOT easier to clean because you can pull them outside the cabinet and clean them in another room.

EDIT: the "I cut a wire" problem: let's use something like a cable tie removal tool.

http://www.cableorganizer.com/act/cable-tie-removal-tool/
 
Last edited:

gpsguy

Active Member
Joined
Jan 22, 2012
Messages
4,472
Thanks for the suggestions. I'm always extremely careful while cutting a cable tie and have been fortunate so far.

I might pick up one of these tools, since I have enough cable ties to last me for the next 10 years and I've had the container for several years.

jgreco said:
... try something like an Xcelite 170M

panz said:
EDIT: the "I cut a wire" problem: let's use something like a cable tie removal tool.

http://www.cableorganizer.com/act/cable-tie-removal-tool/
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
The advantage to the 170M is that it is an extremely versatile tool. It works very well to trim cable ties, cut wire, trim leads after soldering, and anything else that a small side cutter with a sharp set of tips would be good for. That cable tie removal tool is slightly more of a one-trick tool.
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
Sounds like a drink. Any amateur (or professional) bartenders here want to invent said drink?

The Cyberjock Standard:

Take a 40 square centimeter piece of 80 grit aluminum oxide sandpaper, and buff your scrotum. Coat with lemon juice. This summons the Cyberjockian mood. Then, prepare the drink itself:

1 jigger of very, very cheap Scotch
1/4 cup of fortified Manischewitz wine
1/2 jigger turkey gravy
dash of Angostura bitters
garnish with celery, or other hard-to-digest vegetable

Serve lava hot.
 
Last edited:

Knowltey

Patron
Joined
Jul 21, 2013
Messages
430
The Cyberjock Standard:

Take a 40 square centimeter piece of 80 grit aluminum oxide sandpaper, and buff your scrotum. Coat with lemon juice. This summons the Cyberjockian mood. Then, prepare the drink itself:

1 jigger of very, very cheap Scotch
1/4 cup of fortified Manischewitz wine
1/2 jigger turkey gravy
dash of Angostura bitters
garnish with celery, or other hard-to-digest vegetable

Serve lava hot.

Hmm, besides the turkey gravy that doesn't sound half bad and is actually pretty similar to a drink I often get at bars when the place doesn't have all the ingredients for a Singapore Sling or Algonquin.

And yeah as jgreco said don't use compressed air cans on a running system. The air released by those is itself quite cold and can be a sudden temperature change. And then for fans either temporarily disconnect them from the board or make sure they can't spin. I personally just disconnect them so I can walk them out to the allotment and give them the beans.

  • Odd weeks of the month: Perform a SMART "long" test on each drive. (Absolutely essential)
  • Even weeks of the month: Perform a ZFS scrub. (Absolutely essential)

Hmm, I must be really overkill then lol. I have mine set up to do a long test every Sunday at 2AM and a scrub every Monday at 2AM. But on the other hand I have each of my computers set to backup up to the NAS on the days before. one on Friday 2AM and the other on Saturday 2AM, so seemed like a good idea to get a scrub after those just to make sure everything is in order.
 
Last edited:

Yatti420

Wizard
Joined
Aug 12, 2012
Messages
1,437
Any kind of vacuum a bad idea..

Sent from my SGH-I257M using Tapatalk 2
 

jamiejunk

Contributor
Joined
Jan 13, 2013
Messages
134
Since most of my work involves racked gear, usually not being removed from the rack, usually a vac or compressed air is as good as it gets.

How'd you manage to cut a fan wire? Usually that's a sign of using the wrong tool for the job... try something like an Xcelite 170M and you should have no problem with precision cuts.
Just to be clear. Jgreco isn't talking about an air compressor. Those have water and oil in the lines.
 

panz

Guru
Joined
May 24, 2013
Messages
556
I was talking about "dry" compressors. Abac manufactures them (no oil or fluid lubricant).

I agree with @Yatti420 that any kind of direct vacuum is a bad idea. I use the vacuum cleaner only to catch the airborne dust particles; I use it with a good humidifier and an ionizer running in the room.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Since most of my work involves racked gear, usually not being removed from the rack, usually a vac or compressed air is as good as it gets.

Okay as a number of you pointed out this probably leaves a wrong impression. My intended point was that opening up and carefully cleaning a rack mounted server is basically untenable when you have twenty of them in a rack, only an hour of spare time in your maintenance window, and much of the gear needs to remain running.

Typically you don't have filters on the servers and other gear that you put in a data center, and in theory the data center's HVAC filtration maintains the environment as a relatively dust-free environment. However, we deploy gear for many years, and sooner or later it can get a little dusty anyways. People are always tracking more dust and dirt into the environment because it simply isn't a cleanroom environment.

So, first: vac is the preferred method of dealing with this, because it is least likely to be damaging to the gear, but it is also usually less effective (lower rate of airflow). When I see the inlet ports on gear getting covered with dust, a vac is the solution of choice. But as some of you have noted, ESD is a significant issue here. Dust moving along pipes gets you static, as the woodworking guys occasionally worry about.

So you can cope with this a few ways, the expensive way is to buy a "pro" DataVac like this unit:
https://www.metrovacworld.com/DataVac_ESD_Safe_Maintenance_System-DV-3ESD1/overview
and this is a great idea if you're doing this on a daily basis. The filtering is the big thing that'd be hard to replicate, the ESD properties can probably be duplicated pretty cheaply with a small shopvac, a grounding wire, and coating the inside of your hose and tools with a conductive paint such as this one by MG.

But those of us in the industry have a dirty little secret. Specialist gear is never available where and when you need it. We typically take a shopvac, some aluminum foil, and a wrist grounding strap. You fold the aluminum foil over the end of the hose, forming a conductive ring, and fasten it in place with the wrist grounding strap. Then you attach the wrist strap to the rack frame (or any other approperiate ground) and you're good to go.

As for compressed air, this is also a problematic realm. Compressors, by their very nature, cause moisture problems. As you compress air, it warms up, and warm air can hold lots of moisture. The water vapor then hits a cool surface like the inside of the tank or hose and condenses. In the hose this turns into fine water mist being sprayed. This problem is worse because typically we don't have huge compressors. A large tanked compressor with a relatively small draw of air means that the tank is likely to be closer to room temperature and not so able to hold moisture; in such a scenario the air tends to be naturally drier. One alternative I've seen is to use an external portable 3 gallon tank and then you fill that up and actively cool it in a fridge, then hit the drain valve. Instant tank of relatively dry air. Oil is also a problem as panz noted, the usual solution for that is filtration (or purchasing an oil-free compressor). But in all cases you still have the problem of the expanding gas cooling down as it expands.

There's been some improvement with the introduction of what are essentially fan based blowers like the ED-500 but I haven't yet gotten around to trying one. So panz is half right and half wrong, most of us are still using strategies that involve compressors and various mitigations. But not just blasting the crappy-quality air out of a cheap compressor into your beautiful server.
 
Status
Not open for further replies.
Top