Unsure of SATA drive spindown

[Tekkie]

Another crontab job to add I guess...

 

[joeschmuck]

Another crontab job to add I guess...

Do you have the same issue as Jim? Do your drives on the SAS fail to spin down after a set time? If so, can you verify if "camcontrol standby da0 -t 120" spins your drives down after 2 minutes. You can use 900 for 5 minutes, or you get the point. I'm looking to find out if the command works of fails. If it work despite the error messages then it could be worked into FreeNAS easily enough.

 

[Milhouse]

I can confirm that "camcontrol standby da0 -t 120" has no effect on an LSI 9211-8i when connected to Samsung SATA disks (a mixture of 2TB 3.5" and 500GB 2.5" disks) - it would appear that the mps0 FreeBSD driver does not support the ATA idle/standby commands.

"camcontrol stop da0" does work though on all my disks.

 

[joeschmuck]

Thanks for the info. I'm curious if the controller card works on regular FreeBSD 8.2 because ataidle actually works without error messages. But the result here is camcontrol standby doesn't work under the FreeNAS build.

Thanks for the info.

 

[Milhouse]

From tests with FreeBSD 8.2 and even a snapshot of 9.0, neither ataidle or camcontrol worked with idle/standby and the mps0 driver. The problem is the driver, it has no power management support.

One hack/work around might be a daemon script that checks "zpool iostat" every 60 seconds, recording the number of consecutive queries that return zero transfers and when the count exceeds a configurable limit (ie. 30 for 30 minutes) it then "camcontrol stop"s all the drives (any activity would reset the count back to zero). Not nice, but since the mps0 developers don't seem intent on supporting power management (or even discussing it in the mailing list) it's probably the only option.

 

[joeschmuck]

That's interesting and like you said, might be the only option. I suspect you are talking about "zpool iostat". The only problem with it is every time I ran it, it had a write value of 2, never could get 0. Also there is a format associated with the output, not sure how to sift through that to get the read and write data only. Do you have an example command I could punching into the command line that works better?

 

[Milhouse]

I suspect you are talking about "zpool iostat".

Yes sorry, I have corrected my post.

The only problem with it is every time I ran it, it had a write value of 2, never could get 0.

Yeah, not sure why it always shows activity on the first iostat, but then settles down to 0 activity if left to run continuously. This anomaly makes life a little more tricky for the shell scripter... :)

Also there is a format associated with the output, not sure how to sift through that to get the read and write data only. Do you have an example command I could punching into the command line that works better?

I'm not sure if it is possible to control the format of iostat, but a simple grep should handle it - basically you're just looking for 4 space separated zeroes at the end of each line ("....0......0......0......0$") in order to decide if the count should be incremented or reset.

It would be nice if the disk timeout could be extracted from the config database, but each disk has it's own timeout in which case this would only make sense if it were possible to determine which drive belongs to which pool, and maintain individual counts for each disk...

I knocked up this quick script, which seems to work for my setup (my pool is called "share", and I have a working zdb cachefile which is important!). It's not great, and might be more efficient if written entirely in perl or awk, but it does the job and shouldn't tax any system even as is.

Code:

#!/bin/sh
#set -xv

#VERSION: 0.1
#
#AUTHOR:  Milhouse
#
#DESCRIPTION: Created on FreeNAS 8.0.1-BETA4 with LSI 9211-8i HBA
#             Tested on FreeNAS FreeNAS 8.0.1-BETA4 with LSI 9211-8i HBA
#
#  This script will attempt to stop disks in a ZFS pool connected
#  by a SAS HBA controller, using the camcontrol command. In effect,
#  this is a poor mans ATAIDLE.
#
#  The script is designed to run as a "daemon" process, running
#  constantly from bootup. The idea is that it will consume standard
#  input, which will itself be the output of "zpool iostat n" where n
#  is a time interval, usually 60 (seconds).
#
#  A count (--timeout) is specified as an argument for the script, and this
#  quantity is decremented each time there is no iostat activity, and reset
#  whenever there is iostat activity. When the count reaches zero, the disks
#  will be stopped using "camcontrol stop" (unless the --test option is
#  specified).
#
#  The amount of time elapsed before disks are stopped is calculated as
#  the product of the timeout and the value n specified for zpool iostat.
#  For example, a timeout of 15 and an interval of 60 gives 15 * 60, or
#  900 seconds (ie. 15 minutes). The default timeout is 30.
#
#  By default, disks will be stopped asynchronously (ie. simultaneously)
#  however if this is a problem, specify the --sync option to stop disks
#  in sequence, one after the other.
#
#  If a system has multiple pools, run individual commands for each pool.
#
#  If the script is unable to automatically detect disks/devices, specific
#  devices can be specified using the --devices argument.
#
#CHANGELOG:
#
#  0.1: 2011-08-01  - Initial version
#
#
#USAGE:
#  Execute the script with the -h or --help arguments.
#
#EXAMPLES:
#   zpool iostat tank 60 | sh scsi_standby.sh --timeout 15
#   zpool iostat tank 60 | sh scsi_standby.sh --timeout 15 -devices "/dev/da[2-3]"
#   zpool iostat tank 60 | sh scsi_standby.sh --timeout 15 --sync
#   zpool iostat tank 60 | sh scsi_standby.sh --timeout 15 --test --verbose
#

TIMEOUT=30
DEVICES=
ASYNC=Y
DOSTOP=Y
DEBUG=
QUIET=N

while [ $# -gt 0 ]; do
	case ${1} in
		"-t" | "--timeout")	shift 1; TIMEOUT=${1};;
		"-s" | "--sync")	ASYNC=;;
		"-d" | "--devices")     shift 1; DEVICES=${1};;
		"-x" | "--test")	DOSTOP=;;
		"-q" | "--quiet")	DEBUG=; QUIET=;;
		"-v" | "--verbose")	DEBUG=Y; QUIET=;;

		"-h" | "--help" | *)
echo "usage: ${0} [-t --timeout #] [-v --verbose] [-h --help]
-t --timeout   timeout delay (default 30)
-s --sync      stop disks synchronously (default, async)
-d --devices   override device detection by specifying devices eg. "/dev/da[0-4]"
-x --test      do not stop disks (simulation)
-q --quiet     supress all informational and debug messages
-v --verbose   output debug messages
-h --help      display this help";
exit;;
	esac
	shift 1
done

STOPPED=
let COUNT=${TIMEOUT} >/dev/null

# Show config info...
if [ "$DEBUG" -o "$QUIET" ]; then
	echo "`date +'%Y/%m/%d %H:%M:%S'` $0 starting"
	echo

	echo "--------------------------------------------------"
	echo "Timeout interval: ${TIMEOUT}"

	echo -n "ASync Enabled:    "
	[ ${ASYNC} ] && echo "Yes" || echo "No"

	echo -n "Simulated Stop:   "
	[ ${DOSTOP} ] && echo "No" || echo "Yes"

	[ "${DEVICES}" ]&& echo "Pool Devices:    " ${DEVICES}
	[ ! "${DEVICES}" ]&& echo "Pool devices will be determined automatically"

	echo "--------------------------------------------------"
	echo
fi

# Skip 3 lines of "zpool iostat" headers...
for H in 1 2 3; do
	read HEADER
	[ "$DEBUG" ] && printf "%.3d: %s\n" $COUNT "$HEADER"
done

# Main infinite loop...
while [ true ]; do
	read POOL_NAME POOL_USED POOL_AVAIL POOL_OP_READ POOL_OP_WRITE POOL_BW_READ POOL_BW_WRITE

# Determine non-gptid devices if not already known...
	if [ ! "$DEVICES" ]; then
		[ "$DEBUG" -o "$QUIET" ] && echo "Identifying devices for pool \"${POOL_NAME}\"..."

		MYCOMPONENTS=`gpart status -s | sed "s/^ //" | sed "s/  / /g" | cut -d" " -f3 | sort -u`

		MYGUIDS=`zdb -C ${POOL_NAME} | grep "path" | sed "s/.*path='//" | sed "s/p2//" | sed "s/'//" | sed "s#/dev/gptid/##"`

# Extract any non-gptid devices that may be in use...
		DEVICES=`echo "$MYGUIDS" | grep "^/dev"`
# Leaving only gptids for further processing
		MYGUIDS=`echo "${MYGUIDS}" | grep -v "^/dev"`

# Now, for each gpt component (disk), attempt to identify it's corresponding gptid
		for COMPONENT in ${MYCOMPONENTS}; do
			for GUID in ${MYGUIDS}; do
				if [ "`gpart list ${COMPONENT} | grep \"rawuuid: ${GUID}\"`" ]; then
# Add the matched component to the list of devices
					[ ! -z "${DEVICES}" ] && DEVICES="${DEVICES} "
					DEVICES="${DEVICES}/dev/${COMPONENT}"

# Remove the matched GUID so we don't process it again
					MYGUIDS="`echo ${MYGUIDS} | sed "s/${GUID}//"`"

					break
				fi
			done
		done

# Eliminate any duplicates, stop disk in sequence...
		DEVICES=`echo ${DEVICES}| tr -s " " "\n" | sort -u`

		[ "$DEBUG" -o "$QUIET" ] && echo "Pool Devices:" ${DEVICES}
	fi

# If no activity, decrement count, else reset it
	if [ ${POOL_OP_READ-1} = 0 -a ${POOL_OP_WRITE-1} = 0 -a \
	     ${POOL_BW_READ-1} = 0 -a ${POOL_BW_WRITE-1} = 0 ]; then
		let COUNT=COUNT-1 >/dev/null
	else
		let COUNT=${TIMEOUT} >/dev/null
		STOPPED=
	fi

	[ "${DEBUG}" ] && printf "%.3d: %-10s  %5s  %5s  %5s  %5s  %5s  %5s\n" \
		${COUNT} ${POOL_NAME} ${POOL_USED} ${POOL_AVAIL} \
		${POOL_OP_READ} ${POOL_OP_WRITE} ${POOL_BW_READ} ${POOL_BW_WRITE}

# If count reaches zero, stop devices
	if [ ${COUNT} -le 0 ]; then
		let COUNT=1 >/dev/null

		if [ ! "${STOPPED}" ]; then
			[ "$DEBUG" -o "$QUIET" ] && echo "`date +'%Y/%m/%d %H:%M:%S'` ** Stopping devices in pool \"${POOL_NAME}\" **"
			
			for DISK in ${DEVICES}; do
				if [ "$DOSTOP" ]; then
					[ "$DEBUG" -o "$QUIET" ] && echo "camcontrol stop ${DISK}"
					[   "${ASYNC}" ] && camcontrol stop ${DISK} &
					[ ! "${ASYNC}" ] && camcontrol stop ${DISK}
				else
					[ "$DEBUG" -o "$QUIET" ] && echo "#camcontrol stop ${DISK}"
				fi
			done
			STOPPED=Y
		fi
	fi
done

EDIT: See post #79 for the latest version of the above script.

Call it "scsi_standby.sh", then to run it using the following command which will check every 1 second (the zpool interval) and wait 10 zpool intervals before stopping the disks in the appropriate pool (as determined from zdb):

Code:

zpool iostat share 1 | sh /tmp/scsi_standby.sh -t 10 -v

Obviously in a "production" environment the following would be more appropriate (30 x 1 minute intervals):

Code:

zpool iostat share 60 | sh /tmp/scsi_standby.sh -t 30

If you have multiple pools you'll need to run the above command once for each pool.

 

[Milhouse]

The ideal solution would now be a way to schedule a script to run at startup - this is possible with modern versions of cron (in particular, Vixie cron that ships with Ubuntu etc.), by specifying a time of "@reboot" but this probably isn't supported in FreeBSD and certainly not by FreeNAS (yet).

 

[joeschmuck]

You can't do that with Cron via the GUI on FreeNAS? It has periodicity selection of the job. Please keep in mind that I'm not familiar enough with BSD or any other unix/lunix variant.

 

[Milhouse]

You can't do that with Cron via the GUI on FreeNAS? It has periodicity selection of the job.

Definitely not in the GUI, and probably not supported by the version of FreeBSD cron.

Let me know if the script is any good for you. I've realised I switched my drives from gptid to "normal" labels which may not have been such a great idea (especially as I haven't been able to revert back to gptid...) so if you've still got a system using gptid's it would be interesting to know if the script works or if it needs a little bit more attention!

 

[joeschmuck]

Hopefully someone else can help out here. I'm headed out on vacation for 2 weeks in a few days so I'm busy getting ready for vacation. Crazy times. If I get time I'll test it out otherwise I hope someone else can do it. Since my FreeNAS box is not final, I have changed back and forth the gptid but after getting an email from the Devs about why I shouldn't use it, I'm not using it anymore. They use it in their code so switch back if you can.

 

[Milhouse]

Since my FreeNAS box is not final, I have changed back and forth the gptid but after getting an email from the Devs about why I shouldn't use it, I'm not using it anymore.

I switched back, but when I run "zpool status" or "zpool iostat -v" I still see non-gptid devices listed even though I've switched back by removing the gptid entries from /boot/loader.conf... how did you "switch back" as mine seems stuck (not the end of the world, like yours my box is also not final).

 

[joeschmuck]

I switched back, but when I run "zpool status" or "zpool iostat -v" I still see non-gptid devices listed even though I've switched back by removing the gptid entries from /boot/loader.conf... how did you "switch back" as mine seems stuck (not the end of the world, like yours my box is also not final).

Reformatted the drives. Not something most people like to do but I have only a little over 1TB on the drives which is also on my primary NAS. Like I said, my FreeNAS box is strictly for testing and I won't move over to it until I'm certain it's stable.

 

[pauldonovan]

I switched back, but when I run "zpool status" or "zpool iostat -v" I still see non-gptid devices listed even though I've switched back by removing the gptid entries from /boot/loader.conf... how did you "switch back" as mine seems stuck (not the end of the world, like yours my box is also not final).

I'm not sure you can. At least, I had the same issue and haven't solved it. My devices changed back to adaX when I temporarily imported the pool into ZFSguru, because it has the settings in loader.conf to disable gptids. On import back into FreeNAS they stayed like that. I'm not too concerned myself as I only have two drives and don't have room for more! I will use the serial numbers if I need to change one

Paul

 

[Milhouse]

I couldn't fix it either, so blew the entire volume away and am currently in the process of transferring my data. :)

I have updated the script in post #27 to v0.1, and it should now work with zpools consisting of gptid and/or non-gptid devices.

If anyone can test v0.1 I'd be grateful for any feedback (or even patches!) If the script works, please specify your controller and I will include it in a "tested" section; if the script fails, please attach the error and also the output of "zdb | grep path" (as the method used for device detection is most likely the reason for any failures!)

Finally, does anyone have a suggestion as to how this script can be scheduled to run at startup (after all ZFS pools are imported and online)?

 

[sonisame72]

I am using latest freenas 8.01 beta, I am seeing similar issues when I do
ataidle -S 5 /dev/ada0
ataidle -S 5 /dev/ada0

I see no pass/fail confirmation, similar with camcontrol command issus no confirmation

All my harddrives are connected to MB sata ports and I do have them configured to idle after 5 mins through webGui also.

How do I confirm that harddrives do spindown after some inactivity..

sonisame

 

[joeschmuck]

I am using latest freenas 8.01 beta, I am seeing similar issues when I do
ataidle -S 5 /dev/ada0
ataidle -S 5 /dev/ada0

I see no pass/fail confirmation, similar with camcontrol command issus no confirmation

All my harddrives are connected to MB sata ports and I do have them configured to idle after 5 mins through webGui also.

How do I confirm that harddrives do spindown after some inactivity..

sonisame

If you are using the -S parameter they should spin down immediately. Place your hand or ear on the drive you are commanding to spin down, you should notice it. Also the drive will spin right back up if it needs to. Please note that some add-on cards do not relay this command properly but if you're using the MB connectors it should work.

If you have it set via the GUI, the machine must be rebooted because the developers have not made the changes to set this immediately. I have a ticket is for this change.

 

[sonisame72]

If you are using the -S parameter they should spin down immediately. Place your hand or ear on the drive you are commanding to spin down, you should notice it. Also the drive will spin right back up if it needs to. Please note that some add-on cards do not relay this command properly but if you're using the MB connectors it should work.

If you have it set via the GUI, the machine must be rebooted because the developers have not made the changes to set this immediately. I have a ticket is for this change.

I did reboot the machine, but for some reason it seems like the drives are always spinning i.e. I can access the data sitting on them within a second or so. Maybe the swap space on each of drive is getting used, leaving them on all the times.

I will see how it goes or else I will look into shutting my server down each night and turning it back up in the morning using WOL to save on power consumption and extend my green drives lives.

 

[Durkatlon]

I am using latest freenas 8.01 beta, I am seeing similar issues when I do
ataidle -S 5 /dev/ada0
ataidle -S 5 /dev/ada0

I see no pass/fail confirmation, similar with camcontrol command issus no confirmation

All my harddrives are connected to MB sata ports and I do have them configured to idle after 5 mins through webGui also.

How do I confirm that harddrives do spindown after some inactivity..

sonisame

I posted a solution to check spindown using camcontrol commands, here:

http://forums.freenas.org/showthread.php?2068-How-to-find-out-if-a-drive-is-spinning-down-properly.

 

[scsi-toaster]

Big thank you to Milhouse for your Script!
I have a Intel SASUC8i controller with 7 discs attached and the script works fine for me. Is there a solution for autostart of this script yet?