[Low Priority] SSD caching as long as possible, before writing to disk (power saving)

[diskdiddler]

Hi all,

Hope the subject covers it.
In the next 36 months (depending on the budget) I'm probably going to re-build a new FreeNAS system from scratch, likely with FreeNAS 10, intel CPU, 16gb of ECC, nice new case.

I'm curious if I build a machine for example with 3 to 6 physical HDD's and say 2 to 3 SSD's. Is it possible to configure the system so that ALL writes to the system are on the SSD and the drives are only spun up for reads? Is there a way to configure jails, other such stuff so the disks are idle far more often?

I completely realise there's been discussion about disk spin down not being recommended but perhaps with sufficiently sized SSD's we can work around this? Understandably the complexity is significantly more for the system but I'd love to keep power usage lower, disk utilisation and heat lower.

Any thoughts on this, maybe not for now but for the long term?
(it hits 91.4f in my apartment in summer..... indoors!) :/

 

[cyberjock]

The solution if you don't want disks to spin is to not buy disks that spin to start with. You cannot stop writes from going to the pool. You can set horribly high limits on transactions (default is 5 seconds), but ultimately what is in the slog *must* be written to the main pool. It happens on the next transaction, which you could set to like 10 minutes, but you'd create other problems that would "constipate" ZFS.

There's just no way to avoid spinning disks from spinning except to not use them. Assuming that your reads would be entirely within the ARC/L2ARC you could definitely run with a pool that isn't spinning. But the second you need even one byte of data from the pool the entire pool would spinup to handle that one byte.

 

[diskdiddler]

I understand the entire pool would need to spin up to read a byte but say for example, my jails are configured to be on the SSD's, FreeNAS is running from the USB disk, is there a way for me to reduce those spin ups as it currently stands? Even if that means some kind of path substitution or multiple volume / pools (one for SSD, one for the HDD?)

When I sleep for 8 hours or go on holiday for 5 days as I just did, it would be nice if sickbeard, utorrent etc did their thing to the SSD's - the disks only spin up when my weekly schedule backup and weekly shedule disk index runs?

 

[cyberjock]

Well yes, that would work. But as I saw on a friend's box when we were experimenting, if you open up Windows Explorer and you have a network share mapped to the pool, the pool spins up because of that "access" from your workstation. So it's not exactly possible to make sure the pool is asleep the whole time. Then you have things like reporting data requests and nightly maintenance that might spin the disk up. You'd have to work incredibly hard to get it to go to sleep and stay asleep. And all it takes is "that one thing" to run all the hard work. It requires perfect caching of all the data you'll need while you are away.

 

[diskdiddler]

Well yes, that would work. But as I saw on a friend's box when we were experimenting, if you open up Windows Explorer and you have a network share mapped to the pool, the pool spins up because of that "access" from your workstation. So it's not exactly possible to make sure the pool is asleep the whole time. Then you have things like reporting data requests and nightly maintenance that might spin the disk up. You'd have to work incredibly hard to get it to go to sleep and stay asleep. And all it takes is "that one thing" to run all the hard work. It requires perfect caching of all the data you'll need while you are away.

I'm cool with a spinup or two a day - but 24/7 across 6 disks adds up in heat, noise and $$$. Even if explorer fires up the disks at least there would be up to 8 hours downtime at night

I believe the OS as it stands now already supports some basic SSD caching features right?

Bonus question, if I migrate to 9.3 and install some kind of large, fast, reliable, USB2 drive to the system (I have 3 ports free, no SATA :( ) could I consider making these changes under 9.3 now?

 

[cyberjock]

Nothing with USB is particularly reliable except as booting since it uses RAM after bootup. slogs and L2ARCs are a fail for USB.

 

[diskdiddler]

I'm talking about purely expendable data, like /download/ and some logs - if it gets lost I won't be too upset, I could even consider some kind of RAID 1 set with 2 mid range 16/32gb USB sticks?

 

[cyberjock]

I'm talking about purely expendable data, like /download/ and some logs - if it gets lost I won't be too upset, I could even consider some kind of RAID 1 set with 2 mid range 16/32gb USB sticks?

Honestly, if it's so unimportant you want to put it on 'expendable' media, then just delete it. It'll be lost forever the first time there's anything except a perfect disk with perfect power and perfect data signal from the computer to the drive. :P

 

[diskdiddler]

Can someone besides Cyberjock who is 2 shades less paranoid please respond?

 

[cyberjock]

Can someone besides Cyberjock who is 2 shades less paranoid please respond?

I do have to "LOL" at that one. It's not paranoia when it happens regularly.

 

[zambanini]

do not mess with cyberjock, he has the ultimative killswitch ; )

if your data is that unimportant, buy some terrabyte ssds, or create a high ramdisk and do rsync to disks, rofl. oh, there is the part with the budget, well..your problem.

 

[mjws00]

Just build an ssd pool first. .system will end up there. Build the second pool with your spinners, and move the data from pool1 when you feel like it. You might get the odd extra spin-up for maintenance etc... but pool2 isn't going to just keep spinning forever no matter what. It's the manual version of your cacheing idea. Put it in a script with a cron job if you want to automate.

Spinning drives down isn't a bad thing if the intervals are long enough. The problem is it isn't easily done with one pool, when we need logs, persistent storage, samba, etc.

I'd heartily recommend a pool like this SUN video. Grab a big ole usb hub(s) and knock yourself out. Then make sure you let everyone know it works and they should do it as well. See if you can make cyber's head spin right around. ;)

 

[Ericloewe]

Nothing says campy like a bunch of German geeks filling in for Sun's PR department and then dubbing the video over in Denglisch.

Hey, this could be a sport. Who can build the craziest pool using only USB hubs and flash drives?

 

[jgreco]

Nothing says campy like a bunch of German geeks filling in for Sun's PR department and then dubbing the video over in Denglisch.

Hey, this could be a sport. Who can build the craziest pool using only USB hubs and flash drives?

I know a guy who did that. He had a bunch of free flash drives he got from trade shows.

 

[Glorious1]

I've read many times here about the advantages of leaving drives spin 24/7. It's not that I don't believe, but I can't help myself, I want to see if getting the drives to spin down and stay down (except when someone accesses the pool of course) is an option.

mjws00 gave an important clue - make the first pool the SSD or whatever, then the system will use that for its frequent access. I have a small second pool on one disk so I tried setting that disk to Standby in 10 minutes and Advanced Power Management to 127. Sure enough, it stayed spun down indefinitely as long as noone logged into the server. (If they did, everything spun up, even if they were logging into a dataset in the other pool)

Well it's too late for that trick for my main pool. So I tried the System Dataset settings. I had an SSD in a USB enclosure left over from upgrading one, plugged that in, and created a pool on it. In System > System Dataset, I assigned that pool to the System, checking the syslog and reporting database boxes. However, even after rebooting, that drive is empty, the system isn't using it.

BUT, now ALL the disks are spinning down and so far staying that way (though some take a bit longer than the 10 minutes I set for testing). I don't understand where the system is now putting stuff (maybe it's cached and will be written to the SSD later?).

Almost forgot, for this to work, you also have to set the SMART setting Power mode to Standby. Meaning SMART then won't spin up a drive to check it. Oh, and if you use Time Machine, since it automatically backs up every hour, that's obviously going to spin everything up. Another reason for me to not actually do this except as an experiment.

By the way, here is the script I made to check the disks based on other scripts around. It prints a header row of all the ada* drives, then every minute, prints the time and lists Spinning or Standby under each disk. It's a bash script. I gather csh is the popular shell in FreeNAS so maybe it won't be useful, but it works great for me.

Code:

#!/usr/local/bin/bash
# spincheck.sh, checks every minute to see if disk is in standby.
# Must run as root.
# In output of:
# camcontrol cmd $LINE -a "E5 00 00 00 00 00 00 00 00 00 00 00" -r -
# the 10th hex number changes from FF to 00 when disk is spun down"

echo
echo -n "          "
camcontrol devlist | awk -F\, '{print $2'} | awk -F\) '{print $1}' | while read LINE
do
     if [[ $LINE != "ada"* ]] ; then
         break
     fi
     echo -n "$LINE      "
done
echo

while [ 1 ]
do
TIME=$(date "+%H:%M:%S")
echo -n "$TIME  "
camcontrol devlist | awk -F\( '{print $2'} | awk -F\, '{print $1}' | while read LINE
do
        if [[ "$LINE" != "pass"* ]] ; then
            break
        else
        CM=$(camcontrol cmd $LINE -a "E5 00 00 00 00 00 00 00 00 00 00 00" -r - | awk '{print $10}')
            if [ "$CM" = "FF" ] ; then
                echo -n "Spinning  "
                elif [ "$CM" = "00" ] ; then
                echo -n "STANDBY   "
            else
                echo -n "UNKNOWN   "
            fi
        fi
done
echo
sleep 60
done

 

[cyberjock]

The system dataset doesn't mount in .system anymore. Too many people were being silly with permission management and doing the whole pool, which in turn screwed up permissions for the files inside .system. This rendered the server unusable. So now .system actually mounts in /var/db/system. If you do a "df" from the CLI you'll see the mountpoint.

 

[Glorious1]

The system dataset doesn't mount in .system anymore. Too many people were being silly with permission management and doing the whole pool, which in turn screwed up permissions for the files inside .system. This rendered the server unusable. So now .system actually mounts in /var/db/system. If you do a "df" from the CLI you'll see the mountpoint.

OK, yes, I see that. So physically where would /var/db/system be? On the boot device? Or in the first pool created?
Does this mean that System Dataset settings don't do anything anymore?
Sorry, just learning my way around here and don't really know what system dataset and .system are.

 

[cyberjock]

It's on the pool you select. The dataset is simply mounted in /var/db/system. The dataset that /var/db/system points to is on whatever pool you choose from the WebGUI.

 

[IanWorthington]

Nothing says campy like a bunch of German geeks filling in for Sun's PR department and then dubbing the video over in Denglisch.

Hey, this could be a sport. Who can build the craziest pool using only USB hubs and flash drives?

Err... Isn't that rather like the mirrored USB stick boot environment in 9.3...?

 

[Ericloewe]

Err... Isn't that rather like the mirrored USB stick boot environment in 9.3...?

Well, next time asks you whether using USB drives for booting is a good idea, you can point and say "SUN's engineers told me it was safe!"