So, hard drives don't spin down now?

[allanonmage]

I'm setting up TrueNAS and looking around at the settings, and when I went to see about a power down time for the drives, I was linked to this post:

https://forums.freenas.org/index.php?threads/how-to-find-out-if-a-drive-is-spinning-down-properly.2068/

Which is 12-13 years old, locked, and has a few people saying that their drives don't spin down when idle, some even recently. Given there are multiple scripts available on Github, it looks to me like the fix is to run some external code. Am I reading that right? Are these problems edge cases, or a general problem because TrueNAS assumes datacenter conditions?

In reading some other forum posts, lots of people are saying that spinning them down wears them out faster. Which on it's face is counterintuitive. Reading some forum posts, those opinions seem to be infrequently qualified by "many times per day". Has anyone done a study to explain this? For a home NAS that gets used maybe a few hours a day and sits idle most of the time, it would be advantageous to spin the disks down to save power and heat, which seems to be the use case for most users asking for spin downs. Drives have spun down on the desktop when not in use for many years, so I would think that if such a bold claim would have ample research to back it up. I would also like the understand why people think a drive spinning 24/7 is not bad for a drive. Anything with a motor should spin down when not in use to extend the life of said motor.

From this post: " Feb 3, 2016


What's more likely is that some dev inadvertently enabled something that's touching the pool. Since a TrueNAS system never spins down its disks and a FreeNAS system should never do so either, I'm guessing it's more a matter of "no one's noticed" plus "no one's that interested in looking into it.""

If this were the case, why would there be multiple UI elements that say they control these things? That is again counterintuitive, and later posts in that same thread say as much. Ultimately the thread ends up with "run my scripts on Github" though.

 

[Davvo]

I would also like the understand why people think a drive spinning 24/7 is not bad for a drive.

Because NAS/enterprise drives are manifactured with that use in mind... as well as the fact that maintaining a state of motion requires less force/energy than overcoming the inertia of the platters in order to spin them up.

Anything with a motor should spin down when not in use to extend the life of said motor.

Not really. The issue is not spinning them down, but spinning them up: doing so might stress the magnets of the electric motor, thus resulting in a shorter (than what it could have been) life.

The following thread's post by @joeschmuck has a more comprehensive explaination. There is also another thread, again with joe input about the argument, that I can't seem to recall.

Multi_Report Critical Temp 17*C

Installed Multi_Reports and the first tests on sending emails I keep getting the same critical error for HDD temp at 17*C, this also occurred when the temp was at 23*C. O' the 17*C is after a cold boot and the 23*C was when running for a while. SMART overall-health self-assessment test...

www.truenas.com

Anyway, I don't understand what's your question... so here a few FAQs:

Can you spin down drives in TN? Yes.
Is it advisable? Under certain workloads/conditions, yes.
Is that going to reduce the drives' potential/extra lifespan if done responsably? Yes.
Is that going to reduce the drives' normal lifespan if done carelessly? Yes.
Why don't my drives spin down? You have to move the system dataset out of the HDD pool.
Are there useful scripts that can help spindown drives? Yes, look the following resource.

HDD Spindown Timer

Disk spindown has always been an issue for various TrueNAS / FreeNAS users. This script utilizes iostat to detect I/O operations (reads, writes) on each disk. If a disk was neither read nor written for a given period of time, it is considered...

www.truenas.com

Next time, please search in the active parts of the forum before thinking of a dev's mistake.

 

[joeschmuck]

I would agree that some users may benefit from spinning down their drives but you need to know how many time your drive spins up over a period of time and then make a judgement call. No one will stop you from spinning down your drives and we are just trying to provide good sound advice as we know it today.

Has anyone done a study to explain this?

That would be nice for someone to do but I'm not aware of one. A good study would need to take into account several different make/models of drives over the course of time. But the only real time that matters is the warranty. If you bank on that then you have no issues at all, but most of use would like to extend the drive life well beyond the warranty, myself included, which is why my drives are still running with over 43,000 hours as of last night. That feels good to say for a 3 year warranty drive. Start/Stop count is 235 times, Load Cycle Count is 2060. My system is not accessed often either, maybe once a day but on the weekends, quite often.

Best of luck to you no matter which path you take.

 

[allanonmage]

I kept reading around last night, and from what I can tell, years ago people would qualify their advice as "You should not spin up/down your drives too much, for example tens or hundreds of times per day. It's not really the spin up that matters, but the temperature changes associated with changing the spinning state" (basically what joe says in the link above), and that seems to have morphed into "dRiVe nO StOp sPiNnInGg nEvErRrRrR". Spongecase for comedic effect, mostly.

In a datacenter, I can totally see the benefit of leaving the drives spinning all the time. Even workplace servers in some capacities. However, for SOHO, home users, and "low performance file storage", those use-cases would benefit for a 1 or 2 hour idle time to then spin down the drives. In those cases, the next spin up might be 3 - 12 hours later. But only if the OS actually spins them down.

My confusion stems/stemmed from:

1. Many people asking about power management say that the GUI doesn't work.
2. Some people implying that the GUI is broken by design or neglect.
3. No one ever complained that it worked, or even said "it works for me".
4. Many people over the years had/have to run special scripts to get their drives to spin down. Some never figured out how to get their drives to spin down.
5. People staunchly saying that you should never spin down a hard drive, which may make sense in a datacenter, but makes almost no sense anywhere else. The strange part is that I could find no basis for such an opinion, let alone such a staunch one. "Datacenter drives are made that way" is the closest I could find, which seems too general of a reason to me.

I found the spec sheet/product manual (PDF) for the WD datacenter drives that I have, and they are ~5W in use, and ~1W when spun down, with several power states in between. Since the UI for power management is not self evident, which I and others have asked for clarification about, and the fact that people say it doesn't work, the situation looks like a mess, which is why I'm confused.

Is there a difference between idle the common word, and "idle" the technical word as it pertains to hard drive power states and TrueNAS? For example, does TrueNAS put the drive into a spinning-but-idle or spinning-but-low-power state? Or does it keep the drive in its highest powered state? Is that what the not-completely-self-evident power management options are trying to explain?

I've seen a few people mention moving the system dataset to actually achieve a HDD spin down. Isn't that on the boot device? Does it somehow end up on the HDD array?

The difference between 5W and 1W is not a lot, but it adds up when you start multiplying by the number of drives. I have 6 drives right now, and a chassis that will accommodate 15. One day I'd like to max it out. Today, that's a difference of 24W (6W min, 30W max), but eventually it will be 60W (15W and 75W, min and max), which isn't huge, but is enough to make a difference, especially when it's always consuming that power.

 

[joeschmuck]

I laughed when I read your posting and you are correct on many points. That is why I said it is a judgement call, yours to be exact. Many drives will accept the APM power states and some will ignore it and TrueNAS supports this from the GUI, and as far as I'm aware, the commands are sent to the drives properly. This doesn't mean the user gets the desired effect. Depending how your system is configured you may spin your drives down but a few minutes later something happens to make the drives wake up, and it doesn't need to be the System Dataset, which as you have read must be moved off the pool, and yes, by default the System Dataset is created on the pool, BUT there is always an exception. I understand that if your boot drive is 64GB, the System Dataset will be created on the boot drive. That is a very nice feature and that is a fairly new change, well new to me since I only heard about it a few months ago. Also it could be as simple as another computer on your network asking for a status of a shared drive. There are ways to troubleshoot this if you ever have this kind of problem. This is why I recommend you monitor your spinup/stop count so you know exactly what the drives are doing.

For people concerned about power consumption it may make a lot of sense to sleep the drives, or just where power costs a lot of money.

I don't think you will get any arguments out of me about sleeping your drives, it's a personal decision.

But your original posting is a lot of statements of multiple things. but basically complaining about TrueNAS or how many of us users promote constant running (case dependant). There are ways to sleep drives and you may need to use a script to achieve this. iXsystems goal (I don't really want to speak for them) is to sell their product to businesses and turn a profit. They target data centers where drives are apt to run full time. While TrueNAS is free to us, we are the Beta testers for their product. They did include some APM features but the home user may need to take a few steps to make it work as one would expect.

Read the User Guide, I think it will help you out with APM settings.

As I said, if you do sleep your drives, check your SMART data over a period of time to ensure you have the desired effect. When drive sleeping kicked off in FreeNAS, some folks found that they induced failures because the drives were sleeping about 5 minutes and then spinning up again, over and over. So monitor the data. With any luck you will be able to sleep your drives to your satisfaction.

 

[sfatula]

Ok, so, I have an external USB drive hooked up 24/7 that spins up for a backup and then pool is exported and it spins down. The Truenas UI does this, the spin down. It is definitely not spinning. So, works for me.

 

[ChrisRJ]

People staunchly saying that you should never spin down a hard drive, which may make sense in a datacenter, but makes almost no sense anywhere else. The strange part is that I could find no basis for such an opinion, let alone such a staunch one. "Datacenter drives are made that way" is the closest I could find, which seems too general of a reason to me.

You are entitled to your opinion. But making a broad claim (not spinning-down is unreasonable outside the data center) will make those people with multiple decades of professional experience "smile and nod". What is your level of experience with hardware and storage in particular?

 

[allanonmage]

I laughed when I read your posting and you are correct on many points. That is why I said it is a judgement call, yours to be exact. Many drives will accept the APM power states and some will ignore it and TrueNAS supports this from the GUI, and as far as I'm aware, the commands are sent to the drives properly. This doesn't mean the user gets the desired effect. Depending how your system is configured you may spin your drives down but a few minutes later something happens to make the drives wake up, and it doesn't need to be the System Dataset, which as you have read must be moved off the pool, and yes, by default the System Dataset is created on the pool, BUT there is always an exception. I understand that if your boot drive is 64GB, the System Dataset will be created on the boot drive. That is a very nice feature and that is a fairly new change, well new to me since I only heard about it a few months ago. Also it could be as simple as another computer on your network asking for a status of a shared drive. There are ways to troubleshoot this if you ever have this kind of problem. This is why I recommend you monitor your spinup/stop count so you know exactly what the drives are doing.

For people concerned about power consumption it may make a lot of sense to sleep the drives, or just where power costs a lot of money.

I don't think you will get any arguments out of me about sleeping your drives, it's a personal decision.

But your original posting is a lot of statements of multiple things. but basically complaining about TrueNAS or how many of us users promote constant running (case dependant). There are ways to sleep drives and you may need to use a script to achieve this. iXsystems goal (I don't really want to speak for them) is to sell their product to businesses and turn a profit. They target data centers where drives are apt to run full time. While TrueNAS is free to us, we are the Beta testers for their product. They did include some APM features but the home user may need to take a few steps to make it work as one would expect.

Read the User Guide, I think it will help you out with APM settings.

As I said, if you do sleep your drives, check your SMART data over a period of time to ensure you have the desired effect. When drive sleeping kicked off in FreeNAS, some folks found that they induced failures because the drives were sleeping about 5 minutes and then spinning up again, over and over. So monitor the data. With any luck you will be able to sleep your drives to your satisfaction.

OK, cool, so it is supposed to work. That's good info. Sorry for complaining a too much

You are entitled to your opinion. But making a broad claim (not spinning-down is unreasonable outside the data center) will make those people with multiple decades of professional experience "smile and nod". What is your level of experience with hardware and storage in particular?

Enh, I only have 30 years or so of experience with computers, but it is effectively the maximum I could have for my age. I started with MFM hard drives way back in the day.

On a related note, my datacenter drives come with a fancy new "feature" that if you apply voltage to the 3.3V pin, they power down regardless of what you tell them to do. I need a special adapter that removes the 3.3V pin from an ATX power supply so that they will spin up at all.

 

[joeschmuck]

I started with MFM hard drives way back in the day.

Me to, and I use to perform head alignments of MFM and RLL drives. Those were very easy because they were stepper motors and you basically made a fine adjustment of the stepper motor until the heads and tracks were putting out peak signal. Also back in those days you could low level format a drive if your data was going to be deleted. One that was more challenging was a complete rebuild of a modified for submarine use IBM 2316 (ruggedized). See photos for what the disk pack looked like (14" wide, 11 platters and the bottom one was not used, it was support for the sector disk which is the aluminum disk at the bottom and it has slots cut all the way around it, and two slots close together to mark the start of the disk rotationally.) The heads were moved hydraulicly, it was hard sector, 100 cylinders, 20 heads (these were large at almost 1" wide and they did float on air, most of the time ), and so many people had a difficult time tearing one apart and putting it back together for refurbishment every ~90 days. We had two and one would be reworked before every patrol. I was the lucky dog because I actually understood how it worked (all the flip flops and the state they were in (lots of damn flip flops), and O'scopes for me were easy tools and I was mechanically inclined, so easy work. It took a 12 hours day to completed the maintenance, if you didn't cut any corners. The hard part was really one thing, the alignment of the heads to the center of the spindle, it had to be exactly parallel in motion while the heads moved in/out of the pack. A very slight offset meant some of the tracks would not produce adequate signal because the were not perpendicular to the disk tracks. We had the smallest hydraulic pump on the submarine. That was a question everyone was asked for submarine qualifications. I miss those days. Sorry about the sea story.

On a related note, my datacenter drives come with a fancy new "feature" that if you apply voltage to the 3.3V pin, they power down regardless of what you tell them to do. I need a special adapter that removes the 3.3V pin from an ATX power supply so that they will spin up at all.

Is that a new feature? Are you talking about the Pin 3 issue? That isn't new, it's been around for a long time actually, at least 5 years if not longer. If you are good with a piece of tape, you can cover the pin and disrupt that 3.3 volt power delivery. It's not pretty but it does work as a good temporary fix. I don't recall anyone other than Western Digital doing this, but I could be wrong, it has been a while and I'm an old man as my title states.

OK, cool, so it is supposed to work. That's good info. Sorry for complaining a too much

We all complain and have bad days so it's not a problem, unless it's happening a lot and it offends people. I was not offended.

 

[bugsysiegals]

FWIW - I'm running unRAID (considering TrueNAS) which allows me to write new files first to my Cache Pool (2x 4TB NVME), keep my array (3x 8TB HDD) spun down, and then spin them up only to offload files once per day which are 30 days old, refreshing my Jellyfin library, or streaming older files.

I found this thread I was looking to see if TrueNAS can work in the same way and figured I'd share my disk stats since it seems people worry about spinning down disks. These are for 8TB WD EasyStore drives shucked from the external closure. I'm no expert but considering these stats for a consumer drive it seems the whole spin-down lowers lifespan seems to be overhyped?

Power on hours: 42,031 (4y, 9m, 15d, 7h) ... Start Stop Count: 5,594 ... Load Cycle Count: 11,677

 

[Davvo]

I'm running unRAID (considering TrueNAS) which allows me to write new files first to my Cache Pool (2x 4TB NVME), keep my array (3x 8TB HDD) spun down, and then spin them up only to offload files once per day which are 30 days old, refreshing my Jellyfin library, or streaming older files.

I found this thread I was looking to see if TrueNAS can work in the same way

Sadly it cannot.

Power on hours: 42,031 (4y, 9m, 15d, 7h) ... Start Stop Count: 5,594 ... Load Cycle Count: 11,677

That's a reasonable amount though? An eyballed average of 3 spinups per day; the issue is when there are dozen(s) per day. Usually the suggestion is not to exceed 5 spinups per day.

All in all:

[...] it is a judgement call, yours to be exact.

 

[cap]

Is there a way to monitor how often a drive does a spindown?

 

[Davvo]

Is there a way to monitor how often a drive does a spindown?

Monitoring the smartctl Start Stop Count value.

 

[Deogorn]

FWIW - I'm running unRAID (considering TrueNAS) which allows me to write new files first to my Cache Pool (2x 4TB NVME), keep my array (3x 8TB HDD) spun down, and then spin them up only to offload files once per day which are 30 days old, refreshing my Jellyfin library, or streaming older files.

I found this thread I was looking to see if TrueNAS can work in the same way and figured I'd share my disk stats since it seems people worry about spinning down disks. These are for 8TB WD EasyStore drives shucked from the external closure. I'm no expert but considering these stats for a consumer drive it seems the whole spin-down lowers lifespan seems to be overhyped?

Power on hours: 42,031 (4y, 9m, 15d, 7h) ... Start Stop Count: 5,594 ... Load Cycle Count: 11,677

That is the solution I am attempting to implement with TrueNAS but for now it with no avail.

I have 2TB nvme and 2x 12TB HDD. I tried to have a single pool with the SSD as a cache but the HDDs were always active. I split them into two separate pools with a replication task and it seemed at first that it did the job, however, the HDDs spun back up around 30m after they were manually spun down with hdparm. I´ve moved the system data set to the boot pool and I can´t figure out what is accessing the disks.

I have no trouble detecting when the HDDs are spinning up and down since the system is living on my desk right next to my bed. This is the main reason I want them to work only once or twice a week... or maybe once a day and use them as an alarm clock.

I was considering trying unRAID and now that I´ve read your comment I am even a bit more optimistic.

 

[bugsysiegals]

That is the solution I am attempting to implement with TrueNAS but for now it with no avail.

I have 2TB nvme and 2x 12TB HDD. I tried to have a single pool with the SSD as a cache but the HDDs were always active. I split them into two separate pools with a replication task and it seemed at first that it did the job, however, the HDDs spun back up around 30m after they were manually spun down with hdparm. I´ve moved the system data set to the boot pool and I can´t figure out what is accessing the disks.

I have no trouble detecting when the HDDs are spinning up and down since the system is living on my desk right next to my bed. This is the main reason I want them to work only once or twice a week... or maybe once a day and use them as an alarm clock.

I was considering trying unRAID and now that I´ve read your comment I am even a bit more optimistic.

The only think that wakes my disks is the mover which you can schedule when you want it to move files from cache to array, parity checks which you can schedule, and unfortunately radar/sonar which have tasks scheduled to update metadata, etc. and you can’t control how often these run unless there’s some hack I’m unaware of. For now I’ve shut them off and am just streaming media rather than downloading and the array stays offline all day.

 

[Joe-freenas]

Because NAS/enterprise drives are manifactured with that use in mind... as well as the fact that maintaining a state of motion requires less force/energy than overcoming the inertia of the platters in order to spin them up.


Not really. The issue is not spinning them down, but spinning them up: doing so might stress the magnets of the electric motor, thus resulting in a shorter (than what it could have been) life.

The following thread's post by @joeschmuck has a more comprehensive explaination. There is also another thread, again with joe input about the argument, that I can't seem to recall.

Multi_Report Critical Temp 17*C

Installed Multi_Reports and the first tests on sending emails I keep getting the same critical error for HDD temp at 17*C, this also occurred when the temp was at 23*C. O' the 17*C is after a cold boot and the 23*C was when running for a while. SMART overall-health self-assessment test...

www.truenas.com

Anyway, I don't understand what's your question... so here a few FAQs:

Can you spin down drives in TN? Yes.
Is it advisable? Under certain workloads/conditions, yes.
Is that going to reduce the drives' potential/extra lifespan if done responsably? Yes.
Is that going to reduce the drives' normal lifespan if done carelessly? Yes.
Why don't my drives spin down? You have to move the system dataset out of the HDD pool.
Are there useful scripts that can help spindown drives? Yes, look the following resource.

HDD Spindown Timer

Disk spindown has always been an issue for various TrueNAS / FreeNAS users. This script utilizes iostat to detect I/O operations (reads, writes) on each disk. If a disk was neither read nor written for a given period of time, it is considered...

www.truenas.com

Next time, please search in the active parts of the forum before thinking of a dev's mistake.

Ha! You are *only* about 5-7 years behind.

Current (ramp-up speed/power) technology allows to spin down as MUCH AS YOU WANT, as the device is INTELLIGENT ENOUGH to NOT STRESS the components upon spin up!

But that's what happens when people that "want to know", just don't have 100% of the bases and understanding that it takes.

In this case, you do need to have an Electrical or Electronics degree to understand what you meant to pretend to talk about.

I have been spinning down 100% of my spinning hard drives (80+) for the last 12+ years and NONE have failed due to motor or electronic malfunction.
The 4, 5 of the "failed" ones, are still somewhat alive but have bad sectors, which are normal for aging drives.

Now, do come and tell us that "bad sectors happen because you spin the drives down!"...

Or even better yet:
What about all the external USB drives that DO SPIN DOWN all the time!?

You PEOPLE have ZERO LOGIC!!!

 

[Joe-freenas]

That is the solution I am attempting to implement with TrueNAS but for now it with no avail.

I have 2TB nvme and 2x 12TB HDD. I tried to have a single pool with the SSD as a cache but the HDDs were always active. I split them into two separate pools with a replication task and it seemed at first that it did the job, however, the HDDs spun back up around 30m after they were manually spun down with hdparm. I´ve moved the system data set to the boot pool and I can´t figure out what is accessing the disks.

I have no trouble detecting when the HDDs are spinning up and down since the system is living on my desk right next to my bed. This is the main reason I want them to work only once or twice a week... or maybe once a day and use them as an alarm clock.

I was considering trying unRAID and now that I´ve read your comment I am even a bit more optimistic.

Easy. The S.M.A.R.T. parameters checking is what is keeping them up.

Configure smart Power Mode = Standby.

 

[Davvo]

Ha! You are *only* about 5-7 years behind.

Current (ramp-up speed/power) technology allows to spin down as MUCH AS YOU WANT, as the device is INTELLIGENT ENOUGH to NOT STRESS the components upon spin up!

[...]

You PEOPLE have ZERO LOGIC!!!​

I'm basing my statements on manufacturers' warranty bounds and officially stated limits. I do not have an Electronics degree though, and would appreciate if you explained things instead of going around pointing and screaming (CAPS LOCK) at people.​

 

[Kailee71]

@Joe-freenas,
Until you've built up a considerable level of karma and show some humor while you come across all grinchy I suggest not shouting, and not making personal digs. Makes the atmosphere here much more tolerable.

Though I do (still) miss Jock. He did make me laugh.

 

[diversity]

I kept reading around last night, and from what I can tell, years ago people would qualify their advice as "You should not spin up/down your drives too much, for example tens or hundreds of times per day. It's not really the spin up that matters, but the temperature changes associated with changing the spinning state" (basically what joe says in the link above), and that seems to have morphed into "dRiVe nO StOp sPiNnInGg nEvErRrRrR". Spongecase for comedic effect, mostly.

In a datacenter, I can totally see the benefit of leaving the drives spinning all the time. Even workplace servers in some capacities. However, for SOHO, home users, and "low performance file storage", those use-cases would benefit for a 1 or 2 hour idle time to then spin down the drives. In those cases, the next spin up might be 3 - 12 hours later. But only if the OS actually spins them down.

My confusion stems/stemmed from:

1. Many people asking about power management say that the GUI doesn't work.
2. Some people implying that the GUI is broken by design or neglect.
3. No one ever complained that it worked, or even said "it works for me".
4. Many people over the years had/have to run special scripts to get their drives to spin down. Some never figured out how to get their drives to spin down.
5. People staunchly saying that you should never spin down a hard drive, which may make sense in a datacenter, but makes almost no sense anywhere else. The strange part is that I could find no basis for such an opinion, let alone such a staunch one. "Datacenter drives are made that way" is the closest I could find, which seems too general of a reason to me.

I found the spec sheet/product manual (PDF) for the WD datacenter drives that I have, and they are ~5W in use, and ~1W when spun down, with several power states in between. Since the UI for power management is not self evident, which I and others have asked for clarification about, and the fact that people say it doesn't work, the situation looks like a mess, which is why I'm confused.

Is there a difference between idle the common word, and "idle" the technical word as it pertains to hard drive power states and TrueNAS? For example, does TrueNAS put the drive into a spinning-but-idle or spinning-but-low-power state? Or does it keep the drive in its highest powered state? Is that what the not-completely-self-evident power management options are trying to explain?

I've seen a few people mention moving the system dataset to actually achieve a HDD spin down. Isn't that on the boot device? Does it somehow end up on the HDD array?

The difference between 5W and 1W is not a lot, but it adds up when you start multiplying by the number of drives. I have 6 drives right now, and a chassis that will accommodate 15. One day I'd like to max it out. Today, that's a difference of 24W (6W min, 30W max), but eventually it will be 60W (15W and 75W, min and max), which isn't huge, but is enough to make a difference, especially when it's always consuming that power.

I fully agree with you. I also do not understand some opinions are sold as facts.

Anyway I can't be sure but I have a pool that meets all prerequisits. no activity and system pool is not on it.

APM 1 spindown 5 mins on all drives in that mirror.
the TOSHIBA p300 is not being spun down my truenas scale 22. the others are.
hdparm -y does standby the toshibas.

So what I think is going on here is that ix system makes sure to only install hdd in their systems that do get standby'd. and the rest are considered edge cases that are just not worth it to support.

Where can I find the command used to standby the hdd in the source code so that I can check to see what is going on so that I can propose a more robust approach going forward in the future.

too many people struggle with this for too long. all the way back since 2014 i found articles about the same thing with always non satisfying solutions sometimes no solutions