Water Cooling

[Csaba Gajdos]

Hi there,

Has somebody experience with a fully water cooled NAS build?
Is there any hardware recommendation for build like that?
I'm just wondering how can I build the quietest NAS box along the recommended (https://forums.freenas.org/index.php?threads/hardware-recommendations-read-this-first.23069/) hardware parts...

 

[Mirfster]

I always wanted to give this a whirl: Computer Liquid Cooling (Submersion) with 3M Novec :)

There are a lot of "quiet" systems. Guess it depends on how much hardware you are planning on using (Number of Drives, etc). While water-cooling has intrigued me, I am reluctant to use it on my Servers (water leak in a Server Rack = bad things...).

I would suggest staying away from any 1U; since IMHO they tend to be the loudest.

 

[jgreco]

The basic problem with a NAS is the drives; they tend to be noisy. Building a low noise NAS is mostly an exercise in making sure you have good airflow that doesn't require pressure differentials (i.e. don't stack the drives right next to each other, use 120mm fans, etc) and then maybe augment with some noise reduction material.

 

[joeschmuck]

Has somebody experience with a fully water cooled NAS build?

I'm not sure what the purpose of building a water cooled unit would be since FreeNAS itself has very minimal requirements and doesn't produce much heat if you purchase reasonable components. My hardware is virtually silent. But of course you could build a water cooled system but I think the water pump will be much noisier than any other component in a properly designed system.

 

[jgreco]

I was thinking of building a nice workstation out of something and water cooling it. Of course I was talking about that months ago too.

https://forums.freenas.org/index.php?threads/merry-christmas.40144/page-3#post-255114

But I'm a bit closer,


nice! Plenty of space for drives and fun stuff. I'm thinking like an X10SRA, 64 or 128GB RAM, ESXi, and some GPU's that can be passed through to guest OS's for a dual seat workstation. Ever since I got rid of the Sun 3/260 I've missed a nice deskside box.

 

[joeschmuck]

So how many hard drives could you fit into this case with water blocks attached to the drives to pull away heat, or would you just use air cooling for the drives? When I think of a water cooled system I'm thinking of the CPU, Video, Hard Drives, and MB chipset being cooled, but of course that is really up to the needs of the designer. That box does look big and it sounds like a nice project.

 

[jgreco]

So how many hard drives could you fit into this case with water blocks attached to the drives to pull away heat, or would you just use air cooling for the drives?

Well, there's the thing. I wasn't finding what I wanted for just a good solid deskside server chassis. Supermicro kind-of had one in the SC743TQ but it lacked space for some of the stuff I wanted. In the old days I had no problems finding large "fileserver" class stuff, but that seems hard now. What I really want is the ability to stick two DVD (or Blu-Ray) drives in there. Then the system datastores will probably be my normal 8x 2.5" drive bays, which should be workable in some sort of 4x2.5-in-5.25 (brings us to four bays used), and then an HBA for general purpose play, with at least another 4 2.5" bays and then some 3.5" bays (seven 5.25 bays used)... so many times I've wanted to spin up a FreeNAS on some real disks to try this or that, but just lacked free hardware to do so. Two bays free, maybe a water cooling reservoir?

And the cool thing about running ESXi on it would be that I could make it multiheaded or dual seat or whatever you want to call it when you have two keyboards and two GPU's, each bound to different VM's. It'd be cool to be able to run a FreeBSD or OS X desktop alongside the sadly-required Windows.

When I think of a water cooled system I'm thinking of the CPU, Video, Hard Drives, and MB chipset being cooled, but of course that is really up to the needs of the designer. That box does look big and it sounds like a nice project.

Yeah, I'm waffling back and forth on whether the whole water cooling thing is stupid (because it's certainly unnecessary and introducing liquid into a workstation is risk) or a fun thing to do. Since I tend to be in the business of doing the same sorts of things over and over and over and over again for many years at a time, and since this will certainly be my primary workstation for at least five years, the question is whether the fun outweighs the inconvenience/risk/etc of having a bunch of tubes and other stuff inside the machine. I could just go with a conventional air cooled design and that'd probably be the smart thing to do, but that's less play. The opportunity to go and be a little frivolous and do something unusual... hm.

 

[BigDave]

Different and fun? Without a doubt! I envision a hugh amount of inconvenience if you ever need to change out a hard drive.
As far as I know, they don't make a hot swap hard drive water block.
Not to mention, a mixture of Grinch and frivolity would be JUST WRONG man! :p

 

[jgreco]

Different and fun? Without a doubt! I envision a hugh amount of inconvenience if you ever need to change out a hard drive.
As far as I know, they don't make a hot swap hard drive water block.
Not to mention, a mixture of Grinch and frivolity would be JUST WRONG man! :p

What, my strangling Beastie in front of this large audience isn't sufficient to prove my frivolous nature? I'm so disappointed that no one's said anything.

 

[Mlovelace]

The opportunity to go and be a little frivolous and do something unusual... hm.

You should get really frivolous and wait for the 15TB Samsung SSDs to come out. :D

http://www.digitaltrends.com/computing/samsung-15tb-business-ssd/

 

[jgreco]

Yeah, I saw that. Quite the thing.

Right now, the cost differential between HDD and SSD isn't anywhere near as bad as it once was. The price I'm willing to pay for SSD today is the price I was paying for HDD a few years ago. But also the density ... that damn thing is actually a 2.5" SSD according to Samsung!

The implications are staggering. We're barely cramming 3TB into 2.5" platters. Yow.

So if the price to manufacture silicon keeps plummeting, and there seems to be no particular reason to think that it's going to slow down all that much, ...

Right now a 2TB SSD is ~$600. A 2TB HDD (2.5") is ~$100.

We might see the death of the 2.5" HDD in a few short years.

 

[Mlovelace]

I'm curious with nvme taking off like it is, if the sata drive (be it ssd or hdd) will soon be a thing of the past. Not a lot of optical drives shipping these days and it hasn't been that long since everyone had to have a burner in their PC.

 

[Mlovelace]

Yeah, I saw that. Quite the thing.

Right now, the cost differential between HDD and SSD isn't anywhere near as bad as it once was. The price I'm willing to pay for SSD today is the price I was paying for HDD a few years ago. But also the density ... that damn thing is actually a 2.5" SSD according to Samsung!

The implications are staggering. We're barely cramming 3TB into 2.5" platters. Yow.

So if the price to manufacture silicon keeps plummeting, and there seems to be no particular reason to think that it's going to slow down all that much, ...

Right now a 2TB SSD is ~$600. A 2TB HDD (2.5") is ~$100.

We might see the death of the 2.5" HDD in a few short years.

I agree the prices are getting ridiculously cheap. Mushkin just put out a 1TB SSD drive for $200 which getting into the all flash pool realm of pricing.

 

[jgreco]

I agree the prices are getting ridiculously cheap. Mushkin just put out a 1TB SSD drive for $200 which getting into the all flash pool realm of pricing.

They lost that race. SanDisk was blowing out their 960GB unit for $199 last Black Friday.

 

[Fuganater]

This was my last server/NAS. Watercooling is fun but ended up being a pain for a server.

 

[jgreco]

I'm curious with nvme taking off like it is, if the sata drive (be it ssd or hdd) will soon be a thing of the past.

Certainly one possibility. I mean, look at this, this is crazy:

http://www.supermicro.com/products/system/2U/2028/SSG-2028R-NR48N.cfm

But on the other hand, just making *all* the storage faster isn't necessarily the way to go. Storage tiering is a thing for a reason.

Not a lot of optical drives shipping these days and it hasn't been that long since everyone had to have a burner in their PC.

True, but one of my goals in life is to take every optical disc here, create an ISO on the fileserver, and then never need to go hunting for a particular magic disc ever again.

 

[Ericloewe]

Certainly one possibility. I mean, look at this, this is crazy:

http://www.supermicro.com/products/system/2U/2028/SSG-2028R-NR48N.cfm

+1 for the unusual "you can access the inner row of drives by swinging them out" feature

I wonder how they're connecting the PCI-e SSDs. They seem to imply that the single x8 slot is being used, and they have four additional four-lane connectors (non-standard, it seems) available. I'd probably have used one of the x16 slots plus the four connectors, especially since it pushes people towards the expensive form-factor NICs.

Also, switching 112 lanes of PCI-e 3.0 is probably not happening in one chip. They must be cascading them, with four PLX 28-lane switches per backplane - though that means spreading out the 8 lanes into four sets of two, which sounds less-than-ideal...

 

[BigDave]

Certainly one possibility. I mean, look at this, this is crazy:

http://www.supermicro.com/products/system/2U/2028/SSG-2028R-NR48N.cfm

WOW
She's got some ample PSUs on 'er.
1600W Redundant Power Supplies
Titanium Level (96%)

 

[BigDave]

This was my last server/NAS. Watercooling is fun but ended up being a pain for a server.

View attachment 10855

I love my Case Labs box ;)

 

[jgreco]

+1 for the unusual "you can access the inner row of drives by swinging them out" feature

I wonder how they're connecting the PCI-e SSDs. They seem to imply that the single x8 slot is being used, and they have four additional four-lane connectors (non-standard, it seems) available. I'd probably have used one of the x16 slots plus the four connectors, especially since it pushes people towards the expensive form-factor NICs.

Also, switching 112 lanes of PCI-e 3.0 is probably not happening in one chip. They must be cascading them, with four PLX 28-lane switches per backplane - though that means spreading out the 8 lanes into four sets of two, which sounds less-than-ideal...

Yeah, I didn't get a chance to research, but obviously there's a lot going on there. We'll be seeing more and more of the NVMe thing obviously but that strikes me as being on the far side of crazy. ;-)