Quest for the ultimate Freenas chassis.

[Black Ninja]

I have 2 Supermicro 836TQ. One main freenas server other just as backup , same specs like my signature.I can't llike any up to 16 chassis any better.

BUT, now for the next step(bigger freenas) with more drives. I thought about 846TQ with 24 bays , but then why not get a more bay to last longer before run out of space. I came across 847 with 36 bays. It has this unique design like a 2u chassis inside and 4u outside. 24 drive on front and another 12 on the back.

I would appreciate if you guys can share your thoughts or experience with this chassis. I really can't affort to make a mistake on such a expensive chassis. What is it I should be aware ?

P.S. Everyone's input will be appreciated very much.

 

[depasseg]

Are you looking to buy a 3rd server? Or do you only need to add an expansion chassis? I have the JBOD attached to my 2U server and it's running great.

 

[jgreco]

Once you start stacking disks horizontally, as is done in the 847, the second tier of disks gets substantially warmer air than the first. This is really not too good unless you're in a reliably cooled environment like a data center.

 

[Black Ninja]

Are you looking to buy a 3rd server? Or do you only need to add an expansion chassis? I have the JBOD attached to my 2U server and it's running great.

Yes. When build this one with 36 bays, that will be the new main freenas. The other 2 E3-1230 v2 nasses that I have now , will become a backup servers for the 36 bay new nas.

I thought about adding a JBOD empty chassis , but I couldn't bare the risks of the hdd cables leaving the chassis. Plus the risk of powering off the jbod first by mistake instead of the main unit. If everything is in one chasis, there won't be a risk for half of thedrives to dissaper. you know

 

[Black Ninja]

Once you start stacking disks horizontally, as is done in the 847, the second tier of disks gets substantially warmer air than the first. This is really not too good unless you're in a reliably cooled environment like a data center.

How much more warmer compared to the front ? I was really hoping designers made to work properly. By the way I am only running one type of hdd- HGST 4.0TB 5900 RPM Coolspin, but still want to stay away from heat problems. What would you do for more space ?

 

[Mirfster]

I thought about adding a JBOD empty chassis , but I couldn't bare the risks of the hdd cables leaving the chassis. Plus the risk of powering off the jbod first by mistake instead of the main unit. If everything is in one chasis, there won't be a risk for half of thedrives to dissaper. you know

I guess it would depend on your pool/vdev design. If you have a pool that consists only of vdevs that are in the JBOD; then that pretty much negates the issue of external cabling. But, if the pool consists of vdevs that are in the main unit and the external JBOD; well there is more of a risk.

 

[Black Ninja]

I guess it would depend on your pool/vdev design. If you have a pool that consists only of vdevs that are in the JBOD; then that pretty much negates the issue of external cabling. But, if the pool consists of vdevs that are in the main unit and the external JBOD; well there is more of a risk.

You mean like let's say 1 vdev raidz2 with 8 drives. So 6 drives needs to be in main chassis other 2 in jbod ? Something like that you mean ?

This is smart idea I wonder if people designed this way ? How would you make it with 12bay and 24 bay chassis ? It's for media so raidz2 , 8 disk wide is preferable ?

 

[Black Ninja]

Are you looking to buy a 3rd server? Or do you only need to add an expansion chassis? I have the JBOD attached to my 2U server and it's running great.

While still scared to spread drives in second chassis I still would like to know how yours is setup ?:)

 

[jgreco]

How much more warmer compared to the front ? I was really hoping designers made to work properly.

The designers designed it properly, but the users have to use it properly too.

Look at the 847D. You have 90 drives in a 4U space. No amount of design is going to make that work right unless it is deployed in a properly cooled environment, and even then, I'm going to say it probably wouldn't be a good idea to have all the drives spun up.

So for the 36 drive chassis, the back drives are going to be somewhat warmer. How much warmer? I don't know. Put your hand behind your server and guess how much heat is blasting out. Then subtract whatever you'd guess is being used by the mainboard.

 

[Mirfster]

You mean like let's say 1 vdev raidz2 with 8 drives. So 6 drives needs to be in main chassis other 2 in jbod ? Something like that you mean ?

I guess that would work too, but I was thinking more along the lines of an entire separate pool consisting of just drives on the JBOD. Then a different pool that consists of just drives contained in the main Server chassis. Of course, if you require to have all those drives in one pool that changes things.

Example:
FreeNas Server: Houses 12 drives, I use all 12 for a single pool (whatever vdev design I want)
JBOD Box: Houses 24, 36, 48, etc. drives, I use those for a second/different pool (whatever vdev design I want)

Of course, you could get even more detailed if say you had to do 3way mirrors. Then you could design a way to have part of the mirrors reside on the main server as well as the JBOD.

Of course, to properly do this you for sure would want to know the Serial Numbers of the drives, where they are located/which slots, the assigned ID (da0, da1, etc) and then use the manual option when creating the dependent vdevs(s) and eventual pool(s).

 

[depasseg]

While still scared to spread drives in second chassis I still would like to know how yours is setup ?:)

It's not complicated. Lsi external hba, and a cable to the jbod chassis.

I never need to use the power buttons, so it's never an issue.

And I have both redundant power supplies of both the server and jbod plugged into separate circuits and UPS's.

 

[Black Ninja]

The designers designed it properly, but the users have to use it properly too.

Look at the 847D. You have 90 drives in a 4U space. No amount of design is going to make that work right unless it is deployed in a properly cooled environment, and even then, I'm going to say it probably wouldn't be a good idea to have all the drives spun up.

So for the 36 drive chassis, the back drives are going to be somewhat warmer. How much warmer? I don't know. Put your hand behind your server and guess how much heat is blasting out. Then subtract whatever you'd guess is being used by the mainboard.

I see. Then what would you do then to expand ? I had a great Success with 836TQ but need bigger nas ?

 

[Black Ninja]

Of course, to properly do this you for sure would want to know the Serial Numbers of the drives, where they are located/which slots, the assigned ID (da0, da1, etc) and then use the manual option when creating the dependent vdevs(s) and eventual pool(s).

The beauty of Supermicro .. , well just one of many is that you can see the drive serial from the front of the hdd tray.

So two separate pools you are saying. I never really was able to find out how multiple pool will effect freenas resources. Otherwise logically I like separate pools, as much as like separate hdd for boot and for data, as long as is not adding too much overhead to freenas.

 

[depasseg]

If you don't need another server, nor need to expand CPU and RAM beyond the limits of your current system, just get a jbod.

 

[depasseg]

The beauty of Supermicro .. , well just one of many is that you can see the drive serial from the front of the hdd tray.

So two separate pools you are saying. I never really was able to find out how multiple pool will effect freenas resources. Otherwise logically I like separate pools, as much as like separate hdd for boot and for data, as long as is not adding too much overhead to freenas.

I haven't really looked into it, but since the backup pool isn't really "used" I imagine the resource impact is minimal.

 

[Black Ninja]

It's not complicated. Lsi external hba, and a cable to the jbod chassis.

I never need to use the power buttons, so it's never an issue.

And I have both redundant power supplies of both the server and jbod plugged into separate circuits and UPS's.

Do you use one external lsi to backplane with expander ? I assume the jbod chassis is empty and has just the small supermicro module that can power it it ?

For the power delivery I have the same setup: dual psu,ups and circuits.:p

 

[depasseg]

Yep you got it. That's it.

 

[Black Ninja]

Yep you got it. That's it.

Ohh you have the 847 (from your signature). If had 847 I can put the mb/cpu, ram inside and don't even need jbod.

Is your pool spread across chassis, so if you disconnect one you loose your pool ?

How are the temp of the hdd. As jgreco mention , the back drives will have all the heat from the front drives + the heat from the computer (if you have one inside). Can you share for comparison what temp front drives gets vs back ones ?

 

[depasseg]

I have the 847 JBOD which holds 45 drives. I don't think there is room for a MB/CPU due to the drives in the rear. The nice thing about drives is they don't get as hot as a MB/CPU/RAM. I don't spread my pools across enclosures. I use the 847 with a bunch of 2TB drives I got cheap to use as my backup.

As for temps, can you guess which ones are in the back (the 3rd column is Temp in C):

Code:

+------+---------------+----+-----+-----+-----+-------+-------+--------+------+----------+------+-------+
|Device|Ser|Temp|Power|Start|Spin |ReAlloc|Current|Offline |Seek |Total |High |Command|
| | | |On |Stop |Retry|Sectors|Pending|Uncorrec|Errors|Seeks |Fly |Timeout|
| | | |Hours|Count|Count| |Sectors|Sectors | | |Writes|Count |
+------+---------------+----+-----+-----+-----+-------+-------+--------+------+----------+------+-------+
|da12  |WD-| 21 |24157| 2389|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da13  |WD-| 20 |28416| 2384|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da14  |WD-| 19 |27843| 2392|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da15  |WD-| 19 |27744| 2394|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da16  |WD-| 18 |15585| 2401|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da17  |WD-| 18 |24896| 2412|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da18  |WD-| 20 |26652| 2374|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da19  |WD-| 21 |29598| 2410|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da20  |WD-| 19 |27195| 2392|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da21  |WD-| 20 |22843| 2385|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da22  |WD-| 18 |11140| 2400|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da23  |WD-| 18 |11124| 2407|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da24  |WD-| 19 |28480| 2422|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da25  |WD-| 21 |27743| 2422|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da26  |WD-| 21 |26651| 2427|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da27  |WD-| 20 |28483| 2394|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da28  |WD-| 19 |26095| 2407|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da29  |WD-| 18 |15513| 2439|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da30  |WD-| 19 |29603| 2403|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da31  |WD-| 19 |29603| 2410|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da32  |WD-| 19 |25884| 2381|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da33  |WD-| 19 |28410| 2382|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da34  |WD-| 18 |20443| 2394|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da35  |WD-| 21 |27193| 2395|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da36  |WD-| 17 | 6503| 2375|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da37  |WD-| 17 | 1129|  542|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|

 

[jgreco]

I see. Then what would you do then to expand ? I had a great Success with 836TQ but need bigger nas ?

The 846 is as big as you can go without doubling up on drives. The usual solution to go beyond that 24 drive solution is to add another 846 as a JBOD. You can basically keep doing that all day long until you run out of SAS.

If you can keep the front drives operating at a very pleasant temperature, then the back ones will probably be within an acceptable range. You'll have better luck with 5400/5900RPM drives than 7200RPM drives.