Quest for the ultimate Freenas chassis.

[Black Ninja]

I have the 847 JBOD which holds 45 drives. I don't think there is room for a MB/CPU due to the drives in the rear. The nice thing about drives is they don't get as hot as a MB/CPU/RAM. I don't spread my pools across enclosures. I use the 847 with a bunch of 2TB drives I got cheap to use as my backup.

As for temps, can you guess which ones are in the back (the 3rd column is Temp in C):

Code:

+------+---------------+----+-----+-----+-----+-------+-------+--------+------+----------+------+-------+
|Device|Ser|Temp|Power|Start|Spin |ReAlloc|Current|Offline |Seek |Total |High |Command|
| | | |On |Stop |Retry|Sectors|Pending|Uncorrec|Errors|Seeks |Fly |Timeout|
| | | |Hours|Count|Count| |Sectors|Sectors | | |Writes|Count |
+------+---------------+----+-----+-----+-----+-------+-------+--------+------+----------+------+-------+
|da12  |WD-| 21 |24157| 2389|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da13  |WD-| 20 |28416| 2384|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da14  |WD-| 19 |27843| 2392|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da15  |WD-| 19 |27744| 2394|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da16  |WD-| 18 |15585| 2401|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da17  |WD-| 18 |24896| 2412|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da18  |WD-| 20 |26652| 2374|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da19  |WD-| 21 |29598| 2410|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da20  |WD-| 19 |27195| 2392|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da21  |WD-| 20 |22843| 2385|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da22  |WD-| 18 |11140| 2400|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da23  |WD-| 18 |11124| 2407|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da24  |WD-| 19 |28480| 2422|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da25  |WD-| 21 |27743| 2422|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da26  |WD-| 21 |26651| 2427|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da27  |WD-| 20 |28483| 2394|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da28  |WD-| 19 |26095| 2407|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da29  |WD-| 18 |15513| 2439|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da30  |WD-| 19 |29603| 2403|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da31  |WD-| 19 |29603| 2410|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da32  |WD-| 19 |25884| 2381|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da33  |WD-| 19 |28410| 2382|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da34  |WD-| 18 |20443| 2394|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da35  |WD-| 21 |27193| 2395|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da36  |WD-| 17 | 6503| 2375|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da37  |WD-| 17 | 1129|  542|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|

Thank you very much.

I think you are right, you can't put mb in the 45 chassis, in 36 you can put a 2u system.

I am not sure I understand the columns here. Are these drives off right now , or run in a freezer ?:)
What I see is temps from 17-21 Celsius ?! That can't be right ?!

 

[Black Ninja]

The 846 is as big as you can go without doubling up on drives. The usual solution to go beyond that 24 drive solution is to add another 846 as a JBOD. You can basically keep doing that all day long until you run out of SAS.

If you can keep the front drives operating at a very pleasant temperature, then the back ones will probably be within an acceptable range. You'll have better luck with 5400/5900RPM drives than 7200RPM drives.

What can you tell to the people who are afraid of spreading a pool across a chassis in jbod scenarios ?:)

 

[jgreco]

What can you tell to the people who are afraid of spreading a pool across a chassis in jbod scenarios ?:)

Make sure you wire the JBOD's for power the same way as you wire the head unit. SAS multipathing is not a bad thing. Always remember to turn the JBOD's on first, and turn them off last.

 

[depasseg]

What can you tell to the people who are afraid of spreading a pool across a chassis in jbod scenarios ?:)

Enterprises do this all the time. What is the risk you will press the power button on the JBOD or pull the SAS cable from the JBOD? If the answer is very-low, then there isn't really an issue.

And besides, it's easy to add more drives and have a complete backup in case disaster strikes.
And don't forget, you can also add more than one enclosure so you can but your backup all in the same enclosure.

 

[jgreco]

Enterprises do this all the time.

Plus it's very fun to see a rack full of hard drives top to bottom.

 

[depasseg]

Plus it's very fun to see a rack full of hard drives top to bottom.

Especially the blinky lights!

 

[jgreco]

Especially the blinky lights!

I have to say, one of the coolest things about the 24 drive 2.5" 2U storage server here is the way the little blue lights rip around. They're all in a single row because the drives are on edge.

 

[Black Ninja]

Make sure you wire the JBOD's for power the same way as you wire the head unit. SAS multipathing is not a bad thing. Always remember to turn the JBOD's on first, and turn them off last.

For power I assume you mean: dual PSU, dual ups and dual circuits. That I have, just have to buy a super-micro module that can turn chassis on and connect the fans.(never use those just heard about them , I hope they don't run fans at max speed)


I am not very familiar about how the SAS multipathing works, but I assume will need a backplane with build in expander. I never used expanders, being one of these stubborn DABF* people, but always willing to learn something new as long as is not trading convenience for simplicity and reliability.

For the proper turning of turning on device there is a little chance to forget, and another thing: When power goes down , in few mins my main freenas will turn off (ups to nas usb cable) but who will turn off the jbod ? Just wait till ups juice is depleted and cut it of at will ?

* DABF - Direct-Atached backplane fanatic

 

[Black Ninja]

Especially the blinky lights!

I like the lights. But recently I start putting front bezels on my servers , so can't see the light anymore.:-(

 

[jgreco]

I like the lights. But recently I start putting front bezels on my servers , so can't see the light anymore.:-(

That's what they make drills and drill bits for.

 

[Black Ninja]

Enterprises do this all the time. What is the risk you will press the power button on the JBOD or pull the SAS cable from the JBOD? If the answer is very-low, then there isn't really an issue.

Pressing the power button risk can be eliminated. I can disconnect the button or in my case front bezel is covering it, and the way Supermicro power buttons are , you fell on the front of the chassis the chance to engage power button is almost non existent.
While knocking off of cable while you are dig in something in the back like , removing chassis or connecting new ones is more likely situation.But that risk come with the territory I guess.[/QUOTE]

And besides, it's easy to add more drives and have a complete backup in case disaster strikes.
And don't forget, you can also add more than one enclosure so you can but your backup all in the same enclosure.

Not sure I got that. Doing a backup on the same machine on another pool , or .. ?! I do backup on another nas almost identical machine and capacity. How would you do it ?

 

[depasseg]

That I have, just have to buy a super-micro module that can turn chassis on and connect the fans

The SC847 JBOD comes with the controller and everything is all cabled. All you have to do is insert drives, connect the SINGLE (or 2 if you end up multipathing) SAS cable to the back of the chassis and add power.

And if you are multipathing, you will need the E26 backplane and dual ported SAS drives.

 

[Black Ninja]

That's what they make drills and drill bits for. :p

Are you serious. I can't stand a scratch on it, don't have to be perfect to function but that's the only way I would do it.:)

 

[depasseg]

Not sure I got that. Doing a backup on the same machine on another pool , or .. ?!

What I meant was that you can add multiple enclosures (even later in the future). They just daisy chain.

This should reduce the risk of requiring the backup pool to span multiple enclosures. You could put your local backup drives in a single enclosure.

And you would replicate locally from your main pool to the backup pool (and like you, I also have a second system that I also backup to).

 

[Black Ninja]

The SC847 JBOD comes with the controller and everything is all cabled.

Ohh I didn't know that. No wonder is few thousand dollars this chassis.:(

I forgot about SAS drives for the dual path, I't wont help me with sata drives , so I guess I won't be getting redundant multi-path in this project.

 

[jgreco]

For power I assume you mean: dual PSU, dual ups and dual circuits. That I have, just have to buy a super-micro module that can turn chassis on and connect the fans.(never use those just heard about them , I hope they don't run fans at max speed)

http://www.servethehome.com/supermicro-cse-ptjbod-cb1-jbod-power-board-diy-jbod-chassis-made-easy/

I am not very familiar about how the SAS multipathing works, but I assume will need a backplane with build in expander. I never used expanders, being one of these stubborn DABF* people, but always willing to learn something new as long as is not trading convenience for simplicity and reliability.

The idea for multipathing is to avoid single points of failure. So you put two HBA's in your head unit (not a technical requirement, but if an HBA fails, it takes both SFF8088's with it). Now let's look at a slightly more complex example with two JBOD's:

So the first HBA connects to the primary expander ("SIM0") in array 1. That daisy chains over to the primary expander (also "SIM0") in array 2. The second HBA connects to the SIM1 in Array 2. Daisy chains to Array 1.

The point is to make sure that there is always a path to storage. When everything's optimal, there are actually fully redundant paths. Now, in your mind, one at a time, kill:

1) Each HBA
2) Each SIM (SAS expander)
3) Each cable

Convince yourself that there's always a path for data to reach each drive. And also note that if you were to merely connect both HBA's directly to Array 1 and then daisy chain Array 2, that'd result in a topology where the loss of the PSU in Array 1 could render your storage inaccessible. As it stands, you could be running mirrors, one half in Array 1, one half in Array 2, and even the loss of an array doesn't kill your storage. Cool huh.

Now the thing is that multipathing is also a software thing, so there's additional software complexity too. This is just intended to give you a feel for the hardware angle.

For the proper turning of turning on device there is a little chance to forget, and another thing: When power goes down , in few mins my main freenas will turn off (ups to nas usb cable) but who will turn off the jbod ? Just wait till ups juice is depleted and cut it of at will ?

Your NAS should be sending a command to the UPS to power off. That will take out the JBOD unit.

 

[jgreco]

Ohh I didn't know that. No wonder is few thousand dollars this chassis.:(

Yeah, that's another good reason not to go with the larger chassis units. You can actually often find SC846 JBOD's on eBay pretty cheap...

 

[Black Ninja]

What I meant was that you can add multiple enclosures (even later in the future). They just daisy chain.

This should reduce the risk of requiring the backup pool to span multiple enclosures. You could put your local backup drives in a single enclosure.
View attachment 11071


And you would replicate locally from your main pool to the backup pool (and like you, I also have a second system that I also backup to).
View attachment 11072

Yeah. I can add another pool but adding another jbod to existing nas. Not sure how good is to have a backup in the same one pool to another , but if you have another system then it wont hurt.I am still trying to wrap my head about the benefits of doing that.

Your other backup nas you have separate like mine, how you sync the data to it?

 

[Black Ninja]

Yeah, that's another good reason not to go with the larger chassis units. You can actually often find SC846 JBOD's on eBay pretty cheap...

Just both 1u from ebay used again:-(( Now I regret why didn't buy new one. By the time buy everything the way I wanted , endup paying like new ones.

 

[Black Ninja]

http://www.servethehome.com/supermicro-cse-ptjbod-cb1-jbod-power-board-diy-jbod-chassis-made-easy/



The idea for multipathing is to avoid single points of failure. So you put two HBA's in your head unit (not a technical requirement, but if an HBA fails, it takes both SFF8088's with it). Now let's look at a slightly more complex example with two JBOD's:

So the first HBA connects to the primary expander ("SIM0") in array 1. That daisy chains over to the primary expander (also "SIM0") in array 2. The second HBA connects to the SIM1 in Array 2. Daisy chains to Array 1.

The point is to make sure that there is always a path to storage. When everything's optimal, there are actually fully redundant paths. Now, in your mind, one at a time, kill:

1) Each HBA
2) Each SIM (SAS expander)
3) Each cable

Convince yourself that there's always a path for data to reach each drive. And also note that if you were to merely connect both HBA's directly to Array 1 and then daisy chain Array 2, that'd result in a topology where the loss of the PSU in Array 1 could render your storage inaccessible. As it stands, you could be running mirrors, one half in Array 1, one half in Array 2, and even the loss of an array doesn't kill your storage. Cool huh.

Now the thing is that multipathing is also a software thing, so there's additional software complexity too. This is just intended to give you a feel for the hardware angle.



Your NAS should be sending a command to the UPS to power off. That will take out the JBOD unit.

On the first part I can only say "Cool" multiple times
By the way the link you gave me I came across before and that's where I heard of this nice jbod device.


On the second one about the UPS i got confused. The goal is to properly shutdown main nas and jbod before the ups runs out of battery power, I don't care to shutdown UPS unit itselft.
This is what I have now: When power goes down my freenas sense that ups is on battery (usb cable to ups) and in 2 minutes shuts down gracefully freenas and that's it jobs is done. But If I add jbod, who will shutdown jbod after main unit is gracefully shutdown ?