Quest for the ultimate Freenas chassis.

[Black Ninja]

I have the 847 JBOD which holds 45 drives. I don't think there is room for a MB/CPU due to the drives in the rear. The nice thing about drives is they don't get as hot as a MB/CPU/RAM. I don't spread my pools across enclosures. I use the 847 with a bunch of 2TB drives I got cheap to use as my backup.

As for temps, can you guess which ones are in the back (the 3rd column is Temp in C):

Code:

+------+---------------+----+-----+-----+-----+-------+-------+--------+------+----------+------+-------+
|Device|Ser|Temp|Power|Start|Spin |ReAlloc|Current|Offline |Seek |Total |High |Command|
| | | |On |Stop |Retry|Sectors|Pending|Uncorrec|Errors|Seeks |Fly |Timeout|
| | | |Hours|Count|Count| |Sectors|Sectors | | |Writes|Count |
+------+---------------+----+-----+-----+-----+-------+-------+--------+------+----------+------+-------+
|da12  |WD-| 21 |24157| 2389|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da13  |WD-| 20 |28416| 2384|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da14  |WD-| 19 |27843| 2392|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da15  |WD-| 19 |27744| 2394|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da16  |WD-| 18 |15585| 2401|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da17  |WD-| 18 |24896| 2412|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da18  |WD-| 20 |26652| 2374|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da19  |WD-| 21 |29598| 2410|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da20  |WD-| 19 |27195| 2392|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da21  |WD-| 20 |22843| 2385|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da22  |WD-| 18 |11140| 2400|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da23  |WD-| 18 |11124| 2407|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da24  |WD-| 19 |28480| 2422|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da25  |WD-| 21 |27743| 2422|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da26  |WD-| 21 |26651| 2427|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da27  |WD-| 20 |28483| 2394|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da28  |WD-| 19 |26095| 2407|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da29  |WD-| 18 |15513| 2439|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da30  |WD-| 19 |29603| 2403|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da31  |WD-| 19 |29603| 2410|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da32  |WD-| 19 |25884| 2381|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da33  |WD-| 19 |28410| 2382|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da34  |WD-| 18 |20443| 2394|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da35  |WD-| 21 |27193| 2395|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da36  |WD-| 17 | 6503| 2375|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|
|da37  |WD-| 17 | 1129|  542|    0|      0|      0|       0|   N/A|       N/A|   N/A|    N/A|

I am still asking how you got these drive so cold ?! 17-21 Celsius ?! That can't be right ?! Third colum like you said is temp , temp show 17-21 Celsius ?

 

[Black Ninja]

And if you are multipathing, you will need the E26 backplane and dual ported SAS drives.

Wait I minute I change my mind about what I said. I need sas drives because of the dual ports for multipathing, but if I use SATA drives I can still create a redundancy from the HBA point, right ? Like having 2 HBA and if one fails other will still access the drive , cause HBA's will goes from one backplane with expander to another , not to the drive directly.

 

[depasseg]

This is what I have now: When power goes down my freenas sense that ups is on battery (usb cable to ups) and in 2 minutes shuts down gracefully freenas and that's it jobs is done. But If I add jbod, who will shutdown jbod after main unit is gracefully shutdown ?

Once the freenas is shutdown, the only thing left to do on the JBOD is kill the power. So in the case of the UPS shutting off, it will handle that (by just killing power).

 

[depasseg]

Your other backup nas you have separate like mine, how you sync the data to it?

I have 2 ZFS replication tasks. One to the local pool and another to a second device. (look in post #34).

 

[depasseg]

I am still asking how you got these drive so cold ?! 17-21 Celsius ?! That can't be right ?! Third colum like you said is temp , temp show 17-21 Celsius ?

My basement is ~62F. And those drives are probably spun down or idle.

 

[depasseg]

but if I use SATA drives I can still create a redundancy from the HBA point, right ? Like having 2 HBA and if one fails other will still access the drive , cause HBA's will goes from one backplane with expander to another , not to the drive directly.

No. The expanders just pass through both ports so since the SATA drives don't have 2 ports, they won't benefit from the second HBA.

 

[Black Ninja]

Once the freenas is shutdown, the only thing left to do on the JBOD is kill the power. So in the case of the UPS shutting off, it will handle that (by just killing power).

I was afraid you gonna say that. I prefer to keep ups power for router, modem, switches longer by having the bigest power consumers shutdown as soon as power is out, but if there is no other way with jbod then nothing I can do.

 

[Black Ninja]

No. The expanders just pass through both ports so since the SATA drives don't have 2 ports, they won't benefit from the second HBA.

I was afraid of that too. but thanks for explaining it.

 

[Mirfster]

I was afraid you gonna say that. :) I prefer to keep ups power for router, modem, switches longer by having the bigest power consumers shutdown as soon as power is out, but if there is no other way with jbod then nothing I can do.

Well all of that can be controller via NUT and power sections/groups. Your UPS should have that ability (or software); I would think.

You seem to be somewhat/pretty informed, but (no offense intended) seem to be not putting all the pieces together. Which is understandable, because what you are mostly asking about is not in a book (straight forward like). However, I (personally) would think that you have enough knowledge (based on your past postings) to layout a suitable design to accommodate your environment AND understand the drawbacks/cons of each design.

I am no "guru"; but I believe you can design a great environment and make the change from "going by the book" to actually "designing" a great/robust system. I think for the most part, you just have to believe in yourself and be willing to test all aspects before putting them into production. :)

 

[jgreco]

How much support does NUT have for multiple power groups?

 

[depasseg]

I was afraid you gonna say that. I prefer to keep ups power for router, modem, switches longer by having the bigest power consumers shutdown as soon as power is out, but if there is no other way with jbod then nothing I can do.

I should mention that the newer power controllers in the jbod enclosures have their own ipmi so it might be possible to script those for shutdown. You can even buy the newer controller and retrofit an older enclosure.

 

[jgreco]

Yeah, what they really need is something like the APC PDU power synchronization, where a head unit's IPMI would be in charge of subordinate units.

 

[jgreco]

Is this the longest Black Ninja has gone without replying?

 

[Black Ninja]

Well all of that can be controller via NUT and power sections/groups. Your UPS should have that ability (or software); I would think.

You seem to be somewhat/pretty informed, but (no offense intended) seem to be not putting all the pieces together. Which is understandable, because what you are mostly asking about is not in a book (straight forward like). However, I (personally) would think that you have enough knowledge (based on your past postings) to layout a suitable design to accommodate your environment AND understand the drawbacks/cons of each design.

I am no "guru"; but I believe you can design a great environment and make the change from "going by the book" to actually "designing" a great/robust system. I think for the most part, you just have to believe in yourself and be willing to test all aspects before putting them into production. :)

No offence taken. Honestly, just between me and you, I do think I know a lot, but I would never assume I know everything. There is so much stuff to learn and try, you can't have this knowledge in one person. I enjoy sharing knowledge and learning how the others solve a similar problems. This forum is full with great and smart people and I really like to read it and participate in it, not just for the love of freenas I have.

Now for this NUT , which I am not very familiar with and let me tell you why. This is the official way to setup APC ups to shutdown multiple let say windows machine:

1. You need Smart UPS class SMT or above, not SMC.(you need ups that can take NMC card)
2.You need to get a NMC card ( cost half of the ups and is like small pc for management)
3. You install APC software for network shutdown on each client.
4.You got APC web GUI , which is actually you NMC card gui and add ip's of the machines you what to send a shutdown signal.

This is how it should be done. Yes I did read a some hack solutions how to try to do it without NMC card , but I am not going that route.


Now back to the topic we were talking about how you can shutdown jbod after the main unit is gracefully shutdown. If you have an idea please share it, if I am not putting the pieces together you tell me how is done.

 

[Black Ninja]

I should mention that the newer power controllers in the jbod enclosures have their own ipmi so it might be possible to script those for shutdown. You can even buy the newer controller and retrofit an older enclosure.

I'll read more about these new controllers. I didn't know originally they come with the chassis. I thought you are buying them to make you chassis jbod , which is very good idea, that came from the servethehome web site.(the link jgreco post above).

 

[Black Ninja]

Is this the longest Black Ninja has gone without replying? :)

This happens cause I sleep during the night.It is a waste of time if you ask me , but can't go without it.:D

P.S. Or if I am not home for a day or two.

 

[Black Ninja]

Yeah, what they really need is something like the APC PDU power synchronization, where a head unit's IPMI would be in charge of subordinate units.

This gives me idea I could explore: I'll login in the NMC of the UPS and see if I can use some setting to cut the power of "outlet groups 1" .

Well worst case, like depasseg said: I'll shutdown gracefully main unit right after the power loss and jbod will lose power when battery is depleted.

 

[jgreco]

This gives me idea I could explore: I'll login in the NMC of the UPS and see if I can use some setting to cut the power of "outlet groups 1" .

I suggested that back in message #50.

 

[Black Ninja]

I suggested that back in message #50.

Perhaps you did, but I wasn't familiar with NUT since is kind of hack-tool more or less, than official software for managing APC UPS., so I didn't get the hint.;)

I got the idea from post #52

 

[Black Ninja]

I should mention that the newer power controllers in the jbod enclosures have their own ipmi so it might be possible to script those for shutdown. You can even buy the newer controller and retrofit an older enclosure.

That is very interesting. Is this one :CSE-PTJBOD-CB2 ? It seems it has the same connector like SMbus on MB, but then how is going to sen that info over the network.

P.S. I was gonna say I only buy PSU with SMbus and MB that has the connector for it. You know to monitor and alert for the PSU operation and power consumption.