BUILD SuperMicro X10SRL-F + 3 846 Chassis + 72 Disks

[pclausen]

Yeah ok so I had a typo there.. It's -a and not -o. Not sure how that happened. Also don't include the numbers. Just the same line you enter on the command line.

# smartctl -a /dev/da0 | grip -i temperature
# smartctl -a /dev/da1 | grip -i temperature

Don't include the # :) Do that in a file for each device then type the chmod command and run the program as above.

Ok. I'm a complete noob when it comes to UNIX. So here's my file (using vi to edit).

When I execute, it returns nothing

 

[pclausen]

Ya got a power station in back there somewhere? :D

Lol, I wish I could make my own power. You don't want to see my power bill. Hopefully retiring that X7 motherboard will shave some off the bill.

 

[rogerh]

Ok. I'm a complete noob when it comes to UNIX. So here's my file (using vi to edit).

When I execute, it returns nothing

You need a space between the "-a" and the "/dev/". Generally spaces are needed to separate items, though usually it doesn't matter how many spaces.

 

[DataKeeper]

Type.... 'chmod 700 custom_smart_temps' then try it ;) Gotta make executable :p

The best book I ever purchased to start learning Unix/Linux was Linux For Dummies. It'll get you through BSDi for the most part as well. Some syntax differences but that's it. Easy to understand fir a 100% novice and good all around reference.

Vi is a great editor and I love it. That said, I find I use nano for most quick edits.

 

[pclausen]

Oops, my bad. I think I removed those spaces by mistake when I changed the -o to -a. It has been years since I used VI. Works great now. Thanks guys!

 

[SweetAndLow]

Type.... 'chmod 700 custom_smart_temps' then try it ;) Gotta make executable :p

The best book I ever purchased to start learning Unix/Linux was Linux For Dummies. It'll get you through BSDi for the most part as well. Some syntax differences but that's it. Easy to understand fir a 100% novice and good all around reference.

Vi is a great editor and I love it. That said, I find I use nano for most quick edits.

If this was the case they would have seen a permission denied error. The space is most likely the culprit.

 

[pclausen]

Yep, it was definitely the space!

With the ability to monitor disk temps in real-real time, I switched fan profile from Heavy I/O to optimal.

Before doing so, the temps ranged from 27 to 34 C. After switching to optimal, the range moved up to 30 to 39 C. So I switched back to Heavy I/O and plan to leave it there. Heavy I/O is still much quieter than having them run wide open.

 

[Apollo]

Such a beast and yet you are running out of "space":)

 

[marbus90]

if temps don't go above 40C during a scrub, you're fine.

 

[pclausen]

Thanks, I'll keep that in mind when I start doing scrubs. While these 846 chassis are in my basement, they are right behind the wall with my projection screen, so the noise was quite annoying in the past when we were down there watching movies on the big screen.

So I really love the ability to control the fan speed on the main chassis now as well as being able to monitor the power consumption and track it over time.

I'd love to be able to do that with the 2 "drive only" chassis' as well. So I have this old revision power controller board that used to be in the chassis that now has the X10 motherboard:

When I was using it, the 2 rear 3 pin fan connectors were connected to the 2 rear fans, and the 3 mid plane fans were connected to the 4 pin fan connectors on the SAS2 expander backplane. With this setup, it was not possible to adjust any fan speeds, all 5 fans were running at full blast all the time.

Supermicro has a newer power controller board which looks like this:

Is has a pair of 4 pin fan connectors installed. Looking at the PCB, it looks like it would be possible to install 6 additional 4 pin fan connectors! Unfortunately, I suspect that it would take more than just soldering them in to make them work.

Other than the 4 pin fan connectors, a very interesting feature on this power board is the inclusion of a 5 pin Power System Management Bus (I2C) connector. My X10 mobo has this as well and it is what allows me to monitor the power consumption via the IPMI web interface.

There is also a separate 4pin I2C connector. Now I know the normal intent is to connect this to the expander backplane, but I'm not sure what that would do for me since I can't access it anyway. The X10 motherboard has a pair of I2C jumpers (I2C1 and I2C2) that can be set to enable I2C System Management Bus for the PCI-E slots. So I wonder if that means I can get a PCI Express card that has I2C connectors I can then connect to the power controller cards in the external chassis?

My goal is to be able to monitor power consumption in all 3 chassis as well as being able to control fan speed in both external chassis, and I'd like to be able to do it via the IPMI interface.

If that is all a pipe dream, then I guess I'll need some sort of manual fan controller for the fans in the external 846 chassis. Right now I know one of them has 3 pin fans. Not sure yet what the 2nd one has yet. I guess the cheapest solution for the 3pin fan chassis would be to just add a resistor inline with each fan to reduce the speed. I believe they are around 1A @ 12 volt. So that is about 12 ohms and 12 watts. So to drop the speed in half, I would need a 6 watt 12 ohm resistor, right?

A better solution would probably be to get one of these for each 3pin fan:

http://www.ebay.com/itm/PC-Fan-CPU-...682?pt=LH_DefaultDomain_0&hash=item4acc715682

Or these if I upgrade to 4pin fans:

http://www.ebay.com/itm/Mini-Fan-Co...631?pt=LH_DefaultDomain_0&hash=item3a8346e517

Any experience with any of these solutions would be greatly appreciated! Again, my preference would be to be able to monitor control all the chassis from the IPMI interface, but I realize that might not be possible.

 

[marbus90]

There's an even newer revision around which has its own IPMI port -> CSE-PTJBOD-CB3-O-P but it's SAS 12Gbps, so it might not work well with 6Gbps backplanes.

 

[pclausen]

Wow, that is PERFECT for what I need. Thanks!

I don't think it matters that it is intended for SAS3 / 12 Gbps backplanes since I don't intend to connect the power board to it. I just need to connect the ATX and PMBus connectors to the power supply and I'm good.

Looks like I also need this card bracket:

MCP-120-84702-0N

Got both ordered from wiredzone.com

 

[Apollo]

Is has a pair of 4 pin fan connectors installed. Looking at the PCB, it looks like it would be possible to install 6 additional 4 pin fan connectors! Unfortunately, I suspect that it would take more than just soldering them in to make them work.

Other than the 4 pin fan connectors, a very interesting feature on this power board is the inclusion of a 5 pin Power System Management Bus (I2C) connector. My X10 mobo has this as well and it is what allows me to monitor the power consumption via the IPMI web interface.

You are right about the additional 6 pin connector. From the picture, you are missing the connectors, fuses (X15 marking), pull-up resistors for PWM read signal from fan, a 16 pin IC which function is most likely to sample PWM signals from fan (I suspect a microcontroller). One transistor with a few discrete components.
As far as I can tell, the 4 top connectors are all connected together. Same with the 4 on the left. The missing transistor is most likely a N-Channel FET for speed regulation.
It seems now that the missing 16 pin IC is a microcontroller. It will drive the transistor to make the fan spin, and will monitor individual FAN speed.
For such system, I wouldn't be surprised at power-up the microcontroller is testing presence of every fan by running full blast then drops the speed to target settings. This way it can check if and when a fan fails and will trigger the alarm. The board may also include a temperature sensor. I would also think you can program into the system which fan is populated as an extra failsafe.

I would not go with this part:

http://www.ebay.com/itm/Mini-Fan-Co...631?pt=LH_DefaultDomain_0&hash=item3a8346e517
I only see the top components but I suspect there are none on the bottom side which means the regulation is done by the variable resistor. It is not going to last as the power rating for this is minimal.

The other link maybe OK, but why so many openings. I think its a bit better as it may incorporate transistor control similar in the method as the Supermicro board. Except a potentiometer is allowing for speed adjustment.
I doubt there are any failsafe such as buzzer and such.
Also, you will have to place them outside the case if you want to adjust the speed manually. Wouldn't it be cheaper and better to by the complete version of the Supermicro board instead?
With all the money you have put in this system it will be a shame to have a no name brand hardware fail.
You could possibly add the 2 missing connector and fuses, and if you are good at soldeting SMD part you could add the pull up resistor at the left of the fuse. Depeding on the microcontroller firmware you may have access to the 2 connectors fan speed, but no guaranty.
If you are handy, you could also add an extra FET transistr and wire it's Gate pin to the one already in place. Add 4 more connectors, fuses, no need for the pull up resistors and there you go. You will have speed controll over all of the fans, but you will not be able to sense speed on at least the 4 on the left.
It is a long shot but it could work.

 

[pclausen]

Appreciate the thoughts on the JBOD-CB2 board and I think you're right about it needing more components than just the fan headers. Don't know that I want to mess with that. Especially now that there's an IPMI based board. The CB2 can be had for around $50. The IPMI CB3 is closer to $100, but money well spent I think given all the features that it has. Here's what it looks like and it has plenty of 4 pin fan headers!

 

[Apollo]

Appreciate the thoughts on the JBOD-CB2 board and I think you're right about it needing more components than just the fan headers. Don't know that I want to mess with that. Especially now that there's a IPMI based board. The CB2 can be had for around $50. The IPMI CB3 is closer to $100, but money well spent I think given all the features that it has. Here's what it looks like and it has plenty of 4 pin fan headers!

This is definitely the better way to go.
Beside, as you stated, there is a IPMI interface, so you could even adjust the settings remotely while watching your movies.

 

[pclausen]

Haha yes that would be neat! I think what would be even better would be to have the fan speed switch from Optimal to Heavy I/O to 100% as needed to keep disks in each chassis below 40 degrees. I guess I can deal with the semi-automated approach of logging into the IPMIs of the different chassis to set the speeds as needed when I get emails on my phone from SMARTS. Once I get all 3 chassis mounted in the rack, I'll see what the temps are like and what the noise level is from the home theater side. I'm also thinking of rearranging the components in the rack so that the Supermicro chassis are down lower in the rack, right above the UPS units. This should help keep them cooler as well. The audio amps are only on when actually watching movies or entertaining guests with music playing.

 

[Apollo]

If you were to push the idea a notch, you could possibly implement active noise cancellation mind you this might be quite impossible in your situation.

 

[pclausen]

I tested creating a config backup, do a fresh install to a different USB stick, boot from it, and restore my config. Everything went well.

So now I'm thinking about going ahead and doing a badblocks test on all my drives for good measure. I understand that it will destroy the data on the drives, but what about my vpool/datastore configs and permissions? Will they survive? What about my jail configs?

Not a big deal to re-create everything as I just have test data / jails at the moment, but I'll want to document my permission setting and various other jail configs as it was a pain to get everything working correctly the first time around.

Another option might be to run badblocks on just 1 or 2 drives in each vdev and then resilver afterwards? That would also teach me how to deal with failed drives.

That JB3 power board should be here Tuesday according to FedEx tracking. Can't wait!

 

[Apollo]

Well, if you run badblock as intended, you will loose everything. As I understand, jails and some of the Freenas folder and files are also stored on you pool.
I would suggest, you replicate the pool (assuming you don't have much) to an external drive or on another server running ZFS. This way, you can replicate back into your server and restore everything.

 

[pclausen]

Thanks. So basically just create a new single disk vdev in a new zpool, create a datastore within in, and copy the content from my existing datastore into it?