DrKK
FreeNAS Generalissimo
- Joined
- Oct 15, 2013
- Messages
- 3,630
This is a companion piece for my earlier post.
So, I just want to go through here stream-of-consciousness style, and tell everyone what I did during the build process, and give a lot of observations and tips and tricks that I encountered. Hopefully you will find it interesting, and/or someone in the future may have a question or two answered.
1. Arrival of Parts: I had three separate orders, two from Newegg, and one from Amazon (a single item, the pimp LED CPU fan). Most of this was not supposed to arrive until Thursday of this week, but ALL of it arrived by Wednesday, to my delight. As expected from Newegg, my hard drives were more or less just tossed in the box without much attention being paid to securing them very well. Whatever. I'm sure it's fine.
2. Banning of women of children: I declared the dining room to be the official DrKK work area, and no wives (I have just the one wife, but had I had several, they would have been all banned from the dining room), or children, or any associates or guests thereto appertaining were to set foot in the room for any reason whatsoever, under threat of serious sanction, until such time that the build was complete, or Saturday, whichever came first.
3. Step one: Unbox the motherboard, CPU, and RAM. The box the motherboard came in, plus the foam, provides a convenient work surface. RTFM to find out which DIMM slots are to be populated first, and install RAM. No problem, pretty standard. Install CPU. No problem, pretty standard. Nine year old enters dining room at this point, violating aforementioned rule, and is sanctioned appropriately, and told to bring me his monitor from his bedroom, and to grab a 20 ft VGA cable from the cabinet, and a 20ft PSU power cable, because I'm going to need it during testing. Child, sensing a test or trap, states that bringing me the aforementioned items back to the dining room would be a second violation of the dining room prohibition. I inform child that he has a special dispensation to retrieve the said items and bring them to the dining room, but that after that point, he is not to reenter the dining room unless beckoned for further slave labor.
4. Testing the completely unnecessary, ridiculous, pimptastic cooler I totally unnecessarily bought. Fired up massive Cooler Master Hyper 212 LED cooler by itself. Shout to wife: "This thing is {profanity} awesome, woman, check this out." Wife states: "That thing is stupid." I state: "Get out of the dining room, we mock what we don't understand!" Fortuitously, at this moment, I receive a tweet from @Allan Jude , complimenting me on my choice of pimp CPU fan, and stating that he bought the same for himself. I curse my wife's lack of appreciation for LEDs, and praise Allan's name for his excellent taste.
5. Installing CPU cooler. I look at the sorry excuse for thermal compound that Cooler Master provides with the cooler. I debate whether or not to fetch, or send a child to fetch, proper thermal compound from the cabinet. I decide I am too lazy, and the supplied compound will have to do. I pre-tint the cooler's contact surface with a bit of the compound, because any time you have copper heat pipes meeting an aluminum contact plate, there's always imperfections there, so let's fill those in. If the situation is real bad, you may have to "lap" the contact surface; but fortunately that is not necessary with this product. I wipe off the excess (with a coffee filter---why? Coffee filters have no lint, see...). I treat the CPU cap with a light dusting of 91% isopropyl, and apply a small amount of compound as a dot in the center, and apply the massive cooler to the CPU, and make sure the screws are set as strong as they can without risking busting the motherboard.
6. Testing the mobo, CPU, and RAM on bench. This is a step I always recommend, that most people do not do. In this case, I was a little bit nervous, because I strayed from the qualified vendor part for memory, so there was a chance it would not work as expected. I leave my motherboard on the box + foam (do **NOT** use the conductive bag it came in to set it on, because you will fry your motherboard!!), unbox the Seasonic PSU, and plug in only the main 24-pin connector. At this point, the monitor has been brought down, and I carefully connect the long VGA cable between it and the motherboard across the room, as well as the 20 ft power cord. I pull the wireless keyboard and dongle from the HTPC to use for testing. I notice the IPMI light come on, and start flashing. OK, that's good. Then I power up the motherboard. I see the monitor immediately detect the video signal (good), and then a strange thing happens....
7. Strange boot. My X11SSM-F throws about 30 POST codes, goes blank, boots again, throws about 30 POST codes, goes blank, boots again. Jesus! Maybe the RAM doesn't work, after all! But it seems to be getting...better? Hmph. Weird. I hit DEL furiously to get into the BIOS and see what's up. It puts me in BIOS.
8. Setting the BIOS. Nice, I have BIOS v2.0a, which is the latest. So I don't have to install a new BIOS update, which I hate doing. OK, let's see what we got. I turn off virtualization technology (waste of power--about 2 watts or whatever), turn off the parallel and/or serial port(s), etc. etc. etc. I note the boot screen looks way more complex than the X10 mobos...for future reference. Make sure the IPMI is in "failover" mode, so I can access it on the main interface.
9. Testing IPMI. I fetch a 100ft ethernet cord (these are must-haves, if you don't have one, get one), and string the cable to the nearest switch/hub. I inform wife and children that the long green ethernet cord should be considered an extension of the dining room, like an embassy of the Republic of Dining Room, and no one is to touch it, or trip over it, under the threat of serious penalty. I use the Fing App for Android to see what IP address the IPMI got. In this case, it was 192.168.1.196. I go to the main computer to access IPMI. Ugh, it's still got the Java crap on it, @m0nkey_ lied when he said this mobo had the new IPMI on it. I click on "Sensor Readings", because I'm curious about the CPU temperature, because if the cooler is installed correctly, it should be 25ish, and if not, 80ish. :) WHOOPS!!!!!!!!!!! Every sensor shows "Not Present". Every fan sensor, every temperature sensor, every voltage sensor. Not Present. Well, the mobo is sitting there in the BIOS screen? Maybe the sensors don't work in the IPMI until you get past the BIOS screen or something? Interesting, I never knew that. I ask @m0nkey_ about this in Mumble, he claims that the sensors should work when the mobo is in the setup screen, and that he "does it all the time". But then I remember that he was already caught once, and I discard his lies. Fool me once, shame on you. But twice? shame on me! Alright, whatever, we'll assume this sensor thing will resolve itself in due course, and that I put the cooler on correctly.
10. I boot it a few more times, but now it seems to boot as I expect. Then I recall something that I believe @joeschmuck (or someone) said about how the first use with the 2.0a BIOS resulted in a weird boot and inability to boot until there was a full hard reset. Cool. I then put the PSU and Mobo in the case. Along the way I discover an odd motherboard standoff, already pre-installed, with a bumper on it. It does not match any feature of the mobo, and in fact, its presence seems to be a problem actually installing the mobo. I get pliers and remove it---wonder what it was supposed to be for? Anyway, pro tip: The backplate for the case that Supermicro supplies is only punched out for the minimal/lowest model. You will more than likely have to remove several of the punchouts yourself to match your motherboard outputs. I do so.
11. I go downstairs to the main computer area, and go to freenas.org downloads, and I notice, "LOL, they took the Corral totally off the download screen now. lolololol.", download the latest 9.10.2 U3 or whatever it is, and burn it to a CD. Then I get my trusty external USB CD/DVD drive (you DO have one of these, don't you?), bring it upstairs, along with a random SSD out of my "Box O' SSD's"---I choose an older, but quite reliable 64GB Sandisk to hold the OS, at least for now. Installation goes very well (once I make sure the CD precedes the SSD in the boot order), and I note with approval that I can now choose "UEFI" or "BIOS" for booting method during setup. Very satisfied with this feature, I gladly click "UEFI". I remove the CD drive, and boot the machine now (with a knife, of course, shorting out the power jumper pins, because the only thing I've connected thus far is the 24 pin main power cable). Everything boots, I configure my FreeNAS for lagg failover, it gets an IP address, everything's great. I access my (still unconfigured) FreeNAS from another machine, and I pray that IPMI sensors will now be working. I drop to shell and hit and sure enough, everything's fine, my CPU is 23C. I apparently did the cooler right. Strange that IPMI didn't report my sensors earlier; whatever, it's working now. While I'm at it, I set up the SSH service in the GUI, because I'll need that. I don't worry about security or certificates yet. I disconnect all power to the system, and we're ready to finish.
12. Now it's time to connect the various case features to the motherboard and PSU. What a great case this Fractal Design R5 is. For example, they have a nice fan controller for 3 case fans with 3 speed settings. I decide to use this, instead of the mobo fan headers (except the CPU fan, which must be installed to the mobo header). I notice that I have a power LED on the case, and a HDD LED on the case. Why do I need an HDD LED? It'll always be on, pretty much. So I get creative: I attach the HDD LED to the header for the NETWORK ACTIVITY on LAN1. Now my HDD LED is actually a network activity indicator. This case has all kinds of sound-dampening foam and windows you can open and whatnot. I open all that crap up. I have a $1300 NAS, I'm getting every bit of air flow in this case that I can.
13. Time to install the drives. They have a fascinating system for this. Each of the trays has 4 grommets, each of which can be in a front or back position. Arbitrarily, I choose "front", and begin attaching the hard drive to the tray. I notice that the grommets are vibration-dampening, AND, they give a nice bit of clearance to the underside of the drive for airflow. Nice design. I mount my five HDDs, and attach the SATA power and SATA data cables. Pro tip: in a big case like this, if you use a small motherboard (like the X11SSM), standard sizes for things like cables are BARELY going to make it. Plan ahead.
14. I boot. Weird, not a sound from the fans, or drives. Did I screw something up? Turns out I did not. These drives make 0.0000dB of noise booting and operating, and the fans Fractal Design uses in the case are ultra silent 140mm jobs. I drop to shell. shows my boot drive, and 3 of the 5 drives!!?! A quick check shows I did not attach the Molex power to the cable I used to power two of the drives. Whoops. Fixed that. Restart. Camcontrol devlist now shows all 5 drives plus the SSD. Superb.
15. Now I use the CLI to start "SMART conveyance" tests on all the drives. This allegedly checks for "shipping damage", but I am skeptical. It informs me it will take 5 minutes. All drives come back "good" so far. Whatever that means.
16. So I finish the case, perform the cable management, tie off cables, etc. WHOOPS, can't get the side panel on. Turns out, unless you have angled connectors, the "front" setting for the grommets on the drives means the connectors project out too much. Super. So I have undo all the drives and drive trays, and fix the grommets, and reset everything. Side panel now gets on.
17. I remove 100 ft ethernet cable from area, and announce to household that the dining room is free to be accessed.
18. I take the NAS down into the "server room", put it next to my existing NAS, and hook up a proper power cord and ethernet cable to it, which I attach directly to the main LAN router, and I plug the box into a wireless wattmeter from Ubiquiti, then into the wall. I boot. There is a spike to about 80 watts during the initial drive spin up, but the NAS at idle, even with 5 drives, is only drawing 44 watts. NICE!
19. Now we do a SMART long test on all drives. This is going to take 8 hours. I use the CLI to kick off a smartctl -t long, for each drive. If those all go well, we'll actually configure the FreeNAS itself tomorrow, and begin transfering files.
20. OK, let's post this on the FreeNAS forum.
And there we are. Once the long tests finish up this morning, I'm sure I will have a follow-on report for the FreeNAS configuration itself, and the beginning of the file transfers.
So, I just want to go through here stream-of-consciousness style, and tell everyone what I did during the build process, and give a lot of observations and tips and tricks that I encountered. Hopefully you will find it interesting, and/or someone in the future may have a question or two answered.
1. Arrival of Parts: I had three separate orders, two from Newegg, and one from Amazon (a single item, the pimp LED CPU fan). Most of this was not supposed to arrive until Thursday of this week, but ALL of it arrived by Wednesday, to my delight. As expected from Newegg, my hard drives were more or less just tossed in the box without much attention being paid to securing them very well. Whatever. I'm sure it's fine.
2. Banning of women of children: I declared the dining room to be the official DrKK work area, and no wives (I have just the one wife, but had I had several, they would have been all banned from the dining room), or children, or any associates or guests thereto appertaining were to set foot in the room for any reason whatsoever, under threat of serious sanction, until such time that the build was complete, or Saturday, whichever came first.
3. Step one: Unbox the motherboard, CPU, and RAM. The box the motherboard came in, plus the foam, provides a convenient work surface. RTFM to find out which DIMM slots are to be populated first, and install RAM. No problem, pretty standard. Install CPU. No problem, pretty standard. Nine year old enters dining room at this point, violating aforementioned rule, and is sanctioned appropriately, and told to bring me his monitor from his bedroom, and to grab a 20 ft VGA cable from the cabinet, and a 20ft PSU power cable, because I'm going to need it during testing. Child, sensing a test or trap, states that bringing me the aforementioned items back to the dining room would be a second violation of the dining room prohibition. I inform child that he has a special dispensation to retrieve the said items and bring them to the dining room, but that after that point, he is not to reenter the dining room unless beckoned for further slave labor.
4. Testing the completely unnecessary, ridiculous, pimptastic cooler I totally unnecessarily bought. Fired up massive Cooler Master Hyper 212 LED cooler by itself. Shout to wife: "This thing is {profanity} awesome, woman, check this out." Wife states: "That thing is stupid." I state: "Get out of the dining room, we mock what we don't understand!" Fortuitously, at this moment, I receive a tweet from @Allan Jude , complimenting me on my choice of pimp CPU fan, and stating that he bought the same for himself. I curse my wife's lack of appreciation for LEDs, and praise Allan's name for his excellent taste.
5. Installing CPU cooler. I look at the sorry excuse for thermal compound that Cooler Master provides with the cooler. I debate whether or not to fetch, or send a child to fetch, proper thermal compound from the cabinet. I decide I am too lazy, and the supplied compound will have to do. I pre-tint the cooler's contact surface with a bit of the compound, because any time you have copper heat pipes meeting an aluminum contact plate, there's always imperfections there, so let's fill those in. If the situation is real bad, you may have to "lap" the contact surface; but fortunately that is not necessary with this product. I wipe off the excess (with a coffee filter---why? Coffee filters have no lint, see...). I treat the CPU cap with a light dusting of 91% isopropyl, and apply a small amount of compound as a dot in the center, and apply the massive cooler to the CPU, and make sure the screws are set as strong as they can without risking busting the motherboard.
6. Testing the mobo, CPU, and RAM on bench. This is a step I always recommend, that most people do not do. In this case, I was a little bit nervous, because I strayed from the qualified vendor part for memory, so there was a chance it would not work as expected. I leave my motherboard on the box + foam (do **NOT** use the conductive bag it came in to set it on, because you will fry your motherboard!!), unbox the Seasonic PSU, and plug in only the main 24-pin connector. At this point, the monitor has been brought down, and I carefully connect the long VGA cable between it and the motherboard across the room, as well as the 20 ft power cord. I pull the wireless keyboard and dongle from the HTPC to use for testing. I notice the IPMI light come on, and start flashing. OK, that's good. Then I power up the motherboard. I see the monitor immediately detect the video signal (good), and then a strange thing happens....
7. Strange boot. My X11SSM-F throws about 30 POST codes, goes blank, boots again, throws about 30 POST codes, goes blank, boots again. Jesus! Maybe the RAM doesn't work, after all! But it seems to be getting...better? Hmph. Weird. I hit DEL furiously to get into the BIOS and see what's up. It puts me in BIOS.
8. Setting the BIOS. Nice, I have BIOS v2.0a, which is the latest. So I don't have to install a new BIOS update, which I hate doing. OK, let's see what we got. I turn off virtualization technology (waste of power--about 2 watts or whatever), turn off the parallel and/or serial port(s), etc. etc. etc. I note the boot screen looks way more complex than the X10 mobos...for future reference. Make sure the IPMI is in "failover" mode, so I can access it on the main interface.
9. Testing IPMI. I fetch a 100ft ethernet cord (these are must-haves, if you don't have one, get one), and string the cable to the nearest switch/hub. I inform wife and children that the long green ethernet cord should be considered an extension of the dining room, like an embassy of the Republic of Dining Room, and no one is to touch it, or trip over it, under the threat of serious penalty. I use the Fing App for Android to see what IP address the IPMI got. In this case, it was 192.168.1.196. I go to the main computer to access IPMI. Ugh, it's still got the Java crap on it, @m0nkey_ lied when he said this mobo had the new IPMI on it. I click on "Sensor Readings", because I'm curious about the CPU temperature, because if the cooler is installed correctly, it should be 25ish, and if not, 80ish. :) WHOOPS!!!!!!!!!!! Every sensor shows "Not Present". Every fan sensor, every temperature sensor, every voltage sensor. Not Present. Well, the mobo is sitting there in the BIOS screen? Maybe the sensors don't work in the IPMI until you get past the BIOS screen or something? Interesting, I never knew that. I ask @m0nkey_ about this in Mumble, he claims that the sensors should work when the mobo is in the setup screen, and that he "does it all the time". But then I remember that he was already caught once, and I discard his lies. Fool me once, shame on you. But twice? shame on me! Alright, whatever, we'll assume this sensor thing will resolve itself in due course, and that I put the cooler on correctly.
10. I boot it a few more times, but now it seems to boot as I expect. Then I recall something that I believe @joeschmuck (or someone) said about how the first use with the 2.0a BIOS resulted in a weird boot and inability to boot until there was a full hard reset. Cool. I then put the PSU and Mobo in the case. Along the way I discover an odd motherboard standoff, already pre-installed, with a bumper on it. It does not match any feature of the mobo, and in fact, its presence seems to be a problem actually installing the mobo. I get pliers and remove it---wonder what it was supposed to be for? Anyway, pro tip: The backplate for the case that Supermicro supplies is only punched out for the minimal/lowest model. You will more than likely have to remove several of the punchouts yourself to match your motherboard outputs. I do so.
11. I go downstairs to the main computer area, and go to freenas.org downloads, and I notice, "LOL, they took the Corral totally off the download screen now. lolololol.", download the latest 9.10.2 U3 or whatever it is, and burn it to a CD. Then I get my trusty external USB CD/DVD drive (you DO have one of these, don't you?), bring it upstairs, along with a random SSD out of my "Box O' SSD's"---I choose an older, but quite reliable 64GB Sandisk to hold the OS, at least for now. Installation goes very well (once I make sure the CD precedes the SSD in the boot order), and I note with approval that I can now choose "UEFI" or "BIOS" for booting method during setup. Very satisfied with this feature, I gladly click "UEFI". I remove the CD drive, and boot the machine now (with a knife, of course, shorting out the power jumper pins, because the only thing I've connected thus far is the 24 pin main power cable). Everything boots, I configure my FreeNAS for lagg failover, it gets an IP address, everything's great. I access my (still unconfigured) FreeNAS from another machine, and I pray that IPMI sensors will now be working. I drop to shell and hit
Code:
ipmitool sensor
12. Now it's time to connect the various case features to the motherboard and PSU. What a great case this Fractal Design R5 is. For example, they have a nice fan controller for 3 case fans with 3 speed settings. I decide to use this, instead of the mobo fan headers (except the CPU fan, which must be installed to the mobo header). I notice that I have a power LED on the case, and a HDD LED on the case. Why do I need an HDD LED? It'll always be on, pretty much. So I get creative: I attach the HDD LED to the header for the NETWORK ACTIVITY on LAN1. Now my HDD LED is actually a network activity indicator. This case has all kinds of sound-dampening foam and windows you can open and whatnot. I open all that crap up. I have a $1300 NAS, I'm getting every bit of air flow in this case that I can.
13. Time to install the drives. They have a fascinating system for this. Each of the trays has 4 grommets, each of which can be in a front or back position. Arbitrarily, I choose "front", and begin attaching the hard drive to the tray. I notice that the grommets are vibration-dampening, AND, they give a nice bit of clearance to the underside of the drive for airflow. Nice design. I mount my five HDDs, and attach the SATA power and SATA data cables. Pro tip: in a big case like this, if you use a small motherboard (like the X11SSM), standard sizes for things like cables are BARELY going to make it. Plan ahead.
14. I boot. Weird, not a sound from the fans, or drives. Did I screw something up? Turns out I did not. These drives make 0.0000dB of noise booting and operating, and the fans Fractal Design uses in the case are ultra silent 140mm jobs. I drop to shell.
Code:
camcontrol devlist
15. Now I use the CLI to start "SMART conveyance" tests on all the drives. This allegedly checks for "shipping damage", but I am skeptical. It informs me it will take 5 minutes. All drives come back "good" so far. Whatever that means.
16. So I finish the case, perform the cable management, tie off cables, etc. WHOOPS, can't get the side panel on. Turns out, unless you have angled connectors, the "front" setting for the grommets on the drives means the connectors project out too much. Super. So I have undo all the drives and drive trays, and fix the grommets, and reset everything. Side panel now gets on.
17. I remove 100 ft ethernet cable from area, and announce to household that the dining room is free to be accessed.
18. I take the NAS down into the "server room", put it next to my existing NAS, and hook up a proper power cord and ethernet cable to it, which I attach directly to the main LAN router, and I plug the box into a wireless wattmeter from Ubiquiti, then into the wall. I boot. There is a spike to about 80 watts during the initial drive spin up, but the NAS at idle, even with 5 drives, is only drawing 44 watts. NICE!
19. Now we do a SMART long test on all drives. This is going to take 8 hours. I use the CLI to kick off a smartctl -t long, for each drive. If those all go well, we'll actually configure the FreeNAS itself tomorrow, and begin transfering files.
20. OK, let's post this on the FreeNAS forum.
And there we are. Once the long tests finish up this morning, I'm sure I will have a follow-on report for the FreeNAS configuration itself, and the beginning of the file transfers.
Last edited:
