SuperMicro X8DTN+ and SAS-826EL1 Compatible?

[Link Up]

Earlier I wanted to convert an Isilon server box but ran into some issues with the non-standard nature of the beast. I was encouraged to get a pure SuperMicro solution and have ordered one of these:
SuperMicro SC826E1-R800UB 12-bay (I am guessing this is the chassis, ad doesn't say)
SuperMicro SAS-826EL1 expander backplane
SuperMicro X8DTN+ with dual X5675 3.06gHz Xeons and 72GB of ECC RAM
SuperMicro AOC-SIMLP-3+ remote access card (not likely I will need it)
Adaptec 5405Z RAID Controller

I know from my previous research that I won't want to run RAID. I have already bought an LSI 9211-8i as the JBOD controller for the previous box.

Can I use the LSI controller with the SuperMicro and is that a doable configuration?

I also have a 120TB SSD which will be used for boot.

If in fact this server will work with FreeNAS, is there anything else I will need that I haven't thought of and will all this stuff play together? Thanks!

 

[scrappy]

You will be fine. That is basically my setup and I have had zero issues. Also, yes on the LSI card. I suggest buying a used one on eBay from a reputable seller and not those brand new cards from China or wherever they come from as they are usually fakes. Just be sure to flash your LSI card with the IT mode firmware so your disks are directly exposed to the OS.

120TB SSD? Maybe 120GB? ;)

FYI: You don't really need an SSD for boot. It won't hurt anything, but is definitely overkill. I personally use two mirrored SanDisk USB thumb drives connected to the MOBO's internal USB ports.

 

[Link Up]

You will be fine. That is basically my setup and I have had zero issues. Also, yes on the LSI card. I suggest buying a used one on eBay from a reputable seller and not those brand new cards from China or wherever they come from as they are usually fakes. Just be sure to flash your LSI card with the IT mode firmware so your disks are directly exposed to the OS.

120TB SSD? Maybe 120GB? ;)

FYI: You don't really need an SSD for boot. It won't hurt anything, but is definitely overkill. I personally use two mirrored SanDisk USB thumb drives connected to the MOBO's internal USB ports.

Glad to get the vote of confidence in the configuration. Yes, 120GB SSD..duh...

My main reason for going with an SSD is I am so clumsy, and anything sticking out the back is likely to get broken.

I already have a new LSI card from Amazon, both sold by and fulfilled by. Hope it is real, not that Amazon always sells authentic stuff, not that they are at fault.

The LSI card doesn't come in the right mode? I see from your signature you have an IBM card, so maybe you had to flash and I won't?

To be clear, I put an image on a thumb drive, then boot from that and install on the SSD, and then install the LSI card and flash it from the booted SSD?

Is yours also the 3Gbs expander backplane? Wondering if that will be the bottleneck, or if the drives will still be the slowest component? I am having a hard time wrapping my head around the fact that all 12 drives will be connected to a single connector. Wouldn't have needed an 8-port card if I was only going to use one of the connectors.

Thank you sir!

 

[scrappy]

Pretty sure the BPN-SAS2-826EL1 backplane operates at 6Gb/s. If I am wrong, somebody will definitely correct me. And yeah, one SAS cable will totally be sufficient for a twelve bay server. If there is a bottleneck, it is more likely in your network especially if you're using 1000baseT (1Gb/s) networking cards and/or cables which most people still use.

I won't go into detail about flashing your LSI card here because there are many tutorials and guides for the different ways of doing it based on the hardware you have. And no, your card will almost definitely NOT be set to IT mode when you get it. Many people (myself included) flash the card hooked up to a PC with a UEFI based motherboard before installing it into the server. It was quite unnerving the first time doing it, but once you have, you'll realize it's no big deal.

I don't think you understood what I said about the thumb drives. There are literally two USB ports on the motherboard inside the case. Nothing is exposed on the outside to accidentally break off. If you need to change one, you simply slide off the top server case cover.

 

[Link Up]

Pretty sure the BPN-SAS2-826EL1 backplane operates at 6Gb/s. If I am wrong, somebody will definitely correct me. And yeah, one SAS cable will totally be sufficient for a twelve bay server. If there is a bottleneck, it is more likely in your network especially if you're using 1000baseT (1Gb/s) networking cards and/or cables which most people still use.

I won't go into detail about flashing your LSI card here because there are many tutorials and guides for the different ways of doing it based on the hardware you have. And no, your card will almost definitely NOT be set to IT mode when you get it. Many people (myself included) flash the card hooked up to a PC with a UEFI based motherboard before installing it into the server. It was quite unnerving the first time doing it, but once you have, you'll realize it's no big deal.

I don't think you understood what I said about the thumb drives. There are literally two USB ports on the motherboard inside the case. Nothing is exposed on the outside to accidentally break off. If you need to change one, you simply slide off the top server case cover.

Hello again,

I have the SAS-826EL1, not the SAS2... so mine is 3Gbs not, 6. Would that change your mind on "weakest link"? I don't know for sure, but the main purpose of this build will just be to back up my main server, the one that is running 24 hours a day which is down due to the PS crashing during a power on after a power outage.

In my zillion years of working on servers, I have never had to flash a controller. I remember the Compaq servers were always being upgraded, but the upgrades were almost hands-off. Whatever flashing I have done was like with the Compaq where you stuck a floppy in and it was automatic, or like MOBO BIOS which you just run a program from Windows or whatever OS was running.

I don't have the system I am talking about here yet, it was shipped today, but I don't think my similar mobo in my other server has actually connectors to plug into. All I thought they had was header connectors to route to a front panel connector, not actual Type-A connectors you could directly plug into like on the back panel. Looking at a detailed picture, I do see the two internal Type-A connectors. Haven't had those on a mobo before.

The more I learn, the more I realize how much more I have yet to learn.

Thanks for your time to answer the questions..

Lew

 

[scrappy]

It ought to be adequate for backup and general use purposes. Only thing I'd worry about is if that backplane can support hard drives larger than 2TB which can be an issue with older, pre-SAS2 backplanes from what I've read.

Keep researching and learning everything you can. Many problems you'll run into have likely been discussed and solved on these forums.

 

[Link Up]

It ought to be adequate for backup and general use purposes. Only thing I'd worry about is if that backplane can support hard drives larger than 2TB which can be an issue with older, pre-SAS2 backplanes from what I've read.

Keep researching and learning everything you can. Many problems you'll run into have likely been discussed and solved on these forums.

I called SM yesterday and they said the SAS version only limits the speed to 3Gbs, not the drive size. Hope the tech was right.

I am dismayed that there is so much to getting one of these setup. From flashing the card to making a thumb drive to installing on the SSD to adding and configuring drives. It would seem a "mature" product like this would be no different than installing DOS, Windows, or other server installs I have done from the old Netware days up to Windows. Hope I can get there from here.

Thanks

 

[scrappy]

These steps are necessary for the ZFS filesystem to function properly. Flashing your HBA to IT mode allows for it to expose your hard drives directly to the FreeNAS operating system with no layers in between. This is the same for using ZFS on FreeBSD, Illumos, and Linux. ZFS must be able to see and operate the disks directly in order to reliably operate. These extra steps are the price we end users pay to have access to an enterprise-grade filesystem for FREE on DIY hardware. iXsystems (the makers of FreeNAS) do sell pre-built servers where all the difficult stuff is taken care of which is what you might refer to as a "mature" product. ;)

 

[Link Up]

Thanks so much for the further explanation. It happens my SuperMicro arrived today, a day early. I had a pleasant surprise. Instead of a BPN-SAS-826EL1, it is the BPN-SAS2-826EL1 version, i.e. 6Gbs. It is suprising they would undersell their product. I also noticed they have increased the price on the ones they still have to sell, but are still selling them with the wrong backplane identified.

I had another surprise and I haven't figured it out yet. I mentioned it shipped with an Adaptec 5405Z RAID controller which has just one SAS connector. What came in the box was the controller I expected, but it was obviously cabled to have a second controller, or a two SAS controller. There is a second cable running from the backplane to nothing. According to the SuperMicro site, the second port is for cascading, not for a second controller. So how it appears to be cabled doesn't match any of the options in the backplane manual. I don't know if that means I can connect the second port from the LSI to the second port for additional throughput or???

Also, the SuperMicro is about 1000 decibels quieter than the Isilon server. I don't know how it will sound under load, or when it is full of drives, but comparing empty to empty the SM system is much quieter. The Isilon, although the same Mobo, didn't have a fan setting option in BIOS, but this one does. I don't check the setting, but whatever it is set to allowed the fans to settle down and are much quieter. Even the PS fans are quieter, but I don't think redundant. Meaning the Isilon PS's had redundant fans, back to back, and these PS's don't, so right there is a sound savings. I am less concerned about redundant fans in the PS considering there is redundant PS's.

And the last surprise was that it came with a property sticker from where they got it from. Happens the web host is the same web host I have a leased server at. Small world :)

 

[Link Up]

Making progress...made a thumbdrive with Rufus, put in the SSD, booted from the thumbdrive and installed on the SSD. Faked my way through the IT firmware change as the best instructions I found didn't work, so I used Rufus to make a MS-DOS drive and didn't have the 2GB issue the article had. Booted from the thumbdrive into DOS, executed the two SAS2FLSH commands and was done with that. Haven't installed any drives yet so I am studying on the volume stuff. I have to figure out how many drives I am going to put in to start with. Also figuring out how to set up the Windows share...but I am pleased with the hardware and my progress so far. Thanks again!

 

[scrappy]

Sounds like you're well on your way! Progress, whether it's slow, fast, or with a few setbacks is still good nonetheless. The hardest part when starting out (for me, anyway), was flashing the HBA because the PC I used had an ASUS Motherboard with UEFI known to cause problems for this HBA flashing procedure. It took a while make it work. Also, I was worried about bricking it by accident since I had never flashed hardware like it before.

 

[Link Up]

I didn't try the UEFI method, just a DOS boot from thumbdrive and two command lines to type...couldn't have been easier considering the instructions were for a different method.

I see a lot of guides on what the ZFS system is about, but haven't found anything about why to do it what way based on the drives you have. There must be some guidelines on what to do with a given mix of various sizes of drives. Seems I can't see the forest for the trees. I found the one guide on ZFS, but after a 66 page powerpoint and 14 pages of updates, changes, corrections, makes for a big mess. Seems getting it running on the SSD was simple but actually getting the volume built is another matter.

(banging head on wall)

 

[scrappy]

How many drives and what size(s)? Are you using FreeNAS mainly for general file storage?

 

[Link Up]

Hello again Mr. Helpfull!

Here is my situation.

My PC has 6TB RAID w/WD Reds and three BarraCuda 5TB expansion drives attached via USB for primary backup.

I have a ReadyNAS 6 bay NAS that has a bad PS now. I have the parts and need to get it going again. It is my secondary backup and has content not on my PC, primarily my photos as I am a photographer. It all of my digital and scanned photos as well as a large movie and TV series library. I use a Minix media player as well as a Fire and a Roku for viewing my content and transcoding is done there, not in the NAS. I tried Plex and had issues with the product. Not performance, but management. I use Tiny Media Manager to sort, organize, and manage posters, etc. The ReadyNAS just serves the content to one of the players.

I had bought the Ilison to run as a Windows Server with a 12-port 3Ware RAID controller, got it going, but got bogged down, never finished getting it going. It was going to be my backup to the ReadyNAS. When I began running into issues with the Ilison, like noise, so I bought the SuperMicro I am working on now.

So the SuperMicro I just bought will be the backup to the ReadyNAS. Tertiary backup for part of my content, secondary backup for the rest.

The SM, as far as my plans go now, will only be to backup the ReadyNAS, not to perform any other function.

Drives:
ReadyNAS
2-2TB WD Green, 2-3TB WD Red, 2 4TB HGST Deskstars

Additional:
2-2TB WD Green, 6-2TB Hitachi Deskstars

Can be shucked:
1-8TB Seagate BarraCuda
1-8TB Seagate Archive HDD
2-8TB WD Red

My current plan is to shuck the two 8TB WD Reds and swap them for the two 2TB WD Greens in the ReadyNAS giving me four 2TB Greens for the FreeNAS.

Since I don't need the three 5TB drives on the PC, I could use the Seagate 8TB Archive HDD for that purpose, freeing up the three 5TB BarraCudas, but would also need to be shucked. I could put two in the ReadyNAS, swapping out the two 3TB Reds.

If I do all that shucking and swapping around, it would leave:

4-2TB WD Green
6-2TB Hitachi Deskstars
2-3TB WD Reds

Extra if I used the 12 drives above
1-5TB BarraCuda
1-8TB BarraCuda

Some of my logic is based on my minimal knowledge of FreeNAS.

1. There is a preferred limit of 10-11 drives in a pool?
2. The capacity of a pool is based on the smallest drive in the pool? Ie, if I have a pool of 4 or six 2TB drives, to see an increase in space all four or six would need to be upgraded to something larger?

Therefore it might make most sense to have three pools? But that would leave the two WD reds "alone", only being able to mirror them? Should the two reds go with the four greens and have two pools of six drives each? Then upgrade the WD Greens over time, letting me access the other 2TBs "lost" on the 3TB drives?

Since I don't have the second two WD Greens or the two WD Reds until I get the ReadyNAS going and can begin upgrading the drives in it. Makes me think of starting the FreeNAS with just the six Hitachis, then add the other drives when they have been freed up.

Sorry for the diatribe, but to find a solution means taking into consideration the ReadyNas and my PC's storage.

Seems I will have to buy you a beer :)

 

[danb35]

There is a preferred limit of 10-11 drives in a pool?
The capacity of a pool is based on the smallest drive in the pool?

Both of these are true of vdevs, not of pools. A pool consists of one or more vdevs. As you see in my sig, my pool consists of one RAIDZ2 vdev of 6 x 6 TB, one RAIDZ2 vdev of 6 x 4 TB, and one RAIDZ2 vdev of 4 x 3 TB + 2 x 2 TB. Total usable capacity is 48 TB, or 4 x 6 TB + 4 x 4 TB + 4 x 2 TB (not accounting for ZFS overhead, the 80% rule, etc.).

 

[Link Up]

In that I am feeling adventurous, I forged ahead, knowing I could always get back to zero. I plugged in the disk drives, all 8 that were handy, and the system saw them. I told the system to only use the six Hitachis and it made a SZ2? It enabled compression. I haven't used compression before and don't know that I want to.

I then set up a Windows share and was able to view the volume on my PC.

Haven't touched anything else, not the network, not anything so it is as bare as can be, but the fact I have a usable share is making me smile. So, what I created is a poo with one vdev? What I named wasn't a pool, but was a vdev? The webGUI says you are setting up a volume which you name, tell it the drives to include, and the "group" type. If behind the scenes you are creating pools and vdevs, it would be nice if the nomenclature was consistent.

The assumption then is I can add the next 4 drives, create a second vdev, and then add it's space to the current volume, or it can have it's own volume name?

I basically faked it, but I have read the Uncle Fester guide, and the 66-page ZFS doc which basically blew my mind up, what little is left. Fester's guide keeps thing simplistic.

Thanks again!

 

[danb35]

It enabled compression.

Compression is the default, and there's rarely a reason not to use it.

So, what I created is a pool with one vdev?

Yes.

What I named wasn't a pool, but was a vdev?

No, you named the pool. Vdevs don't have names.

The assumption then is I can add the next 4 drives, create a second vdev, and then add it's space to the current volume,

Correct.

or it can have it's own volume name?

No; as above, vdevs aren't volumes and don't have their own names.

it would be nice if the nomenclature was consistent.

Yeah, it would. The FreeNAS devs have, for reasons known only to them, decided to depart from the standard ZFS nomenclature. Presumably they think their chosen phrasing is easier to understand, and it might be, but at the cost of confusion of anyone who knows ZFS.

 

[Link Up]

Maybe I should ask that a different way. Can I create a second volume and then put a vdev in that, or is there only ever one volume and all vdevs go into it?

Thanks again Wiz...

 

[danb35]

Can I create a second volume and then put a vdev in that,

Sure you can. You can have as many pools (volumes) as you like.

 

[Stux]

zero or more pools/volumes, each composed of one or more vdevs, each composed of one or more HDs.

You can add pools at any time, and you can add vdevs to pools at any time.

You can't add/remove HDs to/from a RaidZ[123] vdev.

And you can't remove vdevs from a pool.