Used Supermicro 4U server or homebuilt?

[Death Dream]

You just need one of these internal brackets to put your boot device in. That is what I use. I thought I had already posted that info.
Don't waste a hot swap bay on a boot drive.

View attachment 29266

View attachment 29267

This is the bracket you need.
https://www.newegg.com/Product/Product.aspx?Item=N82E16816101828

I have 12 open bays so I didn't care if I burned them at this point. Nice bracket though!

 

[Bozon]

I have 12 open bays so I didn't care if I burned them at this point. Nice bracket though!

But still hook the boot drives to the onboard SATA connectors, and forget about having them run off of the SAS controller. Don't fight for something that really doesn't make any sense (running the boot drives on the SAS controller.)

 

[Death Dream]

But still hook the boot drives to the onboard SATA connectors, and forget about having them run off of the SAS controller. Don't fight for something that really doesn't make any sense (running the boot drives on the SAS controller.)

Yeah I didn't spend to much time trying to get it to work. Figured it wouldn't.

Now what was lovey was when I tried to transfer my data over from my old FreeNAS server that is taking a crap. First I was just being lazy and tried to transfer the 7TB of data through FTP with a 100Meg cap. Gez that was slow. My old server crashed (panic) and made my main Volume non-importable. Only way I could get to it was in read only. So I transfer the 4 drives to my Supermicro box. corrupted my BIOS once I installed the drives. Couldn't boot off of anything or get into recovery mode. Thankfully IPMI can upgrade the BIOS. Now, import it there as read only and copy paste the files over to my new volume.

Literally, I get past one wall to get hit by another a second later. What should of been an easy process all around has proven just the opposite.

 

[Andrew Ostrom]

Yes, that is essentially what the Bell Lab Journal articles talk about. The early system predated Unix and didn't have much in common with it. It is only when they promised to develop a word processor, so that they could get funding for a better machine, that the development took off. So it was probably the best trojan horse project ever. Sure we will develop a word processor and by the way we developed a computer language, compiler, and operating system also. Because the only real deliverable was the word processor ( nroff ), they really didn't have to deal with operating system bloat.
The biggest issue with most of the other operating systems of the day was complexity. The Unix developers did away with this by defining almost nothing as part of the operating system. All the utilities were stand alone applications that were built on top of the operating system. You want to interface with the operating system, then you'll need to run the shell program, sh. You wan't to copy files, you will need to run the program cp.

This is still true today, nothing makes the GNU people madder than talking about Linux and all of the GNU tools as Linux. They even put it in their acronym definition. GNU -- GNU's Not Unix. Linus develop an operating system, but most of the things we use on it were developed by the GNU people.

Of course, FreeNAS is developed on top of BSD, and that is a different story entirely.

My first job after college was in 1979, at DEC, writing application code on a PDP-11/45 and PDP-11/70 running RSTS/E. A year or so later a college friend's brother was taking a summer school course at Harvard and I was asked to help. He was learning C on a PDP-11 running Unix. That was my first exposure to Unix, and it felt so primitive compared to RSTS/E or RSX-11. By then I had stopped writing code (mostly) and was doing internal network implementation with PDP-11, PDP-10, PDP-20, and the new VAX/VMS systems. (Most groups I was in always seemed to keep an old PDP-8 around, just to run the Lunar Lander game, too.) Later I was in marketing and product management (mostly). In the 80's I was the marketing interface to the ARPAnet, DARPAnet, and HEPnet folks (HEPnet was all the High Energy Physics labs like Los Alamos, Livermore, etc). The HEPnet was a DECnet network and the others ran TCP/IP (Version 2) and we provided the gateway/firewall technology to them.

Honestly, a lot of the early Unix concepts were pretty much copies of stuff we were providing in RSX-11 or RT-11. AT&T was one of our biggest hardware customers (virtually every telephone central office in the U.S. had one or more PDP-11s in it). We made special configurations for them, too (for example we put more hardware in a cabinet (rack) than "standard" to save space), and, in large part, they came up with Unix so they didn't have to buy OS software from us. Our operating systems were also constructed so the "system utilities" were separate, on RSTS/E, for example, they all came as source files that were compiled when you installed the OS. Our user groups had libraries of modified or replacement utilities, which was specifically allowed in the copyright.

The file editor on RT-11 and RSX-11 was called "Runoff" - AT&T 's"nroff" was their clone that (supposedly) didn't infringe on our copyrights. The DEC file manipulation utility, PIP (peripheral interchange program) was pretty closely copied in the various Unix commands, etc. Later on, TOPS-20 and VMS made the command line much more powerful and intuitive, but in many ways the Unix shells are still stuck in the 1970s... The way they mimicked our software was often a bone of contention, but they bought a log of hardware (10s or 100s of millions of dollars) and DEC never really cared about software revenue in those days.

Enough rambling...

 

[Andrew Ostrom]

Just an update on my experience with this server in case someone else is inclined to go the same route. I have to say it was pretty painless, except for one issue (highlighted below).

I bought the server, as I described it in the first post, for $599. Subsequently I:

1) Replaced the supplied with a SAS 9207-8i, in IT mode, that I got on eBay for $59.
2) Removed the Intel quad port NIC - I'm just using the onboard 1GbE port until I upgrade to 10GbE someday soon.
3) Installed 2 120GB SanDisk SSD Plus drives (2x$20.99), connected to onboard SATA with spiffy bundled SATA cable ($8.99) (https://smile.amazon.com/gp/product/B00VI3MFMO/ref=ppx_yo_dt_b_asin_title_o04_s00?ie=UTF8&psc=1 ), and the Supermicro tray ($11.39)
4) I bought 8 Seagate ES.3 3TB SAS drives on eBay for $312
5) Transferred about 15TB of my data from my Windows PC - that freed up 4 3TB SATA drives.
6) Installed another 64GB or ECC memory thanks to the find on eBay by "Death Dream" - $96
7) Bought a lot of 6 3TB IBM/Seagate SAS drives for $145.50, and installed them an 2 of my SATA drives as a second vdev (all raidz2).
8) Transferred the rest of my data, freeing up 4 4TB Seagate IronWolf drives.
9) Bought 5 4TB Seagate ES.3 SAS drives for $299.50, combined with 3 IronWolf SATA drives and installed as a 3rd vdev.

So, in summary, I spent $1,580.36, and now I have a commercial grade server with 32.29TB of total pool space configured as 3 8 drive raidz2 vdevs.

Some observations:

Running memtest on 128GB of memory and badblocks on all those drives took FOREVER. on the 5 new (to me) 4TB drives badblocks took 4 days to finish one pattern, so I decided that was good enough and stopped it. I have spare drives of both sizes, so I'll just swap anything if it fails.

Getting into the BIOS on the Supermicro motherboard is just about impossible. I got in once, and I don't even know exactly how it happened - i haven't been able to get back in again.

I ended up using both the "import disk" fjunction in the GUI and figuring out how to mount my NTFS disks with ntfs-3g - the GUI is pretty useless at providing status, and if you have a crash (I did - we had thunderstorms), it will not re-import the same disk...

Even with all these disks my new (to me) Tripp-Lite UPS says the system is drawing about 440W, so that's really not bad.

Now to my one issue...

When I installed Freenas I used the two SSDs in a mirrored config. Everything went fine and then a day (maybe) later I got an alert that the boot pool was degraded because of an error on the /dev/ada1 SSD. SanDisk doesn't have a diagnostic for BSD, so I pulled the two drives, installed them in a Windows box and ran the SanDisk test utility. I also ran a variety of Windows-based tools to try to stress test the drives. The utilities , and SMART, show no errors. I reinstalled in the Unix box, reinstalled Freenas, and within a day it dropped /dev/ada1 again. I have tried multiple SATA cables, no change. Any ideas?

 

[anmnz]

Installed 2 120GB SanDisk SSD Plus drives (2x$20.99), connected to onboard SATA

I got an alert that the boot pool was degraded because of an error on the /dev/ada1 SSD.

There are multiple reports of that disk model having that problem when FreeNAS is installed on it.

Lots of detail in this bug report:
https://redmine.ixsystems.com/issues/35065

and the threads it references, including:
https://forums.freenas.org/index.php?threads/boot-disk.63829/
https://forums.freenas.org/index.php?threads/freenas-installer-sandisk-ssd-checksum-errors.64049/
https://forums.freenas.org/index.php?threads/transcend-ssd-boot-disk-zfs-checksum-errors.64321/

 

[Andrew Ostrom]

There are multiple reports of that disk model having that problem when FreeNAS is installed on it.

Lots of detail in this bug report:
https://redmine.ixsystems.com/issues/35065

and the threads it references, including:
https://forums.freenas.org/index.php?threads/boot-disk.63829/
https://forums.freenas.org/index.php?threads/freenas-installer-sandisk-ssd-checksum-errors.64049/
https://forums.freenas.org/index.php?threads/transcend-ssd-boot-disk-zfs-checksum-errors.64321/

Thanks. I did a search earlier, but i didn't specify the disk model and I didn't find anything.

 

[tfran1990]

not trying to highjack this thread


I think im going to buy, as suggested by chris
https://www.ebay.com/itm/HP-H220-6G...9205-8i-P20-IT-Mode-From-US-Ship/192639052923
as long as it has FW 20.0.0.7.0(IT mode)
what more do i have to do,just put it in and connect the 8087 and boot?
i already have a pool setup so i would want to preserve everything the way it is.

Death Dream started his setup fresh right?

 

[Andrew Ostrom]

not trying to highjack this thread


I think im going to buy, as suggested by chris
https://www.ebay.com/itm/HP-H220-6G...9205-8i-P20-IT-Mode-From-US-Ship/192639052923
as long as it has FW 20.0.0.7.0(IT mode)
what more do i have to do,just put it in and connect the 8087 and boot?
i already have a pool setup so i would want to preserve everything the way it is.

Death Dream started his setup fresh right?

Don't worry about "hijacking" the thread. I don't know what the 9205-8i is, I can't find it on the Broadcom site. I bought a SAS8207-8i on eBay for $59. it works perfectly with 24 drives, a mix of SAS and SATA. The sell I bought from still has cards available, and they come withe the latest firmware and already in IT mode... I have no interest in this seller, just a customer...
https://www.ebay.com/itm/New-SAS-92...e=STRK:MEBIDX:IT&_trksid=p2057872.m2749.l2649

 

[Bozon]

not trying to highjack this thread...

What! How dare you waste internet bandwidth. Such a thing has never happened before.

 

[tfran1990]

Just ordered mine last night. i went with a LSI 9211-8i P20 dell H200.
some sellers on ebay have negative feedback saying what they got was a fake from some sellers.
I went with storagetekpro, he has good reviews for all the H200 from his e bay shop.

I read Alot of guides for flashing the sas HBA, from what i get out of it i think the first thing i need to do is use a PC other then my FREENAS boot from dos and run the command and see what it says as far a bios version and FW version. what did you do?

I read somewhere in a guide, that if you use 2 hbas in the same system it can cause issues.(long boot time) True?

 

[Andrew Ostrom]

When I bought a card for my Windows system a year or more ago I had to go through all the flashing hassle - I am pretty sure i ended up using the UEFI boot stuff from the Broadcom site, so an operating system isn't needed, just a system with UEFI abillity.

The board I bought recently for my Freenas system came all set up in IT mode with the latest firmware. (Link above). No flashing - i was up and running immediately. It was from a U.S. seller, and it is a genuine card. For $59 I thought it was a pretty good deal.

 

[tfran1990]

i wonder if i can contact dell support via chat and find out of mine is legit.
like i said the seller had 100% feedback with almost 200 sales on the card i bough, nothing but good reviews.

BONUS: i contacted the seller and had them throw in the small bracket for free.

 

[tfran1990]

Ok so today i got an lsi 9211-8i, i ordered a dell H310 but they didnt have anymore so they gave me the LSI version.
I put it in my windows pc just to see if it would work/make sure it was not completely DOA. when i turn on my pc it does not detect a hdd. what would cause this?
is it UEFI bios
I shouldn't matter that it is an 8x card in a x16 slot. Would the format of the drive matter?

If the card does a true pass through it should detect it right?

 

[acp]

Ok so today i got an lsi 9211-8i, i ordered a dell H310 but they didnt have any more so they gave me the LSI version.
I put it in my windows pc just to see if it would work/make sure it was not completely DOA. when i turn on my pc it does not detect a hdd. what would cause this?
is it UEFI bios
I shouldn't matter that it is an 8x card in a x16 slot. Would the format of the drive matter?

If the card does a true pass through it should detect it right?

Check your bios to see what gen your pci-e slots are set to. That card should be a gen 2. You may need to play around. You can also try another machine. Some are pickier than others.

 

[tfran1990]

I connected it to an older AM3 system which i know had a pcie gen 2 slot and it worked.
One thing i did notice was after having the card connected for maybe 20 min it got HOT. i know with some sas cards its a good idea to attach a fan on or close to the HS, has anyone needed to do it with the 9211?

 

[acp]

I have problems with gen 3 boards with gen 1 cards. All my gen 3 boards see my gen 2 cards. Since you know the card is good, maybe update the BIOS to the latest version might fix it. Also flashing the code on the 9211 might help as well.

I have the M1015 and never notice it get hot, but I do have a SuperMicro case which has 3 6000 RPM fans so it moving a lot of air through it. Of course it is not doing anything it shouldn't be getting hot.

 

[Yorick]

Thanks. I did a search earlier, but i didn't specify the disk model and I didn't find anything.

Did disabling TRIM work for you and resolve your checksum errors?

 

[Death Dream]

not trying to highjack this thread


I think im going to buy, as suggested by chris
https://www.ebay.com/itm/HP-H220-6G...9205-8i-P20-IT-Mode-From-US-Ship/192639052923
as long as it has FW 20.0.0.7.0(IT mode)
what more do i have to do,just put it in and connect the 8087 and boot?
i already have a pool setup so i would want to preserve everything the way it is.

Death Dream started his setup fresh right?

That card is not P20 as I found out when I bought it. Complained and spent a few days trying to get it upgraded on my SuperMicro box. Huge pain. Now the seller correctly listed the product as P15 after we had a chat.

 

[Death Dream]

6) Installed another 64GB or ECC memory thanks to the find on eBay by "Death Dream" - $96

:)

Getting into the BIOS on the Supermicro motherboard is just about impossible. I got in once, and I don't even know exactly how it happened - i haven't been able to get back in again.

Spam F8 when you boot the machine. F11 will work too.

 

[Andrew Ostrom]

:)



Spam F8 when you boot the machine. F11 will work too.

I'll give that a try, thanks. I've been trying "del" and F2.