looking for a new storage, advice and opinion, 36 BAY server

[iliak]

currently i have a small freenas server , and in progress of configuring a relatively large(will grow to 500TB raw ssd, currently on 50TB) ssd based ceph storange
and to be safe i am planning for make another dedicated freenas server based on hdd for redundancy

looking for an advice for hardware, i am thinking something like https://www.ebay.com/itm/4U-36-Bay-...295454?hash=item23cc4a95de:g:KrkAAOSw0ntch~S-

will populate it with 10Tb hdds in

• storage
  • each vdev 8 hdds on raidz3
  • 4 vdev max, 4 hdd spare
  • should 4*5*10 = 200TB refunded
• ram will be 12*32GB ecc ddr3 (to might upgrade in the future if ill need to)
• cpu 2x E5-2697 V2

1. what type of lsi card do i need to connect 36 hdds,
2. do i need WAL or SLOG

what do you think ?

 

[G8One2]

I cant say for certain, but i would think any of the LSI HBA shown on that listing, would work if it were flashed to IT mode. You only need 1 LSI card to control the 36 drives.

 

[joeinaz]

Two things:

1. Your plan calls for adding two E5-2697 V2 CPUs ; I believe your motherboard only supports E5-26xx V3 and V4 CPUs.
2. If it were me, before investing in different CPUs I would run the system as is and see if the existing CPUs provide the needed performance.

 

[iliak]

Two things:

1. Your plan calls for adding two E5-2697 V2 CPUs ; I believe your motherboard only supports E5-26xx V3 and V4 CPUs.
2. If it were me, before investing in different CPUs I would run the system as is and see if the existing CPUs provide the needed performance.

Current. my mistake. i miss read the cpu vertion,
I buy most of our system from a refurbished seller, and not directly trough ebay. and we got a lot servers with E5-2697\2695 v2 (with ddr3 is very cost effective for performance for the $ )

this is more what i was looking for (better value with almost the same peformance)https://www.ebay.com/itm/Supermicro...496681?hash=item1eebf193e9:g:yb0AAOSwZq5btouY
https://www.ebay.com/itm/Supermicro...496681?hash=item1eebf193e9:g:yb0AAOSwZq5btouY

ill will order it already assembled and tested, and the price difference to get the best cpu the socket support in V2 is low

 

[sretalla]

Be careful... these both claim to be 36 bay NAS, but the pictures (and physics generally) say that there are 6 disks high in 4U of space and I only see 4 columns of disks... that makes 24, not 36.

I don't see reference to where the other 12 disks will go.

At the very least, the pictures are wrong... these are not the droids you're looking for.

 

[danb35]

Be careful... these both claim to be 36 bay NAS, but the pictures (and physics generally) say that there are 6 disks high in 4U of space and I only see 4 columns of disks... that makes 24, not 36.

I can't follow the eBay links here, but it's probably the same chassis as mine--the other 12 bays are at the rear of the chassis, in 2U of space at the bottom. This is the layout for the 847 chassis; the 846 is a straight 24-bay unit with all the bays in the front.

 

[iliak]

here are the specks from the ebay post:

Performance Specs:
Processor: 2x Intel Xeon E5-2650L V2 Deca (10) Core 1.7Ghz
Memory: 256GB DDR3 (16 x 16GB - DDR3 - REG PC3-10600R (1333MHZ) )
Controller: 1x LSI 9311-8i HBA JBOD FREENASS UNRAID 12GBPS (With expander will run all 36 drives)
NIC: * Integrated Quad Intel 1000BASE-T Ports
No HDD 2x 2.5" HDD KIT or Caddy

Secondary Chassis/ Motherboard specs:
Supermicro 4U 36x 3.5" Drive Bays
Server Chassis/ Case: CSE-847BE1C-R1K28LPB
Motherboard: X9DRi-LN4F+
* Integrated IPMI 2.0 Management
Backplane: 2x Backplane: SAS3 12GB/S
*BPN-SAS3-846EL1 24-port 4U SAS3 12Gbps single-expander backplane, support up to 24x 3.5-inch SAS3/SATA3 HDD/SSD
*BPN-SAS3-826EL1 12-port 2U SAS3 12Gbps single-expander backplane, support up to 12x 3.5-inch SAS3/SATA3 HDD/SSD
PCI-Expansions slots: Low Profile 4 x16 PCI-E 3.0, 1 x8 PCI-E 3.0, 1 x4 PCI-E 3.0 (in x8)
Drive bay: 36x 3.5" Supermicro caddy
Power: 2x 1280Watt Power Supply PWS-1K28P-SQ
Rail Kit: Supermicro Rev B Rail Kit 4U

 

[danb35]

Server Chassis/ Case: CSE-847BE1C-R1K28LPB

Same basic chassis I have, just updated for SAS3 rather than SAS2. Expect the disks in the rear bays will run warmer than those in the front bays.

 

[iliak]

Same basic chassis I have, just updated for SAS3 rather than SAS2. Expect the disks in the rear bays will run warmer than those in the front bays.

We have dedicated room for servers, it freezing in there :). any way they will used mostly for backup, so very low load is expected. (hopefully we will never have to use it )

 

[Constantin]

At that drive quantity, you could also consider the Storinator series from 45drives.com. It's what backblaze allegedly uses.

 

[iliak]

At that drive quantity, you could also consider the Storinator series from 45drives.com. It's what backblaze allegedly uses.

whats it cost?
the supermicro chassis (with the above specks) cost approx 1500$ +shipping, the price is for refurbished

 

[Constantin]

Gotcha. If you want to go used (which is very cost effective) then that chassis is a very good choice.

Some lucky souls got free storinators from backblaze in the past. But you had to be on the RSVP list, i.e. follow their postings like a hawk. Plus, show up in CA to pick the things up (limit 2 per customer).

 

[iliak]

Gotcha. If you want to go used (which is very cost effective) then that chassis is a very good choice.

Some lucky souls got free storinators from backblaze in the past. But you had to be on the RSVP list, i.e. follow their postings like a hawk. Plus, show up in CA to pick the things up (limit 2 per customer).

We are from Israel... the trip there will make if a bit pricey :)

 

[Constantin]

A friend of mine used to finance his trips to the USA via the savings associated with buying suits there vs. Germany. The break even point for him was somewhere around 3 Hugo boss suits vs. a fun weekend in NYC.

Traveling back with a storinator would make a pretty epic story to tell the grandkids one day. Might make for an interesting conversation at the border though.

“Free? You have to pay import duty!!!”:)

 

[Chris Moore]

At that drive quantity, you could also consider the Storinator series from 45drives.com. It's what backblaze allegedly uses.

The folks at Backblaze only bought the model that 45drives developed for a short time and then developed another newer model that they like better because they wanted to go back to the SATA port multiplier backplanes they had used previously. Backblaze has a strange fascination with using SATA port multipliers instead of SAS for drive connectivity and they don't see any advantage in speed or reliability because of the way their infrastructure is designed to allow for an entire server to fail with no service interruption in the cluster of servers that they use. They basically treat each server like a drive in a RAID array with some software they developed in-house to manage the data, parity and rebuild across servers. The last forum post I read on the Backblaze forum was over a year ago, and that was the status then. I don't know what they are doing lately.

 

[Constantin]

Interesting. I wonder if it’s a matter of a lot of dormant data combined with the business model of using shucked media, if necessary.

SAS drives kinda lock you into the $$$ business market and Backblaze is famous for having shucked half a bazillion external drives during the post Thailand-flood HDD shortage.