Large 36 x 12TB Build

[Chris Moore]

The only reason for specific vdev width is to avoid lost space due to padding.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Rob_SAN]

How does the performance compare?

Sent from my SAMSUNG-SGH-I537 using Tapatalk

On paper the 1.6TB card is faster, just seems such a waste of space!

Unless you are not monitoring the system, or the system is at a remote location, there's no need for hot spare drives. Keeping a couple pre-tested cold spares is plenty.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

Will be in a remote DC, ideally need hot spares otherwise it could be a few days before replacement.

 

[Rob_SAN]

Am I the only one that thinks this is crazy? Are there no better options?

Agreed it is! I wish there was a smaller enterprise NVMe PCI card we could use at a good price point.

 

[Chris Moore]

I have a system with 80 drives, 4 disk shelves of 16 plus 16 in the head. All 4 of the disk shelves are on an 8e. Works well.

What about using the Supermicro 60 bay chassis?

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Rob_SAN]

I have a system with 80 drives, 4 disk shelves of 16 plus 16 in the head. All 4 of the disk shelves are on an 8e. Works well.

What about using the Supermicro 60 bay chassis?

Sent from my SAMSUNG-SGH-I537 using Tapatalk

I've been looking at the 60 bay option, very tempting!

 

[Chris Moore]

I've been looking at the 60 bay option, very tempting!

I have one in my data center now and just ordered another one this year. I really like them because I can put more drives in the same number of rack units.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Rob_SAN]

I have one in my data center now and just ordered another one this year. I really like them because I can put more drives in the same number of rack units.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

What have you ordered? the 6049P-E1CR60H?

 

[Chris Moore]

What have you ordered? the 6049P-E1CR60H?

No, we ordered this one:
https://www.supermicro.com/products/system/4U/6048/SSG-6048R-E1CR60L.cfm
The part number you mentioned (6049P-E1CR60H) is the one with a hardware RAID controller and it would not work properly with FreeNAS / ZFS.

This is what our contracting office put out for bid. I can only hope we get what we actually want:

Code:

											SALIENT CHARACTERISTICS
											 Storage Server Supermicro
											Model 6048R-31CR60L or Equal

1.0	GENERAL DESCRIPTION.  The contractor shall provide one each Storage Server,

Supermicro Model 6048R-31 CR60L, or equal meeting the following salient characteristics.

2.0  SALIENT CHARACTERISTICS.   Storage Server, Supermicro Model 6048R-31 CR60L configuration shall meet or exceed the following salient characteristics.

a.	PHYSICAL SPECIFICATIONS:
 -	   4U Storage 2000W Redundant Power Titanium Level
 - Dual socket R (LGA 2011) supports Intel® Xeon® processor E5-2637 v4 15M Cache,
	 3.50GHz
 - 24x 288 –pin DDR4 DIMM slots
 - Up to 3TB ECC 3DS LRDIMM, 768 ECC RDIMM
 - Chassis – 4 U Rackmount; Model CSE-946STS-R2K05P
 - Width: 17.2”; Height: 7”; and Depth: 30.2” with cable management
	 arm installed: 35.2”; Not to exceed -Gross Weight 170.5 lbs; Net Weight 118 lbs
-   60X 3.5” Hot-swap SAS3/SATA3 drive bays; 2x 2.5” rear Hot-swap SATA drive bays;
	 optional 6x NVMe bays
 - Broadcom 3008 SAS3 IT mode controller.
 - Server remote management: IPMI 2.0/KVM over LAN/Media over LAN
	 5x 8cm hot-swap redundant PWM fans
 - 2000W Redundant Power Supplies Titanium Level (96%) Certified
 - Front 3.5” LCD display

b.  CONFIGURATION:
 - Storage Server:  Supermicro 6048R-E1CR60L (1 EA)
 - Processor:  Intel® processor E5-2637 v4 15M Cace, 3.50 GHz (2 EA)
 - 32GB DDR4-2400 2Rx4 LP ECC REG RoHS (8 EA)
 - Data Drives: Seagate 3.5” 12TB, 7.2K RPM, SATA 6GB/s, Cache 256MB, 512E (60 EA)
 - Operating System: Samsung PM863a, 960BG, SATA 6Gb/s, VNAND, 2.5” (1.3 DWPD) w/SED (2 EA)
 - Network Interface:  SIOM 2-port 10G SFP+ with 1U bracket, Intel 82599ES (1 EA)
								 Intel® i350 AM2 2-port RJ45 (1Gv/port) Gen3 x4 Standard LP  (1 EA)
 - Cache Drives:  Intel 3D XPoint DC P4800X 375G PCIe 3.0 HHHL AIC 30DWPD (1 EA) & Intel DC P4600 2TB, NVMe PCIe3.0x4,3D TLC HHHL AIC 3DWPD (1 EA)
 - Include assembly and full burn in testing.

c.  SPARE PARTS KIT:  Additional disk drives, Seagate 3.5” 12TB, 7.2K RPM, SANTA 6GB/s, Cache 256MB, 512E (5 EA).

 

[Stux]

I was reading it should be 6 or 10? If 7 will work ok that's great!

Yeah, 7 is fine. The 6 or 10 thing was theoretically a problem before ZFS started using compression. Now since blocks are all sorts of sizes, it doesn't really tend to matter.

Plug your numbers into the ZFS raid calculator if you want to find out your overheads
http://wintelguy.com/zfs-calc.pl

 

[Ericloewe]

The backup software in use constantly checks the data for consistency,

On the server? That's pure waste with ZFS, since everything is checksummed and regular scrubs will fix anything that doesn't pop up in normal use. If you need better hashes than the default fletcher, you can manually choose sha512 or skein, which are slower but cryptographically secure.

On the clients? That sounds painful.

 

[Stux]

Depends on how the data gets onto the ZFS volume...

 

[Rob_SAN]

No, we ordered this one:
https://www.supermicro.com/products/system/4U/6048/SSG-6048R-E1CR60L.cfm
The part number you mentioned (6049P-E1CR60H) is the one with a hardware RAID controller and it would not work properly with FreeNAS / ZFS.

This is what our contracting office put out for bid. I can only hope we get what we actually want:

Code:

											SALIENT CHARACTERISTICS
											 Storage Server Supermicro
											Model 6048R-31CR60L or Equal

1.0	GENERAL DESCRIPTION.  The contractor shall provide one each Storage Server,

Supermicro Model 6048R-31 CR60L, or equal meeting the following salient characteristics.

2.0  SALIENT CHARACTERISTICS.   Storage Server, Supermicro Model 6048R-31 CR60L configuration shall meet or exceed the following salient characteristics.

a.	PHYSICAL SPECIFICATIONS:
 -	   4U Storage 2000W Redundant Power Titanium Level
 - Dual socket R (LGA 2011) supports Intel® Xeon® processor E5-2637 v4 15M Cache,
	 3.50GHz
 - 24x 288 –pin DDR4 DIMM slots
 - Up to 3TB ECC 3DS LRDIMM, 768 ECC RDIMM
 - Chassis – 4 U Rackmount; Model CSE-946STS-R2K05P
 - Width: 17.2”; Height: 7”; and Depth: 30.2” with cable management
	 arm installed: 35.2”; Not to exceed -Gross Weight 170.5 lbs; Net Weight 118 lbs
-   60X 3.5” Hot-swap SAS3/SATA3 drive bays; 2x 2.5” rear Hot-swap SATA drive bays;
	 optional 6x NVMe bays
 - Broadcom 3008 SAS3 IT mode controller.
 - Server remote management: IPMI 2.0/KVM over LAN/Media over LAN
	 5x 8cm hot-swap redundant PWM fans
 - 2000W Redundant Power Supplies Titanium Level (96%) Certified
 - Front 3.5” LCD display

b.  CONFIGURATION:
 - Storage Server:  Supermicro 6048R-E1CR60L (1 EA)
 - Processor:  Intel® processor E5-2637 v4 15M Cace, 3.50 GHz (2 EA)
 - 32GB DDR4-2400 2Rx4 LP ECC REG RoHS (8 EA)
 - Data Drives: Seagate 3.5” 12TB, 7.2K RPM, SATA 6GB/s, Cache 256MB, 512E (60 EA)
 - Operating System: Samsung PM863a, 960BG, SATA 6Gb/s, VNAND, 2.5” (1.3 DWPD) w/SED (2 EA)
 - Network Interface:  SIOM 2-port 10G SFP+ with 1U bracket, Intel 82599ES (1 EA)
								 Intel® i350 AM2 2-port RJ45 (1Gv/port) Gen3 x4 Standard LP  (1 EA)
 - Cache Drives:  Intel 3D XPoint DC P4800X 375G PCIe 3.0 HHHL AIC 30DWPD (1 EA) & Intel DC P4600 2TB, NVMe PCIe3.0x4,3D TLC HHHL AIC 3DWPD (1 EA)
 - Include assembly and full burn in testing.

c.  SPARE PARTS KIT:  Additional disk drives, Seagate 3.5” 12TB, 7.2K RPM, SANTA 6GB/s, Cache 256MB, 512E (5 EA).

Interesting build!

Are you just having one slog drive?

Out of interest what is the build for?

 

[Chris Moore]

Yes, just one SLOG. It is mostlyfor reading data. Write is less critical.

It will be used to house data for ArcGIS. Something that they call a, "File George Database". When the existing server was ordered, the people making the decision did not understand how the system should be configured. Since then, I was hired and have selected the system, within the budget, that would give them the best performance.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Rob_SAN]

Yes, just one SLOG. It is mostlyfor reading data. Write is less critical.

It will be used to house data for ArcGIS. Something that they call a, "File George Database". When the existing server was ordered, the people making the decision did not understand how the system should be configured. Since then, I was hired and have selected the system, within the budget, that would give them the best performance.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

With the latest release is there a reason to have a mirrored slog?

Thanks, interesting write up!

 

[Ericloewe]

With the latest release is there a reason to have a mirrored slog?

Thanks, interesting write up!

Depends on your level of paranoia. The SLOG could fail at just the wrong time and you would lose the transaction group or two. The majority of data, i.e. Everything that has been committed to the pool will be intact.

 

[Stux]

The SLOG could fail at just the wrong time

Ie between when your system crashes and when it reboots.

I’m of the opinion if someone has that level of paranoia, they should probably be looking at HA.

(High Availability = $$$$ )

 

[kdragon75]

Ie between when your system crashes and when it reboots.

I’m of the opinion if someone has that level of paranoia, they should probably be looking at HA.

(High Availability = $$$$ )

Not necessarily. The ESOS project offers block level storage with HA for the cost of hardware. Granted there is some work to set it up.

 

[kngpwr]

Since it seems like you are focused on PCIe cards for your SLOG, you can sometime find SLOG device cost savings by using adapters like this one: https://www.amazon.com/StarTech-com...F8&qid=1526575450&sr=8-4&keywords=PCIe+to+U.2

These cards tend to be all but completely passive since the NVMe SSD is already a native PCIe device. That helps make them a reliable choice. Avoid the "NVMe or SATA" variants since they tend to use cheap SATA controllers that don't need to be in your system.

The company I work for has shipped thousands of these type adapter in products without issue. All the usual compatibility stuff applies of course. These just save you needing a U.2 bay+cable somewhere in the system. Combine something like this with a "proper" U.2 drive with PLP and you might have a bang for the buck winner. You can often find smaller capacity U.2 drives with PLP for great prices. Write workload issues can be heavily mitigated by over-provisioning the SSDs.

In my personal FreeNAS system I use one of these style adapter with an "older" Intel 750 (SSDPE2MW400G4M2). With sync=always I sustain 400-900MB/sec in writes (varies by block size being written) to an 8x8 WD Red Pro RAIDZ2 array.

Another thing to keep in mind is that more and more enterprise motherboards are being made with OCuLink ports on them. Pick out one of those, move your OS to a SATADOM mirror, get a couple of SFF-8639 to U.2 cables, and repurpose the internal 2.5" bays for some of the U.2 drives on the market and you could get some slots back for future whatever. Some recent chassis are also offering the U.2 bays in the rear near the PSU(s) if that's of interest to you. It would enable hot swap.

Just for full disclosure, one of the tasks I have in my day job is to figure out how to safely cram the most capability into a smaller space and keep it enterprise grade reliable. Your motherboard and chassis combo might make this a lot less important to you than it tends to be to me :)

 

[Chris Moore]

The company I work for has shipped thousands of these type adapter in products without issue. All the usual compatibility stuff applies of course. These just save you needing a U.2 bay+cable somewhere in the system. Combine something like this with a "proper" U.2 drive with PLP and you might have a bang for the buck winner. You can often find smaller capacity U.2 drives with PLP for great prices. Write workload issues can be heavily mitigated by over-provisioning the SSDs.

Do you have some U.2 drives with PLP that you can point out?

 

[kngpwr]

Let's see if this link works: https://ark.intel.com/Search/Featur...face=PCIe NVMe 3.0 x4&SSDFormFactor=2.5" 15mm

These are just the Intels (which I have the most experience with). Some of these can be found with attractive prices on them. I know from working closely with Intel in my job that they are "doing it right" so I tend to stick with what I know. I also have not seen many other companies making U.2 lineups.

Here is Intel's own definition of their EPLDP feature:
"Enhanced Power Loss Data Protection prepares the SSD for unexpected system power loss by minimizing data in transition in temporary buffers, and uses on-board power-loss protection capacitance to provide enough energy for the SSD firmware to move data from the transfer buffer and other temporary buffers to the NAND, thus protecting system and user data"

There are more proper options in the SATA space but that's beyond the scope of the question here.

NOTE: You might find cheaper SATA SSDs floating around that claim to have powerless protection but it can be a half truth. Some companies (*cough* Crucial *cough*) on certain models protect the metadata only and it can basically only tell you with authority what you failed to write :) Not exactly ideal.