X1600 based performance build - will it work

AN602

Cadet
Joined
Mar 13, 2019
Messages
6
Hy there


I use for backups, a plex media Server and filesharing within the intranet a Windows Server 16 VM on my ESXi host. Therefore, my ESXi Host has around 50TB - a Raid6 over 8 5TB HDDs, a raid 5 over 4 2TB Sata SSDs, a 1.92TB FusionIO drive and a 480GB Optane 900p. There is a little bit of a design flaw, since I backup the other VMs on a VM.
Therefore, I wanted to build a dedicated, powerful NAS. But all the Supermicro 12 bay cases are, at least here in Switzerland, when looking for a 6Gb SAS or better backplane, around 450 CHF. An additional major investment for HDDs is also required. Hence those circumstances I still run the setup explained above.

Luckily, I came across an auction of a HP StorageWorks X1600 (/G2), which I got then for 16 CHF, as far as I can tell, it is the same machine as a DL180 G6 (/7). The pre-owner has "modificated" it a lot. Either way the Raid controller is missing. Another luckily occasion is that will get 16 used HGST 4 TB HDDs for free, even though they have around 38'000 hours runtime, they seem to be a good base for a NAS system. - Replacing the filesharing and Plex component, and for backups of the VMs.



Looking around for a good solution, I came across FreeNAS. - I "read" the documentation. - Is there a smaller, "get started"-like article?

It does look, like LSI HBA cards are the best choice, isn't it? - Unfortunately, I don't have such a card. Indeed, they seem to be a bit pricy, even the used ones (in CH, used are around 150 CHF). So, I thought about using a Fujitsu D2616-A22 I have laying around. It is based on the LSI SAS 2108, which I read is a good base for FreeNAS, but needs to be flashed with another firmware. - Questions I have now; will this (card and flashing) work and is there a step by step guide for the flashing (with linking to the needed files)?

Might that backplane of those 12 bays cause any problems, or is there anything I should keep in mind?

I'm not sure how powerful the CPU should be. Technically I could install two X5680. Have them laying around. And even though they are powerful, they also suck a lot of energy out the wall. Considering that the NAS should be running 24/7, I would much more prefer to use one L5640. Would this CPU be enough for the NextCloud Plugin and running Plex in a Jail?

Also, with 48TB I should at least have 48GiB Ram, isn't it? - I'm planning to use 6* 16GiB registered ECC dimms, is that enough for a satisfactory performance? - I mean, if not so, there would be option 1, to install a second CPU and use 12*16GiB dimms, or option 2 to buy 6*32GiB dimms, since that would cost me around 700 CHF, or with other words, even when option 1 results in 150W more consumption, ROI would take 3.5 years. Including the opportunity cost of not having the same amount of power (CPU) and bandwidth (12 vs 6 dimms), and the fact, that initial cost of option 2 is nearly 30 times the initial cost of option 1, I prefer option 1.

Then the next question I have, could I use 2 860 evo 512 GB as the system partition and still use them for L2ARC? - And should I do so? Because I read that you should only use SLC SSDs. But I don't have any more of those. And as I understand it, as larger your ARC/RAM is, the less writing operations happen on the SSD, so it shouldn't be much of a problem for them? The other option I'm thinking of, is to install the FusionIO drive instead of the two Sata SSD. I expecting it to be much more performant, but as it looks like, it does only work with trueNAS is that still so?

Since my Router as also my ESXi Host have HPE 544 QSFP 40 Gbps NICs (because 5 of them were cheaper then 2 10Gbps SFP+ NICs), it would make sense, to use one of them in the NAS, will this work?



It's obvious that this is my first FreeNAS build - so should I have asked something horribly wrong, I'm sorry for that, I'd looked up by myself as much as possible, and those are the remaining questions. Thx for any help.

Kind regards
AN602
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
I've been picking up Dell PERC H310's used off eBay for around $30-$40, depending on deals. These are fine and I can't imagine finding something much cheaper.
 

AN602

Cadet
Joined
Mar 13, 2019
Messages
6
Hi thanks for this tip. But isn't that also a raid controller, so first needed to be flashed for HBA mode?
I mean, the D2616-A22 I have here is a SAS2108, while the PERC H310 seems to be a SAS2008, I don't expect that the 2008 is more powerful or so. So the only motivation to go for it would be that it is simplier to flash to a HBA mode, but is it?

I have looked here[1&2], and it seems that both uses the same tool to get flashed to HBA Mode. But that the 2108 has a more powerful core. I don't know how much/less this affects performance in HBA mode, but I think, ROI would take quit a while, considering that the 2108 only uses like 3W more.
[1] https://www.servethehome.com/lsi-sas-2108-raid-controller-information-listing/
[2] https://www.servethehome.com/lsi-sas-2008-raid-controller-hba-information/
 
Last edited:

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Hi thanks for this tip. But isn't that also a raid controller, so first needed to be flashed for HBA mode?
You can find them already flashed to IT mode with the latest firmware:
https://www.ebay.com/itm/Dell-H310-6Gbps-SAS-HBA-w-LSI-9211-8i-P20-IT-Mode-for-ZFS-FreeNAS-unRAID/162834659601

We also have a guide here on the site to help you if you want to get one that has not been flashed and flash the firmware yourself:

Detailed newcomers' guide to crossflashing LSI 9211/9300/9311 HBA and variants
https://www.ixsystems.com/community...shing-lsi-9211-9300-9311-hba-and-variants.54/

Lowest price from a US shipping location:
https://www.ebay.com/itm/Dell-3P0R3-PERC-H310-PCIe-RAID-Controller/283411361038
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
I am not familiar with flashing that Fujitsu controller.
I'm not sure how powerful the CPU should be. Technically I could install two X5680. Have them laying around. And even though they are powerful, they also suck a lot of energy out the wall. Considering that the NAS should be running 24/7, I would much more prefer to use one L5640. Would this CPU be enough for the NextCloud Plugin and running Plex in a Jail?
I am familiar with these processors. The system I used a couple years ago was running those. I replaced it with something newer to reduce power consumption and heat generated. It should work, but it does take a lot of power to do it compared with a newer system.
Also, with 48TB I should at least have 48GiB Ram, isn't it? - I'm planning to use 6* 16GiB registered ECC dimms, is that enough for a satisfactory performance?
Should be fine. FreeNAS uses RAM for ARC (Adaptive Replacement Cache) so more memory usually equates loosely with more performance, but it isn't strictly needed for video streaming with Plex.
Then the next question I have, could I use 2 860 evo 512 GB as the system partition and still use them for L2ARC?
No, if I am understanding your question properly, the boot drive (mirror set) can only be used for the boot pool and FreeNAS hangles the mirroring. So you could use much smaller disks for the boot pool and for your described use, I don't think L2ARC is needed. It would be better to increase RAM, the L1ARC, so to speak.
as I understand it, as larger your ARC/RAM is, the less writing operations happen on the SSD, so it shouldn't be much of a problem for them? The other option I'm thinking of, is to install the FusionIO drive instead of the two Sata SSD. I expecting it to be much more performant, but as it looks like, it does only work with trueNAS is that still so?
You said you are only using this for backups, so there isn't really a need for all that. You will need fast networking and you need to set your drives up in at least two vdevs to improve IOPS so backups can be completed in a timely manner.
HPE 544 QSFP 40 Gbps NICs (because 5 of them were cheaper then 2 10Gbps SFP+ NICs), it would make sense, to use one of them in the NAS, will this work?
Not sure if that will work in FreeNAS. I have seen a few posts from people that tried, but I don't recall the results. You might do a search of the forum.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Hi thanks for this tip. But isn't that also a raid controller, so first needed to be flashed for HBA mode?

Yes, but it's usually quite cheap.

I mean, the D2616-A22 I have here is a SAS2108, while the PERC H310 seems to be a SAS2008, I don't expect that the 2008 is more powerful or so.

"more powerful"? These things are capable of driving all the SAS ports at a speed higher than FreeNAS is ever likely to need in practice. If you actually try to do full speed on all ports, like SSD doing max SATA 6Gbps speeds, yes, you can probably demonstrate some differences, but in practice you won't notice with HDD.
 

AN602

Cadet
Joined
Mar 13, 2019
Messages
6
I am familiar with these processors. The system I used a couple years ago was running those. I replaced it with something newer to reduce power consumption and heat generated. It should work, but it does take a lot of power to do it compared with a newer system.
That's why I ponder about getting a DL380p G8 14LFF. - I mean, I do have changed a lot of stuff here to used G8 and it's quite nice. The G8 would also allow 128 GB with one CPU based on the dimms I have laying around. - I think of it, for the first "get started"-moment, the X1600 will do a good job, and I could upgrade to a G8 when I can say, it matches my needings, and that I will for sure stick on it.

You said you are only using this for backups, so there isn't really a need for all that. You will need fast networking and you need to set your drives up in at least two vdevs to improve IOPS so backups can be completed in a timely manner.

I will also use it for filetransfer, planning to do so with NextCloud, actually running it on a webhosting offer, and I'm quite happy with it. Regaring that our internet will be upgraded to 500/200 and also that I can only use schools OneDrive for the next 5 months, I think of using NextCloud quite a lot.
Only me having 8 TB of data from my school, many many videos from lectures, but also several smaller datas mostly docx, xlxs, py, vba, jar. My family and me private woudl have round about 12 additional TB. Mostly pictures and music.

No, if I am understanding your question properly, the boot drive (mirror set) can only be used for the boot pool and FreeNAS hangles the mirroring. So you could use much smaller disks for the boot pool and for your described use, I don't think L2ARC is needed. It would be better to increase RAM, the L1ARC, so to speak.

Good to know, so it would be better to use two 64 GB SSD as mirrored boot drives and for L2ARC, if it would be needed, a M.2 SSD with adapter
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
I think of it, for the first "get started"-moment, the X1600 will do a good job
It should be fine for a start so you can see how you like FreeNAS. Upgrade later when you are ready.
Good to know, so it would be better to use two 64 GB SSD as mirrored boot drives
Yes.
for L2ARC, if it would be needed, a M.2 SSD with adapter
You can put a L2ARC in, if you want, it won't hurt anything. You can monitor how it is used in the statistics and decide if you want to keep it because L2ARC is one of the few things you can add to a pool and then take out if you decide you don't need or want it.
 

JaimieV

Guru
Joined
Oct 12, 2012
Messages
742
For the SAS card, I went the easy route of searching eBay for "sas LSI it mode" and picking up one of the Chinese £40 new 9201-8i cards and an American 2nd-hand £60 9207-4i4e. They took a couple of weeks to arrive but neither needed flashing and both arrived on firmware v20.
Looks like ebay.ch isn't as cheap, unfortunately - https://www.ebay.ch/sch/i.html?_nkw=lsi+sas+it+mode
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
both arrived on firmware v20
Also look at the sub-version. It needs to be 20.00.07.00 if I recall correctly.
In a terminal window, the command sas2flash -list should give you output similar to this:
Code:
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

        Adapter Selected is a LSI SAS: SAS2308_2(D1)

        Controller Number              : 0
        Controller                     : SAS2308_2(D1)
        PCI Address                    : 00:03:00:00
        SAS Address                    : 500605b-0-09ef-7220
        NVDATA Version (Default)       : 14.01.00.06
        NVDATA Version (Persistent)    : 14.01.00.06
        Firmware Product ID            : 0x2214 (IT)
        Firmware Version               : 20.00.07.00
        NVDATA Vendor                  : LSI
        NVDATA Product ID              : SAS9207-8i
        BIOS Version                   : N/A
        UEFI BSD Version               : N/A
        FCODE Version                  : N/A
        Board Name                     : SAS9207-8i
        Board Assembly                 : H3-25412-00J
        Board Tracer Number            : SV45308383

        Finished Processing Commands Successfully.
        Exiting SAS2Flash.

Notice the line that says, Firmware Version : 20.00.07.00
Also double check the line above it, wher it says, "IT"
Some of the vendors have not been delivering what they claimed they were delivering.
 

JaimieV

Guru
Joined
Oct 12, 2012
Messages
742
Good point - mine were both 20.00.07.00. I'm new to SAS so haven't gone through the bad firmwares :)
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Good point - mine were both 20.00.07.00. I'm new to SAS so haven't gone through the bad firmwares :)
You might want to take a look at some of these resources if you are new to SAS:

Don't be afraid to be SAS-sy
https://forums.freenas.org/index.php?resources/don't-be-afraid-to-be-sas-sy.48/

Building, Burn-In, and Testing your FreeNAS system
https://forums.freenas.org/index.php?resources/building-burn-in-and-testing-your-freenas-system.38/

Github repository for FreeNAS scripts, including disk burnin
https://forums.freenas.org/index.ph...for-freenas-scripts-including-disk-burnin.28/

Informational videos, mostly about SAS hardware
https://www.ixsystems.com/community/resources/informational-videos-mostly-about-sas-hardware.105/
 

AN602

Cadet
Joined
Mar 13, 2019
Messages
6
Hi, sorry for not answering for so long. Had to do a ton of other things and had have several exams and two essays. So hope to get time to go on with this project at the weekend.

For the SAS card, I went the easy route of searching eBay for "sas LSI it mode" and picking up one of the Chinese £40 new 9201-8i cards and an American 2nd-hand £60 9207-4i4e. They took a couple of weeks to arrive but neither needed flashing and both arrived on firmware v20.
Looks like ebay.ch isn't as cheap, unfortunately - https://www.ebay.ch/sch/i.html?_nkw=lsi+sas+it+mode
Here around nearly noone uses ebay.ch, I personaly only us ebay.com, ebay.co.uk or ebay.de with my swiss adress. But normaly things go over ricardo.ch. And there a not even flashed SAS2008 cost you around 90 Dollars, see https://www.ricardo.ch/de/a/controller-lsi-sas2008-9211-8i-dell-h310-1055500530/

An (great?) news, I got two NetApp DS14MK2 Shelfes, each with 14*1 TB. So a total of 28TB. I read that I could use them with FreeNAS. I hope that FreeNAS supports my LPE12002, I know that FreeBSD does, but also often read, that there is a difference between FreeBSD does and FreeNAS does.
So or so, looking forward to test this also this weekend. Because when it does work as I hope it does, it would be nice to use them for the backups, I mean I could hook them up behind a relay controlled with an arduino, which turns them on when it is time for backups and shuts them down, when the domain server says that the backups are done. - Does FreeNAS autodetect when a ressource is "reconnected"?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
I hope that FreeNAS supports my LPE12002, I know that FreeBSD does, but also often read, that there is a difference between FreeBSD does and FreeNAS does.

No, FreeNAS does not support FC. At least not officially. And really also not even unofficially.

Because when it does work as I hope it does, it would be nice to use them for the backups, I mean I could hook them up behind a relay controlled with an arduino, which turns them on when it is time for backups and shuts them down, when the domain server says that the backups are done. - Does FreeNAS autodetect when a ressource is "reconnected"?

You want to turn off part of your server and hope that FreeNAS will autodetect that?

FreeNAS isn't a PC and doesn't expect portions of its storage to disappear. The reason that disk shelves are powered separately in large arrays isn't so that you can shut off ones you're not using. It's a purely practical matter of how to power large amounts of gear. The fact that you can potentially power gear on and off independently does not mean that an entire disk shelf is designed to be treated in the cavalier manner a PC owner treats a portable external disk drive.
 

AN602

Cadet
Joined
Mar 13, 2019
Messages
6
No, FreeNAS does not support FC. At least not officially. And really also not even unofficially.

Hmmm, ok, at least I don't have to try. - Then I might use them directly at my ESXi Host.

FreeNAS isn't a PC and doesn't expect portions of its storage to disappear.
Never had problems with ESXi>Windows with pnp storage.

The reason that disk shelves are powered separately in large arrays isn't so that you can shut off ones you're not using. It's a purely practical matter of how to power large amounts of gear. The fact that you can potentially power gear on and off independently does not mean that an entire disk shelf is designed to be treated in the cavalier manner a PC owner treats a portable external disk drive.

I do have had an whole HP EVA Rack, an really old one, either it has an defect - and I didn't noticed as such - or it is a powersaving feature; it shutted down some Shelfes, when the Management was connected to the controller, without any loss of performance.
(A damn 42U rack, not even 22TB usable space...)

Fun fact, I first used for backups with ESXi an portable disk, where I exported OVF templates. - Poor students life.
 
Top