Introduction + New Build: plans, questions and more!

[Matthias]

Hi all!

Let me first introduce myself a tiny bit before i fire my plans and questions away .

My name is Matt, and I'm Belgian. A bit of a petrolhead, spacenerd, and IT "enthousiast"..
So i love very fast things!

Some time ago i had a slight cardiac arrest, when i was messing around with my gaming PC setting up RAID Nvme's, when it decided that all my connected HDD's needed changes in their GPT/FS structure too... Resulting in weird things happening when accessing the data...

I managed to retrieve all of my data back (using DMDE) without any issues, so that cardiac arrest was not permanent XD..

Now i know that it's not that smart to fiddle with the controllers, especially when they have drives attached with important data.. Silly me..

I have been following FreeBSD/Freenas way back, some 6-7 years ago i think, and finally decided to build a NAS where i put my data on, away from my experiments..

So, whenever everything arrives from my orders, i will be jumping in the world of ZFS and Trunas Core (or Scale?).

The List:

What	Description	PN	QTY
HDD	Seagate HDD 3.5 EXOS X16 16TB	ST16000NM001G	5​
HDD	WD Red 8TB EFZX		1​
HDD	WD White label 8TB EMAZ		2​
Mobo	Asrock E3C246D4U2-2L2T Intel C246 LGA 1151	E3C246D4U2-2L2T	1​
RAM	Kingston Technology KSM26ES8/16ME 16 GB DDR4 2666 MHz ECC	KSM26ES8/16ME	2​
OS HDD	Samsung evo 870 500GB	MZ-77E500B/EU	2​
HBA	IBM M1015	IBM M1015	2​
Case	Fractal Design Node 804		1​
PSU	Seasonic PRIME GX-650		1​
HDD CBL	DeLOCK Mini SAS SFF-8087 to 4x SATA 7 pin		2​
CPU	Intel Xeon E-2234 processor	CM8068404174806	1​

The plan/use:

Main goal: create a safer and redundant storage pool, with critical files synced to an external drive and cloud. (mostly pictures/docs.. ), all the rest of the data would be painfull to lose, but certainly not critical.

In that way the critical data is on site on a redundant array, could be physically moved taking the external drive (house on fire ), and is backupped to the cloud (house on fire too fast/all drives lost ).

The plan is to put my data, which is currently on the 3x 8TB drives as seperate volumes (windows 11), centralized on the NAS i'll be building.
For now, that's my main goal, but i would like to sync my google services, maybe plex, run the occasional test VM.

Then i will eject those three drives from my desktop to my NAS, and do something with it, not sure what exaclty (maybe a backup for critical data).

So, concerning "the list", i know it's probably overkill, but i rather buy something that's more then sufficient, rather then be bottlenecked..

The questions:

So starting with the mobo:

-> Do i need to flash the internal SATA controller, like recommended for those M1015's IT mode?
Not sure if Truenas will be able to connect to the drives in a direct way. (SATA only board, so probaly not possible too?)
This, off course, in the event i hook up disks to the mobo. I presume it's recommended/preferred to use the HBA's instead of the onboard SATA's?

-> For now i've gone with 2x 16GB of ECC RAM, i hope that's sufficient to start, but can be expanded to 128GB.

VDEV's:
-> 5x 16TB Exos drives
-> 3x 8TB (mix of RED and 2 White's)
-> 2x 500GB Samsung Evo's (OS installation)

So for the 16TB drives, would you guy's opt for RAIDZ1 + hot spare (3 disks + parity disk + hot spare), or a RAIDZ2 (3 Disks + 2 parity disks).
I cannot decide..

I figure that the first is "more secure" in the event that i lose a drive as with the hot spare it would resilver instantly? In the second case i'd have to get spare drives and then rebuild the data. (but imagine this will go faster as it has 2 parity drives to resilver?)

I'd like to hear your recommendations pro's/cons on this.

As for the 8TB drives, a raid 5 or 1:1:1 mirror, not quite sure yet.

(Depending on your thoughts about the onboard SATA's, i will connect all the HDD's through 1x M1015, and then the OS SSD's to the 2nd M1015 or the Mobo directly.)

To Scale or not to Scale?

I understand that Scale is still experimental, but if i understand correctly, is more suited for running VM's.
Also, in the event the OS bricks, it should be a piece of cake retrieving back the VDEV/config of the drives as it is stored on the drives itself right?

If my assumptions are correct, i think i will go for Scale, as my data is very likely not to be at risk. (not 100% fullproof, i know, but what is?)

The build/install checklist:
- Test the Memory (couple of full passes)
- Change the FW on those M1015's and verify it is set to IT mode
- Install OS on 1 drive + basic OS config
- Mount the 16TB drives and run full disk test
- Create The VDEV 'OS' (Mirror OS to second SSD drive)
- Create The VDEV '16TB disks' (RAIDZ1 or 2?)
- copy data to 16TB VDEV
- Remove 3x 8TB from my DT and configure them in Truenas.

This is off course very 'raw' and subject to change.

Future expansion:

I understand it is not that straightforward to expand the VDEV's by simply putting in extra drives, but that it will require a full HDD swap (new VDEV) to accomodate bigger storage capacity. By the time i will use all the space from the 16TB array, there will be larger drives on the market so i can 'juggle' between the 3 disks array and then the 5 disks array, and so on... (or get a larger case/rack to house it all)

With my current components i can house 16 drives through the 2 M1015's, and depending on the mobo controllers, maybe 15 more.
If CPU is not adequate enough -> E-2288G to double the core & thread count, and can expand to 128GB of ECC when needed.

This mobo comes with 2x 10GBe NIC's (Intel® X550-AT2) (and 2x GBe by Intel® i210), so 10 gig is covered. My house still needs an upgrade to these speeds, but in due time when needed. I think that the only route where 10GBe is beneficial or needed, is from my desktop straight to the NAS.

As i will be a 'first timer' to install, explore and configurate Truenas for real (ran freenas in VM's to play around), i hope the ease of Freenas and off course this vast community will come to the rescue when needed .

I will try and update this thread as i progress on the build.

Any advice, comments, tips, suggestions, ... are very welcome, and last but not least, a very happy NE to all of you!

Many thanks,

Greetz,

Matt

 

[Matthias]

Update on the partslist:

What	Description	PN	QTY	STATUS
HDD	Seagate HDD 3.5 EXOS X16 16TB	ST16000NM001G	5	IN
HDD	WD Red 8TB EFZX		1	IN
HDD	WD White label 8TB EMAZ		2	IN
Mobo	Asrock E3C246D4U2-2L2T Intel C246 LGA 1151	E3C246D4U2-2L2T	1	IN
RAM	Kingston Technology KSM26ES8/16ME geheugenmodule 16 GB DDR4 2666 MHz ECC	KSM26ES8/16ME	2	IN
OS HDD	Samsung evo 870 500GB	MZ-77E500B/EU	2	IN
HBA	IBM M1015	IBM M1015	2	IN
Case	Fractal Design Node 804		1	IN
PSU	Seasonic PRIME GX-650		1	IN
HDD CBL	DeLOCK Mini SAS SFF-8087 naar 4x SATA 7 pin		2	IN
CPU	Intel Xeon E-2234 processor	CM8068404174806	1	ETA 25/01

So yeah.. waiting on the last bit!

 

[Matthias]

Created a poll, hoping to get the reactions/comments started!

Thank you all!

 

[Matthias]

Got all the components, and installed TrueNAS Core U.8 succesfully!

I am however getting frustrated getting my M1015's to IT firmware... HELP?

Tried several guides/pages:

Detailed newcomers' guide to crossflashing LSI 9211/9300/9305/9311/9400/94xx HBA and variants

There's hundreds of threads on crossflashing variants of LSI 9211/9300/9305/9311/94xx cards here, on ServeTheHome, and on blogs. But they don't always discuss the subtleties of doing it on an EFI board for users who may never have used EFI...

www.truenas.com

GUIDE FOR DUMMIES : Flash an LSI9240 like the IBM M1015 to a LSI9211-IT mode

Hi there, Here is a straight to the point guide I write because I feel that the others are confusing. This one should work as it even includes the UEFI Shell to boot from. Here is a quick overview of the steps - First make a USB key boot in Freedos, second Drop the files in the USB and Third...

www.truenas.com

+ several others, but with far fewer info.
Also, the files and links were half dead for all the URLs i tried, so i googled all the files from various locations.

-> Cannot get megarec nor sas2flash to work on my mobo. Tried EFI/BIOS.

When going trough BIOS (not uefi for megarec, as i found out its MSDOS only) i get the message "invalid MS DOS version" or command errors that mention Megarec does not exist.
Though i triplechecked the USB (FAT32, all tools/files are present on root).

Through EFI, i can get through, but again no succes in even listing the drives through the sas2flash utility.

Are there better guides to follow?

Thank you

 

[jgreco]

flash the internal SATA controller

No. The internal SATA controller has no firmware. It's just a dumb bit of silicon. An HBA is a small computer all on its own, so you need to be able to talk to it in the proper way. Driver and firmware have to speak the exact same dialect.

I presume it's recommended/preferred to use the HBA's instead of the onboard SATA's?

Either way is fine. Onboard SATA is slightly faster for things like SSD.

I am however getting frustrated getting my M1015's to IT firmware... HELP?

I've found systems where it was difficult/impossible to do the crossflashing. You can try a different PC. You also have the option to buy preflashed cards on sites such as eBay. Since you are essentially "hacking" on a card, the process isn't always easy or familiar to those not used to using embedded system tools.

2x 10GBe NIC's (Intel® X550-AT2) (and 2x GBe by Intel® i210), so 10 gig is covered.

This is not particularly recommended. We've just come off a brief discussion earlier today about this --

Cheap Windows 10 network cards that do 10 Gb over cat6(a)

Hi I recently took the plunge and bought a 10 Gb nic for my NAS and transceivers for my switch. For this I bought fiberoptic parts, however as they have come dramatically down in price I also bought a 10 Gb RJ45 transceiver for my switch thinking that I can use it for my machine. However to do...

www.truenas.com

and the fact is that there aren't a lot of good copper choices. If you're terrified of fiber or don't understand the issues, please do feel free to stop by this thread:

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

The SFP+ stuff remains cheaper and maybe also easier.

 

[Matthias]

No. The internal SATA controller has no firmware. It's just a dumb bit of silicon. An HBA is a small computer all on its own, so you need to be able to talk to it in the proper way. Driver and firmware have to speak the exact same dialect.




Either way is fine. Onboard SATA is slightly faster for things like SSD.




I've found systems where it was difficult/impossible to do the crossflashing. You can try a different PC. You also have the option to buy preflashed cards on sites such as eBay. Since you are essentially "hacking" on a card, the process isn't always easy or familiar to those not used to using embedded system tools.



This is not particularly recommended. We've just come off a brief discussion earlier today about this --

Cheap Windows 10 network cards that do 10 Gb over cat6(a)

Hi I recently took the plunge and bought a 10 Gb nic for my NAS and transceivers for my switch. For this I bought fiberoptic parts, however as they have come dramatically down in price I also bought a 10 Gb RJ45 transceiver for my switch thinking that I can use it for my machine. However to do...

www.truenas.com

and the fact is that there aren't a lot of good copper choices. If you're terrified of fiber or don't understand the issues, please do feel free to stop by this thread:

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

The SFP+ stuff remains cheaper and maybe also easier.

Thank you for your very fast reply!
Appreciate that very much ;)

For the copper networking, i read about that and came to the same conclusion after ordering that mobo.. ah well.. i'm not the networking guy XD.

I hoped to use the big pcie lanes on my mobo for those m1015's, but i'll consider going fiber. And if x4 physical cards exist, it's a no-brainer.

As for the two HBA's, i already bought them so i'd like to get them running properly. Didn't try different system so great suggestion!

Just want to be able to fit a lot of hdd's so i figured those as they are highly recommended.

I did however used the onboard sata's for the 2 ssd's during install. ;)

I'll give it another run tomorrow ;)

Thanks again!

 

[Matthias]

Install sidenote:

When selecting the install target, you can select multiple target drives, as in my 2 listed ssd's.

However the guide (see below) mentions that you map it via system > boot > .... if you want the boot dev's mirrored.

So i went on with just one selected.

Mirroring the Boot Pool |

Mirroring the Boot Pool

www.truenas.com

Is this the same thing, or a different functionallity?

That would be great if it was, mirroring options through the installer.

For the mirror, it worked flawlessly with the guide (i think?), like everything so far with TrueNAS!

Just figured i'd ask and learn!

Thank you!

 

[sretalla]

Is this the same thing, or a different functionallity?

It's the same.

After creating the mirror at install, you can still manage it as if you created it later... you need to be able to manage failed member disks. It would be silly to expect a reinstall just to remake a mirror.

 

[Matthias]

It's the same.

After creating the mirror at install, you can still manage it as if you created it later... you need to be able to manage failed member disks. It would be silly to expect a reinstall just to remake a mirror.

Fantastic!

Thank you ;)

 

[JohnK]

Got all the components, and installed TrueNAS Core U.8 succesfully!

I am however getting frustrated getting my M1015's to IT firmware... HELP?

Haven't done this in years, but when I flashed my two cards, I used this guide. Also, I had to use an older PC to with DOS boot up disk (USB)
servethehome.com/ibm-serveraid-m1015-part-4

 

[Etorix]

-> For now i've gone with 2x 16GB of ECC RAM, i hope that's sufficient to start, but can be expanded to 128GB.

Should be enough. If it's not, then a RDIMM system might have been a better choice.

VDEV's:
-> 5x 16TB Exos drives
-> 3x 8TB (mix of RED and 2 White's)
-> 2x 500GB Samsung Evo's (OS installation)

So for the 16TB drives, would you guy's opt for RAIDZ1 + hot spare (3 disks + parity disk + hot spare), or a RAIDZ2 (3 Disks + 2 parity disks).
I cannot decide..

I figure that the first is "more secure" in the event that i lose a drive as with the hot spare it would resilver instantly? In the second case i'd have to get spare drives and then rebuild the data. (but imagine this will go faster as it has 2 parity drives to resilver?)

In the event of losing a drive, raidz1 has no longer any redundancy. Resilver on 16 TB drives is certainly not "instant" (think 1-2 days), and any issues during the process will result in loss of data.
In the event of losing a drive, raidz2 still has one level of redundancy and can still sustain incidents during resilver.
Too bad you already have the drives, because 6*12-14 TB or 8*8-10 TB would have been worth considering for a similar capacity.

As it stands, I do not see much purpose for three 8 TB drives, short of getting some more drives and build a second NAS to back up the first NAS.

Why 2 M1015? There are 8 SATA onboard, which is enough for 5+2 boot, or up to 8 drives in a pool, booting from M.2 NVMe.
A Node 804 is never going to accommodate 24 drives (2*8 from M1015 + 8 on board)!

To Scale or not to Scale?

I understand that Scale is still experimental, but if i understand correctly, is more suited for running VM's.

Correct. So, for safety, I'd keep the data store on CORE for at least one year or two.

Also, in the event the OS bricks, it should be a piece of cake retrieving back the VDEV/config of the drives as it is stored on the drives itself right?

Keep a copy of the configuration file outside of the NAS itself!

I understand it is not that straightforward to expand the VDEV's by simply putting in extra drives, but that it will require a full HDD swap (new VDEV) to accomodate bigger storage capacity. By the time i will use all the space from the 16TB array, there will be larger drives on the market so i can 'juggle' between the 3 disks array and then the 5 disks array, and so on... (or get a larger case/rack to house it all)

Correct. Expansion is by replacing drives with larger ones and/or adding new vdevs, preferably of the same geometry as the existing vdevs.
So, with a 4-wide or 5-wide vedv as the base unit, there remains a possibility to expand by adding a second vdev. With 5-wide, this then requires a HBA, and then I'd advise to attach the 8 suspended drives to the HBA and the 2 drives on the floor to the motherboard, for ease of cabling. With 8 drives or less, everything on the motherboard and no HBA.

 

[Matthias]

Haven't done this in years, but when I flashed my two cards, I used this guide. Also, I had to use an older PC to with DOS boot up disk (USB)
servethehome.com/ibm-serveraid-m1015-part-4

Will try!

thanks

 

[Matthias]

Should be enough. If it's not, then a RDIMM system might have been a better choice.


In the event of losing a drive, raidz1 has no longer any redundancy. Resilver on 16 TB drives is certainly not "instant" (think 1-2 days), and any issues during the process will result in loss of data.
In the event of losing a drive, raidz2 still has one level of redundancy and still sustain incidents during resilver.
Too bad you already have the drives, because 6*12-14 TB or 8*8-10 TB would have been worth considering for a similar capacity.

As it stands, I do not see much purpose for three 8 TB drives, short of getting some more drives and build a second NAS to back up the first NAS.

Why 2 M1015? There are 8 SATA onboard, which is enough for 5+2 boot, or up to 8 drives in a pool, booting from M.2 NVMe.
A Node 804 is never going to accommodate 24 drives (2*8 from M1015 + 8 on board)!


Correct. So, for safety, I'd keep the data store on CORE for at least one year or two.


Keep a copy of the configuration file outside of the NAS itself!


Correct. Expansion is by replacing drives with larger ones and/or adding new vdevs, preferably of the same geometry as the existing vdevs.
So, with a 4-wide or 5-wide vedv as the base unit, there remains a possibility to expand by adding a second vdev. With 5-wide, this then requires a HBA, and then I'd advise to attach the 8 suspended drives to the HBA and the 2 drives on the floor to the motherboard, for ease of cabling. With 8 drives or less, everything on the motherboard and no HBA.

Thank you for your input! ;)

For the VDEV's, RAIDZ2 it is ;).

Why 2 m1015's? => they were on sale 65€ each, "new from old stock". (so prob not new XD).

Also take in mind that HBA's come highly recommended, everywhere i looked. (with proper setting up ofc) So at the time i was looking around, these popped up.

The node 804 takes in 10 drives, but that case is an easy swap for a bigger unit/rack that holds an mATX mobo.
So i figured with 10 drives, i'd need at least one + 2 mobo ports to start. And if i had been advised to avoid the onboard SATA's, i'd need at least 2, but that's not the case.

Will be running core to get accustomed to it, and will save a config on a different machine.

Thank you!

 

[JohnK]

Will try!

thanks

should also have mentioned that often these "guide" has links to software no longer available on upload sites. You can get the latest software from:
www.broadcom.com/support/

look in current firmware and scroll down to 9211-4 <= that is what I used based upon my cards.

You need to extract the various required files to the Dos boot disk and then follow the guide..

 

[Matthias]

should also have mentioned that often these "guide" has links to software no longer available on upload sites. You can get the latest software from:
www.broadcom.com/support/

look in current firmware and scroll down to 9211-4 <= that is what I used based upon my cards.

You need to extract the various required files to the Dos boot disk and then follow the guide..

Thanks for the info.

However could not find the files you are mentioning, only pdf's about the "9211-4".
As there are 277 downloads under firmware.. i think i'll skip that XD.

 

[JohnK]

Thanks for the info.

However could not find the files you are mentioning, only pdf's about the "9211-4".
As there are 277 downloads under firmware.. i think i'll skip that XD.

haha, finding the right files one their website can be tricky.
You can also just search for in top right corner...
"9211-4i_Package_P20_IR_IT_FW_BIOS_for_MSDOS_Windows"

 

[Matthias]

Got a bit further down the road flashing those m1015's though today!

My mobo went fine in BOIS and UEFI, that didn't matter. What did was the command itself..:

megarec -writesbr 0 sbrempty.bin
megarec -cleanflash 0

It worked changing it in megarec3, as the exe was replaced/updated.
So went on and did all the cmd's, one card at a time, as all the guides mention somewhat the same cmd's.

But i see that the FW file is probably older than the newest available, nor is the ROM installed (not needed).

 

[Matthias]

haha, finding the right files one their website can be tricky.
You can also just search for in top right corner...
"9211-4i_Package_P20_IR_IT_FW_BIOS_for_MSDOS_Windows"

Got it

I do however notice: "2114it.bin", instead of the "2118it.bin" i used earlier.
Or does this not matter?

thanks a million ;)

 

[Matthias]

Got it

I do however notice: "2114it.bin", instead of the "2118it.bin" i used earlier.
Or does this not matter?

thanks a million ;)

NVM! Found it.

(9211_8i_Package_P20_IR_IT_FW_BIOS_for_MSDOS_Windows.zip)
-> 8i

Will try n update the FW and ROMs on both cards.

 

[JohnK]

Ha, my bad, use:
"9211_8i_Package_P20_IR_IT_FW_BIOS_for_MSDOS_Windows'

Long time ago, FreeNas complained badly when your firmware didn't match the install! That forced people to upgrade and sometime downgrade their firmware. You can use the same procedure to upgrade your card to P20, but I don't know if TrueNas is so prescriptive...