What's the best configuration for this hardware?

[WKG]

So, as per my previous thread, I need to rebuild my fileserver from scratch, and I'd like to do it correctly this time, instead of just going with the FreeNAS defaults. Here's what I have for hardware:

Configuration as follows:
Build FreeNAS-9.3-STABLE-201512121950
Platform Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
Memory 49109MB

HP Z800 workstation, dual X5570, 48gb ECC-RAM, AS#460838-002 motherboard, add-on PCIex LSI LSI00303 (9207-4i4e) SATA/SAS HBA
6x 4TB HGST DeskStar 7200 SATA (One currently has all my data backed up to it, so I don't want to use it right now, and a 2nd one has been identified as bad and will likely be RMA'd, so is not currently available)

So with 4x 4TB drives currently available, how should I proceed? Create a VDdev with all 4 drives and then a RaidZ2 array? Then when my other 2x 4TB drives are available (data copied off the one, and the other returned from RMA) what should I do? Just keep them handy as spares, or add them to a separate VDev?

 

[jgreco]

Depending on how much you love your data, you could wait until the RMA drive comes back, and then create a 5 drive RAIDZ2 (12TB) or RAIDZ3 (8TB). Once you copy your data off the existing 4TB drive, then add that as a spare to the pool. This configuration has the benefit that if something goes wrong, the NAS can start remediating the problem immediately.

 

[WKG]

And if I used 5 drives, would this all be in 1x Vdev?

 

[jgreco]

Yes, a virtual device is merely some aggregation of physical devices in a data protection strategy, such as mirroring or RAIDZ.

 

[WKG]

Okay, yes and this would bring me up to 60% space utilization instead of 50% with only 4 drives, according to an online RAID calculator tool I used.

So, another question, in regards to my hardware:

The motherboard I have has 6x SAS on-board connectors with a LSI RAID controller, which I immediately disabled, and an additional 6x SATA (Intel Matrix?) on-board connectors (currently used for CD-ROM, etc). I've added in a LSI00303 (9207-4i4e) SATA/SAS HBA but as I only have the internal cable I can only connect 4 drives. For the 5th and 6th drives, should I use the motherboards onboard SATA Intel Matrix system?

Will FreeNAS allow adding devices to the Vdev / pool that are on different controllers, or is this a bad idea?

 

[jgreco]

So a good question might be "what is that LSI RAID onboard". Can you identify what that is?

The Intel Matrix storage stuff is irrelevant, but I believe that the SATA ports themselves should still be perfectly usable. As long as your LSI HBA is flashed to IT mode, the ports appear to FreeNAS to be almost exactly identical to the mainboard SATA ports. You can move and shuffle disks among them, and, no, FreeNAS doesn't care what you do in that regard. YOU might care if you maybe had that LSI HBA in a PCIe x1 slot or something like that, but the OS isn't going to freak. It ain't Windows after all.

 

[Bidule0hm]

RAID-Z2 + hot spare is stupid 99 % of the time... the spare drive just stay here doing nothing but wearing out... do a RAID-Z3 instead :)

 

[jgreco]

RAID-Z2 + hot spare is stupid 99 % of the time... the spare drive just stay here doing nothing but wearing out... do a RAID-Z3 instead :)

So you propose that he should wipe out his data first?

The problem we were trying to address is that he's got his data on one disk and presumably he wants to keep it. Perhaps he can siphon off his bits and store them in a Ziploc baggie while he's making your suggested RAIDZ3?

 

[Bidule0hm]

That's why I put 99 % and not 100 % :D

Perhaps he can siphon off his bits and store them in a Ziploc baggie while he's making your suggested RAIDZ3?

Yep, should work :)

 

[jgreco]

Yep, should work :)

Works for me, that's how I store my backups!

 

[WKG]

Ha-ha if only it were that easy. But actually, I may just copy the data to a few 1tb drives that I have laying around. I already have the originals backed up, so even if I use some older drives I'm not risking much.

I don't recall the onboard LSI model offhand, but I do know that it doesn't support the 4tb HGST drives that I have, sees them only as 2.5tb or something like that. But now that I think of it, that was when I was trying to build an array using it's hardware raid controller and before I was introduced to FreeNAS. Perhaps the onboard LSI SAS controller can be used as HBA and then FreeNAS will recognize/access the full 4tb?

4tb works fine on the onboard SATA which is why I had planned to us that for any additional drives.

 

[WKG]

From my review of the HP Z800 specs I can find online, looks like the onboard SAS is a LSI 1068E offering RAID 0, 1, 1E/10E. Doesn't look like this controller supports higher than 3tb.

I'm not at home so I can't confirm the exact firmware revision etc of the onboard LSI controller, but is there any reason to believe this controller may be useful in my configuration?

 

[jgreco]

See the LSI sticky. The LSI 3Gbps silicon is limited to 2.2TB disks due to LBA addressing issues. The 1068 is pretty worthless and isn't recommended.

 

[WKG]

So I've switched around the drives and am currently RMA'ing the one 4TB, so in the meantime (mostly for testing) I've created a new pool, RaidZ2 this time, using the 4x4TB HGST drives. I used the wizard and it's defaults, and it's working but seems to be extremely slow? I'm copying data to it via CIFS / Windows, and getting a maximum of 5-6 MB/s, which is far slower than the 30+ MB/s I had with the previous configuration (RaidZ1 I think). Is this because encryption was somehow enabled? I don't see it indicated anywhere, and I didn't choose to add it (as I don't need it).

Or perhaps it's because the RaidZ2 has now defaulted to LZ4 compression and my dual Xeon x5570's can't keep up?

 

[jgreco]

Any vaguely modern CPU can keep up with a reasonable amount of compression at gigabit speeds without issue. You can safely try disabling compression just to make sure, but it's likely something else is amiss.

Did you do any basic burn-in testing of the system? See https://forums.freenas.org/index.ph...esting-your-freenas-system.17750/#post-148773 and in particular the script in that thread which can give some insights into HBA/drive performance.