Suggestions for FreeNAS build

[anmnz]

That certainly is interesting. Must be specifically prohibited by the FreeNAS UI. I stand corrected.

 

[macx979]

I've got 5x 4TB HDDs which I've been planning to make into a single RaidZ2 vdev

This could be done, but in case you like to extend your storage, you need to add another 5 HDDs. If that's ok for you, you could also think about adding a 6th HDDs right from the start using RaidZ2. This increases the usable disk space to raw disk space. Adding a 6th disk gives you roughly 4TB more of useable storage. So basically 100% of this 6th drives is useable.

That's my thinking, but should I be considering pairs of 2 mirrors instead?

well, I think if you are already fine with 10TB (or add the 6th HDDs for 14TB) and don't plan to extend space sooner or later, RaidZ2 would be the better choice.
If you want small incremental upgradeability, plus some performance advantages, maybe mirrors are the better way.

 

[zeebee]

This could be done, but in case you like to extend your storage, you need to add another 5 HDDs.

Yeah I don't think I'd ever be doing this - not enough room in my case for one. I imagine this system will last us quite a while (our 2TB one has lasted us 10 years), but if I wanted to upgrade I'd probably end up getting 5 larger disks and doing the slow replace-one-by-one thing. Hopefully by then they won't be spinning disks either.

you could also think about adding a 6th HDDs right from the start using RaidZ2

It's funny you suggest this, because I originally had the plan to buy 4 drives with RaidZ2, but when I realized I could add one more and get basically the whole disk I bumped it to 5. I'm already beyond the limits of my budget so I think 5 will have to do for now :(

 

[macx979]

t's funny you suggest this, because I originally had the plan to buy 4 drives with RaidZ2, but when I realized I could add one more and get basically the whole disk I bumped it to 5

well, I thought it through for myself and had the same enlightenment. :)
However I decided to go for mirrored vdevs in order to have a nice upgrade path the time I need it.

The configuration for my freenas is almost done on paper now.

existing:
supermicro x9scm-f
xeon e-3 1240v2 (I realized it's not a 1230v2, as I said before - good for me :))
32gb ecc ram
4x Seagate 8TB drives
BeQuiet PurePower 430Watts
2x 5TB Harddrives for Backups
2TB Online Space for Off-Site Backups

to buy:
LSI 9207-8i (seems to be highly recommended and has pci-e 3.0 - to be honest I don't get the numbering scheme of LSI)
2x120GB SSD for OS (smaller SSD aren't cheaper)
Inter-tech 4u4416 or 3u3416 (they are basically identical besides height - which one I go for depends on the spare rack space I have after rearranging it)
Intel x540 T2 2x 10GBit network card
3x Noctua 120mm S12A as a replacement for the case fans
2x Noctua 80mm or 60mm

Is anything wrong with the LSI? If I go for an original LSI instead of OEM I don't need to crossflash which seems to be troublesome sometimes.

Planned setup:
4x 8TB HDDs as mirrored vdevs in one pool
2x 5TB HDDs as mirroed vdev for snapshots
2x 120gb SSD in mirroed vdev for OS

Shall I connect the SSD and/or the 2x5TB HDDs to the HBA or to the onboard SATA controller via breakout cable?
Have I missed anything?

Best

 

[PhiloEpisteme]

LSI 9207-8i (seems to be highly recommended and has pci-e 3.0 - to be honest I don't get the numbering scheme of LSI)

There is a post somewhere with that info as well as resources on other sites. I don't recall offhand. Whether you figure it out via the codes or not what you want to know about any given card is at least the following
- whether it is a RAID card or not (RAID = bad) :)
- how many internal vs external ports it has (8i has 8 internal lanes on 2 ports)
- whether it is SAS, SAS2, or SAS3
- the PCIe version.

You likely don't need a PCIe 3.0 unless you have a lot of drives or are using flash storage for something other than boot devices. Consider that an 8x PCIe 2.0 slot has ~4GB of bandwidth; even accounting for encoding you're looking at ~3.2GB of bandwidth. That is quite a lot.

Shall I connect the SSD or/and to the HBA or to the onboard SATA controller via breakout cable?

I generally keep those on my onboard SATA ports. If they are just boot devices it probably isn't the biggest deal except it saves having to do any configuring for booting from the HBA.

Have I missed anything?

I'd say the build looks pretty reasonable. I use that same board for my backup server, but with a terrible CPU and only 8GB of memory; it does just fine for basic backup work; I'm sure your machine will be much more capable.

 

[macx979]

There is a post somewhere with that info as well as resources on other sites. I don't recall offhand. Whether you figure it out via the codes or not what you want to know about any given card is at least the following

that's exactly what I was looking for but couldn't find. If you know what to search for, I'd appreciate your advise.

You likely don't need a PCIe 3.0 unless you have a lot of drives or are using flash storage for something other than boot devices.

That makes sense. However I recently read that PCIe3.0 gets less hot than 2.0, which would be a benefit for cooling. Furthermore, there's no huge price difference, so I would rather go for something "newer".

I generally keep those on my onboard SATA ports

What do you think about the 2x5TB mirrored backup vdev? onboard sata or HBA? I think, if onboard is fine, that saves me some ports on the HBA for further expansion.

 

[PhiloEpisteme]

That makes sense. However I recently read that PCIe3.0 gets less hot than 2.0, which would be a benefit for cooling. Furthermore, there's no huge price difference, so I would rather go for something "newer".

Never heard anything about temps; but PCIe 3.0 is certainly "better" in that it is faster and has lower overhead. If it isn't a significant cost increase it certainly won't hurt.

What do you think about the 2x5TB mirrored backup vdev? onboard sata or HBA? I think, if onboard is fine, that saves me some ports on the HBA for further expansion.

To me, there isn't really a huge difference; do whatever is most convenient. The only times I'd say go with onboard plugs is if you're worried about SSDs taking up all the bandwidth on the HBA or if you're having issues booting from the HBA.

 

[macx979]

alright, my build is done. Had to wait for some minor but important items to finally finish it.




so far everything works flawless and I already created a Pool and a couple of dataset. I also populated and shared them and this build seems to be very potent.
By measuring performance on an uncompressed dataset I get these results:
Write: 374MByte/s
Read: 458MByte/s

I can't complain about the performance.

After copying data back and forth from all HDD drives I had, I finally managed to wipe all data of the two 5GB drives. Unfortunately, copying all this data was too much for one of these drives and simply broke down.
However I added the remaining drive to the system, in a slot which is not connected to the LSI Controller but directly to the onboard SATA port. I used this cable.
10Gtek® Internes Mini SAS Kabel SFF-8087 zu 4X SATA 0.5-Meter, Mini SAS 36Pin SFF-8087 to 4X SATA Cable

Neither does the HDD being recognized by Freenas nor does it appear in bios settings. So I connected the drive directly via SATA to the onboard SATA ports and then it shows up in freenas.
So, do you have any idea what's the root cause here? Is the wrong cable or just a faulty one or do miss something?

Best

 

[PhiloEpisteme]

Unfortunately, copying all this data was too much for one of these drives and simply broke down.

Was it a new drive? And did you burn it in? It could've just been the new-drive mortality. :)

 

[macx979]

Was it a new drive?

I had two old external 5TB USB drives. I used them to backup my old NAS before moving the NAS drives to my FreeNAS box. This was apparently too much work for one of these drives. :) I lost some data, but nothing serious. So it was rather a old-drive mortality. ;)

What a about the SAS cable should I send it back and get the same cable again or is there any other recommodation?

EDIT: just read that there are forward and backward breakout cable. gotta check this.

 

[zeebee]

all right, my build is done

Looks great - I'm jealous of your fans!

 

[blueether]

I had two old external 5TB USB drives. I used them to backup my old NAS before moving the NAS drives to my FreeNAS box. This was apparently too much work for one of these drives. :) I lost some data, but nothing serious. So it was rather a old-drive mortality. ;)

What a about the SAS cable should I send it back and get the same cable again or is there any other recommodation?

EDIT: just read that there are forward and backward breakout cable. gotta check this.

yep, that looks to for sas controler to sata drive, where you will want a backward one to connect to a sata port on a motherboard

 

[macx979]

Looks great - I'm jealous of your fans!

Unfortunately they're quite expensive if you use it for the entire build. However, especially the 12mm NF-S12A case fans are damn silent if the mainboard fan control is set to optimal. So you don't need an additional fan speed controller like I had to for another build where I used NF-F12.
The noisiest part are the HDD drives. :)

yep, that looks to for sas controler to sata drive, where you will want a backward one to connect to a sata port on a motherboard

This was even mentioned in the description but I kinda ignored it.