Most of my hardware has arrived - recommended reading and best practices when building?

[SRSR333]

Hello everyone -

All the parts that I've ordered have arrived, save for the Node 304 case that I'll have to purchase personally, since it costs a bomb to ship it to where I live, on Amazon. A quick run through of my parts list again:

1. Motherboard: ASRock Rack E3C236D2I (hopefully this Skylake board doesn't suffer the same BMC write issues as the Avoton ones, it's two years newer)
2. CPU: Intel Pentium G4500
3. RAM: Kingston ValueRAM 16 GB DDR4 2133 MHz ECC UDIMM
4. OS drive: SanDisk Ultra Fit 16 GB USB 3.0 drive
5. Storage: 2x WD Red 3 TB drives
6. PSU: Corsair SF 450 SFX 80+ Gold fully-modular PSU
7. Case: Fractal Design Node 304

Right now, I've looked through Cyberjock's presentation, DrKK's 'First Configuration for Small FreeNAS Deployments', jgreco's and qwertymodo's system and drive burn-in guides respectively, and I'm quite certain my NAS system won't be active for another month or so at least.

Meanwhile, I have some questions about how I ought to format my drives. Since I only have two of them, I have only two options: either a) 2 vdevs of one drive each, giving me 6 TB useable or b) one vdev, both drives mirrored.

I'll probably go with b) because everyone says a) is a terrible option, but I have one question. Say I want to expand my storage with more drives in the future. How would I go about planning the new drives in vdevs (given that my case will have space for up to six drives total)?

Next, I understand that the single vdev will contribute to the storage pool. Have you got any suggestions as to how I ought to create my datasets for the use cases mentioned below? I believe that datasets are treated like discrete file systems, where the storage pool-dataset pair is analogous to the traditional disk drive-partition pair.

I plan to use this machine for three use-cases: backups (of the various computers at home, as well as documents and files), family photo and video storage, and finally film and music storage, where I might possibly use Transmission and Plex for the latter.

Next, is there any way to make Windows System Restore, Shadow Copies and File History work natively with the ZFS snapshots feature? All the PCs at home run at least Windows 8.1. We have no Macs around.

Is there any way to secure the shared datasets against a virus attack? For example, some very modern ransomware viruses even attack network-shared folders on Windows. Is there a tried-and-true way to prevent this?

Last few questions: I only have one Ethernet port on my wall where the system will be placed. Does the Intel LAN chipset support IPMI as well? How do I even configure the system with IPMI? I noticed there's a serial COM port on my motherboard - do I want anything to do with this, or leave it well alone?

Besides these, have you guys got any other tips, to be followed while building, installing, setting up? Cable management ideas? Should I bother with FreeNAS 10, which looks wonderful?

Thanks in advance.

 

[Ericloewe]

Say I want to expand my storage with more drives in the future. How would I go about planning the new drives in vdevs (given that my case will have space for up to six drives total)?

The obvious choice would be to add another mirror of two. You could also add a third drive to the original mirror plus a new three-way mirror. Or make a six-way mirror, which is really rather silly.

where the storage pool-dataset pair is analogous to the traditional disk drive-partition pair.

Somewhat. I recommend making a dataset for each set of data that shares some unique property which maps to ZFS properties, like "Documents should have hourly snapshots, Recorded TV doesn't need snapshots or compression and the family pictures should have daily snapshots".

Next, is there any way to make Windows System Restore

What do you mean? The "rollback driver/application changes" kind of thing introduced in Windows ME? No, that's purely local and doesn't rely on backups.

Shadow Copies

One snapshot task per share can be exported by Samba so as to present itself to Windows just like Shadow Copies on an NTFS Windows Server would.

File History work natively with the ZFS snapshots feature?

It'll work, but it's not integrated in any way. Windows will just write whatever structures it generally uses for File History, like it would to a local USB drive. Same as with a Windows Server.

Is there any way to secure the shared datasets against a virus attack?

You'll have to take care of that at the endpoints.

For example, some very modern ransomware viruses even attack network-shared folders on Windows. Is there a tried-and-true way to prevent this?

Yes, snapshots. Just roll back to before your stuff was encrypted and you're good. Even if they were remotely aware of ZFS, they couldn't do anything from a client.

Last few questions: I only have one Ethernet port on my wall where the system will be placed. Does the Intel LAN chipset support IPMI as well?

The Intel i210 can be shared by the host and the BMC, no problem.

How do I even configure the system with IPMI?

You login on the web interface and can just use the Java iKVM application to manipulate the server as you would locally.

I noticed there's a serial COM port on my motherboard - do I want anything to do with this, or leave it well alone?

It's a bog-standard RS-232 port controlled by the Host. You can use it if you need it, but few people need it these days.

Should I bother with FreeNAS 10, which looks wonderful?

Not yet, it's still not close to production-ready. It's just reached a state where your average user can play around with it.

 

[joeschmuck]

Say I want to expand my storage with more drives in the future. How would I go about planning the new drives in vdevs (given that my case will have space for up to six drives total)?

Well you have the proper advice right now for a pair of drives which is a mirror and that will give you just under 3TB of storage for now. As your storage needs go up you will want to add more drives of course but how it the real question. My advice is, because you are only looking at having at the most just over 2TB of active storage, this data would be saved off and then you destroy your pool and add four (4) more 3TB drives and create a RAIDZ2. This would give you less waste of drives as opposed to adding two more vdevs of mirrors.

Also, that USB 3.0 drive you have, toss it in favor of a USB 2.0 drive and save you the grief of a failed USB Flash drive.

Let me ask you this question... What are your storage requirements right now and in 3 years? You should plan for the 3 year mark at a minimum and then double it because lets face it, we always use more than we expected, and this allows you to keep 20% free for performance reasons.

 

[Stux]

6 disks in RAIDZ2 is a good thing, but if you buy 2 disks at a time, you may want to just buy bigger disks each time

So

Start with 2x3 (3 usable)

Add 2x4 for 7 usable total.

Add 2x6 for 13 usable

Replace 2x3 with 2x8 for 18Tb Usable etc.

Or alternatively, buy 4 more 3TB drives when it's time to grow, blow away your pool, make a 6 way z2 pool and restore your files from your backup.

Which gets you 12 usable.

 

[Stux]

You can play musical chairs too.

Partition 3 disks two ways. Make a 6x1.5TB Z2. Gets you 6TB. Copy your 3TB, then replace each 'partition' one at a time with real disks.

Done this. It works.

 

[SRSR333]

Gentlemen, thank you all for your posts. @Ericloewe, thanks especially for your detailed breakdown of my questions and answering them one by one. It's quite interesting that ZFS' snapshots can be presented to Windows as Shadow Copies, need to find out more about that. Is there documentation for this?

As for @joeschmuck's question - I'd probably less than double my needs for storage in the next couple of years. I may purchase another couple of drives (4 TB, this time) in 2018-2019. One thing though - why'd you ask me to ditch my USB 3.0 drive in favour of a 2.0 one? There's also an M.2 slot on my board, I will (in the near future) purchase an NVMe M.2 SSD to use with it.

And finally, @Stux, thanks for your storage config recommendations. For now, I'll use two drives in a mirror config, as I've already reached my budgetary limit. I will keep your ideas in mind as I upgrade a while from now.

Final question: how long can I expect this hardware to last? I don't want to find myself having to toss my > $300 motherboard out in a couple of years just to buy a new one. I might be able to finance a CPU + RAM upgrade to Xeon E3 or E5 and 32 GB, but that will also take a while.

 

[Stux]

Final question: how long can I expect this hardware to last? I don't want to find myself having to toss my > $300 motherboard out in a couple of years just to buy a new one. I might be able to finance a CPU + RAM upgrade to Xeon E3 or E5 and 32 GB, but that will also take a while

Providing the motherboard doesn't expire (and, asrock is in the doghouse at the moment), I'd expect it to last 5-10 years, if it were on a UPS.

The HDs won't last that long, I'd expect them to start failing after 3 years, but that will coincide with you wanting to replace your oldest HDs with bigger ones.

 

[Ericloewe]

It's quite interesting that ZFS' snapshots can be presented to Windows as Shadow Copies, need to find out more about that. Is there documentation for this?

There's a section in the manual: http://doc.freenas.org/9.10/sharing.html#configuring-shadow-copies

Final question: how long can I expect this hardware to last? I don't want to find myself having to toss my > $300 motherboard out in a couple of years just to buy a new one.

The main limitation will be the maximum amount of RAM (hardware failure aside).

There's also an M.2 slot on my board

It's a weird one. Room for very short devices only (of which there are very few), sharing one the SATA ports despite the PCH having two unused SATA lanes (they even seem to be routed to where they placed the M.2 slot!). They do imply it does SATA and PCI-e, though, which is certainly welcome.

 

[Stux]

The main limitation will be the maximum amount of RAM (hardware failure aside).

Yeah, I forgot that that board was Mini ITX. Maybe there will be 32GB DIMMs which are compatible one day since Skylake does support 64GB.

 

[Stux]

It's a weird one. Room for very short devices only (of which there are very few), sharing one the SATA ports despite the PCH having two unused SATA lanes (they even seem to be routed to where they placed the M.2 slot!). They do imply it does SATA and PCI-e, though, which is certainly welcome.

If its is 4x (and I looked into it, but was never sure), there is a possibility of using a U2 adapter and then getting an Intel NVMe U2 SSD, but there won't be much point I suspect with the ram limitations.

 

[SRSR333]

Providing the motherboard doesn't expire (and, asrock is in the doghouse at the moment)

Ah, yeah, I've seen the Avoton boards dying left, right and centre. Hopefully, as already mentioned, this C236 board fares a bit better.

There's a section in the manual: http://doc.freenas.org/9.10/sharing.html#configuring-shadow-copies

Excellent. Time for more reading.

The main limitation will be the maximum amount of RAM (hardware failure aside).

True, but I honestly do not think I will need more than 32 GB of RAM even with the '1 GB per TB' rule-of-thumb of ZFS, just because I won't need that much storage, even if I start shooting in RAW instead of JPG. At most, I'll have 6 × 6 TB drives.

It's a weird one. Room for very short devices only (of which there are very few), sharing one the SATA ports despite the PCH having two unused SATA lanes (they even seem to be routed to where they placed the M.2 slot!). They do imply it does SATA and PCI-e, though, which is certainly welcome.

I noticed that, and it's quite a bummer. But the case I have will anyway only take 6 drives, so it's just as well that I have 6 SATA ports, I suppose. There's a Newegg review of this board where a user simply inserted a longer drive into the slot, and it worked perfectly fine because the slot was near the edge with nothing really blocking it, though the SSD looked quite odd hanging off the edge.

 

[joeschmuck]

One thing though - why'd you ask me to ditch my USB 3.0 drive in favour of a 2.0 one?

This is an easy one. USB 3.0 flash drives tend to overheat. I don't believe they were designed for long continuous operations. USB 2.0 flash drives typically do not produce the higher heat loads. If you really must use a USB 3.0 flash drive then you should buy one that is large enough to dissipate the heat. If you look through the forums you will find a lot of people having failed USB 3.0 drives. I'm not saying everyone has had a bad experience but there are a lot of them out there.

There's also an M.2 slot on my board, I will (in the near future) purchase an NVMe M.2 SSD to use with it.

From what I have been able to find out about your motherboard, it sounds like you already know that you need to purchase an M.2 card with a PCI-E interface, not a SATA interface. I could not locate any 2230 sized SSD PCI-E cards. Have you found anything yet since you stated the "near future"? I found WiFi and such but not a SSD.

 

[SRSR333]

This is an easy one. USB 3.0 flash drives tend to overheat. I don't believe they were designed for long continuous operations. USB 2.0 flash drives typically do not produce the higher heat loads. If you really must use a USB 3.0 flash drive then you should buy one that is large enough to dissipate the heat. If you look through the forums you will find a lot of people having failed USB 3.0 drives. I'm not saying everyone has had a bad experience but there are a lot of them out there.

Damn, I just noticed the super-long thread at the SanDisk forums about the very drive that I purchased overheating like hell and ruining perfectly good USB 3.0 ports. I'm taking a look at the larger SanDisk Extreme.

From what I have been able to find out about your motherboard, it sounds like you already know that you need to purchase an M.2 card with a PCI-E interface, not a SATA interface. I could not locate any 2230 sized SSD PCI-E cards. Have you found anything yet since you stated the "near future"? I found WiFi and such but not a SSD.

Likewise, I've only found Wi-Fi modules that are size 2230. However, as I've mentioned earlier, it is possible to insert a longer drive as the M.2 slot itself is at the edge, and isn't really blocked by anything (not even the PCIe clip, which goes over the plane of the drive). I'm looking for a low-capacity NVMe drive, but I don't think they exist, yet.

 

[Stux]

You'd provably want to use something to help stick the gym stick down

But look into a "u2 hyper kit"

https://rog.asus.com/17342015/hands-on/intel-750-2-5-inch-ssd-with-nvme-and-asus-hyper-kit/

 

[joeschmuck]

Likewise, I've only found Wi-Fi modules that are size 2230. However, as I've mentioned earlier, it is possible to insert a longer drive as the M.2 slot itself is at the edge, and isn't really blocked by anything (not even the PCIe clip, which goes over the plane of the drive). I'm looking for a low-capacity NVMe drive, but I don't think they exist, yet.

I'm a bit skeptical about the standoff not interfering at all unless you just don't install it. You will then need to use something like double sided foam tape to secure it however the problem I see here is when you use this, you need to be careful of ESD. Tape can and does build up a static discharge and it would be a shame to zap a part of your electronics. The point is you still need to find a way to secure the M.2 card into it's slot, it doesn't snap in place and just stay there.

I think if it were me, I'd build an adapter that I could screw into the NUT1 hole and extend another nut to the proper length. It would have to be ridged but there are materials to handle that. Yes, that would be my plan. Sounds pretty easy. A piece of wood could do it. I can see it now.

 

[SRSR333]

I'm a bit skeptical about the standoff not interfering at all unless you just don't install it. You will then need to use something like double sided foam tape to secure it however the problem I see here is when you use this, you need to be careful of ESD. Tape can and does build up a static discharge and it would be a shame to zap a part of your electronics. The point is you still need to find a way to secure the M.2 card into it's slot, it doesn't snap in place and just stay there.

Yeah, that sounds too much like a dirty hack. For now, I''ll stick to using a SanDisk Extreme USB drive, it's fairly big and should dissipate the heat much better. The Ultra Fit I have has apparently ruined USB ports through the heat.

 

[joeschmuck]

The Ultra Fit I have has apparently ruined USB ports through the heat.

It doesn't ruin the ports, the USB flash drive itself dies. Or should I say that I haven't seen any posting where the ports were damaged but then again, I don't know everything.

 

[Stux]

It doesn't ruin the ports, the USB flash drive itself dies. Or should I say that I haven't seen any posting where the ports were damaged but then again, I don't know everything.

When I did some research into this, I heard that too, but... Mine get warm but they're never too hot to hold.

Got them on USB3 ports now. We'll see.

Mines not the ultra. Just regular 16GB usb3 fit for a 5$ or so.

 

[Ericloewe]

The other day, I forgot a USB 3.0 64GB SanDisk Ultra in a USB port for two days or so. I was surprised by how hot it was, after idling for two days.

These USB 3.0 flash controllers are in serious need of proper power and thermal management.

 

[Stux]

Yes, they get hot, but was it too hot to hold? Or just actually really really warm ;)