New Build Help - R630

[errrr404]

Hi all. I have a Dell R630 I'm repurposing to use as a NAS possibly for Plex but mostly for storage and photo/video editing over the network. I'm stuck trying to figure out what kind of drives to use for best performance. Would SATA be good enough or SAS 12gbps stuff? I'm planning on replacing the optical drive with an adapter to a MX500 for boot. Any help is appreciated. Thanks!

Specs:
Dell R630
2x Xeon E5-2667v4
80GB DDR4 Memory
H330 flashed to HBA330 w/ 8 bays
2x 10GBe, 2x 1GBe NIC

 

[nabsltd]

Would SATA be good enough or SAS 12gbps stuff?

No spinning hard disk takes advantage of 12Gbps. Where that number becomes important is with many more bays than you have, all connected through a single cable and an expander. Although really fast disks could push 1.5Gbps, and your 8 bays could then technically saturate a 12Gbps connection, it's only likely to happen for actions like resilvering. With SSDs, 12Gbps SAS would help on an individual disk, but then you'd still be limited in aggregate by the 12Gbps link to the HBA.

12Gbps is great for the HBA, cabling, and any expanders, but individual disks often don't benefit from the difference between a 6Gbps and 12Gbps connection.

 

[errrr404]

12Gbps is great for the HBA, cabling, and any expanders, but individual disks often don't benefit from the difference between a 6Gbps and 12Gbps connection.

That's great to know. I was under the assumption I'd be able to run each disk at full 12gbps speed and it would be a significant speed increase.

My initial thought was four striped SAS 12Gbps mirrored to another four. If I use all SATA SSD's it'd be the same because there's no huge benefit in speed? Ideally I would run SSDs for power savings as well and everything over 10Gbe

 

[HoneyBadger]

The R630 is a 1U system, so those 8 bays will be 2.5" in size - if you're willing to buy SSDs, that will give you the "best performance" certainly, but they don't need to be SAS.

If your budget doesn't allow for SSDs though, you're in a bit of an awkward spot as the only large capacity 2.5" HDDs are SMR (shingled) and the non-SMR drives are smaller and often far higher priced in terms of $/GB compared to a 3.5" model.

 

[errrr404]

Thanks guys. I think I'm going to go with a bunch of MX500's and be done with it. I've read posts about those drives causing errors. Is that still an issue?

 

[HoneyBadger]

Thanks guys. I think I'm going to go with a bunch of MX500's and be done with it. I've read posts about those drives causing errors. Is that still an issue?

The "pending sectors" issue was supposedly corrected in 12.x with a smartmontools update, but they're referenced by @Ericloewe as having "monstrous write amplification" here:

Crucial MX500 pending sectors issue

Hello all, I've been having this issue with FreeNAS 11.3 as well. I have several pools based on Crucial MX500 1Tb drives that started reporting alerts for pending sectors. So far this error seems to be caused by the drive when it's too busy and not a good indicator of a pending failure. Is...

www.truenas.com

Having not used them in a ZFS application I can't expand on that myself, but if that's the case they would consume their write cycles quite rapidly. What about something like the WD Blue SATA line of SSD?

 

[Ericloewe]

The "pending sectors" issue was supposedly corrected in 12.x with a smartmontools update

Worked around, really. All smartmontools can do is ignore pending sectors, but the underlying problem still exists.

For now the safest SATA SSDs are the WD Blues. The Samsung 870s also have an issue (possibly corrected in units shipping today) that leads to what could be seen as very premature wear.

 

[ChrisRJ]

[..] photo/video editing over the network.

The majority of the discussion so far has focused on interface bandwidth and sequential transfer rates. Depending on what exactly your workload looks like, random access may be much more relevant. We are then talking primarily about IOPS and conventional HDDs are not in the game anymore, except perhaps for the most basic of tasks.

 

[errrr404]

I'm not entirely sure as it's pretty new to me, but if it's individual images/clips, then it sounds like random access? WD Blue IOPS is 95K/84K. Seagate Ironwolf 125's are 95K/90K with basically the same sequential except an additional 10MB/s write over the Blue.

In case of UPS failure, should I also be looking into drives with PLP? Random write IOPS seems to drop off pretty significantly. Datasheets for 2TB drives
- Samsung PM863 (mixed use): 510/475MB/s 99K/18K
- Samsung SM863 (write intensive): 520/485MB/s 97K/29K
- Seagate Ironwolf 125 Pro: 545/520MB/s 96K/30K

 

[ChrisRJ]

Well, you never specified precisely how the editing will look like. If you working directly on the NAS, scrubbing the timeline will equate to random I/O for example. Rendering on the other hand will likely be a sequential operation. If, instead, you copy the files to your editing machine before you start, and back to the NAS once you are finished, it's an entirely different game.

 

[errrr404]

Well, you never specified precisely how the editing will look like.

This would be entirely editing off of the NAS over a 10Gbe link, both scrubbing and rendering.

 

[errrr404]

I decided to go with PM863's. Four 1.92tb's on the way. Would it work to have seven SSDs in the pool, and one large internal HDD to backup the pool weekly, or would it be preferred to have eight SSDs and one large external HDD to backup?

 

[ChrisRJ]

An internal HDD is not really a backup. Simply because it is subject to many of the dangers that could also harm the main storage.

In addition you should think about an offsite backup that is suitable for your needs.

 

[Alex_K]

Just making sure that you are taking into account, when connecting backplane with a single SAS-2 (MiniSAS, SFF-8087) cable of from SAS2008, its not 6Gbps, its 4x6Gbps throughtput
Same for SAS-3 (MiniSAS HD, SFF-8643) - its 4x12Gbps

As for backup. I knew of people making offsite backup via hotswap internal slot. Like moving an hdd from slot to safe and back. Not saying I'm recommending it.

 

[errrr404]

Just making sure that you are taking into account, when connecting backplane with a single SAS-2 (MiniSAS, SFF-8087) cable of from SAS2008, its not 6Gbps, its 4x6Gbps throughtput
Same for SAS-3 (MiniSAS HD, SFF-8643) - its 4x12Gbps

4x6Gbps means each drive is full 6Gbps, not splitting the 6Gbps right? I think it's perfect. The board has two SFF-8643 ports too. I can just do it that way if it's faster. Maybe even a hair of power savings by removing the HBA330 card

 

[Alex_K]

It's highly probable that 8bay backplane there uses 2xSFF-8643 for multipathing/redundancy, meaning with SAS-2 you have 6Gbps "limit" per each Two bays
Easy to check: disconnect one of SFF-8643 and if you still would see all 8 disks - thats confirmed.
Or backplane documentation
mprutil and sas2ircu / sas3ircu give hints too.

 

[Ericloewe]

It's highly probable that 8bay backplane there uses 2xSFF-8643 for multipathing/redundancy, meaning with SAS-2 you have 6Gbps "limit" per each Two bays

Nah, I'm relatively sure it's a passive backplane. If you want multipath, you have to shell out for an external disk shelf.

 

[errrr404]

TrueNAS up and running, half of my stuff is transferred over. Haven't really accessed files on it yet except for rearranging directories but so far I'm quite happy. Appreciate all of your help

Currently: 4x 1.92TB SSD's in mirrored vdevs, 91.06 min 99.95 mean hit ratio w/ 144GB memory. Is there some sort of tuning I could/should do or should I just leave it alone for now?

 

[errrr404]

Something kinda odd is happening. I was transferring over gbs of footage from within the same pool, just to a different directory and it felt like it took a very long time. I just tested a test directory with three clips totaling 5.1gb. Laptop to TrueNAS via gigabit ethernet took 44 seconds. TrueNAS pool dir to dir copy took 1 minute and 29 seconds. Wouldn't copying within the same drives be significantly faster?

I'm using 8x Samsung PM863a's. 520/480MB/s seq read/write, 97K/24K rand read/write, 4 mirrored vdevs, 2 drives each

 

[Ericloewe]

Were you doing that from an SMB client or something? It sounds like it's moving through the client over 1GbE.

Server-side copies are supported, but the devil is in the details. Just the other day, someone figured out that GNU cp does some bizarre handwaving to determine whether to manually copy everything or to offload it to the kernel (who can do fancy things like server-side copies), to very mixed results.
Which leads to a relevant question here: Core or Scale?