Unusual usage case - Dell T340 - 8-10x8 TB Sata Drives -I took your advice and waited

J

Julianh

Dabbler
Joined
Jan 19, 2016
Messages
35
Hi all,

I posted the following in October, slightly edited now, then the prices went up, so although I was tempted by another server, I waited, saved up and have just ordered the t340 server, it arrives in a few days.

It's basically a media store for 5000-6000 files 8-30 GB in size, running SMB to connect to an Emby server. Read is all I care about, Write speed doesn't matter at all. The one I bought has 32 GB of Ram. I'm putting a 2.5 GB network card in it, connecting point to point to the Emby server, as that already has one. The rest holds true.

I've read up on L2ARC, would it be worth upgrading the server to 48 GB, as the majority of the files are 7-9 GB only a few are 30-50 GB, HD vs UHD or would I be okay with the 32GB?

I think I'll use Core rather than Scale. If I ever need to transfer the data, I'll copy simply copy it over from the backup to the new solution.

Is there anything you would advice me based on the very simple use for the box?

Thanks

--------

A bit of a weird one, I'm not sure if TruNas is over the top.

I have 60TB of media, accessed by an dedicated physical Emby server running on Ubuntu. My existing solution is multiple servers offering SMB shares. I'm experiencing disk performace bottlenecks, but it doesn't suprise me as it just a set of JBOD arrays per server, so I'm looking at a Dell T340 with 8x 8TB disks in the bays, and using the 3 bays above (DVD etc) for expansion.

I'll use a PCI-e card and NVME to boot from.

I have a totally seperate solution for backing up, it runs like a dog, but it works fine, so I don't need any resiliance in my main storage solution. So I was thinking about a vDev stripe for speed. If it fails, I can live with the downtime, and the 4 days it takes to copy the data back !

The Emby server has about 5 simultaneous users.

I've read the note about flashing the controller here https://www.truenas.com/community/threads/dell-t340-h330-flashing.92183/ so that's one step I need to do.

I've setup TrueNas for my hoome business San, and am very impressed with it, but I'm concerned about the memory I would need for this, as I read it's 1GB per TB, so I'm looking a 64 GB, very expensive.

I was also thinking of adding a 2.5 GB network card to both the EMby and TruNas boxes and simply cabling them directly.

What guidance could you offer me.

Thanks and have a good day as they say :smile:

Julian
 
O

Okeur75

Dabbler
Joined
Nov 16, 2022
Messages
36
From my little experience, what I've been told, and my really similar setup and needs, I would advise the following :
  1. Don't go for 2.5Gb ethernet card, but go for 10G directly (I've been told)
  2. Don't go for a stripe vDev, even if you have a backup (I'm telling you). A backup is a backup and should not be something you rely on to build your primary setup. At the very least I would go for raidz1 even if it's not even recommended anymore.
    1. When you are going to lose one disk, you will put a tremendous load on your surviving disks to copy back your data from your backup. And each time you will lose one disk, you'll be good for a copy of 60TB-ish of data
    2. Also, it means you lose everything not backed-up. So it depends on the frequency of such backup.
  3. The 1GB per TB rule is not "valid" anymore when going for such disk space (I've been told).
    1. I've been recommended a 64GB RAM setup for a 120TB raw disk space + some VMs.
    2. If you are really just doing SMB sharing for Emby, maybe 32GB can be enough. I would try honestly... You can put just one 32GB stick, see how it goes and if the ZFS cache goes crazy because it's not enough, and if so, add another stick.
    3. Go ECC, really...
Edit :
Here is a good read about L2ARC which seems useless in your case : https://www.truenas.com/community/threads/use-cases-for-l2arc.72924/
You will be better with 64GB of RAM first, before thinking about an L2ARC.
Also, in our case, you won't benefit a lot from L2ARC and even the ZFS cache honestly, except if all the streamers are looking at the same movie.

Also, I would advise to divide your media dataset. First time I setup TrueNas I setup a unique dataset for the media called Emby. You lose the flexibility of the dataset when you are doing replication for example, because you have only one dataset to work on (and one set of snapshot).
I would create a "Emby" dataset, empty, with child datasets for the different media you have : TV Shows, Movies, Music, etc.
This will give you more flexibility on how to handle your data, and if you definitely want to treat all these data the same, just configure the parent dataset "Emby" and all the child will inherit from it.
 
Last edited:
J

Julianh

Dabbler
Joined
Jan 19, 2016
Messages
35
Thanks for that :smile:

I've gone ecc. How can I tell if the memory's being hammered and therefore need additional ram? Where are the stats?

What 10GB network card would you recommend?

thanks
 
O

Okeur75

Dabbler
Joined
Nov 16, 2022
Messages
36
Last edited:
Davvo

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
ZFS is known (and used) for its strong releability and data integrity: if you don't care about this you should perhaps look elsewhere.

That being said, what @Okeur75 wrote is almost entirely correct. The only "wrong" information he conveyed is the 1GB of RAM per TB of space being no longer valid: it's more like it being a rule of thumb is left purposefully vague; the fondamental truth is that ZFS loves memory, the more you have the merrier. Here we have an example of what happens when you don't have enough RAM compared to your storage.

First, I suggest you to read the following resource:

ZFS Storage Pool Layout

This amazing document, created by iXsystems in February 2022 as a "White Paper", cleanly explains how to qualify pool performance touching briefly on how ZFS stores data and presents the advantages, performance and disadvantages of each pool...
www.truenas.com www.truenas.com

Now, since you don't want to have redundancy, the cheapest and easiest way to get maximum performance is by striping your HDDs and using a single SSD as metadata vdev. You won't need to add more RAM or L2ARC, and since you don't care about your data integrity (because as you said you have backups) you won't need to give redundancy to the metadata vdev as well.
If one of your drives dies, you lose your pool; if the metadata SSD dies, you lose your pool; if one of your files get corrupted, you lose that file since there is no parity. You could somehow create parity inside the single disks using a "copies" value greater than 0, but you would lose storage efficency and it's not a panacea anyway.

If you want to run emby in this machine you will definitly want a pair of SSDs in a 2-way mirror for your apps' pool (you want redundancy here).

Also, please read the following resources regarding HBAs:

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...
www.truenas.com www.truenas.com

Don't be afraid to be SAS-sy ... a primer on basic SAS and SATA

With the introduction of SAS 12Gbps, seems like "it's time" to do a braindump on SAS. Work in progress, as usual. History By the late '90's, SCSI and PATA were the dominant technologies to attach disks. Both were parallel bus multiple drop topologies and this kind of sucked. SATA and Serial...
www.truenas.com www.truenas.com

Since you will probably want a 10Gbps network to use those speeds (since you are willing to sacrifice resiliency for speed I will get mad if you want to bottleneck your system with a 1Gbps NIC; don't use 2.5Gbps, they are not reliable enough and they would still bottleneck your pool) please read the following resource:

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...
www.truenas.com www.truenas.com

If you changed your mind and you want to have redundancy, we are gonna need to bring up a calculator and start doing math. Please read the ZFS Layout Resource if you wanna go this route.
 
Last edited:
J

Julianh

Dabbler
Joined
Jan 19, 2016
Messages
35
I've been thinking about the network speed. I'll have 8-10 sata drives. In an ideal world, they'll be running at 6 GB/s for 10 that's obviously 60GB/s
a simple 1 GB network, running point to point, as mine will be, no switch, runs at about 80-90MB real world.

So in my case I dont' think I need 2.5 or 10 GB, I'm never going to hit the limits.

Is there anythinhg I've missed?

Thanks
 
Davvo

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
There is no way for a single HDD to reach 6GigaBytes/s, max that one of them usually can do is probably 125 MB/s: take that x10 and you get 1250 MB/s, which is exactly 10 Gigabits per second.
1 Gbps = 125 MB/s.

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to...
www.truenas.com www.truenas.com
 
O

Okeur75

Dabbler
Joined
Nov 16, 2022
Messages
36
Julianh said:
So in my case I dont' think I need 2.5 or 10 GB, I'm never going to hit the limits.
Click to expand...
It depends where are your streamers (locally or through the internet).
If local, and all of them are watching 4k movies (with the right streaming device, not a chromecast at 100Mb/s) you can reach the limit of your 1Gb/s network connection.

For the cost of a 10G card I would go for it, especially between your main and backup NAS first (and between Emby and your main NAS in second). You will happy when your full backup takes 9.6h instead of 96 (theoretically...).

Take the right CAT for your ethernet cable if you want 10Gb/s too.

For the card you linked, becareful as its big sister does not seem to be supported under non-HP system : here
 
J

Julianh

Dabbler
Joined
Jan 19, 2016
Messages
35
Thank you for that Okeur, is there a cheap 10GB card you'd recommend? There dont seem to be any Chelsio cards on ebay.co.uk, I need 2, 1 for emby, 1 for truenas.

I found this, but it's a bit more than I want to spend....

www.ebay.co.uk

Intel X540-T2 Dual Port 10Gb Ethernet PCI-E RJ45 Network Adapter Card Brackets | eBay

Find many great new & used options and get the best deals for Intel X540-T2 Dual Port 10Gb Ethernet PCI-E RJ45 Network Adapter Card Brackets at the best online prices at eBay! Free delivery for many products.
www.ebay.co.uk
 
O

Okeur75

Dabbler
Joined
Nov 16, 2022
Messages
36
From the hardware guide :
nic.png


So I would say no for the 2nd card you mentioned.
All I can recommend is to go through the thread dedicated to 10Gb networking card : https://www.truenas.com/community/threads/10-gig-networking-primer.25749/

I have no personal experience with this kind of card unfortunately.
 
You must log in or register to reply here.

Similar threads

J
Dell T340 8-10 x 8TB drives
Replies
12
Views
1K
ChrisRJ
ChrisRJ
J
Part 2 - Dell T340 8-10 x 8TB drives
Replies
2
Views
532
mrpasc
M
J
Dell t340 internal sata and H330 controller
Replies
2
Views
1K
Julianh
J
J
Dell T340 Boot options
Replies
2
Views
739
Julianh
J
P
Dell T340 H330 Flashing
Replies
2
Views
4K
paret
P
Top