Tuning for huge media center

[ericsmith881]

I've got a (somewhat) unique use case I could use some advice with. I'm building a large media center storage system for my Plex collection. It consists of 12x6TB SAS drives in a PowerVault MD1200 and 8x8TB SATA drives in a PowerVault MD1000. Both are attached to a PowerEdge R710 with 128GB RAM, 2x hex-core 2.93GHz Xeon 5670 using the non-RAID 6GB SAS HBA. The intention is to use this as an iSCSI VMware datastore for my home VMware cluster (2x Dell R620's). Everything is connected via 10-gigabit Ethernet. I have eight free 2.5" drive bays internal to the R710 I could use for SSD's if there is any benefit.

Since this is going to be a media center, it's going to do things a little differently than a typical file/print:

• All files will be large (20GB-80GB).
• Reads will dominate. Writes will only happen when I get new movies which is a couple times a month.
• Sequential performance is all that matters. Random I/O is almost non-existent.

My questions are:

1. How should I tune this for my specific use case?
2. Would adding more RAM to the R710 give meaningful benefits?
3. Is L2ARC worth messing with?
4. What about an SLOG?

 

[ericsmith881]

One more thing I forgot to ask: I'm told dedupe requires about 5GB RAM per TB of storage. For my setup this would be kinda impossible since I have 136TB of disk; the R710 won't support that much RAM even if stuffed to the gills. I also wonder if dedupe would give any benefits since everything is H.264/H.265 encoded already. Any guidance here would be appreciated.

 

[HoneyBadger]

Welcome to the forums!

I'm building a large media center storage system for my Plex collection.
...
The intention is to use this as an iSCSI VMware datastore for my home VMware cluster (2x Dell R620's).

I have a big question here for you that will shape how the whole system should be designed:

Are there other workloads beyond Plex that you're wanting to use through the VMware datastore?

Plex and other "large media file" workloads will run significantly better over an SMB connection as a typical dataset (with a large recordsize like 1M) as compared to an iSCSI/block workload to a VM. Because the two workloads are about as polar opposite as you can get, you really don't want to force one into the other (taking those large 20G media files and chopping them up into little iSCSI/VMFS-sized pieces) as it will hurt efficiency and performance.

On to your questions!

1. How should I tune this for my specific use case?
The ideal setup here is "separate pools" if you have any other non-media workload that's going on this host. Large media files do great in RAIDZ2 with large recordsizes and shared over SMB - iSCSI/block workloads on the other hand need the exact opposite as they're heavily biased towards random I/O - mirrors and small records - which are more expensive both in terms of computing resources and dollars.

2. Would adding more RAM to the R710 give meaningful benefits?
3. Is L2ARC worth messing with?
A general rule of thumb for ZFS is "more RAM never hurts" but for streaming media, unless you're going to re-watch the same episodes over and over, you're unlikely to see any benefit there as you're well over any kind of minimum with 128GB.
However, more RAM will massively help any other random I/O such as other iSCSI/VMware workloads (L2ARC as well, to a lesser degree) and the sky's the limit.

4. What about an SLOG?
No real value to your Plex store, but significant value to your iSCSI/VM workload and potentially considered a "necessity" depending on your tolerance for data loss. See the resource from @jgreco that helps explain this:
Sync writes, or: Why is my ESXi NFS so slow, and why is iSCSI faster?

5. Should I use deduplication and will it help me?
Short answer "No." - long answer "Noooooooooooooooooooo."
You're unlikely to have any duplicate data whatsoever in your media files (unless you plan to store identical copies of the same movie, and even then if they're a few bits off it will likely throw off the whole thing) so it will burn a lot of RAM for no good reason. Even for your iSCSI/block workloads it's discouraged unless you are 100% positive and have tested it out.

 

[ericsmith881]

Welcome to the forums!



I have a big question here for you that will shape how the whole system should be designed:

Are there other workloads beyond Plex that you're wanting to use through the VMware datastore?

Plex and other "large media file" workloads will run significantly better over an SMB connection as a typical dataset (with a large recordsize like 1M) as compared to an iSCSI/block workload to a VM. Because the two workloads are about as polar opposite as you can get, you really don't want to force one into the other (taking those large 20G media files and chopping them up into little iSCSI/VMFS-sized pieces) as it will hurt efficiency and performance.

On to your questions!

1. How should I tune this for my specific use case?
The ideal setup here is "separate pools" if you have any other non-media workload that's going on this host. Large media files do great in RAIDZ2 with large recordsizes and shared over SMB - iSCSI/block workloads on the other hand need the exact opposite as they're heavily biased towards random I/O - mirrors and small records - which are more expensive both in terms of computing resources and dollars.

2. Would adding more RAM to the R710 give meaningful benefits?
3. Is L2ARC worth messing with?
A general rule of thumb for ZFS is "more RAM never hurts" but for streaming media, unless you're going to re-watch the same episodes over and over, you're unlikely to see any benefit there as you're well over any kind of minimum with 128GB.
However, more RAM will massively help any other random I/O such as other iSCSI/VMware workloads (L2ARC as well, to a lesser degree) and the sky's the limit.

4. What about an SLOG?
No real value to your Plex store, but significant value to your iSCSI/VM workload and potentially considered a "necessity" depending on your tolerance for data loss. See the resource from @jgreco that helps explain this:
Sync writes, or: Why is my ESXi NFS so slow, and why is iSCSI faster?

5. Should I use deduplication and will it help me?
Short answer "No." - long answer "Noooooooooooooooooooo."
You're unlikely to have any duplicate data whatsoever in your media files (unless you plan to store identical copies of the same movie, and even then if they're a few bits off it will likely throw off the whole thing) so it will burn a lot of RAM for no good reason. Even for your iSCSI/block workloads it's discouraged unless you are 100% positive and have tested it out.

Excellent and informative response! Thank you. While there are a few other workloads on my home cluster, none of them are remotely performance sensitive. A couple of domain controllers, the big file server running Plex, a couple of Linux machines doing minor stuff like hosting a Minecraft server, and miscellaneous stuff I spin up every now and then as needed. Yeah, it's tremendous overkill but the R620's were cheap and I prefer running my home setup like the corporate environments I've built over the last 20 years. While I agree there may be some advantages to the SMB setup, I prefer the iSCSI due to its sheer flexibility. If I need to do maintenance on something, I just vMotion my stuff around, do my work, and vMotion it back.

I failed to mention have a few disks in the R620's (RAID1 SSD's) set up as DAS that I use as boot disks for all the VM's so they come up quickly. What few "regular" files I have (less than 2TB) are kept there. The big MD1200/MD1000 stuff is strictly for Plex, nothing more.

Sounds like the default setup (other than record size) is probably good as is. If you can think of anything else please let me know.

 

[jayecin]

Since this is going to be a plex server I would suggest a few things. First dont use TrueNAS. TrueNAS only supports igpu transcoding, so you wont be able to use a dedicated GPU to transcode 4k to 1080p etc. I am assuming since this is going to be a huge media center that multiple people will be accessing it from multiple devices. In that case you will most certainly be transcoding media. Since you dont have a dedicated GPU listed I am going to assume you plan on doing it on the CPU. Using both of your CPUs to 100% wont even be able to transcode a single 4k x265 movie to 1080p real time. Those CPUs just arent very powerful and you wont be able to use a dedicated GPU for transcoding. I would suggest you build plex on a linux/windows based OS and add in a dedicated GPU for 4k transcoding, otherwise you will have a very poor experience if someone tries to play a 4k movie on something like a phone, tablet, 1080p monitor/tv.