I need help to configure a TrueNas system.
I am building a system for a TV station. It will only be used for a Premiere Pro media file workflow.
It will have a 220TB spinning rust pool for media files only and a 400GB SSD pool for Premiere Pro projects. I have this separate Premiere project pool to avoid constantly writing a lot of small project files to the spinning rust media file pool.
Some questions:
What would be the best recordsize for the drives in the media only pool?
What would be the best recordsize for the drives in the Premiere projects only pool?
Will the SLOG device for the ZIL do any difference for a media files only workflow?
I guess the amount of metadata will be limited and there might be close to 0 files below 128K?
Will the Special VDEV do any difference for a media files only workflow?
What is a realistic size of a VDEV for my requirements?
Will the Special L2ARC do any difference for a media files only workflow?
Is it possible make a Special VDEV for metadata ONLY - and another Special VDEV for small files only? That way I could use fastest but expensive storage for metadata and cheaper, still fast but big, for the smaller files. Lets say 8TB for files smaller than a recordsize of 1M. Here I assume 1M is a good recordsize for a media only workflow.
Is it possible to tune the transfer speed between ARC and L2ARC? It is by default 8MB/s which is a bit slow for my big media files. I would think around 100 MB/s is a better solution for me and it would ‘t affect overall system performance to bad - I think.
In the future I will have 3 VMs running on the TrueNas servers, they will do some administration and transcoding of the media files with Nvenc.
I plan to add a separate SATA SSD pool for the VMs or perhaps make a separate partition on the Premiere project pool for the VMs. What is the best solution?
With my media workflow requirements, how do I configure the system to have the best balance between L2ARC and Special VDEV. Only one can be based on Optane, which one should it be - if any?
Will the Optane DC P4800X be a better choice than 905p? DC P4800X support reprogramming name spaces, something I am not sure I need (honestly, I don’t know what it is).
A little background information - a lot actually.
The main media format will be AVC-Intra 100, a 100Mbit/second HD format.
There will be a few 4K productions, but they may very well be edited in a proxy format.
The system will be connected to about 25 editing workstations, half of them in full resolution (100 Mbit/s per video track), the rest with a 8Mbit/sec proxy resolution.
Some of the editors will work with the same material, because they do versions of the same story to different platforms.
We will ingest up to 1TB of new material a day and I estimate we don’t touch more than 4TB of media files on the NAS a day. Some files are touched every day.
Previews and temporary’s will be rendered to a local SSD in the editing workstation.
Today we have a similar workflow (except from 4K) with Avid Media Composer and 4 old Avid ISIS 7500. The network is 2x10 GigE and each workstation is connected with 1 GigE. That works perfectly - we never see studdering playback or any kind of lag.
I guess the new system can run on the same or similar network, since it is the same type of media files and workload we will use. I am planning to upgrade the network though. Probably to 2x25 Gig from TrueNas to the switches. Perhaps some of the workstations will get a faster connection to the switch, it depends on how the 4K workflow actuallly evolves. It could be a new NIC or a teamed connection.
I have read a lot about TrueNas, I feel I have good knowledge of the system, but I actually don’t know so much about filesystems, I am more a hardware guy. I don’t know enough about the workload of a media file workflow. I don’t understand if a media workflow like mine will take advantage of a SLOG device for the ZIL, L2ARC and Special VDEV.
I have focused on optimizing the “cache” options supplied by TrueNas. I have around 350 GB ARC. That should be enough to hold the video material that is beeing edited at any time. I will add 4 TB SSD L2ARC. That should be more than enough to hold all video material beiing touched in on day. I will add a SSD Special VDEV for metadata (and probably small files) so the spinning rust can concentrate on delivering high sustained throughput.
The system is being build on a SuperMicro platform.
Actual I am building 2 identical systems for redundancy - the systems will be Rsynced, so I have (most of) the video material backed up.
On a 3’rd Truenas I will store snapshots of the main system for another backup, and I have a LTO library, which will back up all new media files to tape every night.
We have an emergency power generator and a redundant UPS installed.
I feel I am covered on safety. But I also need high uptime, we are doing news production, so actually no downtime at all.
The motherboard is a x11sph-nctf with a Xeon E5-6210U 20 cores@2.5GHz.
The chassis has room for 36 3.5" drives and some 2.5" drives.
Right now it is configured as this:
Network is still 2x10GigE, but i will add faster (probably 2x25Gig) network before we take the system in production. Still under configuration.
2 SuperMicro 64GB SATA-DOMs in mirrored configuration for boot.
25 EXOS 20TB SAS drives in a 3-mirror configuration and a hot spare. This is a pool for media files only. The drives are 4Kn and Ashift -12.
It will be expanded to 33 drives, a total of 220TB in a 3-mirror configuration plus hot spares, before we go into production.
3 Intel DC S3700 400GB SATA SSD in a 3-mirror configuration. This is a pool for storing Premiere Pro project files only (not media files).
384GB RAM - for system and the ARC.
128GB Optane PC4 DDR memory module. I have 2 for mirroring, if possible. This will be used as an SLOG device for the ZIL for the pool with media files.
I need help to configure L2ARC and Special VDEV for metadata for the pool with media files.
I think around 4TB L2ARC would fit the project - it’s about 10 times the ARC. I see 4 options (the budget does unfortunately not stretch for 4 TB Optane DC P5800X - redundant):
1 Samsung M.2 NVME 990Pro 4TB (upcoming).
4 Optane 905p 960GB in a striped configuration.
1 or 2 NVME U.2 drives in a striped configuration - around 4 TB totally.
1 or 2 SAS drives in a striped configuration - around 4 TB totally…
I am not sure how big a Special VDEV for metadata would fit the project - inputs will be appreciated. The rule of thumbs says 0.3% of the pool, that is around 750 GB. The media file pool will ONLY hold media files. I see 4 options (capacity to be determined):
3 Samsung M.2 NVME 990Pro in a 3-mirror configuration.
3 Optane 905p 960GB in a 3-mirror configuration.
3 NVME U.2 drives in a 3-mirror configuration.
3 SAS drives in a 3-mirror configuration.
I only have 4 possible NVME connections - 3 on the motherboard and one 4-lane PCIe to NVME add in card.
All PCIe slots on the motherboard will be used for GPU or network except for a single 4-lane PCIe, which can be used for NVME.
So I can only add 4 NVME drives to the motherboard. I prefer the Special VDEV to be a 3-mirror configuration so it is self-healing.
That is why I also am thinking of using some SAS drives for for the Special VDEV.
Unfortunately I can’t go all-Optane.
Did I miss something? Any comments are appreciated.
Thank you.
I am building a system for a TV station. It will only be used for a Premiere Pro media file workflow.
It will have a 220TB spinning rust pool for media files only and a 400GB SSD pool for Premiere Pro projects. I have this separate Premiere project pool to avoid constantly writing a lot of small project files to the spinning rust media file pool.
Some questions:
What would be the best recordsize for the drives in the media only pool?
What would be the best recordsize for the drives in the Premiere projects only pool?
Will the SLOG device for the ZIL do any difference for a media files only workflow?
I guess the amount of metadata will be limited and there might be close to 0 files below 128K?
Will the Special VDEV do any difference for a media files only workflow?
What is a realistic size of a VDEV for my requirements?
Will the Special L2ARC do any difference for a media files only workflow?
Is it possible make a Special VDEV for metadata ONLY - and another Special VDEV for small files only? That way I could use fastest but expensive storage for metadata and cheaper, still fast but big, for the smaller files. Lets say 8TB for files smaller than a recordsize of 1M. Here I assume 1M is a good recordsize for a media only workflow.
Is it possible to tune the transfer speed between ARC and L2ARC? It is by default 8MB/s which is a bit slow for my big media files. I would think around 100 MB/s is a better solution for me and it would ‘t affect overall system performance to bad - I think.
In the future I will have 3 VMs running on the TrueNas servers, they will do some administration and transcoding of the media files with Nvenc.
I plan to add a separate SATA SSD pool for the VMs or perhaps make a separate partition on the Premiere project pool for the VMs. What is the best solution?
With my media workflow requirements, how do I configure the system to have the best balance between L2ARC and Special VDEV. Only one can be based on Optane, which one should it be - if any?
Will the Optane DC P4800X be a better choice than 905p? DC P4800X support reprogramming name spaces, something I am not sure I need (honestly, I don’t know what it is).
A little background information - a lot actually.
The main media format will be AVC-Intra 100, a 100Mbit/second HD format.
There will be a few 4K productions, but they may very well be edited in a proxy format.
The system will be connected to about 25 editing workstations, half of them in full resolution (100 Mbit/s per video track), the rest with a 8Mbit/sec proxy resolution.
Some of the editors will work with the same material, because they do versions of the same story to different platforms.
We will ingest up to 1TB of new material a day and I estimate we don’t touch more than 4TB of media files on the NAS a day. Some files are touched every day.
Previews and temporary’s will be rendered to a local SSD in the editing workstation.
Today we have a similar workflow (except from 4K) with Avid Media Composer and 4 old Avid ISIS 7500. The network is 2x10 GigE and each workstation is connected with 1 GigE. That works perfectly - we never see studdering playback or any kind of lag.
I guess the new system can run on the same or similar network, since it is the same type of media files and workload we will use. I am planning to upgrade the network though. Probably to 2x25 Gig from TrueNas to the switches. Perhaps some of the workstations will get a faster connection to the switch, it depends on how the 4K workflow actuallly evolves. It could be a new NIC or a teamed connection.
I have read a lot about TrueNas, I feel I have good knowledge of the system, but I actually don’t know so much about filesystems, I am more a hardware guy. I don’t know enough about the workload of a media file workflow. I don’t understand if a media workflow like mine will take advantage of a SLOG device for the ZIL, L2ARC and Special VDEV.
I have focused on optimizing the “cache” options supplied by TrueNas. I have around 350 GB ARC. That should be enough to hold the video material that is beeing edited at any time. I will add 4 TB SSD L2ARC. That should be more than enough to hold all video material beiing touched in on day. I will add a SSD Special VDEV for metadata (and probably small files) so the spinning rust can concentrate on delivering high sustained throughput.
The system is being build on a SuperMicro platform.
Actual I am building 2 identical systems for redundancy - the systems will be Rsynced, so I have (most of) the video material backed up.
On a 3’rd Truenas I will store snapshots of the main system for another backup, and I have a LTO library, which will back up all new media files to tape every night.
We have an emergency power generator and a redundant UPS installed.
I feel I am covered on safety. But I also need high uptime, we are doing news production, so actually no downtime at all.
The motherboard is a x11sph-nctf with a Xeon E5-6210U 20 cores@2.5GHz.
The chassis has room for 36 3.5" drives and some 2.5" drives.
Right now it is configured as this:
Network is still 2x10GigE, but i will add faster (probably 2x25Gig) network before we take the system in production. Still under configuration.
2 SuperMicro 64GB SATA-DOMs in mirrored configuration for boot.
25 EXOS 20TB SAS drives in a 3-mirror configuration and a hot spare. This is a pool for media files only. The drives are 4Kn and Ashift -12.
It will be expanded to 33 drives, a total of 220TB in a 3-mirror configuration plus hot spares, before we go into production.
3 Intel DC S3700 400GB SATA SSD in a 3-mirror configuration. This is a pool for storing Premiere Pro project files only (not media files).
384GB RAM - for system and the ARC.
128GB Optane PC4 DDR memory module. I have 2 for mirroring, if possible. This will be used as an SLOG device for the ZIL for the pool with media files.
I need help to configure L2ARC and Special VDEV for metadata for the pool with media files.
I think around 4TB L2ARC would fit the project - it’s about 10 times the ARC. I see 4 options (the budget does unfortunately not stretch for 4 TB Optane DC P5800X - redundant):
1 Samsung M.2 NVME 990Pro 4TB (upcoming).
4 Optane 905p 960GB in a striped configuration.
1 or 2 NVME U.2 drives in a striped configuration - around 4 TB totally.
1 or 2 SAS drives in a striped configuration - around 4 TB totally…
I am not sure how big a Special VDEV for metadata would fit the project - inputs will be appreciated. The rule of thumbs says 0.3% of the pool, that is around 750 GB. The media file pool will ONLY hold media files. I see 4 options (capacity to be determined):
3 Samsung M.2 NVME 990Pro in a 3-mirror configuration.
3 Optane 905p 960GB in a 3-mirror configuration.
3 NVME U.2 drives in a 3-mirror configuration.
3 SAS drives in a 3-mirror configuration.
I only have 4 possible NVME connections - 3 on the motherboard and one 4-lane PCIe to NVME add in card.
All PCIe slots on the motherboard will be used for GPU or network except for a single 4-lane PCIe, which can be used for NVME.
So I can only add 4 NVME drives to the motherboard. I prefer the Special VDEV to be a 3-mirror configuration so it is self-healing.
That is why I also am thinking of using some SAS drives for for the Special VDEV.
Unfortunately I can’t go all-Optane.
Did I miss something? Any comments are appreciated.
Thank you.
