oneskinnydave
Cadet
- Joined
- Aug 25, 2022
- Messages
- 2
Hey All,
Long time lurker - and just started to get my ideas formulated for a new TrueNas build. I work in post production, we currently have an ~80TB (all spinning drives) server that works well - but we're starting to get 4k/UHD broadcast masters (usually ProRes 444/HQ or DNxHD HQX typically) and we typically work at full resolution because of our turnaround/project timelines (we do short form work, nothing long form). We rarely use much more than 20TB or so of space on active projects.
We have about 5-6 systems that connect direct via 10gbe to our current server and typically see 400-600mb/s throughput. With the way the world is moving towards hybrid on/off prem workflows, I think I want to build a truenas system that would have a "hot" storage pool (all nvme/ssd) for our highly active projects, and a "warm" pool - not quite nearline, but doesn't need the crazy throughput of an SSD/NVME setup - similar to what we have now. I don't need that fast of write speeds - and currently reading up on all things cache in truenas....it's really the read speeds that I'm trying to keep as high as possible on the SSD pool.
Ok - enough rambling - here is what I'm looking at for a server setup:
Mobo: Supermicro X11/12 board - deciding between a newer single CPU with the highest clock speed I can afford, or and older 2xCPU
CPU: Xeon - but still up in the air if 2 CPU is worth if - or just go for the highest clock single processer
RAM: 256-512GB - probably start with 256 and see how it handles
Drives - ~16 SSDs for the "hot" pool - ~16 HHD for the "warm" pool
Cache - 1 or 2 NVME drives to test different cache options
Network - debating between 25GB or 100GB card coming off the server - 25GB would feed a 10gbe switch - or 100GB card would feed 4x systems directly at 25gb and a 10gbe switch. It's a nominal price difference I think...but not even sure I could saturate much more than 10gbe with TrueNas anyway?
Usual workload is tv show files in the 20GB - 100GB range - camera raw files are reasonable enough and don't need high throughput - and no more than 4-5 people pulling from those files at the same time. We are starting to initiate creating proxy files of those shows which greatly reduces throughput needs - but I'm also interested in having a sync process running to some bucket storage or provider for our remote workflow. We backup onsite to a QNAP system hourly and cloud backup nightly. Our remote sync, backups, and proxy creation is all run on other systems, but it would be kind of nice to have TrueNas handle all that for me!
I'll probably be building this system and testing a bunch of difference setups before it goes into production, but didn't know if I have overlooked anything on my initial thoughts - or people have done similar setups and found some things unessecary or not - like cache sizes/speeds, vdev striping/sizes, CPU choices, etc. ZFS is a little new to me so I feel like I keep seeing conflicting information out there on what does what! I'm probably going to hire someone to help out with tweaking once I get everything in place as well just to make sure I haven't made any huge mistakes - but would love to see if there's anyone out there building or built something similar. I see a lot of lower budget builds trying to get by with the minimum - that's not what I'm after, but I also think there is a performance limit on truenas/zfs in general that investing in somethings might not be worth the cost!
Thanks for any insight anyone can offer - I appreciate it!
Long time lurker - and just started to get my ideas formulated for a new TrueNas build. I work in post production, we currently have an ~80TB (all spinning drives) server that works well - but we're starting to get 4k/UHD broadcast masters (usually ProRes 444/HQ or DNxHD HQX typically) and we typically work at full resolution because of our turnaround/project timelines (we do short form work, nothing long form). We rarely use much more than 20TB or so of space on active projects.
We have about 5-6 systems that connect direct via 10gbe to our current server and typically see 400-600mb/s throughput. With the way the world is moving towards hybrid on/off prem workflows, I think I want to build a truenas system that would have a "hot" storage pool (all nvme/ssd) for our highly active projects, and a "warm" pool - not quite nearline, but doesn't need the crazy throughput of an SSD/NVME setup - similar to what we have now. I don't need that fast of write speeds - and currently reading up on all things cache in truenas....it's really the read speeds that I'm trying to keep as high as possible on the SSD pool.
Ok - enough rambling - here is what I'm looking at for a server setup:
Mobo: Supermicro X11/12 board - deciding between a newer single CPU with the highest clock speed I can afford, or and older 2xCPU
CPU: Xeon - but still up in the air if 2 CPU is worth if - or just go for the highest clock single processer
RAM: 256-512GB - probably start with 256 and see how it handles
Drives - ~16 SSDs for the "hot" pool - ~16 HHD for the "warm" pool
Cache - 1 or 2 NVME drives to test different cache options
Network - debating between 25GB or 100GB card coming off the server - 25GB would feed a 10gbe switch - or 100GB card would feed 4x systems directly at 25gb and a 10gbe switch. It's a nominal price difference I think...but not even sure I could saturate much more than 10gbe with TrueNas anyway?
Usual workload is tv show files in the 20GB - 100GB range - camera raw files are reasonable enough and don't need high throughput - and no more than 4-5 people pulling from those files at the same time. We are starting to initiate creating proxy files of those shows which greatly reduces throughput needs - but I'm also interested in having a sync process running to some bucket storage or provider for our remote workflow. We backup onsite to a QNAP system hourly and cloud backup nightly. Our remote sync, backups, and proxy creation is all run on other systems, but it would be kind of nice to have TrueNas handle all that for me!
I'll probably be building this system and testing a bunch of difference setups before it goes into production, but didn't know if I have overlooked anything on my initial thoughts - or people have done similar setups and found some things unessecary or not - like cache sizes/speeds, vdev striping/sizes, CPU choices, etc. ZFS is a little new to me so I feel like I keep seeing conflicting information out there on what does what! I'm probably going to hire someone to help out with tweaking once I get everything in place as well just to make sure I haven't made any huge mistakes - but would love to see if there's anyone out there building or built something similar. I see a lot of lower budget builds trying to get by with the minimum - that's not what I'm after, but I also think there is a performance limit on truenas/zfs in general that investing in somethings might not be worth the cost!
Thanks for any insight anyone can offer - I appreciate it!
