NOOB needs advice for first server build

[K1LLA_KING_KONG]

Hello TrueNAS team

Im a first time server builder and (obviously) a first time TrueNAS user
I am building this from an old PC, hopefully, the hardware is enough. Where it may be lacking I can upgrade if necessary.

Server objectives: small business media storage for video / photo editing purposes and all PC back up
Ideally using files directly from the server to Premier pro other other computers would be the goal, but I understand that may not be possible with my hardware.
I intend to upgrade the target main editing PCs with 2.5Gbe NIC

Heres my current server hardware:
Gigabyte G1.Sniper B6 https://www.gigabyte.com/id/Motherboard/G1Sniper-B6-rev-10/sp#sp
16GB RAM DDR3 (standard / non ECC)
i7 4790 (4 core) with Intel HD graphics 4600

Drives to buy:
5 or 6 x 10TB Toshiba N300
1x 1TB NVME Samsung 970 Evo

Upgrades:
2.5Gbe NIC

Upgrades to expand later:
1. LSI HBA SAS Card

Mobo details:

1 x PCI Express x16 slot, running at x16 (PCIEX16)
* For optimum performance, if only one PCI Express graphics card is to be installed, be sure to install it in the PCIEX16 slot.
(The PCIEX16 slot conforms to PCI Express 3.0 standard.)

1 x PCI Express x16 slot, running at x4 (PCIEX4)
* The M.2 socket shares bandwidth with the PCIEX4 slot. When the M.2 socket is populated, the PCIEX4 slot will operate at up to x2 mode.

2 x PCI Express x1 slots
(The PCIEX4 and PCI Express x1 slots conform to PCI Express 2.0 standard.)

2 x PCI slots

4 x SATA 6Gb/s connectors
2 x SATA 3Gb/s connectors
1 x M.2 Socket 3 connector
* Support for M.2 PCIe SSDs only.
* The M.2 socket shares bandwidth with the PCIEX4 slot. When the M.2 socket is populated, the PCIEX4 slot will operate at up to x2 mode.

With 6x Sata I can install up to 6 HDD's before needing an HBA SAS card

What I don't know about is VDEV's and how to see these up.

1. Is there an optimal amount of drives I should use? If I use 5x10TB vs 6x10TB?
2. Does the NVME act as a boot drive or a cache?
3. Do I need more than 1 SSD? If so I could add a SATA SSD

So before I go to purchase all these drives is there anything missing or I could do better?

TIA

 

[Arwen]

A few things.

You need a separate boot drive, which needs to be 16GB or larger. If buying new, I'd suggest 32GB or even 64GB. Your 1TB NVMe drive is not just over-kill, it would be a waste, (though if bootable, would work fine as a boot drive). You can't share the boot drive;

I have to waste an entire drive just for booting?

With the advent of TrueNAS Scale, we seem to be getting a large number of people from the Linux community who are not quite grasping what TrueNAS is. TrueNAS is not a "Linux distro" and it is not designed to try to meet your arbitrarily spec'd...

www.truenas.com

Some people use small SATA SSDs for boot device(s). And if running short of SATA ports, then use a USB to SATA adapter.

ZFS L2ARC, (aka Cache), drives are not as useful with low amounts of memory, like your 16GBs. Generally you want to max out your system board's memory before adding a L2ARC / Cache drive.

There are 2 basic ZFS vDev types, RAID-Zx and Mirrors. (Skipping the exotic DRAID and non-redundant single disk...). Mirrors need to be in pairs, so 4 or 6 drives. RAID-Zx can have other numbers, though no more than say 10 to 12 per vDev. And the minimum would be;

1. RAID-Z1 - 3 disks minimum, with single parity
2. RAID-Z2 - 4 disks minimum, with dual parity
3. RAID-Z3 - 6 disks minimum, with triple parity

In theory, you can have 2 disk RAID-Z1 or 5 disk RAID-Z3. But, they make little sense.

Using TrueNAS as a direct storage for editing video & photo files might be a bit too slow. Thus, some people suggest using ZFS Mirrors instead of RAID-Zx.

As for the 2.5Gbits/ps Ethernet, they are not recommended here. Jumping to 10Gbit/ps Ethernet over optical with used equipment is generally reliable and cheap.


I would suggest you read up on TrueNAS and ZFS before going any further. You've asked some of the right questions, but you might get bit by some implementation details later.

 

[2twisty]

You're going to want your video files for editing on flash, and you're going to want 10G connections to the editing PCs if you plan to operate directly off that server. I don't know what size video you're editing, but with 4k being rather common, you're going to see a slowdown in your workflow and scrubbing through videos imported into premier will not be great unless you have really fast access to the data and fast network connections.

It also depends on how many simultaneous editors you have working. The more editors the bigger the load. Just go check out Linus Tech Tips videos on what they had to do to their servers to support all their 8k video workstations. You're not likely anywhere near that level, but he is running 100G out of his servers to feed all those editors.

If you can't afford that level of speed, then it may make sense to have your editors copy the source clips to their workstations and copy the completed rendered videos back, or use proxy clips.

Mixing the 6Gb/s and 3Gb/s SATA ports with drives in the same pool will result in all the disks effectively at 3Gb/s. Definitely get that SAS card sooner than later.

I don't think you have enough RAM. I would go with 32GB minimum, preferably 64. Definitely add a cache and SLOG to improve performance if you're going to stick with disks rather than flash. You probably should also be using mirrors rather than RaidZx because you want to be able to read and write data very quickly if you are editing directly off that box.

I also think that your i7 may be a bit weak, but I would be more concerned that the motherboard you have just doesn't give you enough PCIe to really add the kinds of stuff you'd want.

 

[K1LLA_KING_KONG]

A few things.

You need a separate boot drive, which needs to be 16GB or larger. If buying new, I'd suggest 32GB or even 64GB. Your 1TB NVMe drive is not just over-kill, it would be a waste, (though if bootable, would work fine as a boot drive). You can't share the boot drive;

I have to waste an entire drive just for booting?

With the advent of TrueNAS Scale, we seem to be getting a large number of people from the Linux community who are not quite grasping what TrueNAS is. TrueNAS is not a "Linux distro" and it is not designed to try to meet your arbitrarily spec'd...

www.truenas.com

Some people use small SATA SSDs for boot device(s). And if running short of SATA ports, then use a USB to SATA adapter.

ZFS L2ARC, (aka Cache), drives are not as useful with low amounts of memory, like your 16GBs. Generally you want to max out your system board's memory before adding a L2ARC / Cache drive.

There are 2 basic ZFS vDev types, RAID-Zx and Mirrors. (Skipping the exotic DRAID and non-redundant single disk...). Mirrors need to be in pairs, so 4 or 6 drives. RAID-Zx can have other numbers, though no more than say 10 to 12 per vDev. And the minimum would be;

1. RAID-Z1 - 3 disks minimum, with single parity
2. RAID-Z2 - 4 disks minimum, with dual parity
3. RAID-Z3 - 6 disks minimum, with triple parity

In theory, you can have 2 disk RAID-Z1 or 5 disk RAID-Z3. But, they make little sense.

Using TrueNAS as a direct storage for editing video & photo files might be a bit too slow. Thus, some people suggest using ZFS Mirrors instead of RAID-Zx.

As for the 2.5Gbits/ps Ethernet, they are not recommended here. Jumping to 10Gbit/ps Ethernet over optical with used equipment is generally reliable and cheap.


I would suggest you read up on TrueNAS and ZFS before going any further. You've asked some of the right questions, but you might get bit by some implementation details later.

Thanks for your response, well the speed factor is something I would like to optimise. The reason Im leaning towards Truenas is because I'm told its faster than Unraid, which I was first researching.

For speed what OS what you recommend for maximum speed?

I will look into a 10GB network SFP set up now

 

[K1LLA_KING_KONG]

Thanks for your response.
We are mainly doing 1080p (95% of all media) and max 4k (5%)
When you say 'flash' do you mean SSD / NVME? So I would move the actively used files to an SSD, then after use move back to HDD? Or maybe I'm misunderstanding

Yes I was thing Sata ii would slow it down, can go via Pcie with a SAS card
RAM upgrade, sure thats easy.

add a cache and SLOG to

 

[2twisty]

I mean to put your video data onto SSD, preferably in a striped mirror for maximum speed.

How much video data do you have?

How big (typically) are your source video files?

How many simultaneous users will be accessing these files?

SATA SSD will likely be fine for your use. I would use a minimum of 4-6 drives in striped mirror. This is where I'd put the data that you are actively working on. For older things, you can put them on slower storage, like hard disks.

I guess you're also asking about why to use cache and SLOG?

The cache will speed up read operations of frequently used files. If you don't frequently use the same files, the cache will help but not maximally. If you do frequently use the same files, you want to maximize your RAM (since that's the first level cache) so that at least 2-3 of your most frequently used files will fit, and then add an additional SSD (or NVMe, since it's faster) as cache (2nd level) that will hold about 10x of your frequently used files (you'll have to figure out how big those are and size the RAM/Cache appropriately. Research this more here to find out the best way to determine the size of your cache -- these numbers are just off the top of my head and I am definitely NOT an expert in that area.

The SLOG will speed up your writes if you use sync writes (recommended) because you are writing the incoming data to the faster NVMe SLOG, which is then flushed to disk. The SLOG does not have to be huge, since your transaction groups are generally small. Maybe a 16 or 32G optane drive if you can find one.

You will want to read up on SLOG performance and sync writes and why they are a good idea. But the short version is this:

If you don't use sync writes, the server will tell the client that all the data has been received and is "written" even if it is still in RAM waiting to go to disk. This will improve perceived write performance, but at the possible threat of losing data if there is a power/system failure while that data is still in RAM, leading to corruption. Sync writes make sure that the data is fully written to non-volatile storage before it tells the client that the write is complete. If you're writing direct to disk (no SLOG), then you are at the mercy of the speed of the disks. If you have an SLOG that is NVMe (or Optane), it can write to that storage much faster.

Now, if you're going to put all this data onto SSD/NVMe, the SLOG will not gain you as much, since the write speed of the SSD/NVMe is so much faster than disks. But as long as your SLOG is faster than the disks it writes to, they are beneficial. Optane is faster than NVMe, NVMe is faster than SATA SSD, and SATA SSD is faster than hard disks.

Another important thing to consider if you're using an SLOG is to use a device that doesn't have a DRAM cache (very rare these days) or one that has a battery/capacitor that can supply power long enough to complete the write operation to the nonvolatile NAND in the event of a power failure. Most recommend Optane drives since they are significantly faster than even standard NVMe drives.

Intel is no longer making Optane drives, so if you want one, GET ONE. Hopefully in the future someone will license the Optane technology from Intel and resume manufacture. They are not terribly expensive right now, I found this 16GB optane for $22 on Amazon just now.

As for the TrueNAS vs UnRAID:

UnRAID is a lot easier to set up and use. However, it's not at all designed for speed, and until very recently did not support ZFS, which is TrueNAS's "killer app." (ZFS is available in many other places, but TrueNAS makes it a lot easier to set up).

TrueNAS (and ZFS) require a lot of planning to maximize performance. And believe me, it can be quite a learning curve to the uninitiated. I am still on that learning curve myself, and it's *starting* to flatten out for me, but there is still a long way to go to be as knowledgeable as some of the long time gurus here on this site.

Let me say this from very personal experience: Trust the knowledge of the gurus. (NOTE: I am NOT one) They can sometimes come off a bit prickly when they see the same question asked for the 345,765,876th time, so understand that they have gone through everything you are going through, everything I am going through and more.

Sometimes the ideal configuration for your use case isn't just some "old hardware" that you slap together. Sometimes it is. However, since you are looking for performance primarily, you need to carefully heed the wisdom of these gurus and try to avoid buying things that you will only wind up replacing later after you have "learned your lessons." Don't ask me how I know this :)

 

[K1LLA_KING_KONG]

I mean to put your video data onto SSD, preferably in a striped mirror for maximum speed.

How much video data do you have?

How big (typically) are your source video files?

How many simultaneous users will be accessing these files?

SATA SSD will likely be fine for your use. I would use a minimum of 4-6 drives in striped mirror. This is where I'd put the data that you are actively working on. For older things, you can put them on slower storage, like hard disks.

I guess you're also asking about why to use cache and SLOG?

The cache will speed up read operations of frequently used files. If you don't frequently use the same files, the cache will help but not maximally. If you do frequently use the same files, you want to maximize your RAM (since that's the first level cache) so that at least 2-3 of your most frequently used files will fit, and then add an additional SSD (or NVMe, since it's faster) as cache (2nd level) that will hold about 10x of your frequently used files (you'll have to figure out how big those are and size the RAM/Cache appropriately. Research this more here to find out the best way to determine the size of your cache -- these numbers are just off the top of my head and I am definitely NOT an expert in that area.

The SLOG will speed up your writes if you use sync writes (recommended) because you are writing the incoming data to the faster NVMe SLOG, which is then flushed to disk. The SLOG does not have to be huge, since your transaction groups are generally small. Maybe a 16 or 32G optane drive if you can find one.

You will want to read up on SLOG performance and sync writes and why they are a good idea. But the short version is this:

If you don't use sync writes, the server will tell the client that all the data has been received and is "written" even if it is still in RAM waiting to go to disk. This will improve perceived write performance, but at the possible threat of losing data if there is a power/system failure while that data is still in RAM, leading to corruption. Sync writes make sure that the data is fully written to non-volatile storage before it tells the client that the write is complete. If you're writing direct to disk (no SLOG), then you are at the mercy of the speed of the disks. If you have an SLOG that is NVMe (or Optane), it can write to that storage much faster.

Now, if you're going to put all this data onto SSD/NVMe, the SLOG will not gain you as much, since the write speed of the SSD/NVMe is so much faster than disks. But as long as your SLOG is faster than the disks it writes to, they are beneficial. Optane is faster than NVMe, NVMe is faster than SATA SSD, and SATA SSD is faster than hard disks.

Another important thing to consider if you're using an SLOG is to use a device that doesn't have a DRAM cache (very rare these days) or one that has a battery/capacitor that can supply power long enough to complete the write operation to the nonvolatile NAND in the event of a power failure. Most recommend Optane drives since they are significantly faster than even standard NVMe drives.

Intel is no longer making Optane drives, so if you want one, GET ONE. Hopefully in the future someone will license the Optane technology from Intel and resume manufacture. They are not terribly expensive right now, I found this 16GB optane for $22 on Amazon just now.

As for the TrueNAS vs UnRAID:

UnRAID is a lot easier to set up and use. However, it's not at all designed for speed, and until very recently did not support ZFS, which is TrueNAS's "killer app." (ZFS is available in many other places, but TrueNAS makes it a lot easier to set up).

TrueNAS (and ZFS) require a lot of planning to maximize performance. And believe me, it can be quite a learning curve to the uninitiated. I am still on that learning curve myself, and it's *starting* to flatten out for me, but there is still a long way to go to be as knowledgeable as some of the long time gurus here on this site.

Let me say this from very personal experience: Trust the knowledge of the gurus. (NOTE: I am NOT one) They can sometimes come off a bit prickly when they see the same question asked for the 345,765,876th time, so understand that they have gone through everything you are going through, everything I am going through and more.

Sometimes the ideal configuration for your use case isn't just some "old hardware" that you slap together. Sometimes it is. However, since you are looking for performance primarily, you need to carefully heed the wisdom of these gurus and try to avoid buying things that you will only wind up replacing later after you have "learned your lessons." Don't ask me how I know this :)

How much video data do you have? Currently only about 7TB, but plan to increase to 4k footage

How big (typically) are your source video files? 100-300MB @ 1080, will be 4x larger as we go 4k

How many simultaneous users will be accessing these files? 2


OK so i have sorted 10gb SFP+ card and a SAS HBA card, I will also increase to max RAM capacity of 32GB

Currently, I have 5x 10TB HDD's, 1x NVME 1TB, 1x SSD sata 250gb

I understand I should swap out the NVME 1TB for something much smaller as to not waste it.

5x HDD's in RAID-Z2 = approx 30TB usable storage

In my case I still have room for 3 sata SSD's. As you mention here I can create a fast pool in ZFS Mirror

A fast pool (SSD) and slow pool (HDD) whereas the actively used files can be kept in the SSD pool, and then later stored permanently in the HDD pool.

Regarding L2ARC & SLOG, the reading I've done suggests both are irrelevant to my system specs / use. (need 64GB+ RAM)

Does that all sound remotely correct?

 

[2twisty]

I'd get 1 more HDD and go with a 2x3 striped mirror. This will be much faster for those big read operations.

If you want to go with slow/fast pools, stick with the 5 drives and pick up 4x 2TB SSDs. Use NVME if you can, but that will require another card in an X16 slot that can be bifurcated. Server boards commonly have this feature, but a lot of desktop boards don't. If you can't use NVMe, go with 4 SSDs. Then put them in a 2x2 striped mirror. You'll have to find some way to cram it in the case, but SATA SSDs are small and can be mounted with sticky tape.

I would grab 2 of those Optane SSDs I linked earlier and use it for SLOG on both arrays unless you're using an all NVMe array. If you have room for more NVMe after you add your SSDs for storage, I'd add the 250 as L2ARC on whichever pool has the most frequently read files.

Having 2 pools will require that people properly put data on the fast pool when they are working on it, and to move it back to the disk pool when they are done. If it were my setup, I'd have a directory called "WIP" for Work In Progress on the flash. Then you create and do all your work on a given video in that one. When done, move it to the disk array in a folder called "Finished Work." It then becomes sort of an "archive."

If you have standard video production resources like intros, outros, mattes, lower thirds, bugs, etc, you can put them on the flash as well. However, you'd need to be careful on how your editors use them, or those links could break as you move resources around on the network. You'd have to play with it to see what I mean.

When I create a video project, I try to have all the resources for that video in the same folder that contains my video work files so that I can grab that entire folder and move it elsewhere and then open it up in my video editor and not spend 20 minutes remapping all the resources. This does result in having multiple copies of a file on the server (like intros, etc) but it makes the video project more portable, and I was OK with that, since in my case, those duplicated resources did not eat up much space in comparison to the unique resources for a given video project.

DO NOT use this as a justification for turning on deduplication. That feature eats enormous amounts of RAM and system resources, and is usually NOT a good move in most cases.

 

[K1LLA_KING_KONG]

hus, some people suggest using Z

I'd get 1 more HDD and go with a 2x3 striped mirror. This will be much faster for those big read operations.

If you want to go with slow/fast pools, stick with the 5 drives and pick up 4x 2TB SSDs. Use NVME if you can, but that will require another card in an X16 slot that can be bifurcated. Server boards commonly have this feature, but a lot of desktop boards don't. If you can't use NVMe, go with 4 SSDs. Then put them in a 2x2 striped mirror. You'll have to find some way to cram it in the case, but SATA SSDs are small and can be mounted with sticky tape.

I would grab 2 of those Optane SSDs I linked earlier and use it for SLOG on both arrays unless you're using an all NVMe array. If you have room for more NVMe after you add your SSDs for storage, I'd add the 250 as L2ARC on whichever pool has the most frequently read files.

Having 2 pools will require that people properly put data on the fast pool when they are working on it, and to move it back to the disk pool when they are done. If it were my setup, I'd have a directory called "WIP" for Work In Progress on the flash. Then you create and do all your work on a given video in that one. When done, move it to the disk array in a folder called "Finished Work." It then becomes sort of an "archive."

If you have standard video production resources like intros, outros, mattes, lower thirds, bugs, etc, you can put them on the flash as well. However, you'd need to be careful on how your editors use them, or those links could break as you move resources around on the network. You'd have to play with it to see what I mean.

When I create a video project, I try to have all the resources for that video in the same folder that contains my video work files so that I can grab that entire folder and move it elsewhere and then open it up in my video editor and not spend 20 minutes remapping all the resources. This does result in having multiple copies of a file on the server (like intros, etc) but it makes the video project more portable, and I was OK with that, since in my case, those duplicated resources did not eat up much space in comparison to the unique resources for a given video project.

DO NOT use this as a justification for turning on deduplication. That feature eats enormous amounts of RAM and system resources, and is usually NOT a good move in most cases.

Without adding a NVME expansion card. I have just 1 M.2 port, so I assume this is best kept for the boot drive. Although this shares bandwidth with the second 16x slot (which is operating at 4x) so that leaves just 2 lanes each for NVME and 1 card (SAS HBE or SPF+)
Both SAS HBA and SPF+ are recommended for 8x, so I'm already half what it should be, then with a NVME it will be 1/4 bandwidth.

I will not know the effects of this until testing, but I think without changing the mobo I better not use the NVME.

I can use 1 sata SSD as a boot drive
4x sata SSD as a fast pool
And then id have no more for L2ARC or SLOG

I can plan for this on the upgrade in the future with a more server dedicated mobo

Current board PCIE details:

EDIT:

So iv been held back by the limited option for mobos with ECC ram support and enough PCIE 8x+ slots

So iv decided to just go with regular ram and a mobo with at lease 3 slots of 8x pcie

So I can have 3 cards installed

SAS HBA, SPF+ and 4x NMVE SSD

With this combo I can have 16 sata drives, and 4x NVME SSD
I can allocate 2 sata for L2ARC or SLOG

Now I think I'm getting somewhere