Storage for small Renderfarm

[TobiT92]

Hi everyone,

I did spent the last few days on reading on the whole FreeNAS topic and still am very overwhelmed as I am new to many things. I decided to create a thread to get help from experienced people to get a hint in the right directions, and what is best for my specific needs.

(I am willing to read on all the necessary things by myself, of course, I'd just love to know what exactly is good for my needs before I read several other guides that cover things that are not important to me at all, like 24/7 uptime for example)


So, let's begin.

Background:
I am a 3D Animator (Motion Designer) and have a small render Farm of old workstations that can render an animation simultaneously and therefor speed up render times. However, depending on the project, the project files and cached animation data can be as large as 500GB, with individual files as big as 10GB and need to be accessed by up to 4 render nodes at a time.


What do I need this thing to do:

I need 2 Volumes (not sure if Volume is the right term):
first Volume: Big Space available (8-16TB). Provide fast read access to large (several GB) cache files for up to 4 machines over network. Speed is king here. Security is NOT a priority here, as the cached data can be recreated easily, so Parity can be removed for performance.

Second Volume: Host small files like textures and project files, needs Parity to prevent data loss. Speed is not a top priority, shouldn't be less than 1Gbit/s for single access though, more is always welcome of course. Volume should be around 2-4TB of usable space.

The 2 Volumes will be linked as Network Drives in Windows 7 on each of the render machines and shall act like normal local harddrives.


What do I NOT need:

-24/7 uptime guarantee. The files are only accessed when rendered or when I actively work on the second Volume. The NAS will be shut down when it is not needed.
-Any form of security for the big cache drive
-Anything fancy that goes up and beyond my needs stated above.



So hopefully this gives a good idea of what I am looking for.


Hardware:

• Base:

I have 2 workstations currently unused, I'd like to use one of those for the NAS if possible:

Option 1:
HP Z420 workstation
Mainboard: HP Z420 specific mainboard
CPU: (1x) Xeon E5-1620 - i.e. 4 Cores 8 Threads @3,6Ghz

Option 2:
HP z800 workstation
Mainboard: HP Z800 specific mainboard
CPU: 2x Xeon X5670 - i.e. 12 Cores 24 Threads @2,96Ghz

• RAM:

DDR 3 ECC Ram - as much as needed (z420 tops out at 64GB, Z800 at 192GB)

• HDDs for Storage:

Bunch of WD Red 2TB/4TB drives

• OS Drive:

either USB (Z800 only 2.0) or small SSD

• NIC:

2x DualPort 10Gbit SFP+, resulting in 4 SFP+ ports, directly attached to the 4 render nodes. Not sure which NICs to get, they are hard to find on ebay Germany...




Questions I have:

1. Is FreeNAS the right option for my use case?
2. Which one of the workstations shall I use, or are neither of those enough for my needs?
3. Are any other hardware picks a bad fit?
4. Any other hardware I need (SSDs for Cache?)
5. What kind of Volumes shall the two Volumes be? For the big fast Volume I thought about Raid0 (or the equivalent on FreeNAS), for the second Volume I thought about Raid5 or 6, maybe Raid1.
6. Is it probably better to create one big volume that has both parity and speed for all my needs?

Again, I'm happy if you just point me in the right direction of what terms I shall google and will read on my own...


Thanks so much!
Tobi

 

[BigDave]

I'm happy if you just point me in the right direction of what terms I shall google and will read on my own...

https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/

http://forums.freenas.org/threads/slideshow-explaining-vdev-zpool-zil-and-l2arc-for-noobs.7775/

https://forums.freenas.org/index.php?threads/zfs-primer.38927/

 

[Dice]

Welcome to the Forums.

1. Most definitely.
2. Both WS CPU's should do the trick. The 1620 is probably the more powerful among them. RAM is the deciding factor to me. The more the merrier. FreeNAS is a cookie monster extraordinare when it comes to RAM. More ram also enables you to use a larger L2ARC cache drive.
3. Do some searches on different candidates for your 10Gbit solution, you'll find some good to read threads. I believe Chelsio is often used.
4. Once you've enough RAM, I believe your workload can benefit well from a L2ARC cache disk. Search for threads discussing sizing of L2ARC in relation to RAM. Aim for maxing out RAM prior to getting L2ARC.
5. The equivalent to Raid 0 in FreeNAS terminology is <stripes>. Since your declaration of non-interest in data integrity, you are ready. For the second volume, since you do not require a lot of space you could setup either a 4x drive raidz2 pool, or 2 vdevs of mirrors forming a pool. More intel on those decisions are to be found in the links @BigDave provided.
6. You could definitely run numerous pairs of mirrors in a single pool. That would cover both aspects but require more hardware. Remember, speed improves by adding additional vdevs.

Cheers.

 

[TobiT92]

Thank you very much to both of you.

I've read through all the 3 links BigDave provided and am certainly more educated now.

Those guides and threads gave me a very good understanding of what is "good" and what is "bad", but still leave me a bit puzzled about how much more improvement what measurements will do.

So, for example, the RAM question. I know, the more the merrier, but at what point does more Ram not equal in more speed?
Are there any guides that give me a good understanding of what exactly are the depending factors here?

For example, things that come to my mind:
4 clients are requesting the same file that is 10GB large, does that mean that more RAM than 10GB (plus the 8GB for the FreeNAS system) will not make any difference in this specific case?
4 Clients are requesting 4 individual files, each 10GB, does that mean that more RAM than 40GB (plus system used RAM) will not make any difference?
4 Clients are requesting many files, summing up to a total of 100GB. Does now the speed increase up to 100GB RAM even if those 100GB are not requested at the same time?



The dual Xeon machine would be a total waste of computing power for just providing data, but provides more RAM. If the difference of 64GB vs 192GB is not dramatic in a real world scenario like mine, I'd rather use the dual Xeon workstation as rendernode.


@Dice

Thank you so much for answering my questions.
I'm looking for 2x Chelsio S320E, but unfortunately they are not as common here as in the US.

I think that your suggestion of using one pool of several mirrors will be the best use case for me, as read speeds are much more important than write speeds in my case. And with striped mirrors, read speeds of all disks can be added together (in theory), right?

Thanks!
Tobi

 

[Dice]

the RAM question. I know, the more the merrier, but at what point does more Ram not equal in more speed?

Difficult to say. From my experience, it is not a linear function. Rather, the system works pretty fine, until it suddenly tanks. For an idle system, a lot less RAM is required. Moving a couple small files around, requires minimally more. However, when the load becomes heavier ...the mileage may vary quite a bit. Ie, the system may work decently for your sized files, ...but may crumble if you hit the box multiple times etc.. It is difficult to predict.
The ZFS motto pretty much is - throw more cheap hardware at the problem, and you'll still be considerably better off vs commercial hardware solutions.

The dual Xeon machine would be a total waste of computing power for just providing data, but provides more RAM. If the difference of 64GB vs 192GB is not dramatic in a real world scenario like mine, I'd rather use the dual Xeon workstation as rendernode.

For sure, FreeNAS requires a boatload of RAM, and rather limited CPU power.
I'll be a little conservative in my predictions regarding CPU usage when using multiple clients and 10Gbe network.......

In my testing heavy of usage as 2 clients over SMB, combined with the load of running scrubs, I worst case scenario break 15% load on my i3-6100 CPU. IIRC, the system could completely stall out on 48GB of RAM though (that's when I regressed from X11 and got myself an X8 dual dual 5630 xenon box, only because it was the cheapest way to get +144GB of RAM in one box. Completely disregarded all aspects of computational power).

And with striped mirrors, read speeds of all disks can be added together (in theory), right?

Yes.