Will lots of RAM help w/ this workload?

[Justin Aggus]

I have the following workload:

files are constantly being written,
80% of files are never read
19% of files are read during the first month after being written
-------of this 19%:
-------90% of files are read only once
-------10% of files are read more than once
1% of files are read after the first month
-------of this 1%:
-------90% of files are read only once
-------10% of files are read more than once

What would get cached in the arc/l2arc with a workload like this?
I could easily have enough arc to cache 1 month worth of files, but do files have to be read once before they are cached?

It seems to me, depending on the caching algorithm, my cache hit ratio could be anywhere from ~2% to ~95%.

 

[anodos]

More RAM will increase the size of ZFS transaction groups, which may dramatically increase write performance (depending on workload). Maybe give more specifics about what exactly you're trying to accomplish and the hardware you're wanting to use?

 

[Justin Aggus]

I should have clarified, I'm only concerned with read performance.
File sizes vary, but on average fairly small, about 100kb.

Can any reads come from ram, if files are only read once?

EDIT:
I plan on using CIFS,
I could use ISCSI/NFS to a VM to SMB, but that seems like a waste.

 

[anodos]

I should have clarified, I'm only concerned with read performance.
File sizes vary, but on average fairly small, about 100kb.

Can any reads come from ram, if files are only read once?

EDIT:
I plan on using CIFS,
I could use ISCSI/NFS to a VM to SMB, but that seems like a waste.

How many files are we talking about? How many users? Gigabit networking? What CPU? What size of zpool? How many vdevs and what type (mirror, raidz2, etc)

 

[Justin Aggus]

~200 million files, 20tb to start
2 million file writes per month
500gb writes per month.
nothing gets deleted ot changed.

If zfs will cache files after writing them then something like z2 with a 1tb l2arc

if zfs won't cache them, then just r10.

I'll probably put a 10gb nic to help with backups.

Current gen xeon quad core.

 

[anodos]

Extra ram will help with SMB performance even if you're not opening files. Most SMB clients like to walk the file tree and read file metadata. Just use dtrace on an smbd process. When you scale this across millions of files, you can see a significant performance hit as this starts to consume IOPs. Look into optimizing samba for small file performance.

 

[depasseg]

then something like z2 with a 1tb l2arc

This will help your problem, but will also require a lot of RAM. IIRC, there is a 1:5 or 1:8 RAM:L2ARC requirement.

 

[Ericloewe]

128GB of RAM for that much L2ARC would be pushing it. I'd probably gravitate towards 256GB of RAM.

 

[Stux]

ARC will at a minimum cache the metadata, which sounds like it would be worthwhile.

 

[MatthewSteinhoff]

Will lots of RAM help w/ this workload?

Easy question! Of course, yes.

Now ask the more important, harder to answer question: Is lots of RAM required to meet the performance requirements and stay within budget?

At 200 million files, I'm going to assume you've already figured out best-practice directory structures and don't have an insane number of files in any one directory that no amount of engineering could make reasonably fast. If you do have especially bloated directories, that's one area will additional RAM will greatly help by caching metadata.

I'd start out with the rule-of-thumb amount of RAM with no L2ARC then see if you like the performance. If so, great, done. If not, look for bottlenecks, add RAM and check the performance. Adding a L2ARC is the last thing I'd do since it sounds as though files are rarely read and those that are aren't likely to be cached anyway.

We have a 20TB pool - 2 x 6 x 3TB RAIDZ2 - that stores our security video feed: 25 cameras each dumping 5MP of video around the clock; about 750GB of writes a DAY though only 20,000 files. In a small corner of that pool, we also have rarely accessed archive data.

This pool seems to most closely match your useage case though with far fewer files. We don't have a SLOG or L2ARC on this pool and the entire server has just 96GB of RAM. Performance is great: no meaningful delays when opening directories in the archive with a lot (10,000) of files. Security video plays without buffering and pulling files out of the archive happens just as fast as from other pools (details in signature).

Cheers,
Matt

 

[Justin Aggus]

I don't know if it's best practice, but I definitely wouldn't call it insane:

Root folder has about 20,000 folders,
One level down each folder has anywhere from 1-2000 folders (over 1000 is typical)
one more level down each folder has 1-1000 files (Most only have ~10 files)

Files are being accessed by a program directly 99% of the time, (No directory browsing)

I'm currently using a Synology NAS, running BTRFS, performance is ok when using the application.
Trying to open the root directory from windows explorer takes forever. - But we don't usually do that.

The big problem with the Synology is backups.
Before BTRFS backups were impossible (took about 1 month)
After upgrading to BTRFS, backups via snapshot replication were possible, (NAS would have to run at 100% for about 4 hours per night to backup 15gb worth of data)

However, occasionally the snapshot replication would fail, and the NAS would get stuck at 100% usage until rebooted - not fun.

Recently, the synology snapshot replication always fails, and I need to do another 1-month full backup,
so I decided to try some alternatives instead. (FreeNAS)

 

[MatthewSteinhoff]

(NAS would have to run at 100% for about 4 hours per night to backup 15gb worth of data)

I think you'll be very happy with FreeNAS if that's where you're coming from.

Creating and sending a 27G snapshot from our primary server to the replicated server takes under five minutes. That's actually moving 27G across the wire from server to server. The lower-powered, receiving server hits 20% CPU utilization during the reception. The sending server tops out at 4% CPU utilization.

Cheers,
Matt