Mini R - Add On Card ??

[StarTrek133]

Hi Everyone,

Some time ago I got a Truenas Mini R and it has a pci express expansion slot .. My thought was to get a M2 card for it and put some M2 Drives in to use as a cache ..

So wanted the Community's option on if this is a good idea or bad idea and also what add on card should I use ..

Thanks for the help and advice ...

 

[HoneyBadger]

Hey @StarTrek133

I don't believe the PCIe slot on the R supports bifurcation, so you'll only be able to use a single M.2 device unless you purchase an adapter that has a PCIe switch in it (append "PLX" to your search terms and it may help you find one, but figure on them being around the USD$150 range.) It's also a PCIe x4 3.0 slot so each of the devices would be limited to x2 (about 2GB/s)

Were you thinking of L2ARC (read cache) or SLOG (write "cache", but isn't actually)?

 

[StarTrek133]

Hey @StarTrek133

I don't believe the PCIe slot on the R supports bifurcation, so you'll only be able to use a single M.2 device unless you purchase an adapter that has a PCIe switch in it (append "PLX" to your search terms and it may help you find one, but figure on them being around the USD$150 range.) It's also a PCIe x4 3.0 slot so each of the devices would be limited to x2 (about 2GB/s)

Were you thinking of L2ARC (read cache) or SLOG (write "cache", but isn't actually)?

I only figured I would need one anyways .. seeing how 1tb M2's are cheap enough one would be more than enough ...

As for L2ARC or SLOG , I honestly don't know I figured when I installed the software it would pick for me ... I am new to Truenas so still learning ...

If it helps all the Mini R is going to do is be a file server .. its going to house my videos, movies, anime and such ..

 

[danb35]

If it helps all the Mini R is going to do is be a file server .. its going to house my videos, movies, anime and such ..

What makes you think you'd benefit from either sort of cache in the first place? For what you're describing, SLOG won't do anything for you--that's only used for sync writes, which aren't used with normal file-sharing applications. So that leaves L2ARC. Generally that isn't going to be useful without at least 64 GB of RAM, but even then, do you need it? The Reporting screen in the UI will give some information, and if your ARC hit ratio is north of 90% there really isn't any reason to do that.

I went through something similar a few months ago: I changed the motherboard in my NAS to one that had two m.2 NVMe slots on it, and was wondering what to do with them. The best answer I found was "nothing":

What to do with two m.2 slots?

I recently upgraded the motherboard in my NAS, mainly to get a usable IPMI experience, but also because my previous X9 board was getting kind of long in the tooth. The new board is running fine, so woohoo. But it has two onboard m.2 slots that I'm not using, and I'm wondering what I might do...

www.truenas.com

I might yet set up a pool for my apps there, but haven't been especially motivated to, since the current SATA SSDs are working pretty well.

 

[StarTrek133]

What makes you think you'd benefit from either sort of cache in the first place? For what you're describing, SLOG won't do anything for you--that's only used for sync writes, which aren't used with normal file-sharing applications. So that leaves L2ARC. Generally that isn't going to be useful without at least 64 GB of RAM, but even then, do you need it? The Reporting screen in the UI will give some information, and if your ARC hit ratio is north of 90% there really isn't any reason to do that.

I went through something similar a few months ago: I changed the motherboard in my NAS to one that had two m.2 NVMe slots on it, and was wondering what to do with them. The best answer I found was "nothing":

What to do with two m.2 slots?

I recently upgraded the motherboard in my NAS, mainly to get a usable IPMI experience, but also because my previous X9 board was getting kind of long in the tooth. The new board is running fine, so woohoo. But it has two onboard m.2 slots that I'm not using, and I'm wondering what I might do...

www.truenas.com

I might yet set up a pool for my apps there, but haven't been especially motivated to, since the current SATA SSDs are working pretty well.

Thanks for the info Dabd35 ...

The Mini R does have 64gb of ram already .. I have always built my network and gear to be a little over kill because I would rather spend a little bit more now and have it for a long time then have to redo everything every 2 years or so ..

Here is a pic of the Arc screen from my current TrueNas box ... if I am reading it right , from the first graph to me it looks like have a cache drive might be worth it ...

But as I am new to all this I am not sure ... I would rather better safe and then sorry ... Plus with the cost of a card and an M.2 drive being as cheap as they are I am not out a of money if it doesn't really help or only helps out a little ..

 

[mb17]

Please don't take this as "expert" advice but I will do my best.

Every workload is different and planning ahead for every scenario can be costly. If you want the short answer, try without and see if you even need it.

From your screenshots, your ARC memory is still filling up from a recent reboot. The longer your system is online and being used, the more that will be stored in your ARC.

Also note that ARC will always have dips. I can speak from experience of other SAN vendors too. Any sort of read cache/index/ARC is going to store information from the most active data that is being used. (Every storage vendor has a different logic to how they do this.) Until you have your unit fully loaded and running for days, you can only speculate how well the 64gb is going to work for you.

My recommendation for you is to let it run for at least a few weeks. Unless you are feeling a performance issue, just let it keep going for these few weeks. Then go in and look at your ARC. First thing look at the averages over more than a few minutes. Take at least 1 hour and look at that for the average. If you are >90% hit ratio average, L2ARC isn't probably for you. Me personally, for home use data, I would go as far as saying until you get below 80-85% it isn't worth it. I also ignore the short term dips because no mater what, you will always have these. Even if you have 1tb RAM, if you put enough stored data behind it, something will always cause it to dip if the data being accessed is old and large.

Now if you are doing things like VM storage, then 90%+ is a must and you want to be looking at averaged in both the 1hr and 1day ranges. I have an aging SAN at work from another vendor. It is a hybrid setup with 42x 1tb disks and 9 SSDs (I forget the size). It was getting to about 88% cache hit ration and we were filling it in performance. I was already getting a new SAN in place at this time so I moved a few heavy servers off and the percentage got up to 91-92% and the servers felt as if I doubled their speed. So that 90% ration can be felt. But again, this is for VM workload and not just file access.

 

[StarTrek133]

Please don't take this as "expert" advice but I will do my best.

Every workload is different and planning ahead for every scenario can be costly. If you want the short answer, try without and see if you even need it.

From your screenshots, your ARC memory is still filling up from a recent reboot. The longer your system is online and being used, the more that will be stored in your ARC.

Also note that ARC will always have dips. I can speak from experience of other SAN vendors too. Any sort of read cache/index/ARC is going to store information from the most active data that is being used. (Every storage vendor has a different logic to how they do this.) Until you have your unit fully loaded and running for days, you can only speculate how well the 64gb is going to work for you.

My recommendation for you is to let it run for at least a few weeks. Unless you are feeling a performance issue, just let it keep going for these few weeks. Then go in and look at your ARC. First thing look at the averages over more than a few minutes. Take at least 1 hour and look at that for the average. If you are >90% hit ratio average, L2ARC isn't probably for you. Me personally, for home use data, I would go as far as saying until you get below 80-85% it isn't worth it. I also ignore the short term dips because no mater what, you will always have these. Even if you have 1tb RAM, if you put enough stored data behind it, something will always cause it to dip if the data being accessed is old and large.

Now if you are doing things like VM storage, then 90%+ is a must and you want to be looking at averaged in both the 1hr and 1day ranges. I have an aging SAN at work from another vendor. It is a hybrid setup with 42x 1tb disks and 9 SSDs (I forget the size). It was getting to about 88% cache hit ration and we were filling it in performance. I was already getting a new SAN in place at this time so I moved a few heavy servers off and the percentage got up to 91-92% and the servers felt as if I doubled their speed. So that 90% ration can be felt. But again, this is for VM workload and not just file access.

Thanks for the advice MB17 ..

yeah I am not sure seeing how I am still new to all this ... my goal for the mini r is just use at as a NAS / File Share .. that's it , everything else Plex , Shoko , and such will be running on another windows systems and just accessing the data on the mini r with smb share ..

Only thing I plan on setting up on the mini r is backups to my Synology ..

Also I don't leave my systems running all the time , cant afford the power bill if I do that ... seeing how this is just running out of my condo ... so everything auto shutdowns at 230am and gets turned back on when I start work around 9am ..

I was just trying to plan all this out before setting up the mini r ... that way I wouldn't have to redo everything later on if I do find out that i need the cache drive ..

So .....

 

[mb17]

Someone that knows more than me would have to confirm this thought.

If the L2ARC is persistent between boot cycles then it might work for your scenario.

One other thought I have for you is drive spin down. Not sure how TrueNAS Core/Scale handles this one but it might be a better option if you want to maintain a running system. The board in this server is fairly low power. I would guess in the 15-30w range at idle if the HDD's were spun down. Might be worth looking into. I personally would rather spend the extra money to not have to always be turning my systems on and off at home. Then again I do not know how your electricity costs compared to mine.

Your signature says 6 WD Red Pro drives which spec sheet shows 3.6w idle. Running this through an online calculator, at idle the total cost for the 6 HDD's to run at idle is about 29$ USD if your electricity costs are $0.15 per kWh. Might be cheaper to keep it running and utilize the ARC cache than buy SSD's and PCIE board so you can turn it off every night.

Plus keep in mind that you still need to let the ZFS processes run for drive health so you need to do regular SMART scans and scrubs.

 

[StarTrek133]

Someone that knows more than me would have to confirm this thought.

If the L2ARC is persistent between boot cycles then it might work for your scenario.

One other thought I have for you is drive spin down. Not sure how TrueNAS Core/Scale handles this one but it might be a better option if you want to maintain a running system. The board in this server is fairly low power. I would guess in the 15-30w range at idle if the HDD's were spun down. Might be worth looking into. I personally would rather spend the extra money to not have to always be turning my systems on and off at home. Then again I do not know how your electricity costs compared to mine.

Your signature says 6 WD Red Pro drives which spec sheet shows 3.6w idle. Running this through an online calculator, at idle the total cost for the 6 HDD's to run at idle is about 29$ USD if your electricity costs are $0.15 per kWh. Might be cheaper to keep it running and utilize the ARC cache than buy SSD's and PCIE board so you can turn it off every night.

Plus keep in mind that you still need to let the ZFS processes run for drive health so you need to do regular SMART scans and scrubs.

Hey MB17,

My setup in my signature is what I am currently running .. I have not switched over to the mini r yet ... I am having backup issues which I am currently working on fixing .. Have to get good backups before I go to switching over the mini r ..

When I move to the mini r , that will have 12 x 6tb hard drives and I was told by someone at Truenas that I should do zfs3 which will give me up to 3 hard drives lost before loosing any data ...

So while I am working on my backup issue , I am trying to figured out if I need anymore parts for the mini r before getting it ready to be deployed ..