Drive upgrades or server upgrade?

[nattan]

Hello all,

Usage:
I am currently running FreeNAS-11.0-U2 (e417d8aa5) on a embedded CPU C2750 with 32GB eec ram and (6) 3TB seagates in raid-z2 storing mostly high quality video copies from blurays or dvds. Currently running into the 80% storage issue and hitting major walls when playing back in plex. Currently I am not sure if the video trans-codes are being throttled due to disk lag or just the fact that its a low power atom board. Only own a few 4k movies but they play back fairly well without buffering most of the time, rest is 720/1080p content.

Questions:
-Will just upgrading the drives to a higher capacity help? or is the 80% just kind of a warning and not really a performance hit?
-If I did decide to upgrade, would it be better to replace the disks 1 by one or just make a new raid-z2 and copy the data over? Last time I checked the current raid-z2 is about 15% fragmented
-Would it be better to just upgrade the whole server? I am thinking 8TB x6 so id need at least 48GB of ram unless it only factors in usable storage space ( unsure)... I would prefer to keep the server costs lower, but if I need to upgrade to have a smoother experience I would do that also.

 

[Chris Moore]

A faster processor might be able to help you out but it depends on what you see in the utilization (CPU load) when you are playing video. My system only gets above 20% when something needs a lot of transcode.
New drives will be faster because of the new technology in the drive.
You might want to try this and see if is better after. The system board and CPU can always be changed later.
I did, at different times, both options. It is faster, in my opinion, to do one drive at a time. I did one a day for a week.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[nattan]

A faster processor might be able to help you out but it depends on what you see in the utilization (CPU load) when you are playing video. My system only gets above 20% when something needs a lot of transcode.
New drives will be faster because of the new technology in the drive.
You might want to try this and see if is better after. The system board and CPU can always be changed later.
I did, at different times, both options. It is faster, in my opinion, to do one drive at a time. I did one a day for a week.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

I looked back at my usage and the only thing that seemed high was the pending I/O requests.

Usage during transcode

Similar results over 6 drives just took ada0 for example:

Edit:
It also looks like ram gets very tight when trans-coding and my arc hit is currently around 65%

So new question: Should I worry about Arc hit since my drive can more than saturate the 1gbps network?

 

[Chris Moore]

It also looks like ram gets very tight when trans-coding and my arc hit is currently around 65%
So new question: Should I worry about Arc hit since my drive can more than saturate the 1gbps network?

I think that your memory and CPU utilization look fine. If I were in your position (80% pool capacity) I would focus on installing larger drives unless your current build will support adding another 6 drive vdev to the pool...
Several years ago, when I ran up against the capacity limit, I added another 6 drives so that I am now using 12 drives in two vdevs of 6 drives each. If you don't have room for the additional drives, you would likely be just as well served to switch to higher capacity drives. You mentioned 8 TB drives, but right now it looks like the best cost to TB ratio is with 6 TB drives. It is down to what you need for the next few years. How quickly is your data growing and how much money do you want to invest in the storage.
Where I work, we are looking at buying a new server with 10 TB drives even though the latest thing is 12 TB and 14 TB will probably be on the market by the time the procurement package gets through the system. Sometimes the latest technology is just needlessly expensive.

 

[Chris Moore]

I am thinking 8TB x6 so id need at least 48GB of ram unless it only factors in usable storage space ( unsure)...

You should be fine with the 32GB that you already have. When you get a new system board (at some future date) you can get more memory then. Memory cost is crazy right now.

 

[nattan]

I think that your memory and CPU utilization look fine. If I were in your position (80% pool capacity) I would focus on installing larger drives unless your current build will support adding another 6 drive vdev to the pool...
Several years ago, when I ran up against the capacity limit, I added another 6 drives so that I am now using 12 drives in two vdevs of 6 drives each. If you don't have room for the additional drives, you would likely be just as well served to switch to higher capacity drives. You mentioned 8 TB drives, but right now it looks like the best cost to TB ratio is with 6 TB drives. It is down to what you need for the next few years. How quickly is your data growing and how much money do you want to invest in the storage.
Where I work, we are looking at buying a new server with 10 TB drives even though the latest thing is 12 TB and 14 TB will probably be on the market by the time the procurement package gets through the system. Sometimes the latest technology is just needlessly expensive.

I dont mind spending money on storage, since most of the stuff is replaceable I am considering going back to raid-z1, while its a pain in the butt to slowly copy all the media again it is not out of the question to do so if i loose it.
if i end up replacing the drives with 6x 6/8/10TB drives which ever is cheapest at time of ordering... I worry that the 32GB of ram wont be enough.

Currently based off the few vendors I trust for storage: it looks like the cheapest 10TB drive is $300
https://pcpartpicker.com/products/i...ort=ppgb&S=3000000,12000000&t=7200&m=19,34,38


as far as how fast my data grows, it varies but it took me around 3-4 years to get to 8TB consistently used. I try to keep my blurays around 50GB per movie but some are larger. my DVDs are around 4GB each and my recorded tv makes it on the server for about a month then is deleted. The biggest file increase would be my windows images, got a few new devices and those are around 200GB each with all their programs installed and that pushed me to the 80% buffer recently.

I have also considered getting archival grade drives and just running them as single drives, 1 for movies, 1 for tv. then having a raid array for my windows images and other work related files probably 1TB data total.

Edit: but that will still probably have performance issues when playing back on plex

 

[Chris Moore]

The archival (SMR) if I recall correctly, do not work well with FreeNAS.

Edit: but that will still probably have performance issues when playing back on plex

More drives usually equates to better performance. The reason is because the data rate of the drives adds up. So, if you have 6 drives and you are using RAID-z2, you are almost getting the speed of 4 of those drives (minus some overhead) when you are reading or writing to the pool. So, for talking purposes, if a drive has a mechanical speed of 150 MB/s, four of those would give 600 MB/s minus the amount that goes to overhead. The actual speed is usually much less than these figures might lead you to beleive because of things that are out of scope for what I am trying to explain, but if you use a single drive, you only get the speed of that one drive which would be much less.
In addition to the loss of speed, you would also loose many of the robust data protection features of FreeNAS and the ZFS file system. ZFS depends on spreading parity and checksum information across multiple (at least two) disks.