Help on my FreeNas setup

[RodrigoLopes]

Hello everyone,

This post isn't about a problem but about a request for help.

I have been reading a lot of topics arround here and i have been seing that everyone talks about ECC memory and i don't have that in my current setup.
I have been upgrading my setup for a year now (piece by piece), i would like if anyone with more experience than me could rate and help me with it.

I just started to having problems with a disk (Degraded) after a scrub and a reboot everything is fine (or not 49 damaged sectores and 7 checksum errors).

I don't use it for any mission critical JOB, it server only has storage for family photos, pc backups and some media. obviously i don't intend to loose data.

Next purchase are going to be 2 new disks: HDD 3.5" Seagate IronWolf 1TB 5900RPM 64MB SATA III


My current setup is:
Board: Motherboard Micro-ATX Gigabyte A320M-S2H V2 GA-A320M-S2H V2
CPU: Processador AMD Ryzen 3 2200G Quad-Core 3.5GHz c/ Turbo 3.7GHz
Memory: G.SKILL NT 8GB (2x4GB) DDR4-2400MHz CL17 Memória RAM G.SKILL NT 8GB (2x4GB) DDR4-2400MHz CL17
Case: Micro-ATX Cooler Master MasterBox Q300L
PSU: Nox Hummer X500W 80 PLUS Bronze Semi Modular
Boot Disk: SSD 2.5" Kingston A400 120GB TLC SATA

 

[PhiloEpisteme]

Hi @RodrigoLopes, I'm not sure exactly what feedback you're looking for so I provided my thoughts on a few aspects of what you posted. Other folks may have a different perspective than mine so certainly don't take my thoughts as gospel.

I just started to having problems with a disk (Degraded) after a scrub and a reboot everything is fine (or not 49 damaged sectores and 7 checksum errors).

If you're seeing checksum and sector reallocation errors you're probably best off to replace the disk. The purpose of the SMART tests are to notify you of issues with drives before they flat out die so you can replace them early and reduce data loss risk. The purpose of the scrubs are to identify and correct any issues with data using parity information on the disks. If you see issues and you have the means to replace/resilver the drive immediately I would suggest you do so. Waiting to replace the disk increases the likelihood of the disk flat out failing and/or multiple disks going bad at the same time.

Next purchase are going to be 2 new disks: HDD 3.5" Seagate IronWolf 1TB 5900RPM 64MB SATA III

Would you mind editing your above post to fix the formatting for readability? Also, what do you plan to do with these disks? What are the current disks you're using? What is the current pool layout? What is your planned future pool layout?

Memory: G.SKILL NT 8GB (2x4GB) DDR4-2400MHz CL17 Memória RAM G.SKILL NT 8GB (2x4GB) DDR4-2400MHz CL17

If this were a new build I would suggest you go with the largest modules your board supports. FreeNAS likes memory and the more you give it the more performant it will be. Using the largest supported modules allows you to grow your total available memory without tossing old modules. Just something to keep in mind with future upgrades.

Boot Disk: SSD 2.5" Kingston A400 120GB TLC SATA

I hope you're keeping backups of your system config and any encryption keys. It will save you huge headaches and possibly your data to do so if your boot drive dies.

I have been reading a lot of topics arround here and i have been seing that everyone talks about ECC memory and i don't have that in my current setup.

A lot of people strongly prefer ECC memory in their systems, myself included. I think it is important to understand the risk you're exposing your data to if you don't. Will your system burst into flames if you don't use ECC memory? No. Does ECC memory provide a bit more data integrity throughout the entire system? Yes. Is it more expensive? Perhaps a bit. Is it WAY more expensive? Typically not. Whether it is right for you, your budget, and your data is your choice. I think the only mistake would be to forego ECC memory and be ignorant of the differences between ECC and non-ECC builds despite being encouraged to investigate it.

Board: Motherboard Micro-ATX Gigabyte A320M-S2H V2 GA-A320M-S2H V2

I'm not sure what advice or upgrades you're looking to do but a lot of folks prefer server boards over consumer boards. For many people the slight increase in price is worth the features such as IPMI, design for 24/7 operation, etc; the lack of non-useful functions such as rgb, sound cards, etc; and the benefit of supporting ECC memory and server processors. If you're ever considering upgrade you could pick up a used motherboard off ebay from a reputable seller and get a quality board for not too much money. For example, the X10 generation supermicro boards can be had used on ebay and often come with a nice feature set. As above, sticking with the board above won't murder your data; but does lack some very nice functionality that you may consider in the future.

obviously i don't intend to loose data.

Keep in mind that no amount of parity in a FreeNAS system is a replacement for a backup. If you have any data you truly do not want to lose you're going to want to maintain a backup, ideally off-site. My personal strategy is to run a main server in my home with a separate backup server in my home that the main server backs up to regularly. Additionally, I rotate disks out of my backup server off-site regularly so that in the event of a catastrophe at my home I do not lose all of my data; only new data since the last disk rotation. Your preferred backup system may look different than mine depending on your budget, risk tolerance, how much joy you get building a machine, etc. I built my backup on the cheap by buying all used for the backup server, including the HDDs. I regret the used HDDs because several have failed already; but I do not regret the money I saved on the used chassis, cpu, memory, board, etc.

 

[RodrigoLopes]

Hi @RodrigoLopes, I'm not sure exactly what feedback you're looking for so I provided my thoughts on a few aspects of what you posted. Other folks may have a different perspective than mine so certainly don't take my thoughts as gospel.


If you're seeing checksum and sector reallocation errors you're probably best off to replace the disk. The purpose of the SMART tests are to notify you of issues with drives before they flat out die so you can replace them early and reduce data loss risk. The purpose of the scrubs are to identify and correct any issues with data using parity information on the disks. If you see issues and you have the means to replace/resilver the drive immediately I would suggest you do so. Waiting to replace the disk increases the likelihood of the disk flat out failing and/or multiple disks going bad at the same time.

Would you mind editing your above post to fix the formatting for readability? Also, what do you plan to do with these disks? What are the current disks you're using? What is the current pool layout? What is your planned future pool layout?


If this were a new build I would suggest you go with the largest modules your board supports. FreeNAS likes memory and the more you give it the more performant it will be. Using the largest supported modules allows you to grow your total available memory without tossing old modules. Just something to keep in mind with future upgrades.


I hope you're keeping backups of your system config and any encryption keys. It will save you huge headaches and possibly your data to do so if your boot drive dies.


A lot of people strongly prefer ECC memory in their systems, myself included. I think it is important to understand the risk you're exposing your data to if you don't. Will your system burst into flames if you don't use ECC memory? No. Does ECC memory provide a bit more data integrity throughout the entire system? Yes. Is it more expensive? Perhaps a bit. Is it WAY more expensive? Typically not. Whether it is right for you, your budget, and your data is your choice. I think the only mistake would be to forego ECC memory and be ignorant of the differences between ECC and non-ECC builds despite being encouraged to investigate it.


I'm not sure what advice or upgrades you're looking to do but a lot of folks prefer server boards over consumer boards. For many people the slight increase in price is worth the features such as IPMI, design for 24/7 operation, etc; the lack of non-useful functions such as rgb, sound cards, etc; and the benefit of supporting ECC memory and server processors. If you're ever considering upgrade you could pick up a used motherboard off ebay from a reputable seller and get a quality board for not too much money. For example, the X10 generation supermicro boards can be had used on ebay and often come with a nice feature set. As above, sticking with the board above won't murder your data; but does lack some very nice functionality that you may consider in the future.


Keep in mind that no amount of parity in a FreeNAS system is a replacement for a backup. If you have any data you truly do not want to lose you're going to want to maintain a backup, ideally off-site. My personal strategy is to run a main server in my home with a separate backup server in my home that the main server backs up to regularly. Additionally, I rotate disks out of my backup server off-site regularly so that in the event of a catastrophe at my home I do not lose all of my data; only new data since the last disk rotation. Your preferred backup system may look different than mine depending on your budget, risk tolerance, how much joy you get building a machine, etc. I built my backup on the cheap by buying all used for the backup server, including the HDDs. I regret the used HDDs because several have failed already; but I do not regret the money I saved on the used chassis, cpu, memory, board, etc.

Hi @PhiloEpisteme this was the kind of insight that i was looking for i really apreciate your time answering it even with that vague title.

I will try to answer some of the questions you made about my question ;D....

Regarding the disk i'm trying my luck because both disk's where old used disks i had. (It's normal that they are starting to fail), regarding the new disks i want to replace those old ones and use RAID1 betwen the new ones.
I will try to buy them as fast as possible just trying to raize the budget again.

Regarding the memory what are the modules you reffer to? the speed? Aparently in my country it is complicated to find ECC memory (DDR4 at least) and when you find those the prices sky rocket normaly three times more money than normal ones :(

Regarding the board i too wanted a board from a server but since getting parts in my contry via Ebay and Amazon is a bit tricky cause of customs i have a leg behind my back there, that's why i'm using normal parts... and for what i saw it supports ECC but doesn't like it very much.

Once again thanks a lot for you insight, it helps a lot since i'm a beginner with this kind of stuff, but amazed by it :D

Regards,

Rodrigo

 

[PhiloEpisteme]

Regarding the disk i'm trying my luck because both disk's where old used disks i had. (It's normal that they are starting to fail), regarding the new disks i want to replace those old ones and use RAID1 betwen the new ones.

Sorry, I still do not understand. It would helpful to me if you specified exactly what your current setup is like. By that I mean, how many pools do you have, what types of vdevs do you have, what size and number of disks are in each vdev. Then, if you could specify exactly what kind of vdev you plan to put those two new drives into and whether they will extend an existing pool or be used to create a new pool. As an example, my main system looks like this. For your current system layout you could simply provide the output of zpool status here.

Also, a bit of a nit, using phrases like RAID1 can be a bit confusing. There is no such thing as RAID1 in zfs, there is an analog called a mirror vdev. There IS a RAIDZ1 in zfs which is more like the traditional RAID5. You can see how the terminology can be confusing. To avoid confusion it is best to stick with only zfs terms. This primer is quite helpful.

Regarding the memory what are the modules you refer to? the speed?

"A module" just refers to a stick of ram. For example, your machine has 2x 4GB ram so you have 2 4GB modules.

Apparently in my country it is complicated to find ECC memory (DDR4 at least) and when you find those the prices sky rocket normally three times more money than normal ones :(

Some places the prices are a lot higher. The price may make a huge difference in your decision to use ECC or not.

Regarding the board i too wanted a board from a server but since getting parts in my contry via Ebay and Amazon is a bit tricky cause of customs i have a leg behind my back there, that's why i'm using normal parts... and for what i saw it supports ECC but doesn't like it very much.

That is totally reasonable. Different parts of the world have different prices and availability of items. The board supports ECC memory but says it will work in "non-ECC" mode, so there wouldn't be much point in purchasing ECC memory for this board.

Once again thanks a lot for you insight, it helps a lot since i'm a beginner with this kind of stuff, but amazed by it :D

No problem, happy to help. I think checking out the primer I linked above would be useful. You may also find the following information explaining pools, vdevs, etc useful. There is extra stuff there about SLOGs and L2ARC devices that you may not yet need to worry about. The main takeaway is the following.

ZFS uses pools to store data. Pools are composed of one or more vdevs. vdevs are made of disks. If a single vdev within a pool fails, the entire pool is lost, this redundancy at the vdev level is important. vdevs can be of several types. A single-disk, a mirror of n disks, RAIDZ1, RAIDZ2, and RAIDZ3.
RAIDZ1 vdevs allow for 1 disk failure within the vdev without losing the vdev.
RAIDZ2 vdevs allow for 2 disk failures within the vdev without losing the vdev.
RAIDZ3 vdevs allow for 3 disk failures within the vdev without losing the vdev.
Mirrored vdevs of n disks tolerate n-1 disk failures within the vdev without losing the vdev.
single-disk vdevs are lost if the drive fails.
You could have a pool composed of 1x RAIDZ2 vdev of 6 disks. 2 of those 6 disks could fail and you would still have your pool, but if a 3rd failed you would lost your entire pool.
You could also have a pool of say 2x RAIDZ3 vdevs each composed of 6 disks. Either of the vdevs could lose up to 3 disks and you're okay; but if any single one of the two vdevs loses 4 or more disks you lost that vdev and thus the pool, even if the second vdev is 100% healthy.

Understanding the above points is key to putting together a good system. If any of it is confusing check out the primers I linked above; do some googling; search these forums; or ask questions. It can seem very complicated at first, but once you see the big picture it makes a lot of sense.

 

[RodrigoLopes]

Sorry, I still do not understand. It would helpful to me if you specified exactly what your current setup is like. By that I mean, how many pools do you have, what types of vdevs do you have, what size and number of disks are in each vdev. Then, if you could specify exactly what kind of vdev you plan to put those two new drives into and whether they will extend an existing pool or be used to create a new pool. As an example, my main system looks like this. For your current system layout you could simply provide the output of zpool status here.

Also, a bit of a nit, using phrases like RAID1 can be a bit confusing. There is no such thing as RAID1 in zfs, there is an analog called a mirror vdev. There IS a RAIDZ1 in zfs which is more like the traditional RAID5. You can see how the terminology can be confusing. To avoid confusion it is best to stick with only zfs terms. This primer is quite helpful.


"A module" just refers to a stick of ram. For example, your machine has 2x 4GB ram so you have 2 4GB modules.


Some places the prices are a lot higher. The price may make a huge difference in your decision to use ECC or not.


That is totally reasonable. Different parts of the world have different prices and availability of items. The board supports ECC memory but says it will work in "non-ECC" mode, so there wouldn't be much point in purchasing ECC memory for this board.


No problem, happy to help. I think checking out the primer I linked above would be useful. You may also find the following information explaining pools, vdevs, etc useful. There is extra stuff there about SLOGs and L2ARC devices that you may not yet need to worry about. The main takeaway is the following.

ZFS uses pools to store data. Pools are composed of one or more vdevs. vdevs are made of disks. If a single vdev within a pool fails, the entire pool is lost, this redundancy at the vdev level is important. vdevs can be of several types. A single-disk, a mirror of n disks, RAIDZ1, RAIDZ2, and RAIDZ3.
RAIDZ1 vdevs allow for 1 disk failure within the vdev without losing the vdev.
RAIDZ2 vdevs allow for 2 disk failures within the vdev without losing the vdev.
RAIDZ3 vdevs allow for 3 disk failures within the vdev without losing the vdev.
Mirrored vdevs of n disks tolerate n-1 disk failures within the vdev without losing the vdev.
single-disk vdevs are lost if the drive fails.
You could have a pool composed of 1x RAIDZ2 vdev of 6 disks. 2 of those 6 disks could fail and you would still have your pool, but if a 3rd failed you would lost your entire pool.
You could also have a pool of say 2x RAIDZ3 vdevs each composed of 6 disks. Either of the vdevs could lose up to 3 disks and you're okay; but if any single one of the two vdevs loses 4 or more disks you lost that vdev and thus the pool, even if the second vdev is 100% healthy.

Understanding the above points is key to putting together a good system. If any of it is confusing check out the primers I linked above; do some googling; search these forums; or ask questions. It can seem very complicated at first, but once you see the big picture it makes a lot of sense.

I will try to read about it more and try to answer your question to help me too ahah..

Regarding the zpool here is the output:

Show : Output

root@FreeNas:~ # zpool status -v
pool: Disco1
state: ONLINE
scan: scrub repaired 0 in 0 days 01:50:09 with 0 errors on Sun Sep 29 01:50:09 2019
config:

NAME STATE READ WRITE CKSUM
Disco1 ONLINE 0 0 0
gptid/9e7a1438-d164-11e8-8139-90fba6464c11 ONLINE 0 0 0

errors: No known data errors

pool: Disco2
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub repaired 0 in 0 days 00:34:58 with 2 errors on Mon Sep 30 19:30:41 2019
config:

NAME STATE READ WRITE CKSUM
Disco2 ONLINE 0 0 0
gptid/6c15d0b7-d16f-11e8-b52e-90fba6464c11 ONLINE 0 0 0

errors: Permanent errors have been detected in the following files:

<metadata>:<0x83>

pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0 days 00:00:05 with 0 errors on Fri Sep 27 03:45:05 2019
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
ada1p2 ONLINE 0 0 0

errors: No known data errors

As of now i only have two disks as standalone, i want to buy new ones and after you explanation i will change my purchase from 2*1TB to 2*2TB since i want to use RAIDZ1 vdevs (allow for 1 disk failure within the vdev without losing the vdev.)
I have two pool's each pool has a disk.

One with 1TB and the other with 500GB.

Regarding the memory module i only have one module and i will put one more. The board can only hold up to 16GB (2*8GB).

Regarding the matter of price i have the dnownside since here everything related to Tech is expensive as hell....

I will start by reading what you linked from the forum, the google part i already started.

Once again thanks a lot

 

[PhiloEpisteme]

i will change my purchase from 2*1TB to 2*2TB since i want to use RAIDZ1 vdevs

I didn't fully explain above, RAIDZ1 does not work with 2 disks. You can find information on minimum disk requirements for the different types of vdevs. If you want only two disks with 1-drive failure capabilities you could do a 2-disk mirror instead.

As of now i only have two disks as standalone

From this and the output above you have two pools, each having a single vdev, each with 1 disk. This is not ideal because if either of those disks fail you will lost all of the data on the disk.

You could, if you chose, have 1 pool with 2 vdevs. For example

Code:

Pool: 2 striped vdevs
  vdev1: 2-disk mirror
    1TB disk
    1TB disk
  vdev2: 2-disk mirror
    2TB disk
    2TB disk

In the above example either vdev could have 1 disk failure and you're okay (you should replace the disk ASAP) and you'd have a total of 3TB of usable space.

I will start by reading what you linked from the forum, the google part i already started.

Great idea. Do some reading and get a better idea of what the different vdev types are and report back with questions etc.