A new NAS setup/advice on feasibility of first TreuNAS setup

[nemesis1782]

Hi,

I've been using Synology NAS for quite sometime and currently have a 2415+ which I got working again after a stressfull day of digging through the internet (C2000 issue :( ).

Now while being stressed out of my mind I was also looking to replace what I have and for a long time wanted to get familiar with ZFS. A dead NAS is always a good starting point for that right :P Since my current is back I do have sometime to figure out what would be best in my case without braking the bank.

I have looking into TreuNAS and I'm quite intrigued. However for ZFS I seem to be finding a lot of conflicting information which concerns me. Some topics in which i find conflicting information are:
- stability
- performance
- usability

The consus is however that ZFS has some benefits but does need more carefull planning and the penalties can be high. This will be fairly long post however feel free to wiegh in on just a part of it.

The use case:
So my use case is fairly wide. I need would like the following in decending order of importance:
- Reliable backups for my important data set (currently have this setup to 2 external synologies and to a Azure share all of these are of course encrypted)
- Shares for Documents/Photos/Audio/Video/Binaries
- At least 30TB of usable storage capacity with at least a 1 disk tolerance level
- Plex server
- iScsi
- Docker/Kubernetes capability
- VMWare (or other virtualization) capability
- Anything else that pops in my head :P

So basicly a homelab mascurading as a NAS :P

What do i find important:
Stability and reliability (I like fiddling with systems, however my wife is a user and shee needs to stay happy!)
Performance (IO, sequential as well as random read)
As low risk as possible rebuilds
1 disk failure recovery, Anything more then a 1 disk failure recovery is just a cherry on top
Power usage and cost

The hardware (For the first two planned steps):
Mainboard: Gigabyte GA-Z68X-UD4-B3
CPU: Intel I7 2700K
Memory: Corsair Vengence LP 4x4GB 1600MHZ DDR3 (Non ECC)
Video card: MSI 560TI (will be replaced with somthing low power if it ever goes into production)
Power Supply: Corsair TX750
Network: Initially the one onboard 1GB in the future this might change to a 4 port 1GB (with LAG) or 1 port 10GB PCI card
Disk controller: SATA 6x on board + SATA 2x on board + SATA 2x on board + eSATA 2x on board (Which would give 10 drives, I might extend this with a 8 port LSI at some point)
Disks: Not sure yet have to rummage around for the test setup. In my Synology I have 5x 6TB WD60EFRX-68L0BN1 and 2x 10TB ST10000NM0478-2H7100 which may get migrated. I'm also thinking of adding 3 more 10TB ST10000NM0478-2H7100 for the final setup
SSD's: 240GB kingston as a boot drive and maybe to offload somethings on, 2x1TB Samsung 860 which are in use in the Synology, I might also add some small SSD's for the IO heavy side operations of ZFS.

The Plan (Test):
Download TreuNAS throw some small orphaned disks I have lying around at it and see how far I get.

The Plan (First actual setup):
Delete everything on the test setup and restart with what I've learned. Create 1 VDEV 5x10TB in ZFSR1 add a chache of 2x1TB SSD and 2 smaller SSD's for the IO heavy ZFS operation (do not remember the correct terminology).
- Configure
- Configure backups
- Migrate the data.
- Have a database running on it.
- Have Plex running.
- Get Docker up and running and set it as the master for my Kubernetes Cluster (with 3 raspbarry pi4 slaves).
- Setup Radar/Sonar/Sabnzbd/etc

The Plan (Future setup):
A multi Xenon system with at least 128GB of ECC RAM. There isn't much more to say about this yet ;)

My questions and possible ceveats:
1. ZFS disk space (in)effinciencies: From what I've read ZFS is far less efficient then the alternatives when it comes to usable disk space for the same cost point

1.a. 80% max pool usage is something I see floating around often. Above 80% ZFS seems to loose perfomance drastically. This means in a 5 disk setup you loose 1 disk right off the batt. However if it is that this is required that is fine. What confuses me is that ZFS does not seem to protect itself against this. I mean there must be gaurds that make sure this does not happen right?

1.b. Resilver time and risks. Because of the long resilver time and the high stress on the disks due to resilvering I read that a 2 Disk parity is almost a requirement.

1.c. VDEV width so basicly if you performance you want VDev's which aren't very wide (so have few disks). However each VDEV has their own parity loss which makes it very inefficient. Also adding multiple VDEVs into one pool is not really what you want since if one fails the pool fails. Meaning you have to split up the available data making managing the storage much more cumbersome and inefficient

1.Conclusion/Confusion: Taking all this in it feels like for a 5 disk VDEV you'd loose 4 disks (assuming 3 disk parity + disk due to the 80% factor). That is a steep price!

2. ZFS Performance: From what I read this is either super or extremely bad depending on whom you ask.

2.a. 80% pool size does performance degrade from 80% or does it start at a lower percentage.

2.b. I read that ZFS has two high IO processes I forget what they're called. How large should the SSD's be to mitigate this and what would be a recommended IO speed for these SSD's be?

3. Memory:

3.a. I have 16GB memory which should be way to little for the pool size I'm aiming for is this correct?

3.b. I have non ECC memory. Can I get definitive answer on if you ECC or not. Some say Yes some say No. Can we get a consensus (of course I understand ECC is better and why). The question is will it function well without it.

For now I'll leave at that. Any input would be appriciated, preferably with your reasoning and references included.

Thank you all for taking to read this regards,

Davy Vaessen

 

[Arwen]

I would suggest looking at more server oriented boards. Anything with IPMI and no audio or PS2 would be a better server. Plus, with network IPMI, you can remote manage almost completely. Even from the console.

2.
ZFS performance is generally related to both hardware used, and it's configuration. Meaning a pool with 14 disks in a single RAID-Z3 will generally perform less well than 2 x 7 disk RAID-Z2.

2.a
80% is for non-ZVol / iSCSI. Mirror pools and 50% is suggested for virtual machines that use zVols / iSCSI.

3.a
ZFS used to "suggest" 1GB of memory per 1TB of disk. BUT, this is just a suggestion. TrueNAS requires 8GBs for it's self. Any VMs will need more.

3.b
ZFS will function completely fine without ECC memory most of the time. That said, my new NAS, (parts are still arriving), will have ECC memory, But, my old media server, my 2 laptops and newish desk top don't have ECC memory, (and all use ZFS for root & data).


From some of your comments, you need to read up on ZFS pools & vDevs a bit more. Their is nothing stopping you from having multiple pools in the same NAS. Nor multiple vDevs in a pool, (though it's suggested that these be similar to each other).

 

[sretalla]

1. ZFS disk space (in)effinciencies: From what I've read ZFS is far less efficient then the alternatives when it comes to usable disk space for the same cost point

See here: https://wintelguy.com/zfs-calc.pl

What confuses me is that ZFS does not seem to protect itself against this. I mean there must be gaurds that make sure this does not happen right?

Are you confused that your car doesn't have a feature to prevent you from driving it off a cliff?

TrueNAS will send alerts and if you're using TrueCommand, you get predictive analysis.

1.b. Resilver time and risks. Because of the long resilver time and the high stress on the disks due to resilvering I read that a 2 Disk parity is almost a requirement.

Highly recommended for RAIDZ of more than 1TB disk size

1.c. VDEV width so basicly if you performance you want VDev's which aren't very wide (so have few disks). However each VDEV has their own parity loss which makes it very inefficient. Also adding multiple VDEVs into one pool is not really what you want since if one fails the pool fails. Meaning you have to split up the available data making managing the storage much more cumbersome and inefficient

Yes, that's all true... perhaps you're not happy about it, but if you want to drive a Ferrari, don't show up to the dealership with $500.

In your use case, you're not going to be happy with RAIDZ1 or RAIDZ2 for Block storage/iSCSI, so the SSD option you're talking about (perhaps mirror, depending on your tolerance for downtime) will be a must.



It sounds like you're probably looking for TrueNAS SCALE, rather than CORE, so beware that SCALE is currently ALPHA.

I hope it all works out well for you. Best of luck.

 

[ChrisRJ]

What I have chosen to do with my virtualization server (XCP-ng, but the principle applies to others as well) is run things off of a local SSD and use the NAS only for backup. With hourly delta backups I have a maximum loss of data of one hour and that is acceptable to me. Your mileage may vary, of course.

With this approach I could build (just done in October 2020) for high reliability, while keeping the price down.

 

[nemesis1782]

@Arwen: Thnx for the reply. Looking at server oriented boards is a given indeed however for my first tests this is what I have to work with.

I'm curious why would the lack of audio and PS2 make it a better server?

For 2. That's what I gathered not clear on the why yet. It seems counter intuitive to me. Also how much of a difference are we talking about, 1, 5 10, 25, 50 or 80%? Also you would loose 1 disk of usable space and have more risk right?

For 2.a I didn't understand you answer would you mind rephrasing it?

For 3.a ok I understand it's just a suggestion but there is probably a reason for it. The question is if I only have 8GB left for 50TB of raw storage would that even work and what would be the side effects/issues I can expect.

As for reading I will definitely be doing that however there is a lot to process and a bit of back and forth helps me a lot. So thank you for your reply!

As for pool VDEV. From what I understand. Multiple for a pool means increased performance and one large blob of disk space but higher risk since there is no redundancy/parity/checksums on the pool level. Multiple pools means that you need to divide stuff and plan usage. Furthermore it'll be inefficient because a pool will never fit exactly to your needs it just seems wasteful to me.

 

[nemesis1782]

@ChrisRJ: Saw a similar comment from you in another thread. You might be right.

Currently I use raspbarry pi4's for my virtualization needs (if you can call docker virtualization, but that is a whole other matter). However it would be nice to have a AMD64 system in the Kubernetes cluster so I can remove the AMD64 emulation from the PI's. The reason I do it this way is that it fits my current needs and uses little power. I will eventually have to bite the bullet and go full server at some point though and I'll definitely take your advice under consideration.

 

[nemesis1782]

@sretalla: Yeah I found that same calculator. This is what gave me many of the numbers.

Are you confused that your car doesn't have a feature to prevent you from driving it off a cliff?

No, but I would be confused if the car would become unstable at 120 km/h and it isn't limited well below that ;) (assuming it's a road legal vehicle)

TrueNAS will send alerts and if you're using TrueCommand, you get predictive analysis.

Ok that's great! Can you limit the amount of storage allotted for usage in a pool. So the pool cannot be utilized more then 80%?

Can you elaborate on why the limitation of 80% exists? Assuming that you know the reason.

Well it's so much not wanting to pay for that extra bit of gold plating. However the question is if the benefit outweighs the cost and I need more information to make a decision. So thanx for providing.

In your use case, you're not going to be happy with RAIDZ1 or RAIDZ2 for Block storage/iSCSI, so the SSD option you're talking about (perhaps mirror, depending on your tolerance for downtime) will be a must.

My plan was a single RAIDZ1 VDEV 5x10TB 2 small SSD's for SLOG (mirror) and 2 TB SSD's for hot block caching. But from your answers and my own research I see that this probably a bad idea.

Maybe a zpool with 3x(2 6TB drives in mirror) and 2x (2 10TB drives in mirror) would be better. Due to the write performance not sizing with the number of disks in the VDEV.

I assumed TreuNAS Scale would be paid and I do not have the budget for enterprise level software licenses. Seems I made the wrong assumption I'll look into it, thnx. I think that might indeed fit better from what I saw.

Anyway I'm still in the learning/testing phase it'll be sometime before it goes to production. I fixed the synology so there is no rush.

 

[Arwen]

@nemesis1782, you have a lot of questions. I won't be able to get to them all.

I'm curious why would the lack of audio and PS2 make it a better server?

It won't make it a better server, but most server boards designed as server boards don't include audio or PS2. So it's a bit of a check if a board is designed as a server board. Other non-server items would be far too many USB ports. I've seen some desk top boards with 10 USB ports. Not all that useful for a NAS server.

For 2.a I didn't understand you answer would you mind rephrasing it?

The 80% full rule is related to fragmentation / contiguous space. So it's uncertain when performance will start to degrade.

Further, if you use virtual machines built on zVols, (a type of ZFS container for raw data), it is suggested to use Mirror vDevs, not RAID-Zx. Further, since the write characteristics is different, they suggest keeping any pool with zVols at 50% or less.

For 3.a ok I understand it's just a suggestion but there is probably a reason for it. The question is if I only have 8GB left for 50TB of raw storage would that even work and what would be the side effects/issues I can expect.

YES. ZFS will work with 8GBs of memory and a pool of 50TBs. The only side effect is that ZFS can't cache more of the pool in memory, so potentially a little slower. But, if your use case does not use cached items, then maybe no impact. That's it.

As for pool VDEV. From what I understand. Multiple for a pool means increased performance and one large blob of disk space but higher risk since there is no redundancy/parity/checksums on the pool level. Multiple pools means that you need to divide stuff and plan usage. Furthermore it'll be inefficient because a pool will never fit exactly to your needs it just seems wasteful to me.

ZFS redundancy is at the vDev level, not the pool level. So a pool made up of 4 disks, 2 in a Mirror vDev and 2 in a second Mirror vDev, has redundancy of 1 disk per vDev. So you can loose a disk in the first vDev, either one. And loose a second disk in the second vDev, either one. And not loose any data.

Or, if you want better reliability for those 4 disks, you can use RAID-Z2 which would allow you to loose ANY 2 disks, and not loose data. You would give up some performance because a pool with 2 vDevs can perform better than a single vDev pool.

Multiple pools are useful for different uses. For example, bulk backup storage can use 6 disks in a RAID-Z2. And the VM images would do better on 4 disks in a Mirrored pool of 2 vDevs.

Ok that's great! Can you limit the amount of storage allotted for usage in a pool. So the pool cannot be utilized more then 80%?

Yes, and no. You create file systems, (aka Datasets), inside a pool for different uses. You can set quotas to prevent one file system from using up all the space. But, ZFS allows you to over-subscribe these "quotas". If you need hard, file system reservations, you use a combination of quota and "reservations". Reservations actually reserve storage for that dataset. No other dataset can use that space.

Example, pool is 100TB. One dataset is 50TB, so you "quota=50T" and "reservation=50T". No chance of it growing beyond 50TB. Then another dataset is 30TB, so "quota=30TB" and "reservation=30TB". Thus, 50+30 = 80TB, which is 80% of 100TB. It's not quite that simple, as you datasets can then have problems getting to that last part.

Quotas and reservations are a bit hard concept. Took me a while when I first started with ZFS.

Can you elaborate on why the limitation of 80% exists? Assuming that you know the reason.

It's not a "limitation" per say, it's a suggestion. Performance will be reduced as ZFS has to work harder to find places to put data. It's worse if you have a highly fragmented pool. At some point, (92% I think, could be wrong), ZFS will transition to space saving means, which is even slower.

To be fair, larger pools can be less affected. A 100TB pool with just large video files written to it, can probably go to 99TBs written without problem. It's a combination of too many things, so the "rule" / "suggestion" to keep your pool at 80% and you should have decent performance.

I assumed TreuNAS Scale would be paid and I do not have the budget for enterprise level software licenses. Seems I made the wrong assumption I'll look into it, thnx. I think that might indeed fit better from what I saw.

Both "TrueNAS Core" and "TrueNAS Scale" are free. Just note that TrueNAS Scale is still Alpha quality code. A simple backup server for now, would probably be fine. But, complex pool arrangements, lots of shares, dockers, etc... may run across problems.

In someways, the "free" versions of TrueNAS Core & TrueNAS Scale are testing for the Enterprise versions. (RedHat did the same, with Fedora.) I don't mind "testing", as long as I know the risks, (very minimal with TrueNAS Core), and I have the choice to use or not use. (iXsystems is not the only free NAS software vendor...)

 

[adrianwi]

I love this kind of post. I want reliability, stability, performance, etc. but I'm planning on building a non-optimal system and don't want to be disappointed when it won't do what I wanted.

My use case is not that dissimilar and my server delivers all of those requirements, but I built it based on the recommendations of the forum and around what is known to work well with Free/TrueNAS.

Good luck

 

[nemesis1782]

I love this kind of post. I want reliability, stability, performance, etc. but I'm planning on building a non-optimal system and don't want to be disappointed when it won't do what I wanted.

I love these kinda replies. Let's not read what he wrote and make some assumptions to make myself feel superior.

Also per defenition you get a certain reliability, stability and performance. The question is what am I comfortable with on having and what am I willing to spend. Further more I've clearly stated that the system specified is my first setup and to get my feet wet. You know, learn, ask why certain limitations are what they are so I do not have to regurgitate what someone else told me once upon a time ;)

I see a lot of these kind of toxic replies around these forums and the ZFS forums are even worse. If you have nothing constructive to post please refrain from replying!

My use case is not that dissimilar and my server delivers all of those requirements, but I built it based on the recommendations of the forum and around what is known to work well with Free/TrueNAS.

Yeah I'm sorry but the requirements are really vague! Also looked at the actual ix NAS systems and they're no where near what people are claiming you need.

Good luck

I'll take that in a positive light and say thank you ;)

 

[nemesis1782]

Ok so I needed to update the BIOS of my mainboard to get things up and running and.. um.. I bricked it :P So I threw that plan out of the window!

Luckily I ran into a nice server system which I'll be getting soon. Might not be perfect but it's ok and the price was great :)

Specs:
Dell PowerEdge R720XD LFF Generation 12
- 12 3.5inch bays in the front
- Dell 5720 QP1GB Network DC
- DELL PERC H710 512MB BBU mini
- 1100W Platinum power supply
- 64GB 12800R 1600MHz / 8x 8GB DDR3
- 2x 8 Core Xeon E5-2650v2 / 2.6Ghz 20MB ( 8C 16T 95W )

What would be the best way to update my original post? What's the convention on this forum, remove old info and rewrite or leave and add to it?

 

[Arwen]

In general, ZFS works best with a HBA, Host Bus Adapter, for disk access. Not RAID cards. I don't know that your "DELL PERC H710 512MB BBU mini" is a RAID card, but it appears to be so from it's description, "512MB" & "BBU", (as in Battery Backed Up).

ZFS was specifically designed to control disks directly. ZFS can and DOES work with SAN attached disk LUNs that already have RAID-something backing them. However, most home and small office users don't have "real" SAN disk arrays. So, back to HBAs.

But, the problem arises when people use PCIe or builtin to the mother board, RAID cards / chips. These can re-order writes, claim writes have been written, (hence the 512MB battery backed up cache...), and even perform RAID functions slower than ZFS. Even RAID cards that say they can expose the disks as JBOD tend to hide SMART or the real disk sizes.

So, we highly recommend using HBAs, or RAID cards that can be changed into HBAs with firmware or drivers. For example, some of the most common LSI / Broadcom / Avago cards can have IR, (a light weight, mirror only, Integrated RAID). But another version of the same card's firmware, IT, (Integrated Target), turns it into a HBA. In these forums you will see references to LSI and IT firmware, that's what it's about.


As for some people not being helpful at times, we get newbies that insist that things must work one way, and can't work any other way. For example, insisting on the 1GB of memory per 1TB of disk as a "rule" not "suggestion", because they read it from an authoritative source. (I am not meaning you, @nemesis1782, you asked for clarification, not stated it was a rule.) Thus, some people on this forum get tired of the same questions from new users.

 

[nemesis1782]

@Arwen: Yeah, sorry I do often have a lot of questions. I appreciate you taking the time to respond in a well formulated and thoughtful manner. If I commit to something new I do have the annoying habit of needing to know how things work and why.

It won't make it a better server, but most server boards designed as server boards don't include audio or PS2. So it's a bit of a check if a board is designed as a server board. Other non-server items would be far too many USB ports. I've seen some desk top boards with 10 USB ports. Not all that useful for a NAS server.

Well true and as you can see above I've been forced to revise my plans. I will be going with a server board. Looking at the specs for the TrueNAS mini I can comfortably say that it would've been fine, especially to test TrueNAS and ZFS on. If I wouldn't have bricked it and barring any possible incompatibility/missing drivers for FreeBSD.

The 80% full rule is related to fragmentation / contiguous space. So it's uncertain when performance will start to degrade.

Ok, that sounds plausible at least the fragmentation part. With contiguous do you mean sequential? Is there a way to get rid of the fragmentation, since if you have a dataset that changes often fragmentation will get much worse over time. I'll read up on this in the coming days atm this still seems off to me and it sounds like a design oversight. However that may very well be due to my still limited knowledge on the subject.

Further, if you use virtual machines built on zVols, (a type of ZFS container for raw data), it is suggested to use Mirror vDevs, not RAID-Zx. Further, since the write characteristics is different, they suggest keeping any pool with zVols at 50% or less.

That's a steep cost! So that basically leaves you with 25% of your capacity being useful. I might need to create multiple different zPools each with a specific goal in mind. I'm still not convinced that being limited to ZFS is a good thing. For many use cases it just seems wasteful and badly equipped.

YES. ZFS will work with 8GBs of memory and a pool of 50TBs. The only side effect is that ZFS can't cache more of the pool in memory, so potentially a little slower. But, if your use case does not use cached items, then maybe no impact. That's it.

Yeah, so if I understand this correctly you're talking about the ZIL (write) and ARC (Read) right? From what I read you can offload part of this to SSDs (not sure yet if they need to be mirrored as well). This is something I am planning to setup and test thoroughly.

ZFS redundancy is at the vDev level, not the pool level. So a pool made up of 4 disks, 2 in a Mirror vDev and 2 in a second Mirror vDev, has redundancy of 1 disk per vDev. So you can loose a disk in the first vDev, either one. And loose a second disk in the second vDev, either one. And not loose any data.

Or, if you want better reliability for those 4 disks, you can use RAID-Z2 which would allow you to loose ANY 2 disks, and not loose data. You would give up some performance because a pool with 2 vDevs can perform better than a single vDev pool.

Multiple pools are useful for different uses. For example, bulk backup storage can use 6 disks in a RAID-Z2. And the VM images would do better on 4 disks in a Mirrored pool of 2 vDevs.

Yeah exactly. From what I read especially write performance suffers in a RAID-Z2 (and with suffers I mean does not scale with the sizeof the set).

Yes, and no. You create file systems, (aka Datasets), inside a pool for different uses. You can set quotas to prevent one file system from using up all the space. But, ZFS allows you to over-subscribe these "quotas". If you need hard, file system reservations, you use a combination of quota and "reservations". Reservations actually reserve storage for that dataset. No other dataset can use that space.

Example, pool is 100TB. One dataset is 50TB, so you "quota=50T" and "reservation=50T". No chance of it growing beyond 50TB. Then another dataset is 30TB, so "quota=30TB" and "reservation=30TB". Thus, 50+30 = 80TB, which is 80% of 100TB. It's not quite that simple, as you datasets can then have problems getting to that last part.

Quotas and reservations are a bit hard concept. Took me a while when I first started with ZFS.

I never liked over provisioning. Seen to many things go wrong because of it. Thnx for the insights I'll look into and once I get the hardware see how it works and what I can break :)

It's not a "limitation" per say, it's a suggestion. Performance will be reduced as ZFS has to work harder to find places to put data. It's worse if you have a highly fragmented pool. At some point, (92% I think, could be wrong), ZFS will transition to space saving means, which is even slower.

To be fair, larger pools can be less affected. A 100TB pool with just large video files written to it, can probably go to 99TBs written without problem. It's a combination of too many things, so the "rule" / "suggestion" to keep your pool at 80% and you should have decent performance.

Ah yeah that makes sense. Thnx. For me a large part of my data is Video and static so that is the case.

Both "TrueNAS Core" and "TrueNAS Scale" are free. Just note that TrueNAS Scale is still Alpha quality code. A simple backup server for now, would probably be fine. But, complex pool arrangements, lots of shares, dockers, etc... may run across problems.

In someways, the "free" versions of TrueNAS Core & TrueNAS Scale are testing for the Enterprise versions. (RedHat did the same, with Fedora.) I don't mind "testing", as long as I know the risks, (very minimal with TrueNAS Core), and I have the choice to use or not use. (iXsystems is not the only free NAS software vendor...)

I'll go for TrueNAS Scale right off the bat. Seems it'll be release ready soonisch. Before I switch to a new setup I will be testing it thoroughly and putting it through it's paces. So that'll take a fair amount of time.

Yeah I understood that, I think the model they've setup is quite nice. Also you could always choose to just upgrade/update at a later point when you know it's stable.

Well thnx again for all the insights. I'll keep you posted!

 

[nemesis1782]

@Arwen: thnx again :)

In general, ZFS works best with a HBA, Host Bus Adapter, for disk access. Not RAID cards. I don't know that your "DELL PERC H710 512MB BBU mini" is a RAID card, but it appears to be so from it's description, "512MB" & "BBU", (as in Battery Backed Up).

ZFS was specifically designed to control disks directly. ZFS can and DOES work with SAN attached disk LUNs that already have RAID-something backing them. However, most home and small office users don't have "real" SAN disk arrays. So, back to HBAs.

But, the problem arises when people use PCIe or builtin to the mother board, RAID cards / chips. These can re-order writes, claim writes have been written, (hence the 512MB battery backed up cache...), and even perform RAID functions slower than ZFS. Even RAID cards that say they can expose the disks as JBOD tend to hide SMART or the real disk sizes.

So, we highly recommend using HBAs, or RAID cards that can be changed into HBAs with firmware or drivers. For example, some of the most common LSI / Broadcom / Avago cards can have IR, (a light weight, mirror only, Integrated RAID). But another version of the same card's firmware, IT, (Integrated Target), turns it into a HBA. In these forums you will see references to LSI and IT firmware, that's what it's about.

Indeed it is a RAID card however it can be flashed to or used as a HBA as you describe. I did read that TrueNAS can have some issues if you do not flash it to IT. I know it's a bit wasteful but did not have any choice and the price was right! I can always sell the 710 and buy a simple LSI HBA.

Indeed a RAID card abstracts the disks from the OS and presents a single block device with a lot of magic under the hood. With RAID it also depends on the use cases. The good RAID controllers like the one that is in the DELL are considered to be very stable, safe and proven. Data redundancy and safety is useless and wasteful when you do not need it. Also the way RAIDZ works if you have a system that performs a lot of writes you do not want that.

I am still skeptical that ZFS cannot run on a RAID config (pure theoretical and I understand what the risks would be) and from what I read till now traditional RAID has advantages over ZFS depending on the use case. I am thus a bit disappointed that TrueNAS binds users to ZFS and supports no alternatives (for which there may very well be very good reasons).

For instance (pure conjecture) having a mirror on the RAID card configured and putting that in a vDEV and building a vPool on top of that might perform far better. The RAID controller will manage the mirroring and provide some caching. The vDEV will provide the checksums. Of course you add a second point of failure and remove (most) of the advantage of Copy on write. However in essence a disk does the same! A disk has a cache, when you write something to said disk it may store it in cache until it has the time to process it. Not sure if ZFS circumvents this risk of disk cache.

As for some people not being helpful at times, we get newbies that insist that things must work one way, and can't work any other way. For example, insisting on the 1GB of memory per 1TB of disk as a "rule" not "suggestion", because they read it from an authoritative source. (I am not meaning you, @nemesis1782, you asked for clarification, not stated it was a rule.) Thus, some people on this forum get tired of the same questions from new users.

I understand. No worries, as a software dev/architect I run into this issue quite often (every time I open stack overflow :P). It just bugs me. I understand it can get frustrating for the more experienced, however we tend to forget how we ourselves started out ;) Also I do understand where their confusion comes from there quite a few "vets" preaching it as a rule almost religiously (not sure if it's on this forum as well, but at least on reddit). Un helpful posts will demotivate possible valuable future contributors.

I may have reacted a bit to harshly though, no offense was intended, if any offense was incurred I do apologize!

 

[Redcoat]

You may find this helpful: https://fohdeesha.com/docs/perc/:

 

[Arwen]

In regards to your bricked system board, some boards have removable BIOS flash chips. You could see if that is the case, and if you can order a replacement. Even if you don't want to go down that route, you could possibly sell the board on E-Bay with that known condition, and point to the vendor site saying the BIOS flash chip is removable.

With contiguous do you mean sequential? Is there a way to get rid of the fragmentation, since if you have a dataset that changes often fragmentation will get much worse over time. I'll read up on this in the coming days atm this still seems off to me and it sounds like a design oversight.

- Yes, I mean sequential.

- Fragmentation can be gotten rid of by ZFS Send to another place, (like backup disk), wiping the original data, and ZFS Receiving the data back again. But, in general, if you stay at the 80% suggestion, you should not have problems.

- In some ways, fragmentation is a design flaw of ZFS. The mythical BPR, (Block Pointer Re-write), has been close to, or the top of, ZFS user's wish list since close to the beginning of mass use of ZFS. It will probably never be implemented due to complexity and the extensive time needed to both write the code and test it thoroughly.

That's a steep cost! So that basically leaves you with 25% of your capacity being useful. I might need to create multiple different zPools each with a specific goal in mind. I'm still not convinced that being limited to ZFS is a good thing. For many use cases it just seems wasteful and badly equipped.

People that NEED iSCSI presented LUNs tend to want decent performance. Thus the suggestion for 50% usage, multiple vDevs, and Mirrored vDevs.

However, MANY home users are quite satisfied with zVols on RAID-Z2. They don't need top end performance. Understand it, and if you want external iSCSI LUNs, start with a RAID-Z2 and see if it works. If not, pay to get better.

Yeah, so if I understand this correctly you're talking about the ZIL (write) and ARC (Read) right? From what I read you can offload part of this to SSDs (not sure yet if they need to be mirrored as well). This is something I am planning to setup and test thoroughly.

More or less.
- The ZIL, (ZFS Intent Log), is actually on disk for synchronous writes. Whence on the ZIL, the synchronous write can return as complete. Later, ZFS will take a copy out of memory and flush it to disk in a ZFS write transaction. That clears the need for that specific ZIL entry.

- SLOG, (Separate intent LOG), does not really work as a write cache. It's only use is for synchronous writes when the stable media, (like spinning disks), is slower than a SLOG device, (usually SSD).

- L2ARC can not be mirrored as far as I know. Their is really no reason as it's data already on stable media. But, the "RULE", is max out your RAM before adding L2ARC. That's because L2ARC requires RAM for it's index / table of contents.

- SLOGs can be mirrored for data safety, but today don't have to be. Note that the mirror part only comes into play IF you get a crash, AND you loose your only SLOG during recovery, then the data that the SLOG was trying to save is lost. So if you have a bad SLOG during normal operation, you simply remove and replace. No problem. Basically a SLOG is a W/O, (Write Only), device. Unless you have a crash, it's not really read from. Thus, another need to have high endurance SSDs.

Last, SLOGs should have either direct writes, or capacitor backed cache. This is referred to as PLP, (Power Loss Protection). If you get a crash due to power loss, having a SLOG that looses writes that ZFS was told are complete, is bad.

 

[Arwen]

I am still skeptical that ZFS cannot run on a RAID config (pure theoretical and I understand what the risks would be) and from what I read till now traditional RAID has advantages over ZFS depending on the use case. I am thus a bit disappointed that TrueNAS binds users to ZFS and supports no alternatives (for which there may very well be very good reasons).

For instance (pure conjecture) having a mirror on the RAID card configured and putting that in a vDEV and building a vPool on top of that might perform far better. The RAID controller will manage the mirroring and provide some caching. The vDEV will provide the checksums. Of course you add a second point of failure and remove (most) of the advantage of Copy on write. However in essence a disk does the same! A disk has a cache, when you write something to said disk it may store it in cache until it has the time to process it. Not sure if ZFS circumvents this risk of disk cache.

Let's start off with facts. ZFS can and will run fine on a hardware based RAID card.

The main problems comes in to play are these:
- You have to rely on the hardware RAID card and it's drivers for telling you, and potentially recovering from, bad disk blocks or bad disks.

- If the RAID card fails, you generally have to replace with a similar model. (Or server with similar model of RAID card.) You can't just put the disks into another server and import your ZFS pool.

- OS side SMART won't work

- RAID cards can and do limit writing speeds. This is because ZFS bundles writes into transactions that will write across multiple disks. They can be larger than a hardware RAID card's battery backed up cache memory.

- RAID cards can and do limit reading speeds. ZFS can trigger reads across all your data disks. This is especially noticeable during a scrub, which can read for hours. Even days. (But a ZFS scrub can't fix anything because it does not control the redundancy... see Note 1)

- Last, ZFS will run fine on the hardware based RAID card, UNTIL IT DOESN'T! Their have been KNOWN cases of ZFS finding bugs in hardware RAID card firmware, (or in user implementations), the HARD way, meaning data loss. For example, not setting the hardware RAID card to automatically change write cache from write later to write through during battery failure. Then a crash occurs and you've lost data that was supposed to be "securely written".


In your theoretical example, that is somewhat what another vendor did. They used a check summing file system, BTRFS, on a type of a RAID-5/6 from MD-RAID, (or hardware RAID, I don't remember). They invented a way to detect problems via BTRFS and figuring out where to fix them in RAID. In essence, inventing a new RAID method, (and one with new code for failure).

ZFS does use the disk cache, using a feature called write barriers. If I understand it correctly, it means after a bunch of writes are sent to the disk, which may go into disk cache first, a barrier is made. No new writes can occur to disk media, except from that prior written cache data. New writes to the disk can occur, to the write cache, but not the same write barrier. Meaning, ZFS expects disk writes to be completely written when the disk tells ZFS the write is complete. So, ZFS write transaction groups need to be written in order, even if the elevator seek suggests more optimal writing would be faster.

That said, disks with known problems on write barriers should have their write cache disabled.

The difficultly is that ZFS has no way to fix any problems it sees, (see Note 1). The ZFS checksums are somewhat useless.


One point you overlook is that ZFS can "scrub" the pool. And if redundancy is available, fix it LIVE. So the next scrub will not find any problems. This does assume that the underlying media has spare disk blocks available. Or that the underlying media did not have an actual bad block, just one with bit rot, (meaning one or more bits changed without action by the disk).


Note 1 - All ZFS metadata is redundant even on a single disk vDev, (less important metadata like directory entries have at least 2 copies, and upper level metadata has 3 copies). I've actually seen checksum errors on a single disk that did not have any redundancy, be corrected. Took me a while to remember about metadata redundancy. The "copies=" is an interesting feature of ZFS, and can be used for data as well, for single disk data redundancy.

 

[adrianwi]

I love these kinda replies. Let's not read what he wrote and make some assumptions to make myself feel superior.

Also per defenition you get a certain reliability, stability and performance. The question is what am I comfortable with on having and what am I willing to spend. Further more I've clearly stated that the system specified is my first setup and to get my feet wet. You know, learn, ask why certain limitations are what they are so I do not have to regurgitate what someone else told me once upon a time ;)

I see a lot of these kind of toxic replies around these forums and the ZFS forums are even worse. If you have nothing constructive to post please refrain from replying!


Yeah I'm sorry but the requirements are really vague! Also looked at the actual ix NAS systems and they're no where near what people are claiming you need.


I'll take that in a positive light and say thank you ;)

You've been here less than a week, are questioning why TrueNAS uses ZFS, think using a RAID controller will be fine and are planning on building a non-standard system because you think you know better. I'm not sure it's me making 'myself feel superior', although I certainly feel happier that my data is well protected with my non-iX but recommended hardware.

It's not toxic trying to point out to someone where they are trying to make things work in a way they don't, but hey, it's your data so do with it as you please. I think you'll need more than good luck.

 

[nemesis1782]

You may find this helpful: https://fohdeesha.com/docs/perc/:

Thnx, this is indeed what I found and meant. Still not sure if I want to do this or just sell the card and buy a cheap LSI though.

For my initial test setup I will leave it as is. Once I go to production I will of course have to do something about this.

 

[nemesis1782]

You've been here less than a week, are questioning why TrueNAS uses ZFS, think using a RAID controller will be fine and are planning on building a non-standard system because you think you know better. I'm not sure it's me making 'myself feel superior', although I certainly feel happier that my data is well protected with my non-iX but recommended hardware.

I would advice re-reading the whole thread carefully and trying to not make to many assumptions this time. If you have some actual feedback and are willing to discus this in a civil manner I am open to it.

Now a few of your (incorrect assumptions):
- His user account is new, so he must be new! I rarely post on a forum. There are only two main reasons I post on a forum 1. Share information, which I will only do once I am confident I have the appropriate knowledge to do so. 2. To ask questions to increase said knowledge.
- He is new so he must not know what he's talking about. I might be new to TrueNAS, however I have experience in many IT related fields. My professional experiences include but are not limited to: Software Engineering, Manufacturing Execution Systems, Network Design and configuration, Data center design and configuration, VOIP, Enterprise data solutions, DBA, Kubernetes, etc. Where you are correct is that I have little experience with TrueNAS or ZFS.
- Oh no he's questioning my [enter your favorite religion, in your case TrueNAS]. I am skeptical by nature, that is how I am wired. That is also one of the things that makes me very good at my job. I am not saying it is wrong I have explicitly stated on numerous occasions in fact that it is probably due to a lack of knowledge on the sbuject. Hence why I'm here, asking these question and explaining where I'm skeptical. To LEARN!
- Oh he's creating a production system on non standard hardware.
--> I clearly stated TEST system haven't I?
--> Where o where does my test system not conform is non standard according to the minimum specs outlined by ix systems?
-------> https://www.ixsystems.com/blog/how-to-install-truenas-core/
-------> https://www.truenas.com/docs/hub/initial-setup/install/FirstTimeInstall/
-------> https://www.freenas.org/hardware-requirements/
-------> https://www.truenas.com/community/threads/hardware-recommendations-read-this-first.23069/
--> My production system which is now replacing my test system since I bricked it ticks all the requirements for a small to medium business setup, doesn't it?

It's not toxic trying to point out to someone where they are trying to make things work in a way they don't, but hey, it's your data so do with it as you please. I think you'll need more than good luck.

I'm sorry you have not provided ANY usable feedback just some remarks without any justification or clarification. How is the following not toxic?

Good luck

By the way this last time I'll reply to one of your posts if it does not contain any actual information. Please keep the discussion on topic. If you have a issue with me either message me and we can discus it or inform a mod so they can take appropriate action.

Do not pollute the thread with it!