Looking for advice on a 100TB Zvol

[Antairus]

I know this will be a Loaded Question, but I am new to FreeNAS and ZFS.

I have a server with 100TB that I am going to bust up into at least 4 20 TB Sections for iSCSI.

Should I just create a 100Tb Zfs2 and split that up or is there a better way?

Thanks.

 

[Arwen]

First off, for iSCSI storage, you should not get above 50% used. With 4 iSCSI volumes at 20TB each, that's 80TB used. So you would need a pool of 160TB to be at 50% used. And that's not including 5 or 10% overhead.

Read up on ZFS terminology. Zfs2 is meaningless. I'd guess, (perhaps wrongly), you meant ZFS' RAID-Z2. That's a poor choice for iSCSI volumes. The recomendation for iSCSI volumes is to use Mirrored vDevs, the more, the better. Meaning don't just use 12TB disks to get to your storage goal. More spindles is more important than size.

Next, performance iSCSI installations are a combination of other things as well. Like amount of RAM and the Ghz of the CPU.

Last, if your setup not read mostly, then a high speed SSD for a SLOG, (Seperate write intent LOG), may be desired.

 

[Antairus]

First off, for iSCSI storage, you should not get above 50% used. With 4 iSCSI volumes at 20TB each, that's 80TB used. So you would need a pool of 160TB to be at 50% used. And that's not including 5 or 10% overhead.

Read up on ZFS terminology. Zfs2 is meaningless. I'd guess, (perhaps wrongly), you meant ZFS' RAID-Z2. That's a poor choice for iSCSI volumes. The recomendation for iSCSI volumes is to use Mirrored vDevs, the more, the better. Meaning don't just use 12TB disks to get to your storage goal. More spindles is more important than size.

Next, performance iSCSI installations are a combination of other things as well. Like amount of RAM and the Ghz of the CPU.

Last, if your setup not read mostly, then a high speed SSD for a SLOG, (separate write intent LOG), may be desired.

Yes Raid-Z2 is what I meant. I will look into VDevs. Why is getting above 50% on the Raid-Z2 a bad thing? seems like a waste of space.
Its dual Xeon 2603 v4, and 32 Gig.

 

[tvsjr]

And if you're doing iSCSI, there's a good chance you're planning to run VMs... which means you need to consider a SLOG. And you're going to want substantially more RAM.

 

[nightshade00013]

Ok, going to chime in here so lets hold off on all the advice until the OP knows what they are planning to do.

@Antairus A few things need to be figured out before you go jumping in and buying a bunch of stuff or making decisions on how it's all going to go. First of all what are your purposes and goals for the system. Approx budget capabilities would also be helpful as would ALL the parts you have on hand already and intend to use. You also need to read up on how FreeNAS and ZFS do things/work.

For sure read and understand this https://forums.freenas.org/index.php?resources/terminology-and-abbreviations-primer.37/

A ton of info is available here as well though some will only apply in specific situations https://forums.freenas.org/index.php?resources/categories/hardware.1/

A lot of information is available here though it can become hard to follow.
http://doc.freenas.org/11/freenas.html

I am sure some others will have some recommended reading as well which I would encourage them to post as right now I am minutes from going to sleep.

Right now I have a feeling that you threw a bunch of money at something without enough knowledge about how it is all supposed to work together for a particular purpose. That is going to make advice hard to give without us knowing what you want to do and what you have to do it with. Once you get some things figured out part way post it up and we will try and help straighten it all out.

 

[Antairus]

The hardware has been purchased. We are moving away from another iSCSI solution that was all command line.
We want the GUI and reporting that comes with FreeNAS.
What we are doing is running VMs on an ESXi 6.5 box for cameras.

The FreeNAS (Storage) is just for archiving recorded data. It will do nothing else. We are doing this over 10gig NICs. This is directly from the VM BOX.

Before the LSI 9361 was set up in a RAID6 and it was split up in Centos for iSCSI. I will read up on your links. Any help is appreciated. Thanks.

 

[tvsjr]

Read @jgreco 's treatise on SAN vs NAS protocols here: https://forums.freenas.org/index.ph...res-more-resources-for-the-same-result.28178/
And remember, you're now dealing in ZFS world, which is a slightly different type of filesystem than what you've run in the past. It's paranoid about your data - but that paranoia comes with a price.

So, my first question is, do you *need* to run iSCSI? A file-based protocol like CIFS or NFS (depending on if your systems are Windows or *nix based) would be a substantially better choice.

You would also be advised to follow the forum rules and give us a complete rundown of your system configuration. That'll make everything easier.

 

[Antairus]

I have read all your suggestions. It is starting to make a little more sense.
However I am stuck with what I already have.
This is something I was thrown into, so I am on a learning curve.
I know my co workers that run the cameras want a GUI and Reporting. This is something they have never had before on a SAN.
Most cameras servers are run directly off of Server 2016 and The Mega Raid Storage Manager Does all the reporting they need.
So I suggested FreeNAS to them. Now its on my shoulders to figure it out. The previous person that did the CentOS Left the company.
The problem with the previous way is that drives would fail and nobody would know. This has lead to entire SANs being lost.
I know this is just a 36TB Server, However the 100TB is one of the SANs that were lost. I have to do both. I do not have the exact specs on the 100TB.

100TB
Intel Motherboard
Xeon CPU
32 Gig
LSI HBA
Intel 4X 1Gbps
Flash boot
That's all I can remember from looking at it briefly.
I do not remember the drive size or quantity.

I figure if I can get the below sorted out I can apply it to the 100TB that's just sitting there in limbo.

Here is this particular setup, including the hypervisor.

ESXi Server
Supermicro CSE-846BE16-R920B
Supermicro X10DRL-I
2x Xeon E5-2680v#
32gig ECC 2133 DDR 4
18 Seagate 15K SAS 600G
LSI 9361-8i W/BBU
Intel X520
Flash Boot

FreeNAS Server
Supermicro CSE-846BE16-R920B
Supermicro X10DRL-I
2x Xeon E52603v4
32gig ECC 2133 DDR 4
2x WD SSD Green 120GB
8x Seagate SAS 6TB
LSI 9300-8i
Intel X520
Flash Boot

I know I can get this up and running with the limited knowledge I have. I also know that it is not the correct way to do it.
This is what we are going to be doing in the future and I want to fully understand it. I don't want to be the "well just reload it that's the nature of it" Guy.

 

[tvsjr]

So first, I'm curious, what sort of "reporting" are they looking for? FreeNAS has a decent reporting engine, but it's all primarily aimed at the administrator keeping the system running, not at an end user type.

Your FreeNAS server is a very competent configuration. How many simultaneous streams are you trying to write to the pool? Depending on the quantity of simultaneous streams, you may need to go to striped mirrors to keep performance up. But, if it's a test system, you can play around with it. Start with a single 8-disk RAIDZ2 vdev and see how things behave.

I would definitely suggest trying to make CIFS/NFS work as your protocol, versus dealing with iSCSI. You will get rid of many of the stumbling blocks. If you aren't running iSCSI and storing VMs, you most likely won't need to worry about a SLOG.

Your LSI card needs to be flashed to be a pure HBA - otherwise known as IT mode. Basically, if it gives you the options to configure a RAID array, it's not in IT mode. FreeNAS relies on direct access to the disks, not through an abstraction layer.

With only 32GB RAM, an L2ARC will probably make things worse, not better, for you. Try the system out and, if you start seeing memory pressure issues (cache thrashing, etc.) then upgrade your memory. When you get above 64GB, then you can start *considering* L2ARC.

Your 100TB box may be a slightly bigger challenge, but it should work. Knowing quantity and types of drives installed will be required if you want help designing your pool.

 

[Antairus]

So first, I'm curious, what sort of "reporting" are they looking for? FreeNAS has a decent reporting engine, but it's all primarily aimed at the administrator keeping the system running, not at an end user type.

Your FreeNAS server is a very competent configuration. How many simultaneous streams are you trying to write to the pool? Depending on the quantity of simultaneous streams, you may need to go to striped mirrors to keep performance up. But, if it's a test system, you can play around with it. Start with a single 8-disk RAIDZ2 vdev and see how things behave.

I would definitely suggest trying to make CIFS/NFS work as your protocol, versus dealing with iSCSI. You will get rid of many of the stumbling blocks. If you aren't running iSCSI and storing VMs, you most likely won't need to worry about a SLOG.

Your LSI card needs to be flashed to be a pure HBA - otherwise known as IT mode. Basically, if it gives you the options to configure a RAID array, it's not in IT mode. FreeNAS relies on direct access to the disks, not through an abstraction layer.

With only 32GB RAM, an L2ARC will probably make things worse, not better, for you. Try the system out and, if you start seeing memory pressure issues (cache thrashing, etc.) then upgrade your memory. When you get above 64GB, then you can start *considering* L2ARC.

Your 100TB box may be a slightly bigger challenge, but it should work. Knowing quantity and types of drives installed will be required if you want help designing your pool.

Reporting Drive Failures. Where these servers are installed, no one notices a drive failure.
You know how many places I have been and the server is screaming because a drive has failed. I say hey you have a bad drive. Response.. Oh is that hat that is?
So we need that Email notification.

iSCSI is easy to setup and use I am just learning that there may be more overhead in using it.

The LSI card in the Storage Server is a 9300-8i That is an HBA.


I will try and get a better rundown of the 100TB unit.
Come to think of it, I dont even thinkits a Xeon.

 

[tvsjr]

Yuck... it's hard to believe *any* storage solution doesn't have notification. That said, no storage solution is 100% hands off - you still need to get in and look around from time to time. You also need to make sure you schedule things like SMART short and long tests, pool scrubs, etc. I would suggest a complete read-through of the manual.

 

[nightshade00013]

Reporting Drive Failures. Where these servers are installed, no one notices a drive failure.
You know how many places I have been and the server is screaming because a drive has failed. I say hey you have a bad drive. Response.. Oh is that hat that is?
So we need that Email notification.

iSCSI is easy to setup and use I am just learning that there may be more overhead in using it.

The LSI card in the Storage Server is a 9300-8i That is an HBA.


I will try and get a better rundown of the 100TB unit.
Come to think of it, I don't even thinkits a Xeon.

iSCSI is easy to set up but because FreeNAS doesn't know what is being written and read it has more issues dealing with the data which is the reason for the massive overhead. Using NFS alleviates a lot of that overhead as rather than dealing with just blocks of data it can deal with the files so it will have less problems with fragmentation.

Some discussion about iSCSI happened here and the masters have said

There is no "fix" accept to throw more hardware at it. That is the problem with copy on write file systems.

https://forums.freenas.org/index.php?threads/iscsi-performance-freenas-9-3-esxi6.39459/

Just because the LSI9300 is a HBA does not mean that it can't do raid. It can have two different firmware's installed on it and for FreeNAS to be happy it needs the IT mode firmware preferably the same version as the driver FreeNAS is using. https://www.broadcom.com/products/storage/host-bus-adapters/sas-9300-8i#downloads

FreeNAS can work wonders and survive a lot of things that would make some systems have issues. However it does have it's own set of quirks and you have to understand them to make a system work well. It's the same with a lot of storage systems out there and the people here have dealt with it for a long enough time to know what the best possible practices for most situations are. I have learned a ton in the last few years and have a lot more to learn from guys like @jgreco and @cyberjock just to name a few of the many. So the best thing I can suggest is to read, learn, ask questions, and provide us with as much information as you can about the situation that is why I asked everyone to hold off in the beginning.

 

[Antairus]

iSCSI is easy to set up but because FreeNAS doesn't know what is being written and read it has more issues dealing with the data which is the reason for the massive overhead. Using NFS alleviates a lot of that overhead as rather than dealing with just blocks of data it can deal with the files so it will have less problems with fragmentation.

I was just reading about that before i came back there to take a look.

Some discussion about iSCSI happened here and the masters have said https://forums.freenas.org/index.php?threads/iscsi-performance-freenas-9-3-esxi6.39459/

Just because the LSI9300 is a HBA does not mean that it can't do raid. It can have two different firmware's installed on it and for FreeNAS to be happy it needs the IT mode firmware preferably the same version as the driver FreeNAS is using. https://www.broadcom.com/products/storage/host-bus-adapters/sas-9300-8i#downloads

Yes True. Sorry. I will look into that. Thanks.

FreeNAS can work wonders and survive a lot of things that would make some systems have issues. However it does have it's own set of quirks and you have to understand them to make a system work well. It's the same with a lot of storage systems out there and the people here have dealt with it for a long enough time to know what the best possible practices for most situations are. I have learned a ton in the last few years and have a lot more to learn from guys like @jgreco and @cyberjock just to name a few of the many. So the best thing I can suggest is to read, learn, ask questions, and provide us with as much information as you can about the situation that is why I asked everyone to hold off in the beginning.

LOL. Understood and I am. I have learned a few things already.
I guess one thing I am not understanding is if I create a large RaidZ2 and break it up why there is a problem?
is there an article you can point me to? Nevermind. Just found out why at the bottom of the ZFS Primer.

I think I have figured out we are going to use an SMB share. The files are huge, and from what I understand I will get a little performance gain.

 

[Antairus]

Yuck... it's hard to believe *any* storage solution doesn't have notification. That said, no storage solution is 100% hands off - you still need to get in and look around from time to time. You also need to make sure you schedule things like SMART short and long tests, pool scrubs, etc. I would suggest a complete read-through of the manual.

Yes. This has been a huge issue. LSI/Broadcomm/Avago have the MegaCLI that will do it if it wants to. I have never got it to work reliably.
I will read some of the Manual but a complete read through?

 

[tvsjr]

Yes. This has been a huge issue. LSI/Broadcomm/Avago have the MegaCLI that will do it if it wants to. I have never got it to work reliably.
I will read some of the Manual but a complete read through?

Umm, yes? Combined with reading on here for awhile? This stuff is hard... FreeNAS is enterprise-class software. It's actually surprisingly user-friendly. But you still need to know what you're doing. And you're not fooling around with a simple little 2-drive affair backing up a few gigs of relatively unimportant data and your pr0nz library. You're talking about a system that your business/employer apparently relies on for day-to-day operation. Not everything is an iPhone, where it's designed for an 8-year-old to pick it up and operate it fully without a manual.

By the way... reading the manual and the stickied forum posts/resources here will a. help you do things right the first time and b. prevent snarky comments from experienced posters when you ask a question that's clearly answered in one of the aforementioned resources :)

 

[Antairus]

Umm, yes? Combined with reading on here for awhile? This stuff is hard... FreeNAS is enterprise-class software. It's actually surprisingly user-friendly. But you still need to know what you're doing. And you're not fooling around with a simple little 2-drive affair backing up a few gigs of relatively unimportant data and your pr0nz library. You're talking about a system that your business/employer apparently relies on for day-to-day operation. Not everything is an iPhone, where it's designed for an 8-year-old to pick it up and operate it fully without a manual.

By the way... reading the manual and the stickied forum posts/resources here will a. help you do things right the first time and b. prevent snarky comments from experienced posters when you ask a question that's clearly answered in one of the aforementioned resources :)

Yes Yes. Probably not front to back but whats pertinent at the time. I have been reading on ZFS for about 3 hours. Manual to links, links to Wiki.
Interesting stuff. Very Resilient.

 

[Antairus]

Is a 120gig WD green SSD good enough for a SLOG?

 

[tvsjr]

Is a 120gig WD green SSD good enough for a SLOG?

A proper SLOG doesn't need to be large (5 seconds at the highest data rate you can write... so if you have dual 10gig links, that's 2.5GB/sec * 5 sec = 12.5GB) but it needs to be ultra-fast and have power loss protection. Unfortunately, the PLP usually pushes you into enterprise-class drives.

The best choice out there is probably the Optane DC P4800X... but that's $1500-ish. A PCIe solution is preferable to SATA for speed, but many of us are running SATA devices and doing just fine.

I still say you need to consider how to get away from iSCSI LUNs and go to simple file sharing. You won't need a SLOG and you won't encounter all the other CoW pitfalls.

 

[Antairus]

A proper SLOG doesn't need to be large (5 seconds at the highest data rate you can write... so if you have dual 10gig links, that's 2.5GB/sec * 5 sec = 12.5GB) but it needs to be ultra-fast and have power loss protection. Unfortunately, the PLP usually pushes you into enterprise-class drives.

The best choice out there is probably the Optane DC P4800X... but that's $1500-ish. A PCIe solution is preferable to SATA for speed, but many of us are running SATA devices and doing just fine.

I still say you need to consider how to get away from iSCSI LUNs and go to simple file sharing. You won't need a SLOG and you won't encounter all the other CoW pitfalls.

Ok. That’s what I am doing. I am sure I read that last night somewhere. I am so used to iSCSI that’s what I continually think. Thanks.

 

[Antairus]

So this is what I have for testing. is this a correct configuration for an SMB/CIFS Share?