TrueNas Core setup with iscsi and proxmox

[dkmcgowan]

I'm running TrueNAS Core 12 on a Dell R720xd. I have 12 drives in the system, 5.46TB, so I have a total of about 65TB. I have a separate 256GB SSD for the OS storage and boot.

Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz x 2
256 GB of RAM

I set a lot of this up based on some research, reading these forums. I am just curious and looking things over again.

I have 2 pools.

vdev0 - RAIDZ2 - 21TB
vdev1 - RADZ2 - 21TB

Mostly defaults on those pools, lz4 compression, dedup off.

I have a dataset on vdev0 with an NFS share.

I have a zvol on vdev1 at almost 17TB (80%) with an iscsi share in disk mode.

I have a dedicated fiber NIC that is the share for the iscsi with a second R730 Dell server running my proxmox. I share the iscsi over a cable to cable network on the fiber line. The NFS is shared across the main network and goes through the switch.

I use the NFS for VM images, so ISO files, and also backups. I use the iscsi primarily to store video files for my Plex server.

I have read about pitfalls of raidz2 vs mirrors when it comes to iscsi, but I have not had any performance issues up to this point. The NFS volume works just fine as well.

I guess my question is did I choose a decent setup? I know I got have just done one large NFS and it most likely would have been fine, thinking I should have, but I wanted to play with iscsi, the separate NICS, get that traffic off the switch, etc...

I'm using about 8TB of the 17TB on the iscsi at this point, kinda worried about when I want to take away from the NFS vdev0 and add more storage to iscsi, not sure how to do that or if it's possible, not researched that yet.

I didn't want to do mirrors because I was willing to give on some performance for more storage. This is generally just home usage at this point, so a couple of TV's running, maybe 1 person remote at times, nothing major.

All the compute, local storage, buffer cache, is all on the second server, the R730, so this R720 and TrueNAS is just doing storage.

I was curious about the 80% thing as well. Was I supposed to do that on the zvol or fill that up and just monitor my actual storage from proxmox? I think I did it right on the zvol, but wasn't sure.

Also curious about expanding things and if I made some bad mistakes along the way.

Thanks!
David

 

[sretalla]

I was curious about the 80% thing as well. Was I supposed to do that on the zvol or fill that up and just monitor my actual storage from proxmox? I think I did it right on the zvol, but wasn't sure.

You would possibly be better off keeping it to 50%...

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

You're already on the wrong side of the line by using RAIDZ so you probably won't want to be over-filling your block storage pool to make it even worse.

 

[dkmcgowan]

Yeah, that's what I was thinking after reading more. I don't have a ton of data right now so now is the time to change things. I'm debating on switching out the vdev1 to a mirror and rebuilding the iscsi drive, or just taking all 12 drives into a raidz2 NFS share... I keep reading that for my use case I most likely will not notice any difference in NFS vs iscsi, will be easier to manage, and more storage...

 

[jgreco]

NFS vs iscsi, will be easier to manage, and more storage

Block storage works the same way for NFS as it does for iSCSI. There is nothing about NFS that would give you "more storage". You should be avoiding RAIDZ in both cases.

 

[dkmcgowan]

Didn't know that, for some reason I thought Raidz2 was very common with NFS... I know there are benefits to mirrors, but didn't realize raidz was that bad, seems your are saying raidz is bad pretty much in all cases and to always use mirrors, whether it's for smb, nfs, iscsi doesn't matter?

 

[jgreco]

Raidz2 was very common with NFS

Of course RAIDZ2 is very common with NFS. That's not the issue here. @sretalla already provided a link to a resource that explains the issue in significant detail, as it relates to you. Please do go read it.

 

[dkmcgowan]

ok, found this and read it https://www.truenas.com/community/r...and-why-we-use-mirrors-for-block-storage.112/

Now it seems like I should maybe go to mirrors and do it all in NFS and should have never done any raidz...

It is mainly files, so not VM data or anything, that's why when I first read about this stuff I thought raidz was ok... I use this for music, video, iso, backups, and document storage primarily, no VM storage or anything like that

 

[dkmcgowan]

Of course RAIDZ2 is very common with NFS. That's not the issue here. @sretalla already provided a link to a resource that explains the issue in significant detail, as it relates to you. Please do go read it.

Thanks, just read it, lots of info there, trying to consume it

 

[jgreco]

It is mainly files, so not VM data or anything

Okay then I'm confused, I guess. I saw

I have a zvol on vdev1 at almost 17TB (80%) with an iscsi share

But I missed

I use the iscsi primarily to store video files for my Plex server.

Probably because the common theme in these things is that people optimistically use iSCSI with RAIDZ2 and then are horrified at the results, and I betcha I see several of these a week. I don't always fully re-read the beginning of a thread for every reply, either. My bad, sorry. You were less screwed up than I thought.

Your results may not be terrible for performance, if you're not actually doing overwrites (common for databases and VM storage), but you are still potentially going to be hurting because of the way RAIDZ allocates parity for data blocks on a zvol and some other random suboptimalities.

Is there some particular reason you have chosen this difficult path? Because you were correct when you said, above,

Raidz2 was very common with NFS.

ZFS does two things very well. Mirrors for ANY kind of storage are always highly performant, and very flexible, but suck at space efficiency. RAIDZ is great for large file storage, think "archival", "backups", "iso's", but NOT "large VM disk files" or anything with lots of random I/O or file overwrites. And also RAIDZ is less performant. Those are the two big ZFS use cases. All other use cases have to basically decide whether they more closely resemble mirror or RAIDZ use cases.

The problem with what you are doing is that iSCSI zvol usage is pre-optimized towards mirrors. It is not IMPOSSIBLE to make zvols work okay with RAIDZ, but it's difficult. If it makes MY head hurt, then that's not going to be good for average users to try.

So my question is, why are you trying to store video files this way? What's wrong with NFS (for FreeBSD/Linux) or SMB (for Windows)?

 

[dkmcgowan]

Thanks for the feedback, learning more for sure... I read a lot when I started a year ago and landed on my VM storage with local mirrored SSD's, so that works great and is fast...

These two vdevs are just for storage, so I did one in NFS and one in iscsi, so 6 drives each on a 12 drive system because I ready making them to large (> 8 drives) was not good...

I ended with one with NFS and one with iscsi, both raidz2 because no VM or databases...

Why did I pick iscsi? I was just learning, that's all, which is kinda why now I want to change and set it up right...

So I'm thinking about switching them both to mirrors, would lose 1 drive per 6 drive pool, so 5.46TB, not that bad, and then setup NFS on both of them, use one for documents, files, iso, backups, and then the second for my media storage, that seems like it would be the easiest setup...

 

[dkmcgowan]

I was going to add I know my use case actually fits with raidz but I care more about performance than storage, so that pushes me towards mirrors, not to mention I like the idea of faster recovery on a failure