Best setup for TrueNAS iSCSI storage device

[modernpaul]

I am needing a storage device to store our backup data on. The application I will be using is ARCserv which does a backup and then just keeps that original full and adds on the incrementals to make new fulls that will be backed up to tape on a weekly basis.

Since it will be using the original data backed up perpetually I want to make sure to stay clear of bitrot so I am thinking about foregoing the RAID enclosure with any of the RAID options and going with an iSCSI based TrueNAS system. I am going to need to back up around 50TB of data so I am planning on a system that will have around 100TB of usable space for future growth.

Would it be ok for me to do a RAIDZ2 or will the performance will be horrible? I will have the TrueNAS server and the backup server hosting the software connected via a 10GB ethernet or SFP+ connection. There will be SAS drives around 10 16TB drives in there (some for spares). The OS will be on a Raid1 set of drives and I am looking at 64GB per processor with dual Xeon 10 core processors.

Question is what suggestions would you make to improve performance on this device? I haven't ordered it yet but I am looking at a SuperMicro based 45bay 3.5" enclosure.

I also have 2 devices from ixSystems already ordered that I will be using as NAS servers and will be backing it up to this storage via the backup software along with some other Windows servers. ixSystem units are pretty pricey so I wanted to go with a non supported one with this. Any suggestions?

 

[ChrisRJ]

What is the position of your system integrator on this? They should have an in-depth understanding of your requirements, after all.

 

[Arwen]

For block storage, (aka iSCSI), you should keep the pool under 50% used. So for the short term, 50TB used on a 100TB pool is fine. However, you don't have your built in space for growth.

Yes, Mirrored vDevs are more appropriate for iSCSI block storage. It has to do with block size in the container and stripe width of the vDevs & pool, plus potentially smaller writes from foreign OS. Others have better way of describing this.

This proposed solution to bit rot is not quite one I would use. If ARCserve does not have a FreeBSD or Linux server version, then perhaps you either accept that you can't get bit rot detection and correction in that manor. Or write something within the constraints of the OS that ARCserve runs under. (Like keeping a separate set of checksums for the files and having a "scrub" program / script verify them when you want to.)

 

[modernpaul]

What is the position of your system integrator on this? They should have an in-depth understanding of your requirements, after all.

They don't know TrueNAS at all so they are not helpful there. As far as bitrot it seems like most people don't even think about it. From what ARCserve has told me most people just have it in a RAID 5 or 6 type of environment.

 

[modernpaul]

For block storage, (aka iSCSI), you should keep the pool under 50% used. So for the short term, 50TB used on a 100TB pool is fine. However, you don't have your built in space for growth.

Yes, Mirrored vDevs are more appropriate for iSCSI block storage. It has to do with block size in the container and stripe width of the vDevs & pool, plus potentially smaller writes from foreign OS. Others have better way of describing this.

This proposed solution to bit rot is not quite one I would use. If ARCserve does not have a FreeBSD or Linux server version, then perhaps you either accept that you can't get bit rot detection and correction in that manor. Or write something within the constraints of the OS that ARCserve runs under. (Like keeping a separate set of checksums for the files and having a "scrub" program / script verify them when you want to.)

Well ARCserve only runs under windows but the client can run on Linux but not FreeBSD. So you are saying just give up and accept the idea of bit rot then? I won't be able to write anything as I am not a programmer nor do we have any in our company that would be able to do that. There has to be another option or am I just going overboard about the idea of bit rot? I have experienced it though and the idea that the Full backup will always be based on the same static original data is making me a bit more worried about it. If the data was written freshly each month or couple of months that would be a different story.

 

[Arwen]

Well ARCserve only runs under windows but the client can run on Linux but not FreeBSD. So you are saying just give up and accept the idea of bit rot then? I won't be able to write anything as I am not a programmer nor do we have any in our company that would be able to do that. There has to be another option or am I just going overboard about the idea of bit rot? I have experienced it though and the idea that the Full backup will always be based on the same static original data is making me a bit more worried about it. If the data was written freshly each month or couple of months that would be a different story.

I too have had bit rot damage files that were irretrievable, (before I started using ZFS). Pretty annoying.

Sometimes there is no good solution to a problem.

In your case, using iSCSI off a TrueNAS server, to MS-Windows could be a bit Frankenstein like. Meaning, if you do have problems, it may be both hard to identify the cause. And possibly just as hard or harder to fix. Always plan on trouble shooting and repair when implementing something in production. We get several people a month here in the forums asking for help on either common failures they don't know how to fix. Or they designed in odd bits and then can't trouble shoot or fix the problem easily because of the odd bits.

Further, without a good understanding of ZFS & TrueNAS, you could loose your ARCserve backup. Either temporarily until you fix a minor problem, like the network between them. Or fix the TrueNAS server, which may require much longer down times. We have seen people implement iSCSI in a less then ideal way, then have problems.

I am not trying to be negative. Or suggest you not implement TrueNAS in this manor. It's that I want you prepared as best you can be.


Good luck.

 

[NugentS]

Do you have to use iSCSI? Can't you use NFS/SMB/WebDAV (insert as required) as the ArcServ storage? That would be easier to debug
On a large array like that with large disks use Z2 or Z3 in case a disk fails during resilver, following a previous disk failure.

 

[HoneyBadger]

They don't know TrueNAS at all so they are not helpful there.

From the perspective of your SI/vendor, they don't need to "know TrueNAS" but they need to "know storage" to the degree of being able to tell you the I/O behavior of their application. In general terms though, a backup workflow is "large, contiguous writes to files - in the case of versioned/containerized backups, new files every time - with minimal to no reads."

This speaks to the suggestion posed by @NugentS - is iSCSI a requirement specifically? Targeting a file-based export from TrueNAS will greatly simplify the system and reduce the "layers of abstraction" as well as take out the potential for corruption at the NTFS level - you'll simply be writing to ZFS over a NAS protocol. Most backup suites support a remote path (typically SMB if running Windows, NFS if running as a Linux-based "appliance") - some more information about your backup software could go a long way here.

 

[IOSonic]

Do you have to use iSCSI? Can't you use NFS/SMB/WebDAV (insert as required) as the ArcServ storage? That would be easier to debug
On a large array like that with large disks use Z2 or Z3 in case a disk fails during resilver, following a previous disk failure.

I don't know much about ARCserv individually, but if it uses enterprise features like changed block tracking or speeds up restores by only restoring changed blocks, he'll need a block storage target