Ubuntu, mysql, nvme ssd and Truenas

[john60]

I would appreciate some advice on how to incorporate Truenas capabilities into my environment. I'm a newbie and waiting for a server to arrive to start experimenting with Truenas.

What I am hoping for is that Truenas will be able to give me:
- rollback on files
- resiliency of SSD/disk failures
- automate backup to offsite
without killing my file access performance

My hobby application (deep learning) has a few operating modes.
Mode 1: training: Ubuntu app reads about 1 TB of data across a million files as fast as possible continuously for a week, feeding them to multiple GPUs for training where GPU load is 100% for the week and room gets nice and hot. Files on nvme SSD helps performance. So does storing the most frequently access files to a large part of my 256G memory.

• Deficiency #1: snapshots of this data is only 5, based on external devices and google drive backup. Would love more snapshots. Would love to have these snapshots available for perusing online. Right now its pull a disk from my safe, plug it in, look at the files, get next disk...
• Deficiency #2: dropbox and google drive is painfully slow and accomplished with sketchy apps. On the plus side, only modified files are uploaded.

Mode 2: dataset creations: Some files are updated (small percentage). The user interface needs a snappy response reading a bunch of files, so nvme SSD is a plus but maybe an overkill.

Mode 3: MySQL operation nvme SSD makes database operation nice and fast, not sure if it's necessary.

• Deficiency is the failure of nvme would mean data loss up to the last dump.
• Would love a resilient volume. Not sure nvme necessary for SQL storage.

Mode 4 My-SQL db-backup: Every week or so a backup of the SQL database to external disk/google drive.

My thoughts are (but I have zero Truenas experience):

• mode 2, put 1st nvme on truenas and mirror to hard disk. Change mode 2 (file creation) to use the Truenas hosted data over the network.
• mode 1, have Truenas dump data to a 2nd nvme for training so it can directly use the 2nd nvme. Training only needs a read only view of this data.
• mode 3, move SQL disk to 1st nvme and access over the lan. SQL dumps onto a disk-based volume over the lan.

Alternative thought:

• proxmox truenas on a VM on same server as Mode 1,2,3 where truenas mirrors across 2 nvme's. I am assuming virtual lans between both VM will be infinite bw and not latency.
• An external truenas based on hard disks mirrors or backups the VM based Truenas.
• External truenas does dumping to offsite.

Question: How would a Truenas expert set this up?

 

[john60]

I just read about zfs experimental on ubuntu 19.x and 20.x
Could Truenas snapshot from ubuntu zfs, and therefore keep nvme SSD on ubuntu and gain Truenas functionality?
This would be really nice since nvme as so fast and would be a shame to loose this speed.

Another thought/question: If Truenas runs as a VM in proxmox and same server as ubuntu vm, would a mount from ubuntu VM have unlimited bw since no physical ethernet cable? If so, this would be like having a robust ZFS files system for ubuntu rather than an experimental version.
Then replication to a harddisk Truenas would be for recovery in case of an nvme pair failure. Google/dropbox would be for site recovery if I get hit my lightning.

Again, I am a Truenas newbie, so would love some experts opinions.

 

[sretalla]

If Truenas runs as a VM in proxmox and same server as ubuntu vm, would a mount from ubuntu VM have unlimited bw since no physical ethernet cable?

The driver is the limit there (10Gbit is the theoretical maximum if you can set it correctly on both, reality likely slower than that, but still fast-ish).

this would be like having a robust ZFS files system for ubuntu rather than an experimental version.

You can install the same OpenZFS version on Ubuntu if that's your only goal.

mode 2, put 1st nvme on truenas and mirror to hard disk. Change mode 2 (file creation) to use the Truenas hosted data over the network.

A mirror will only be as fast as the slowest disk... not a great result in this case. Maybe local replication from NVME to HDD with frequent (ideally small) snapshots would be sufficient.

Mode 4 My-SQL db-backup: Every week or so a backup of the SQL database to external disk/google drive.

Cloud sync tasks can do that.


Overall, you've correctly identified that you have several different workloads and that you'll need to match those to the capabilities of the different pools that you can create in order to have the best out of your hardware.

ZFS will do a good job only if you give it the right working conditions, so make use of things like a metadata VDEV to speed up the access to large numbers of small files while not necessarily needing to store them all on fast media.

 

[john60]

The driver is the limit there (10Gbit is the theoretical maximum if you can set it correctly on both, reality likely slower than that, but still fast-ish).

My goal is to have efficient snapshots from my nvme drive so it integrates nicely into Truenas.
What is your opinion of rsync from (nvme on ubuntu ext4) to (truenas zfs hard drive)
vs
(truenas nvme on a VM) to (truenas zfs hard drive)?
If similar, no need to mess with a VM for Truenas.

You can install the same OpenZFS version on Ubuntu if that's your only goal.

Documentation made ZFS on ubuntu not fully backed. Hence the idea of a Truenas VM to effectively give a zfs mount.

ZFS will do a good job only if you give it the right working conditions, so make use of things like a metadata VDEV to speed up the access to large numbers of small files while not necessarily needing to store them all on fast media.

I will need to find documentation of this "metadata".

Thank you for the information, I appreciate it.