Plans to build a 12 drive SSD array

[Windows7ge]

Try a ramdisk slog. Will show if a slog would make a difference.

https://forums.freenas.org/index.php?threads/testing-the-benefits-of-slog-using-a-ram-disk.56561/

From my understanding SLOGs are only good for sync writes. We used the argument --direct=1 which is suppose to stop the ram from caching the file but what I'm going to do is write a script to test a series of configurations using varying block-sizes and I'll have them write a file larger than the amount of ram I have which should stop RAM caching from happening at all. I'll post back with the results.

 

[Windows7ge]

Performance testing is finished:


Sync standard | Sequential Reads/Writes | iodepth 8 | size 400gb | direct 1 | runtime 4 minutes
4K 8K 16K 32K 64K 128K 256K 512K 1M 2M 4M
writes: 375MB/s | 528MB/s | 843MB/s | 868MB/s | 744MB/s | 1508MB/s | 1571MB/s | 1598MB/s | 1623MB/s | 1638MB/s | 1556MB/s
reads: 295MB/s | 586MB/s | 386MB/s | 745MB/s | 533MB/s | 636MB/s | 721MB/s | 800MB/s | 877MB/s | 886MB/s | 857MB/s

Sync standard | Random Reads/Writes | iodepth 8 | size 400gb | direct 1 | runtime 4 minutes
4K 8K 16K 32K 64K 128K 256K 512K 1M 2M 4M
writes: 244MB/s | 409MB/s | 749MB/s | 136MB/s | 254MB/s | 967MB/s | 1192MB/s | 1289MB/s | 1377MB/s | 1450MB/s | 1500MB/s
reads: 7MB/s | 22MB/s | 23MB/s | 45MB/s | 89MB/s | 173MB/s | 280MB/s | 407MB/s | 489MB/s | 491MB/s | 492MB/s


From my understanding most drives that you don't manually configure default to 4k. How would I go about changing the block size manually? Would it need to be done on each drive individually or on the pool itself? Before taking on that endeavor can it even be said that as a file server that this will show any performance increase at all? If I were running VMs or a Database then I should expect yes but I'm not.

 

[Stux]

Numbers seem very low for 12 SSDs. How is the pool configured?

 

[Windows7ge]

Numbers seem very low for 12 SSDs. How is the pool configured?

raidz2

 

[Stux]

raidz2

12 way?

 

[Windows7ge]

12 way?

Yes. 9600GB usable 1920GB parity.

I have to ask. What do you think about block size? I've been told and read that the block size of the pool is usually 128K and isn't something worth changing but what about the drives themselves? According to S.M.A.R.T. mine are 512 bytes logical, 4096 bytes physical. If I'm just copying or moving data to/from a network share would increasing the sector-size of the drives help results?

 

[Stux]

If you want to help the results, try dual 6-way raidz2

As I mentioned, I see better results with 6 HDs. Namely over a gig/s

 

[Windows7ge]

If you want to help the results, try dual 6-way raidz2

Unfortunately before assembling the array I tested RAID0 across all 12 drives and only saw a brief peak above 900MB/s (the actual in use performance) all other performance numbers were below it so a 12 drive RAID60 I don't think would change the results much. Not to mention losing 4 disks. What I find most strange is that an SSD can do 500+MB/s yet this array is behaving like each drive is pushing less than 100. Each drive has its own SATAIII port. I'm starting to think per drive block size doesn't matter because of how low of latency SSDs are. I'm completely lost. Talking to others it's assumed this is the best ZFS has to offer full SSD arrays. Bummer. Unless I'm willing to drop FreeNAS and use a different file system then this is the best I'll see.

 

[Windows7ge]

I will say though sequential read performance is around 440,000 IOPS. They scaled fairly well.

Addressing another issue but one of which should be much easier to solve is how to setup the backup array? sretalla mentioned snapshots & replication. Both of which I have no idea how to use. Should I just refer to the FreeNAS User Guide and come back if I have questions that it doesn't answer?

 

[Chris Moore]

Addressing another issue but one of which should be much easier to solve is how to setup the backup array? sretalla mentioned snapshots & replication. Both of which I have no idea how to use. Should I just refer to the FreeNAS User Guide and come back if I have questions that it doesn't answer?

This guide might help:
http://thesolving.com/storage/how-to-schedule-periodic-snapshots-of-a-zfs-volume-with-freenas/

 

[Windows7ge]

This guide might help:
http://thesolving.com/storage/how-to-schedule-periodic-snapshots-of-a-zfs-volume-with-freenas/

That was helpful for the first 1/2 of it but from my understanding this is only helpful for when I want to roll back to an earlier version of the files on the array. Now should I lose the array either from some sort of software corruption or an HBA going bad and corrupting data I don't know if snapshots help that. From my understanding that's where replication would come into play and create a copy of all the data. I have my old mechanical array to back up to. Do correct me though. I've never used these tools before.

 

[Chris Moore]

That was helpful for the first 1/2 of it but from my understanding this is only helpful for when I want to roll back to an earlier version of the files on the array.

With proper use, you can mount a snapshot to retrieve individual files that have become corrupted or delted or you can (as you say) roll the entire snapshot back to an earlier time. This is useful but only if the pool is still working. For disaster recovery, if you want to make a copy on another system, you have a couple options. If the other system is not a ZFS file system, you can't use snapshots, so the first question should have been, what kind of system is the backup target? Windows, Linux, Mac?

 

[Windows7ge]

With proper use, you can mount a snapshot to retrieve individual files that have become corrupted or delted or you can (as you say) roll the entire snapshot back to an earlier time. This is useful but only if the pool is still working. For disaster recovery, if you want to make a copy on another system, you have a couple options. If the other system is not a ZFS file system, you can't use snapshots, so the first question should have been, what kind of system is the backup target? Windows, Linux, Mac?

My plan would be to use the HDD 8x2TB raidz2 array that I was using previously and to have both on the same server. I'm well aware having a backup on the same system isn't ideal but I'd say it's better than no backup at all.

 

[Stux]

Then you just replicate one to the other.

 

[Windows7ge]

Then you just replicate one to the other.

The Replication Task only speaks of copying the snapshots. Does it actually copy all of the data within the array?

Also it has parameters that I don't fully understand such as:
Recursively replicate child dataset's snapshots
Delete stale snapshots on remote system
Setup mode
I assume remote host would be 127.0.0.1 for this configuration
Would encryption even be worth enabling if it's writing to a second array on the same system?

 

[Stux]

If you have have a periodic snapshot covering all the data, and replicate that, then it will periodically replicate all the data in the array.

 

[Windows7ge]

If you have have a periodic snapshot covering all the data, and replicate that, then it will periodically replicate all the data in the array.

Using the guide Chris Moore linked I pointed the snapshot to cover the whole array. As for replication, so it does effectively act as a backup just not in the traditional sense. That's fine then. I'll just need to figure out how to set it up since it seems it was designed with the express intention of being used with separate network devices.

 

[Stux]

I believe you can specify localhost. It’d be more efficient to use a local replication command directly, and that’d operate at disk speed, but you get a lot of benefits from the FreeNAS layer, even though the replication would be significantly slower going through networking layers.

 

[Windows7ge]

I believe you can specify localhost. It’d be more efficient to use a local replication command directly, and that’d operate at disk speed, but you get a lot of benefits from the FreeNAS layer, even though the replication would be significantly slower going through networking layers.

I set it up the best way I could figure out. I'll give it until tomorrow then remote in and check if any files made it to the array. I don't expect the files to look normal I just want to see that something was copied there successfully. That should mean that it's working.

 

[Windows7ge]

I believe you can specify localhost. It’d be more efficient to use a local replication command directly, and that’d operate at disk speed, but you get a lot of benefits from the FreeNAS layer, even though the replication would be significantly slower going through networking layers.

Well look at that. I didn't even have to wait until tomorrow to know it failed.

Replication Storage -> 127.0.0.1:Storage-Backup failed: Failed: No ECDSA host key is known for [127.0.0.1]:8086 and you have requested strict checking. Host key verification failed.

Any ideas?