transparent file migration between slow and fast devices/storage?

[relume]

Hello

I have read the ZFS primer about caching and performance improvements with ZIL/SLOG and/or L2ARC. While those caching "concepts" are valid for their itended purpose, I would like to ask if it would be feasible to implement with FreeNAS a transparent usage based file migration between slow an fast devices/storage?

The background of my question is twofold:

1. We have supermicro twin server nodes that can accommodate maximal 3 * 3.5" or 6 * 2.5" HDs.
   
2. We have some 4TB of data increasing (file based) company data (yes we are a small office), only a smaller part (0.5 - 1.0TB) of this project based data is "hot", whereas the remaining data is "cold/tepid". For these reason it would be convinient, that "hot" data that is used actually would be hold in a fast storage (SSD based) and the "cold/tepid" data would be hold on a slow storage (HD based). As our data is organized by year and project a manual move of the data between fast and slow storage is not very convenient. Thus a transparent usage based file migration would be very interessting.

So our ideal storage would be composed of one SSD (about 1 or 2TB) and two HD (about 8 or 10TB as a mirror) where "hot" data (files) hold on the SSD is transparently also hold on the HD-mirror and moved away from the SSD if a file is not used for a certain time (to be determined from 1 to serveral days). In the other direction, if a file on the HD-mirror is "cold/tepid" and used more than once in a hour or day the file is copied/migrated to the fast SSD based storage. Would this be feasible with FreeNAS based on the ZFS?

Many thanks in advance for any suggestions


P.S. A second server node with a RaidZ1 (HD based) configuration would act as an replication storage.

 

[Chris Moore]

Hello

I have read the ZFS primer about caching and performance improvements with ZIL/SLOG and/or L2ARC. While those caching "concepts" are valid for their itended purpose, I would like to ask if it would be feasible to implement with FreeNAS a transparent usage based file migration between slow an fast devices/storage?

The background of my question is twofold:

1. We have supermicro twin server nodes that can accommodate maximal 3 * 3.5" or 6 * 2.5" HDs.
   
2. We have some 4TB of data increasing (file based) company data (yes we are a small office), only a smaller part (0.5 - 1.0TB) of this project based data is "hot", whereas the remaining data is "cold/tepid". For these reason it would be convinient, that "hot" data that is used actually would be hold in a fast storage (SSD based) and the "cold/tepid" data would be hold on a slow storage (HD based). As our data is organized by year and project a manual move of the data between fast and slow storage is not very convenient. Thus a transparent usage based file migration would be very interessting.

So our ideal storage would be composed of one SSD (about 1 or 2TB) and two HD (about 8 or 10TB as a mirror) where "hot" data (files) hold on the SSD is transparently also hold on the HD-mirror and moved away from the SSD if a file is not used for a certain time (to be determined from 1 to serveral days). In the other direction, if a file on the HD-mirror is "cold/tepid" and used more than once in a hour or day the file is copied/migrated to the fast SSD based storage. Would this be feasible with FreeNAS based on the ZFS?

Many thanks in advance for any suggestions


P.S. A second server node with a RaidZ1 (HD based) configuration would act as an replication storage.

The functionality you describe is usually a feature of very expensive SAN systems like the one we have at work where data is automatically moved from tape to disk to SSD and back again based on demand.

You might figure that is not available on a free software packag.

Also, if you are not using 10GB network equipment, you will see no advantage for this.

If you insist on using the equipment you have, which is not ideal, your options are limited.

You can use six 2.5 inch drives at 4TB each in a RAID-Z2 array and get about the speed of an SSD. More drives will be faster, so you are doing yourself a disservice to use the system you have as it is not quite right.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Arwen]

When I worked at Storage Tek they had ASM, which basically was what @Chris Moore said, huge amount of tape with disk cache in front for hot data It did support 3 levels, so we could have used SSDs in front of the spinning disks.

Sun Microsystems had the product too, but they called it SAM-FS.

Note that this was server side, it ran under Solaris SPARC. Don't know if any other OS supported it.

 

[Chris Moore]

Sun Microsystems had the product too, but they called it SAM-FS.

Note that this was server side, it ran under Solaris SPARC. Don't know if any other OS supported it.

The SAN we have at work is by SUN/Oracle and although I participated in the lengthy process of installing it, I have no idea what the total price tag was, but I know it was very high.

So our ideal storage would be composed of one SSD

You wouldn't get a lot of speed from a single SSD, but the array of six drives I described should get you redundancy and consistent transfers in the 200MB/s range

 

[Chris Moore]

PS. The usable storage should be around 15TB.

 

[Adrian]

Back in the late 1960s ICT (later ICL) had an OS with a two level file store - GEORGE 3, running on 1900 series machines. Files were backed up to (reel to reel) tape. If free storage ran low some on line files would be deleted, and retrieved from tape when accessed. Files could also be retrieved asynchronously. Well written batch jobs started with retrieve commands for every file they were going to use.

 

[relume]

Hello

Thank you very much for your valuable considerations.
It seems that I have looked for a sports car feature, but want to afford only an Italian Ape. ;)

best regards