NAS of Alexandria
Cadet
- Joined
- May 12, 2020
- Messages
- 2
Hi, I work for a small business, so we don't have much budget, and are trying to make the most of it with everything going on.
We currently process some data on workstations, shuttling data around via sneakernet with external SSDs, and make backups to external HDDs and cloud storage. We'd like to move the data storage to a nas, as we're virtualizing the workstations for software flexibility going forward and to enable work from home for our employees. We believe the NAS would help with some automation, such as performing some processing and scheduled backups to cloud and external HDDs and emailing when the external drives are full, and ready to be sent off. (We keep 3 mirrors, onsite, with the client, and at an off-site location)
Our data workflow would look something like this:
-Data arrives from field and is transferred from a workstation to nas (10-500GB, in 3-5GB files)
-A machine (vm or real) will take this data and perform a first pass process and data expands about 10-15x and is sent back to the bad (now as 30-70GB files, average ~5tb total)
-the now expanded dataset is read 1-3 times by other machine's software and derivative data is saved (additional 5-20GB files)
-second pass process occurs and creates 20-100GB of output data, which may be read as many as 20 times by other processes taking metrics and analytics from the data.
-the finalized data is written to the backups with metadata files containing info about the client,job results, and some other documents, and distributed to the client and backup locations.
We would like all of the transfers to be as close to the 10G (and in cases of link agg, 20&40G) wire speeds as possible to minimize transfer/wait times
Hardware we already have available for this, that after checking forums should be compatible:
the fiber, SFPs and cards
An existing supermicro 3u box containing an X9 motherboard with dual xeon e5-2600 v2 series (I'll check exact processor if necessary), 128gb ram, a LSI SAS3 HBA (IT mode) and SAS3 backplane.
Hardware we're considering:
Either 4 10TB SAS3 drives or 10 4TB SAS3 drives
Maybe an SSD cache, if advised (seems to be a consensus that more RAM is preferred)
I'm currently thinking that if we get the 10 4TBs we might put them in a mirror of two raidz1 vdevs, 5 disks each for some speed improvement
I'm aware an all flash or hybrid array would be better, but AFAIK it's out of the budget.
If this works well enough for now (given its limitations), and we make our growth targets, we are considering moving onward to a TrueNAS. Is it difficult to migrate?
Hope this wasn't too much of a read, I wanted to be thorough. I apologise for any mistakes or misunderstanding on my part.
We currently process some data on workstations, shuttling data around via sneakernet with external SSDs, and make backups to external HDDs and cloud storage. We'd like to move the data storage to a nas, as we're virtualizing the workstations for software flexibility going forward and to enable work from home for our employees. We believe the NAS would help with some automation, such as performing some processing and scheduled backups to cloud and external HDDs and emailing when the external drives are full, and ready to be sent off. (We keep 3 mirrors, onsite, with the client, and at an off-site location)
Our data workflow would look something like this:
-Data arrives from field and is transferred from a workstation to nas (10-500GB, in 3-5GB files)
-A machine (vm or real) will take this data and perform a first pass process and data expands about 10-15x and is sent back to the bad (now as 30-70GB files, average ~5tb total)
-the now expanded dataset is read 1-3 times by other machine's software and derivative data is saved (additional 5-20GB files)
-second pass process occurs and creates 20-100GB of output data, which may be read as many as 20 times by other processes taking metrics and analytics from the data.
-the finalized data is written to the backups with metadata files containing info about the client,job results, and some other documents, and distributed to the client and backup locations.
We would like all of the transfers to be as close to the 10G (and in cases of link agg, 20&40G) wire speeds as possible to minimize transfer/wait times
Hardware we already have available for this, that after checking forums should be compatible:
the fiber, SFPs and cards
An existing supermicro 3u box containing an X9 motherboard with dual xeon e5-2600 v2 series (I'll check exact processor if necessary), 128gb ram, a LSI SAS3 HBA (IT mode) and SAS3 backplane.
Hardware we're considering:
Either 4 10TB SAS3 drives or 10 4TB SAS3 drives
Maybe an SSD cache, if advised (seems to be a consensus that more RAM is preferred)
I'm currently thinking that if we get the 10 4TBs we might put them in a mirror of two raidz1 vdevs, 5 disks each for some speed improvement
I'm aware an all flash or hybrid array would be better, but AFAIK it's out of the budget.
If this works well enough for now (given its limitations), and we make our growth targets, we are considering moving onward to a TrueNAS. Is it difficult to migrate?
Hope this wasn't too much of a read, I wanted to be thorough. I apologise for any mistakes or misunderstanding on my part.
