Question - How to build a fast FreeNAS system ? - Multiple IT-Mode SAS controllers.

[tjones-cdatribe-nsn.gov]

Question - How to build a fast FreeNAS system ? - Multiple IT-Mode SAS controllers.

Would multiple IT-Mode SAS ( 12-Gig transfer ) controller cards ( lets say 4 different controller cards ) - where each controller card has their own physical disk drives give me a thermotical four times 12-Gig ( 48-Gig ) possible max transfer rate if my ZFS file-system is configured as one massive ZFS pool ?

I'm considering building another couple of FreeNAS system ( in addition to my existing dozen FreeNAS systems ). I am wanting speed and the ability Rsync 50 to 500TB+ TB daily in less than 24 hours to another FreeNAS system.

So what hardware is the fastest for what I want to do ?

North Idaho Tom Jones

 

[tjones-cdatribe-nsn.gov]

Friggen spell check , I wanted to type " … a theoretical four times … "

 

[Samuel Tai]

At the speeds you're talking about, you need to start worrying about PCI-E contention between controllers, considering a single 16x link tops out at 32 GB/s, and channel limits may force you to slice that to 8x/8x or 4x/4x/4x/4x to handle multiple slots.

 

[Ericloewe]

A few things:

• A single typical SAS3 controller has 8x 12 Gb/s bandwidth to the disks
• You'd need serious SSDs to saturate that to the point of needing more controllers and you can probably forget about expanders, at which point NVMe seems to make a lot more sense overall
• rsync is always going to be painfully slow... If you can live with data flowing in a single direction, ZFS replication would be much faster
• rsync would probably be rather CPU-bound at that sort of throughput
• Four disks is hardly what I or anyone would call a "massive" pool. Less than 12 disks is a small pool, 36 is a medium pool, 72 is large and 100+ is massive.
• 47.something Gb/s puts you in "very expensive" territory when it comes to networking. 40 GbE and 25 GbE are pretty common for the high-end, but going beyond that gets really expensive really quickly

One question I have is what kind of insane workload needs 48 Gb/s of writes 24/7.

 

[tjones-cdatribe-nsn.gov]

Re: One question I have is what kind of insane workload needs 48 Gb/s of writes 24/7

I've got 50 1 TB+ servers. Each server has at least two different types of daily backups to two locations. Now I am being asked to create an additional two new 256 TB servers. We are in the process of migrating all databases from dozens of departments into one common database server. The second server is a development clone server for testing program updates prior to moving the updates to the live server. So , I am wanting to build a FreeNAS system that can perform a full Rsync of these servers in a nightly maintenance window ( and also have the throughput for Backup-Exec and/or Veem backup software to also do the same. I generally rotate through a dozen full backup sets prior to re-writing over older backups.
Sooo, I am looking for ideas how to build the fastest FreeNAS systems where each FreeNAS system will have about 96 16-TB drives. ( or larger/more ). I plan on building at least two of these systems this year.
EDIT: I've also got some FreeNAS S3 systems with millions of files that also need to be daily backed up. A simple "find . -print" can take hours.

 

[garm]

Now I haven’t used anything on this scale, but holy IOPS Batman.. there are some real potential for Lean here.. instead of copying entire systems daily, start using snapshots? ZFS send/receive will be a lot more efficient then going through all those systems file by file

 

[tjones-cdatribe-nsn.gov]

… ZFS send/receive will be a lot more efficient then going through all those systems file by file

I have nothing against lean and more efficient.
Not on my watch - but I've seen some pretty costly problems come up when multirole things break, get infected or there is a need to go back in history weeks or months or years to find something. So I am all for two or more different types of backups and locate those backups in two or more locations. If everything in a server room burns to the ground, then you have two other remote locations to quickly bring everything back on-line. I never want to be the admin that everybody blames when things are at their worst.

 

[garm]

I don’t think we are talking about the same things

 

[Ericloewe]

How much data actually changes daily? Using snapshots and replication would eliminate much of the pain rsync inflicts when dealing with a lot of data.

 

[garm]

Reading this makes me think of a Swedish car manufacturer who got bought by an American giant. Prior to being bought they had developed a new drivetrain with astonishing efficiancy. Coming from the aviation industry, the car development was rapid and always pushing the envelope. On being acquired by their new owners, the head of development traveled with some key engineers to motor city to show case their new platform and drivetrain. The feedback they got from the Americans where “why such a complicated engine? Extend the range by fitting a larger tank!”

where I work we don’t use ZFS in large scale, but I’m going to amalgamate my own experience with ZFS from the past decade and my professional situation to give a theoretical picture of the glory that is ZFS. A Copy-on-Write file system has many benefits, one is the ability to stop old blocks from being deleted as new blocks are written due to changes to existing files or the creation of new files. The process by which this achieved is called snapshots, it’s a frozen point in time that is accessible for as long as the snapshot exists, regardless of what is done afterwards.

the systems I administer (I’m not in ops, I’m an admin) is not HA by design, that has been tacked on later. This means we have multiple single points of failure that we need to eliminate with creative systems design. All our datacenters are geographically separated, when multiple servers are being used we split them up between the datacenters so that if one site failed, we have enough redundancy to continue uninterrupted on a single site. Backups are stored in the opposite site as well. Now, we have very big systems running with very large storage needs. To shuffle that data around several kilometers between sites simply isn’t feasible. It would take us weeks to replicate a whole site, that is instead done the old fashioned way, by truck. To backup our servers, we take periodical snapshots and send those to the backup target location. Let’s say we generate 10 TB of data in a day, our site-to-site connection is 10 Gbps fiber optics. We are a global company with 16-18 hours of operations per day. Data is generated in a bell curve with the most data generated around the 12 hour mark. if we saturate our site-to-site (this wont happen as services also utilize the link) we can send 10TB in just over 2 hours. Sending snapshots between sites every 15 minutes is no issue at all, while sending a complete system image would take several days per system, at least the bigger once.