I am working on a storage project and I was wondering if anyone has some good ideas of what kind of app servers to run and how they should be networked (possibly dell?)? I'm also interested about which Nics to install on the app servers and freenas servers (I assume all copper NIC's?). to start with I plan on having 4 4u storage servers running freenas each having 24bays with 3TB per bay. on top of that will be 2 app servers and a layer 3 switch. I'm not exactly sure what this initial install will look like at the networking level in order to scale the storage and app servers properly. I am starting with a half rack 21u. I will need to add storage (freenas) servers on an ongoing basis to increase storage, but not for a year or so (half rack, full rack etc). any help would be greatly appreciated!
-
Important Announcement for The TrueNAS Community.
The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums
multiple large freenas servers in a colo.
- Thread starter rastrent
- Start date
- Status
jgreco
Resident Grinch
- Joined
- May 29, 2011
- Messages
- 18,680
You've provided very little to work with.
What's the application? What sort of "app servers" would normally be considered appropriate for the application? What's your upstream bandwidth? How much bandwidth are you expecting to push? Do you need multiple network segments? Redundancy? What do you think you need a layer 3 switch for? What sort of growth are you expecting? What other complications are there?
The point is that if you are getting a 10 megabit feed from an ISP in their colo center, that's very different than if you're setting up multiple gigabit links to various providers in a carrier-neutral colo and running BGP and a load balancer. It's different in pretty much every way. You can't start asking networking questions until it is clear what sort of networking requirements exist.
What's the application? What sort of "app servers" would normally be considered appropriate for the application? What's your upstream bandwidth? How much bandwidth are you expecting to push? Do you need multiple network segments? Redundancy? What do you think you need a layer 3 switch for? What sort of growth are you expecting? What other complications are there?
The point is that if you are getting a 10 megabit feed from an ISP in their colo center, that's very different than if you're setting up multiple gigabit links to various providers in a carrier-neutral colo and running BGP and a load balancer. It's different in pretty much every way. You can't start asking networking questions until it is clear what sort of networking requirements exist.
thanks for the help. let me clarify as much as I can.
the application is object storage (cloud storage) without replication outside of the rack itself, for now. I was thinking to use Unbuntu server to run on two dell app servers (2 for failover) to manage the users and their storage. I was thinking to use a 100mbps connection, it comes with unlimited usage. It is a carrier neutral colo. I'm not sure what the upload speed would actually be. as far as pushing bandwidth is concerned I chose a 100mbps connection anticipating large bandwidth usage and I can order a larger connection if need be as the user demand grows.
I expect to segment the users into 2 groups meaning one of the four 4u servers will be a segment for light users and the other three will be another segment for the heavy users. all of the servers are raid running 60. I assumed a switch would be needed to mange the app servers traffic? I expect to have hundreds of users for the first 12 months scaling up to the very low thousands over 12 to 24 months (5k).
the application is object storage (cloud storage) without replication outside of the rack itself, for now. I was thinking to use Unbuntu server to run on two dell app servers (2 for failover) to manage the users and their storage. I was thinking to use a 100mbps connection, it comes with unlimited usage. It is a carrier neutral colo. I'm not sure what the upload speed would actually be. as far as pushing bandwidth is concerned I chose a 100mbps connection anticipating large bandwidth usage and I can order a larger connection if need be as the user demand grows.
I expect to segment the users into 2 groups meaning one of the four 4u servers will be a segment for light users and the other three will be another segment for the heavy users. all of the servers are raid running 60. I assumed a switch would be needed to mange the app servers traffic? I expect to have hundreds of users for the first 12 months scaling up to the very low thousands over 12 to 24 months (5k).
jgreco
Resident Grinch
- Joined
- May 29, 2011
- Messages
- 18,680
Okay, so, here, let me hit you with this.
You're planning to set up four storage servers. Assuming your "RAID 60" translates into maybe RAIDZ2 with two groups of 10 drives (8 data, two parity) plus four spares, which may be excessively aggressive for your application. That's 16 data drives worth of space, or about 42TB of usable space per storage server, or 168TB total.
At 100Mbps, or a peak of 12.5MBytes/sec possible on your inbound connection, that's a maximum rate of about 1TB per day that can be written. Your space - ASSUMING you were able to max out your connection 24/7 starting right away - would take you about half a year to fill.
So, some free advice from someone who does this stuff for a living: rethink your business and growth plans. You are probably not well-served by going big to start. Get two 4U storage servers to start out, because you should never have just one of anything unless it's something unimportant. Doing that means that in the worst case scenario, you might be adding two more storage servers within a year, but you might not, and why waste money spinning drives that are empty? Save the capex and you will probably be able to make it stretch further in a few months, like when 4TB drives are the price 3TB's were just a little while ago.
As for the rest, pick up a pair of inexpensive managed 1Gbps switches. We're partial to the Dell 5324's for this sort of thing, they're solid and cheap on eBay. You will hopefully eventually outgrow them, at which point they'll still be useful devices for some parts of your network.
Have a competent programmer do your app development, and this specifically means NOT doing it in some crummy interpreted language like PHP. At least not for the heavy lifting.
Eventually you may want to contemplate ditching the app servers and using virtualization to run the NAS and app stuff on the same boxes; FreeNAS is lightweight on CPU and can be easy on the memory. That means a lot of spare CPU capacity can be floating around as you get up to several dozen storage servers. It helps to contemplate that up front, but you don't need to actually do anything about it. Easier to start things up the conventional way.
You're planning to set up four storage servers. Assuming your "RAID 60" translates into maybe RAIDZ2 with two groups of 10 drives (8 data, two parity) plus four spares, which may be excessively aggressive for your application. That's 16 data drives worth of space, or about 42TB of usable space per storage server, or 168TB total.
At 100Mbps, or a peak of 12.5MBytes/sec possible on your inbound connection, that's a maximum rate of about 1TB per day that can be written. Your space - ASSUMING you were able to max out your connection 24/7 starting right away - would take you about half a year to fill.
So, some free advice from someone who does this stuff for a living: rethink your business and growth plans. You are probably not well-served by going big to start. Get two 4U storage servers to start out, because you should never have just one of anything unless it's something unimportant. Doing that means that in the worst case scenario, you might be adding two more storage servers within a year, but you might not, and why waste money spinning drives that are empty? Save the capex and you will probably be able to make it stretch further in a few months, like when 4TB drives are the price 3TB's were just a little while ago.
As for the rest, pick up a pair of inexpensive managed 1Gbps switches. We're partial to the Dell 5324's for this sort of thing, they're solid and cheap on eBay. You will hopefully eventually outgrow them, at which point they'll still be useful devices for some parts of your network.
Have a competent programmer do your app development, and this specifically means NOT doing it in some crummy interpreted language like PHP. At least not for the heavy lifting.
Eventually you may want to contemplate ditching the app servers and using virtualization to run the NAS and app stuff on the same boxes; FreeNAS is lightweight on CPU and can be easy on the memory. That means a lot of spare CPU capacity can be floating around as you get up to several dozen storage servers. It helps to contemplate that up front, but you don't need to actually do anything about it. Easier to start things up the conventional way.
jgreco
Resident Grinch
- Joined
- May 29, 2011
- Messages
- 18,680
You're going to be sucking in data at speed (presumably). Do you want that to happen on a poorly-written, CPU-intensive PHP script that's needing to run through the interpreter for every kilobyte of data read/written, and starts eating 100% of all four of your cores when you are trying to run a few tens of megabits, or would you rather have a smooth, supple, well-designed bit of code that's barely consuming any CPU to do this very mundane task?
The point is, don't cheap out on the important stuff. Look at FreeNAS for example. It's one of the most horridly slow bits of interpreted code I've seen running on a blazing fast machine. That's okay, because that's mainly for administration and configuration. The *important* bits, the stuff that moves files around, that's all compiled C code, some of it even in the kernel itself, so it is going about as fast as reasonably possible on the available hardware. It won't stop me from cursing at the WebGUI that it feels as slow as an old 486. Point is, FreeNAS is basically a layer on top of FreeBSD that is configuring well-written high performance services. FreeNAS as a project would be dead if it was trying to implement those services in Python all by itself.
In that same manner, you need to make sure that what you're doing is going to work out going forward. It's probably fine to design lots of your infrastructure on something like PHP, but make sure you are getting well-written, compiled, efficient code to handle the heavy lifting of shoveling data around.
The better your code, the smaller and more efficient your app servers can reasonably be (and/or the more they can handle).
The point is, don't cheap out on the important stuff. Look at FreeNAS for example. It's one of the most horridly slow bits of interpreted code I've seen running on a blazing fast machine. That's okay, because that's mainly for administration and configuration. The *important* bits, the stuff that moves files around, that's all compiled C code, some of it even in the kernel itself, so it is going about as fast as reasonably possible on the available hardware. It won't stop me from cursing at the WebGUI that it feels as slow as an old 486. Point is, FreeNAS is basically a layer on top of FreeBSD that is configuring well-written high performance services. FreeNAS as a project would be dead if it was trying to implement those services in Python all by itself.
In that same manner, you need to make sure that what you're doing is going to work out going forward. It's probably fine to design lots of your infrastructure on something like PHP, but make sure you are getting well-written, compiled, efficient code to handle the heavy lifting of shoveling data around.
The better your code, the smaller and more efficient your app servers can reasonably be (and/or the more they can handle).
- Status
