David Buchanan
Dabbler
- Joined
- Dec 19, 2014
- Messages
- 10
Hi All,
Let me start off by saying that I've tried to do my research, so hopefully my post comes across as such and not as a new user asking stupid questions. :)
I'll give you a little back ground as to why I'm looking into FreeNAS and using de-dup.
We have a HP Blade Center backed by a Fibre Channel P2000 G3 with 140TB of storage. We have one bare-metal blade running Server 2012 R2 with 70TB of storage running Windows Data De-duplication (Post processing). This server stores around 130TB of Veeam backups on this 70TB volume.
We are at the stage where Veeam is producing a very large amount of I/O as well as the Windows dedup with it's post processing de-dup.
Our long term strategy is to move to something like a HP StoreOne. However this is 6-12 months off, plus I'm looking into other alternatives. This has lead me to FreeNAS (Which I currently use in a number of smaller sites).
So, here is my FreeNAS plan:
HP BL460 G7 Blade with 2 x Intel Xeon E5645 (6 core with HT, 24 threads total) and 196GB of Memory.
Currently the server has a 4GB SD card in it, however I believe it is now recommended to replace this with at least an 8GB one.
Because of the volume of data and the fact that such large amounts of it are similar we need to run de-dup (Although I'm open to suggestions on compression). Just to confirm, FreeNAS is in-line data de-duplication?
Based on my research it's recommended to have somewhere between 2GB-10GB of memory per 1TB of data. What I'm not sure on is this per TB of written data (eg: 70TB) or the amount of data before being de-dup (eg: 130TB)?
If it's 70TB then the 196GB would be enough if I use the 2GB per TB, it puts me somewhere around the 2.5GB per TB. (Leaving some memory for other system services). If it's 130TB then I have less than 1.5GB per TB.
My biggest concern here is if we run out of memory then the volume can't be mounted. However, based on this article (https://blogs.oracle.com/bonwick/entry/zfs_dedup) it appears that some of the de-dup table can be moved to L2ARC and then to disk at the cost of performance, is this correct for FreeNAS? I'd rather take a hit on performance than not have access to any data.
The next part which I'm a little more unsure on is the best way to set-up FreeNAS with the P2000. I know that FreeNAS likes direct access to the disks, however I'm not sure how that's possible with a SAN. I assume I'd just create a LUN and assign it to the server as I would any other set-up and FreeNAS would use it without direct access to the drives. Is this correct?
For a system with 100TB+ of data would it be recommended to use L2ARC?
I welcome any advice on the above design good or bad. Hopefully someone with more experience with FreeNAS in the enterprise space can provide me with some insight.
Thanks,
David.
Let me start off by saying that I've tried to do my research, so hopefully my post comes across as such and not as a new user asking stupid questions. :)
I'll give you a little back ground as to why I'm looking into FreeNAS and using de-dup.
We have a HP Blade Center backed by a Fibre Channel P2000 G3 with 140TB of storage. We have one bare-metal blade running Server 2012 R2 with 70TB of storage running Windows Data De-duplication (Post processing). This server stores around 130TB of Veeam backups on this 70TB volume.
We are at the stage where Veeam is producing a very large amount of I/O as well as the Windows dedup with it's post processing de-dup.
Our long term strategy is to move to something like a HP StoreOne. However this is 6-12 months off, plus I'm looking into other alternatives. This has lead me to FreeNAS (Which I currently use in a number of smaller sites).
So, here is my FreeNAS plan:
HP BL460 G7 Blade with 2 x Intel Xeon E5645 (6 core with HT, 24 threads total) and 196GB of Memory.
Currently the server has a 4GB SD card in it, however I believe it is now recommended to replace this with at least an 8GB one.
Because of the volume of data and the fact that such large amounts of it are similar we need to run de-dup (Although I'm open to suggestions on compression). Just to confirm, FreeNAS is in-line data de-duplication?
Based on my research it's recommended to have somewhere between 2GB-10GB of memory per 1TB of data. What I'm not sure on is this per TB of written data (eg: 70TB) or the amount of data before being de-dup (eg: 130TB)?
If it's 70TB then the 196GB would be enough if I use the 2GB per TB, it puts me somewhere around the 2.5GB per TB. (Leaving some memory for other system services). If it's 130TB then I have less than 1.5GB per TB.
My biggest concern here is if we run out of memory then the volume can't be mounted. However, based on this article (https://blogs.oracle.com/bonwick/entry/zfs_dedup) it appears that some of the de-dup table can be moved to L2ARC and then to disk at the cost of performance, is this correct for FreeNAS? I'd rather take a hit on performance than not have access to any data.
The next part which I'm a little more unsure on is the best way to set-up FreeNAS with the P2000. I know that FreeNAS likes direct access to the disks, however I'm not sure how that's possible with a SAN. I assume I'd just create a LUN and assign it to the server as I would any other set-up and FreeNAS would use it without direct access to the drives. Is this correct?
For a system with 100TB+ of data would it be recommended to use L2ARC?
I welcome any advice on the above design good or bad. Hopefully someone with more experience with FreeNAS in the enterprise space can provide me with some insight.
Thanks,
David.