hescominsoon
Patron
- Joined
- Jul 27, 2016
- Messages
- 456
FreeNAS: A Worst Practices Guide
There are many best practices guides for managing storage solutions out there, but a lot of how you administer your storage depends on your specific use case and what you’re trying to accomplish. While we have created a best practices for FreeNAS, we also decided to take a look at what you don’t...
www.ixsystems.com
Has FreeNAS dedupe performance & memory requirement improved?
Hello 2 years ago, at the OpenZFS summit Matt Ahrens presented this. Video --> https://www.youtube.com/watch?v=PYxFDBgxFS8. He mentioned that his employer, Delphix, did not implement dedupe but would like interested parties to take up his idea of using logs instead of dedupe hashtables. I was...
www.truenas.com
Reddit - Dive into anything
www.reddit.com
If you want to go back to SMR that's your call...but it is HIGHLY reccommended you do not for obvous reasons. What i would consider is instead of a 2.5 chassis server build a new one with 3.5 drives. Then you can get truly high capacity CMR drives that will also run much much cooler.
Straight from ix systems site:
Setting up Deduplication without Adequate Planning
Deduplication is a much-desired feature for storage solutions. On any given system, more than half your data may be duplicates of data elsewhere in your storage pool, causing a greater storage consumption. Deduplication reduces capacity requirements significantly and improves performance by tracking duplicate data with a ‘deduplication table’, eliminating the need to write and store duplicate information. ZFS stores this table on disk, which means that, if the host has to refer to the on-disk tables regularly, performance will be substantially reduced because of the slower speeds of standard spinning disks.
This means you need to plan to fit your entire deduplication table in memory to avoid major performance and, potentially, data loss. This generally isn’t a problem when first setting up deduplication, but as the table grow over time, you may unexpectedly find its size exceeds memory. This splits the deduplication table between memory and hard disk, turning every write into multiple reads & writes, slowing your performance down to a crawl. In an enterprise environment, this can cause significant productivity decreases and angry staff workers. If this happens, the best solution is to add more system memory so that the pool will be able to import back to memory. Unfortunately, this can sometime take days to perform, and, if your hardware already has maxed out its memory capabilities, would require migrating the disks to a whole new system to access the data.
The general rule of thumb here is to have 5 GB of memory for every 1TB of deduplicated data. That said, there may be instances where more is required, but you will need to plan to meet the maximum potential memory requirements to avoid problems down the road. To get a more precise estimate of the required memory for deduplication do the following: run the ‘zdb -b (pool name)’ command for the desired pool to get an idea of the number of blocks required, then multiply the ‘bp count’ by 320 bytes to get your required memory. If it’s less than 5GB, still use the 5GB per terabyte of storage rule. If it’s higher, go with that number per terabyte.
For must use cases, it is recommended to just utilize lz4 compression for data consumption savings, as there’s no real processing cost. In fact, due to of the advances in CPU speeds, compression actually improves disk performance because writing uncompressed data to disk takes longer than compressed data. To be safe, always use compression instead of deduplication unless you know exactly what you are doing.
