I don't know if you're a home user or a mission critical business user. Obviously, this suggestion only applies to a non mission critical situation.
Create 2 dirs under your current dir and move half the files into each one. Then try and see if setting perms on a reduced number of files (dir1) works. If you want to find the exact number or files it craps out on, you could keep resetting permissions (of dir1), moving over some more files from dir2 to dir1 until the problem rears it's head. And yes, if you do this test, you should reset perms between each iteration.
You mentioned 800GB+, but that's file sizes. It seems more likely it has to do with number of files. How many are we talking? Also, what version of FreeNAS are we talking? How many hard drives? Raid-Z1? And out of curiosity, why are you changing perms in the console instead of the GUI?