ZFS happily will eat all the ram it is given, but will it know to share some of it when other jail/process need them?
More or less. If you have a truly unmanageable process, as
@Bidule0hm suggests, the NAS will start to swap idle processes out to disk. This isn't horrible, but if allowed to continue (such as a fork bomb or memory leak) you can actually run out of swap space too, at which point the system will resort to killing what it thinks might be the problem process. However, if it inadvertently kills your GUI or the middleware or a protocol sharing daemon, your NAS starts to develop some badness.
The default process memory limit in FreeNAS is 32G. You can artificially limit process sizes within a jail by installing some appropriate resource limits, possibly in the jail startup script, so that your jail processes are limited to some much more reasonable number.