You shouldn't need to reboot - the memory change will take effect immediately. Check your reporting pane in the GUI, or swapinfo
or top
from a shell to see if it has reduced the used portion.
If Ombi can run in a jail, that will be more efficient memory-wise than a bhyve VM.
As far as the amount of jails to pick and choose - you no longer have the option of "add more RAM" and have to go with "use less" - perhaps each individual plugin can be tuned, for example by reducing concurrent downloads or adding other limits in the newsgroup/torrent scrapers, lowering Plex transcode quality.
But if all else fails, you're going to have to either move some of them to another physical host, or do your best Thanos impression and snap a few of them out of existence.