@lysunht I'm having the same issue today. Did you by chance figure out the issue or find a solution?
I had originally added an ESXi server to VMware-Snapshots while I was rebuilding my vCenter server. After finishing up my vCenter server, I added it too and confirmed it was working before attempting to remove my ESXi server. However upon trying to remove my ESXi server, I get the same error.
Version: TrueNAS-12.0-RELEASE (October 19th)
Here's a static image of just the error in Chrome's dev tools:
And the steps I'm taking to produce the error:
I dumped the stack trace to /tmp/log with this code:
Code:
314 # + PATCH
315 import traceback
316 with open("/tmp/log", "a") as fh:
317 fh.write("Traceback (most recent call last):\n")
318 traceback.print_stack(file=fh)
319 # - PATCH
And here's the stack trace (numbers are a bit offset from
src/middlewared/middlewared/main.py due to my own added code, and new commits since the release):
Code:
Traceback (most recent call last):
File "/usr/local/bin/middlewared", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1583, in main
Middleware(
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1427, in run
self.loop.run_forever()
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
self._run_once()
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
handle._run()
File "/usr/local/lib/python3.8/asyncio/events.py", line 81, in _run
self._context.run(self._callback, *self._args)
File "/usr/local/lib/python3.8/site-packages/aiohttp/web_app.py", line 458, in _handle
resp = await handler(request)
File "/usr/local/lib/python3.8/site-packages/aiohttp/web_middlewares.py", line 119, in impl
return await handler(request)
File "/usr/local/lib/python3.8/site-packages/aiohttp/web_middlewares.py", line 109, in impl
return await handler(request)
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1369, in ws_handler
await connection.on_message(x)
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 318, in on_message
traceback.print_stack(file=fh)
It seems that
middlewared/middlewared/main.py#L313-L316 is where the check that `message` has a key `method`. Outside of that, I'm really unfamiliar with the code as a whole so I'm trying to piece this all together. Instances of objects and such makes it more work to reverse engineer.
Anyone with more experience is more than welcome to poke around.
