Bluefin Apps: Error "[emptyDirVolume] A dict was expected" when adding Memory Backed Volume

TheNetStriker

Dabbler
Joined
Jan 27, 2022
Messages
11
I've just upgraded to Bluefin and got an error when editing one of my existing apps which has a "Memory Backed Volume" configured. Here is the error message:

Error: [EINVAL] values.emptyDirVolumes: Item#0 is not valid per list types: [emptyDirVolume] A dict was expected

If I remove the Memory Backed Volume the error is gone. But my application needs this memory volume.
Is this a bug and is there a workarround for this?
 

TheNetStriker

Dabbler
Joined
Jan 27, 2022
Messages
11
Sorry I didn't think this would ne necessary. This seems to me my like a problem between the UI and the Backend API. My guess is that the UI does send the value in an incorrect format or there is a faulty validation on the API side.

Here are my server specs:

CaseSupermicro SC846
MainboardSupermicro X8DTE-F
CPU2x Intel Xeon X5690 SLBVX 6x 3.46 GHz
Memory12x Samsung 16GB 4Rx4 PC3L 8500R-07-10-F0-D2 M393B2K70CM0
BackplaneBPN-SAS-846A
SAS Controller1 x Fujitsu D2607-A21 SAS2 HBA IT MODE JBOD (Firmware 20.00.07.00)
1 xLSI SAS 9201-16i
NetworkMellanox ConnectX-3 (PSID: IBM1080111023)
USB 3.2 PCIDelock PCI Express x4

The ix-applications folder and docker appdata are on my main ZFS Pool.

I'm trying to deploy Shinoby (OpenSource CCTV) which is not available in TrueCharts, so I've configured the app manually.

I've captured the Websocket message that is sent when saving the app in the UI. Here you can see all details if the app I'm trying to set:

Code:
{
   "id":"f3ae8a67-f832-0cf7-1653-af3a9923eed7",
   "msg":"method",
   "method":"chart.release.update",
   "params":[
      "shinobi",
      {
         "values":{
            "enableUIPortal":true,
            "portalDetails":{
               "portalName":"Web Portal",
               "protocol":"http",
               "useNodeIP":true,
               "port":9090
            },
            "workloadType":"Deployment",
            "image":{
               "repository":"registry.gitlab.com/shinobi-systems/shinobi",
               "tag":"dev",
               "pullPolicy":"IfNotPresent"
            },
            "updateStrategy":"Recreate",
            "containerCommand":[
              
            ],
            "containerArgs":[
              
            ],
            "containerEnvironmentVariables":[
               {
                  "name":"SSL_ENABLED",
                  "value":"false"
               },
               {
                  "name":"TZ",
                  "value":"Europe/Zurich"
               }
            ],
            "externalInterfaces":[
              
            ],
            "dnsPolicy":"Default",
            "dnsConfig":{
               "nameservers":[
                  
               ],
               "searches":[
                  
               ],
               "options":[
                  
               ]
            },
            "hostNetwork":false,
            "hostPortsList":[
              
            ],
            "portForwardingList":[
               {
                  "containerPort":8080,
                  "nodePort":9090,
                  "protocol":"TCP"
               }
            ],
            "hostPathVolumes":[
               {
                  "hostPath":"/mnt/Pool01/appdata/shinobi/config",
                  "mountPath":"/config",
                  "readOnly":false
               },
               {
                  "hostPath":"/mnt/Pool01/appdata/shinobi/customAutoLoad",
                  "mountPath":"/home/Shinobi/libs/customAutoLoad",
                  "readOnly":false
               },
               {
                  "hostPath":"/mnt/Pool01/appdata/shinobi/database",
                  "mountPath":"/var/lib/mysql",
                  "readOnly":false
               },
               {
                  "hostPath":"/mnt/Pool01/appdata/shinobi/videos",
                  "mountPath":"/home/Shinobi/videos",
                  "readOnly":false
               },
               {
                  "hostPath":"/mnt/Pool01/appdata/shinobi",
                  "mountPath":"/home/Shinobi/plugins",
                  "readOnly":false
               }
            ],
            "emptyDirVolumes":[
               "/dev/shm/streams"
            ],
            "volumes":[
              
            ],
            "gpuConfiguration":{
              
            },
            "tty":false,
            "stdin":false,
            "securityContext":{
               "privileged":false,
               "capabilities":[
                  
               ],
               "enableRunAsUser":false
            },
            "enableResourceLimits":false
         }
      }
   ]
}


And this is the response from the server. There is also an exception Trace that should help to find the problem:

Code:
{
   "msg":"changed",
   "collection":"core.get_jobs",
   "id":1964,
   "fields":{
      "id":1964,
      "method":"chart.release.update",
      "arguments":[
         "shinobi",
         {
            "values":{
               "enableUIPortal":true,
               "portalDetails":{
                  "portalName":"Web Portal",
                  "protocol":"http",
                  "useNodeIP":true,
                  "port":9090
               },
               "workloadType":"Deployment",
               "image":{
                  "repository":"registry.gitlab.com/shinobi-systems/shinobi",
                  "tag":"dev",
                  "pullPolicy":"IfNotPresent"
               },
               "updateStrategy":"Recreate",
               "containerCommand":[
                  
               ],
               "containerArgs":[
                  
               ],
               "containerEnvironmentVariables":[
                  {
                     "name":"SSL_ENABLED",
                     "value":"false"
                  },
                  {
                     "name":"TZ",
                     "value":"Europe/Zurich"
                  }
               ],
               "externalInterfaces":[
                  
               ],
               "dnsPolicy":"Default",
               "dnsConfig":{
                  "nameservers":[
                    
                  ],
                  "searches":[
                    
                  ],
                  "options":[
                    
                  ]
               },
               "hostNetwork":false,
               "hostPortsList":[
                  
               ],
               "portForwardingList":[
                  {
                     "containerPort":8080,
                     "nodePort":9090,
                     "protocol":"TCP"
                  }
               ],
               "hostPathVolumes":[
                  {
                     "hostPath":"/mnt/Pool01/appdata/shinobi/config",
                     "mountPath":"/config",
                     "readOnly":false
                  },
                  {
                     "hostPath":"/mnt/Pool01/appdata/shinobi/customAutoLoad",
                     "mountPath":"/home/Shinobi/libs/customAutoLoad",
                     "readOnly":false
                  },
                  {
                     "hostPath":"/mnt/Pool01/appdata/shinobi/database",
                     "mountPath":"/var/lib/mysql",
                     "readOnly":false
                  },
                  {
                     "hostPath":"/mnt/Pool01/appdata/shinobi/videos",
                     "mountPath":"/home/Shinobi/videos",
                     "readOnly":false
                  },
                  {
                     "hostPath":"/mnt/Pool01/appdata/shinobi",
                     "mountPath":"/home/Shinobi/plugins",
                     "readOnly":false
                  }
               ],
               "emptyDirVolumes":[
                  "/dev/shm/streams"
               ],
               "volumes":[
                  
               ],
               "gpuConfiguration":{
                  
               },
               "tty":false,
               "stdin":false,
               "securityContext":{
                  "privileged":false,
                  "capabilities":[
                    
                  ],
                  "enableRunAsUser":false
               },
               "enableResourceLimits":false
            }
         }
      ],
      "description":null,
      "abortable":false,
      "logs_path":null,
      "logs_excerpt":null,
      "progress":{
         "percent":0,
         "description":"",
         "extra":null
      },
      "result":null,
      "error":"[EINVAL] values.emptyDirVolumes: Item#0 is not valid per list types: [emptyDirVolume] A dict was expected\n",
      "exception":"Traceback (most recent call last):\n  File \"/usr/lib/python3/dist-packages/middlewared/job.py\", line 426, in run\n    await self.future\n  File \"/usr/lib/python3/dist-packages/middlewared/job.py\", line 461, in __run_body\n    rv = await self.method(*([self] + args))\n  File \"/usr/lib/python3/dist-packages/middlewared/schema.py\", line 1152, in nf\n    res = await f(*args, **kwargs)\n  File \"/usr/lib/python3/dist-packages/middlewared/schema.py\", line 1284, in nf\n    return await func(*args, **kwargs)\n  File \"/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/chart_release.py\", line 537, in do_update\n    config, context = await self.normalise_and_validate_values(\n  File \"/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/chart_release.py\", line 338, in normalise_and_validate_values\n    dict_obj = await self.middleware.call(\n  File \"/usr/lib/python3/dist-packages/middlewared/main.py\", line 1306, in call\n    return await self._call(\n  File \"/usr/lib/python3/dist-packages/middlewared/main.py\", line 1255, in _call\n    return await methodobj(*prepared_call.args)\n  File \"/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/validation.py\", line 53, in validate_values\n    verrors.check()\n  File \"/usr/lib/python3/dist-packages/middlewared/service_exception.py\", line 70, in check\n    raise self\nmiddlewared.service_exception.ValidationErrors: [EINVAL] values.emptyDirVolumes: Item#0 is not valid per list types: [emptyDirVolume] A dict was expected\n\n",
      "exc_info":{
         "repr":"ValidationErrors()",
         "type":"VALIDATION",
         "extra":[
            [
               "values.emptyDirVolumes",
               "Item#0 is not valid per list types: [emptyDirVolume] A dict was expected",
               22
            ]
         ]
      },
      "state":"FAILED",
      "time_started":{
         "$date":1671185380205
      },
      "time_finished":{
         "$date":1671185381512
      }
   }
}
 

TheNetStriker

Dabbler
Joined
Jan 27, 2022
Messages
11
Update: I guess I found the problem. I was able to set the memory volume using the restful api.

The problem is that the UI sends:

Code:
"emptyDirVolumes":[
  "/dev/shm/streams"
]


But what it should send is:
Code:
"emptyDirVolumes":[
  { "mountPath": "/dev/shm/streams" }
],
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Update: I guess I found the problem. I was able to set the memory volume using the restful api.

The problem is that the UI sends:

Code:
"emptyDirVolumes":[
  "/dev/shm/streams"
]


But what it should send is:
Code:
"emptyDirVolumes":[
  { "mountPath": "/dev/shm/streams" }
],

Greetings!

Sorry you're having trouble. Awesome that you debugged it. Could you PLEASE take a few moments to go to the top bar, look for "Report a Bug", and create a bug report? You may reference this forum thread in the bug report if that is more convenient for you.

Anytime someone solves a problem like this, it is best to ensure that this is put into the hands of the developers so that your debugging effort is not duplicated in the future.

Thanks.
 

TheNetStriker

Dabbler
Joined
Jan 27, 2022
Messages
11

Attachments

  • Jira.PNG
    Jira.PNG
    988.5 KB · Views: 82

samyapsul

Cadet
Joined
May 7, 2022
Messages
6
Update: I guess I found the problem. I was able to set the memory volume using the restful api.

The problem is that the UI sends:

Code:
"emptyDirVolumes":[
  "/dev/shm/streams"
]


But what it should send is:
Code:
"emptyDirVolumes":[
  { "mountPath": "/dev/shm/streams" }
],
Thanks for providing this. I was able to add a memory-backed volume via websocket with the correct dictionary format.
 

li_chang

Dabbler
Joined
May 31, 2017
Messages
35
Update: I guess I found the problem. I was able to set the memory volume using the restful api.

The problem is that the UI sends:

Code:
"emptyDirVolumes":[
  "/dev/shm/streams"
]


But what it should send is:
Code:
"emptyDirVolumes":[
  { "mountPath": "/dev/shm/streams" }
],
Thanks for providing the temporary solution. I was able to add Memory Backed Volumes via Burpsuite for my Frigate app.
 

Flachzange

Dabbler
Joined
Jul 10, 2022
Messages
16
Hi guys, thanks for bringing up a workaround. I am struggeling with the same issue (also Shinobi). However, I have not used the Rest API nor websocket before. What is the correct API call to update the app properties? I can find the websocket call in the documentation but for some reason I cannot get a working websocket connection. What ist the REST equivalent?

Thanks!

Edit: Followed the hint from @li_chang with Burpsuite. That did the trick for now.
 
Last edited:

Klyoku

Cadet
Joined
Jun 24, 2016
Messages
8
Hi guys, thanks for bringing up a workaround. I am struggeling with the same issue (also Shinobi). However, I have not used the Rest API nor websocket before. What is the correct API call to update the app properties? I can find the websocket call in the documentation but for some reason I cannot get a working websocket connection. What ist the REST equivalent?

Thanks!

Edit: Followed the hint from @li_chang with Burpsuite. That did the trick for now.
Got the same issue but can't figure out how to use Burpsuite, or through the built-in REST API. Could you explain how?
 

Flachzange

Dabbler
Joined
Jul 10, 2022
Messages
16
With Burpsuite and its internal browser you can intercept the websocket calls from the GUI and you can change the wrong syntax manually before sending it to the truenas backend. If I remember correctly ,Burpsuite offers you a video tutorial after the first start on intercepting things. That's rather straight forward. The only thing I noticed is that changing the content manually should not take too long, otherwise the backend would not accept it anymore.
 

sammael

Explorer
Joined
May 15, 2017
Messages
76
Hi,
could y'all imagine I'm about 5 years old and try to explain in more of a step-by-step manner how to add an memory backed volume through the REST thingy please? I want to add a memory backed volume that would mount inside the pod at /tmp/cache

I read through the thread and didn't understand most of it sorry ("With Burpsuite and its internal browser you can intercept the websocket calls from the GUI and you can change the wrong syntax manually before sending it to the truenas backend." reads as Klingon to me).

I'm just trying to duplicate an existing pod with memory backed volume to try out a new beta image before committing to updating the original pod and failing as the ui won't let me add the memory backed volume.

Thanks!
 

Klyoku

Cadet
Joined
Jun 24, 2016
Messages
8
With Burpsuite and its internal browser you can intercept the websocket calls from the GUI and you can change the wrong syntax manually before sending it to the truenas backend. If I remember correctly ,Burpsuite offers you a video tutorial after the first start on intercepting things. That's rather straight forward. The only thing I noticed is that changing the content manually should not take too long, otherwise the backend would not accept it anymore.
Thanks for the hint. this worked well for me.

For others finding this thread via search, use Burpsuite's proxy function and intercept the request. Here are the steps I took:
  1. Under the proxy tab of Burpsuite, open up its browser. Keep intercept disabled
  2. Using Burpsuite's browser to connect to truenas web client
  3. Edit/Create apps as usual, but not click save
  4. Go back to Burpsuite, enable intercept
  5. Go back to truenas, click save on the app that was being edited
  6. A JSON should pop up on Burpsuite, edit it as needed - be warned, not sure what happens to truenas if you make a mistake here
  7. Click "forward" when done. You'll see other requests but I just ignored them
  8. Disable intercept. I believe this automatically forwards all the requests that were queued
  9. You should see truenas does its usual thing
 

sammael

Explorer
Joined
May 15, 2017
Messages
76
Thank you for the detailed instructions, it made seemingly daunting task rather easy! The memory backed volume now appears under the frigate app, but the pod didn't recreate as normally when editing an app, so I just stopped and started it - that should be enough to propagate the change I assume?
 

davejohncole

Cadet
Joined
Jan 10, 2023
Messages
1
I tried fixing this via my REST API and failed miserably. One thing that did work was to edit the deployment via k3s. I deployed it with /dev/shm/streams mounted on disk and then edited the deployment like this:
Code:
# k3s kubectl get deployments --all-namespaces
# k3s kubectl edit deployment -n ix-shinobi shinobi-ix-chart

Then edited the streams volume to look like:
Code:
    volumeMounts:
      - mountPath: /dev/shm/streams
        name: ix-host-path-shinobi-4
    volumes:
    - emptyDir:
        medium: Memory
        sizeLimit: 2Gi
      name: ix-host-path-shinobi-4            

But I bet that this edit will be lost on the next upgrade.
 

sammael

Explorer
Joined
May 15, 2017
Messages
76
But I bet that this edit will be lost on the next upgrade.
It will, until it gets fixed in truenas itself, but your method is much quicker than messing with the burpsuite, I was looking for a way to edit the thing from shell as I test beta releases of frigate and need to redeploy often, thanks!

edit: If you're like me and do not understand how anyone can type anything with vi or vim, you can use this to switch to nano in t-nas shell
Code:
EDITOR=nano k3s kubectl edit deployment -n ix-frigate frigate-ix-chart
 
Last edited:
Top