Issues with Proxmox and Zoraxy as reverse proxy

[tobychui]

For anyone using Proxmox and want to expose the proxmox management web-ui using Zoraxy, I need some inputs regarding the setup.
In simple words, proxmox user when using Zoraxy as reverse proxy might encounter 501 - Not Implemented error when trying to use certain feature of proxmox management UI. One of such example is "start" and "stop" vm.

I was previously trying to add proxmox support in Zoraxy but with no success. Here is an example of request header Zoraxy received from web browser when I press the start/stop button

{
  "Accept": [
    "*/*"
  ],
  "Accept-Encoding": [
    "gzip, deflate, br"
  ],
  "Accept-Language": [
    "en-US,en;q=0.5"
  ],
  "Cookie": [
    "PVEAuthCookie=PVE%3Aroot@pam%3A65DE9998%3A%3APu/u+vEo0CGCqrdvEBvPUs2OsJd6OqUvK8ycyTyErwY+c+udX4bi3RHqSd6qOeMa+aIRQv9TfpESctWjVjfvrWth9nDDzFnZv1zwotY/k3hYpOsKuvcm+QtVDbxIkSHHV+5f0UxQp0zZ1NqJO8AUJXsMMN7bbJFLvsvzHOBjkrTDMMw+XOpY7XNbCy2vKjQgRZ2/Kfnv5rZEGnpNIXnINoHjJ0FPpK16+9bWfI0nR0NOyCwlvO10DXZT6ywet5WQwJZ/HOa0jN8N1pvtNh8kW78kPHTDMrLq1eWfPBwFXohNGn/5zOAUqTRUV+YNWY+qI6PibKBJHmTsJaFS0PBHOw%3D%3D"
  ],
  "Csrfpreventiontoken": [
    "65DE9998:psURpq1uyJukNU9tepAAWPcZdwIn1Bsxq/V6x86uPB4"
  ],
  "Referer": [
    "https://vm.localhost/"
  ],
  "Sec-Fetch-Dest": [
    "empty"
  ],
  "Sec-Fetch-Mode": [
    "cors"
  ],
  "Sec-Fetch-Site": [
    "same-origin"
  ],
  "Te": [
    "trailers"
  ],
  "User-Agent": [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0"
  ],
  "X-Forwarded-Host": [
    "vm.localhost"
  ],
  "X-Forwarded-Server": [
    "zoraxy-0dc57054-6e06-42b1-ac84-c6d29aa5208d"
  ],
  "X-Requested-With": [
    "XMLHttpRequest"
  ]
}

And the following is sent to the Proxmox server by Zoraxy (proxied request). Note that the forwarded host header and forwarded server was removed for testing purpose. However, I also tested with both header in-tact as well as keep / remove the Te header.

{
  "Accept": [
    "*/*"
  ],
  "Accept-Encoding": [
    "gzip, deflate, br"
  ],
  "Accept-Language": [
    "en-US,en;q=0.5"
  ],
  "Cookie": [
    "PVEAuthCookie=PVE%3Aroot@pam%3A65DE9998%3A%3APu/u+vEo0CGCqrdvEBvPUs2OsJd6OqUvK8ycyTyErwY+c+udX4bi3RHqSd6qOeMa+aIRQv9TfpESctWjVjfvrWth9nDDzFnZv1zwotY/k3hYpOsKuvcm+QtVDbxIkSHHV+5f0UxQp0zZ1NqJO8AUJXsMMN7bbJFLvsvzHOBjkrTDMMw+XOpY7XNbCy2vKjQgRZ2/Kfnv5rZEGnpNIXnINoHjJ0FPpK16+9bWfI0nR0NOyCwlvO10DXZT6ywet5WQwJZ/HOa0jN8N1pvtNh8kW78kPHTDMrLq1eWfPBwFXohNGn/5zOAUqTRUV+YNWY+qI6PibKBJHmTsJaFS0PBHOw%3D%3D"
  ],
  "Csrfpreventiontoken": [
    "65DE9998:psURpq1uyJukNU9tepAAWPcZdwIn1Bsxq/V6x86uPB4"
  ],
  "Referer": [
    "https://vm.localhost/"
  ],
  "Sec-Fetch-Dest": [
    "empty"
  ],
  "Sec-Fetch-Mode": [
    "cors"
  ],
  "Sec-Fetch-Site": [
    "same-origin"
  ],
  "User-Agent": [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0"
  ],
  "X-Requested-With": [
    "XMLHttpRequest"
  ]
}

I highly suspect that there is something screwed up in Proxmox build-in API server which is written as perl module. However I am not expert in perl nor I can contribute anything to the proxmox repo, so I think I will just share my idea here and wait for someone who can contribute some idea what might be the issue.

If you are currently facing the same issue and in need for a quick (and dirty) solution, setup an Nginx vm inside your proxmox node and point Zoraxy http rule to your nginx proxy.

Any idea or insights will be welcomed.

[tugsi]

Hi,
I´m at the same problem.
Always used NPM and wanna give Zoraxy a try, but Proxmox is a must have for me.

Zoraxy is new for me and the documentation is where? ;)

In NPM I use this Advanced Configurations:
location / {
proxy_pass https://Proxmox-IP:8006;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header Upgrade $http_upgrade;
}

Perhaps it could be implemented in the Custom Headers in Zoraxy?
The proxy_set_header Upgrade $http_upgrade; is a must for opening Shell/Console.

[tugsi]

I don´t know, what else is missing.
I just tested my config with a friend and it also works with NPM.
He is direct with a ddns-Domain connected.

[tworth04]

Not sure if anyone has noticed but you can select a vm/lxc and once selected, you can click on the console dropdown menu to the right and novnc does work out of the box...the default however is xterm.js and it doesn't work at all.

[tobychui]

@tworth04 Interesting findings. From what I currently know, this should be an issue within the internal router use by Proxmox rejecting request forwarded by Zoraxy. However, websocket (usually) didn't check for origin, that is why novnc works, but xterm.js doesn't works is kind of weird as I thought both of them are based on websocket.

You got any error message when the xterm.js load / when typing into the terminal window?

[tworth04]

No nothing i see error wise so far...

looks like it works when launching new windows? so for instance..if you click on updates...and refresh..generally it launches a sub window to show it's process...that doesn't work..but if you hit upgrade, that pops open a new browser entirely and that works.

[tworth04]

i'm curious if this would be something similar that's happening here
gravitational/teleport#32039

[KmdKeen]

I'm in the same boat! Wanna switch vom NPM to Zoraxy, but xterm.js shows "Connection failed (Error 501: )". noVNC works, but I like xterm more. A solution for this would be awesome because I like Zoraxy a lot!

[franckleroy91]

Got the same issue, I migrated all my hosts to zoraxy and kept one Nginx just for Proxmox.
I follow this discussion to know when a solution will be found.
Thanks for your great job !

[KmdKeen]

Awesome! Updated to 3.1.0 and now Proxmox GUI is working like a charm! Thanks @tobychui for fixing this! Now zoraxy is my proxy of choice!

[scroguard]

i have encountered the same issue. if i hit F12 in firefox and view the network logs when loading the page, i do see a 501 error regardig xterm.js:

i am also unable to make novnc work.

[scroguard]

i will try to test it this week. i'll respond as soon as i have had a chance to do so.

[scroguard]

i found some time today, i gave it a test and it still throws the same error.

[tugsi]

@scroguard You got time to try out the latest v3.0.5 and see if the issue has been fixed?

I´ve got also some time to test.
No step forward, NoVNC can be used, but xterm.js not and also the Shell isn´t ok.

[franckleroy91]

Hi,

just tried ; still 501 error on console.

Franck

[tobychui]

Hmmm ok, that is weird, I think I run out of clues why this could be happening.

[KemMuammer]

Is it now possible to do the get it working with the new update?

[tobychui]

It is already on the 3.1.0 branch. For Debian you can copy the link by right clicking the binary of choice and use wget to download it.

[tugsi]

@tobychui Ok, for understanding, I already used the Branch 3.10 in Github yesterday, if you have insert the API already there, then the problem is still alive.. if you fixed it today into 3.10, then I will test it later, back from work

[tugsi]

@tobychui Back from work, make a test and yes ignore my question above, you've fixed inside 3.10, so now I can upload my cert and everything went fine 👍🏻 👍🏻
One little question I got, I´ve done all on a little test-LXC inside Proxmox, to make it happened, I clone it from github in /opt/zoraxy, focused on branch 3.10 and make a build.
Unfortunately I didn´t make a backup from my conf, so first i deleted the opt/zoraxy in order to get a new clone, so I lost my conf (what a fool I am *LOL)
So here is my question, the build is made in zoraxy/src, is everything I made inside zoraxy saved under /conf/ ?

[tobychui]

@tugsi Yes, most of the configs are in the conf folder with some of the legacy settings will left inside the sys.db file. In fact you can just do git pull in the project file and cd ./src/ with go build. That way you do not need to delete everything.

[Groumf44]

I think I got it figured out and patched in here. This was partially caused by chunked encoding but some of the error was actually from the golang net/http package itself.

If anyone is free this weekend, feel free to test out the nightly build and let me know if it is working now 👍🏻

Hello
I just updated and all is working now
Thank you very much for your work dude !
Your product is now perfect for me and my usage <3

[morvane98]

hello

i discover zoraxy and i just saw that there was a bug between zoraxy and proxmox.
i have this bug and i would like to know if it has been fixed?
i installed zoraxy under proxmox in a debian12 LXC container
i am in version 3.1.1

thanks

[tugsi]

Hey @morvane98
please tell us, what happened or better what not happened.
I use zoraxy in a debian12 LXC on Proxmox and everything went fine.
So a little more information please.

[morvane98]

well when I connect to proxmox via my subdomain, I arrive well under proxmox but for example when I want to go to the shell I do not have the command prompt and the interface cannot connect whereas if I directly enter the ip address of my proxmox everything works well

[tugsi]

ok, that was the bug before v3.1, are you sure being on v3.1.1 (shown on the Dashboard down under left).
which Version of Proxmox? I have 8.2.6
do you use the Shell-Link left under the Name or use the Button right above (NoVNC,Spice,xterm.js)?
If you use left, please make a try using the option Spice or anything else.
And last but not least, how do you use your subdomain, through cloudflare or something else or only directly ?

[morvane98]

I am in 3.1.1 and proxmox8.2.6 and I use the left menu "shell".
when I enter proxmox via its ip address it works but not with the subdomain proxmox.xxxx.ovh.
and so I am at ovh I have configured my domain and subdomain at ovh

[morvane98]

[morvane98]

this is the bug

[tugsi]

Ok, let us check your settings for proxmox.xxx.ovh
My Upstream has checked TLS and Skip Websocket

[morvane98]

[tugsi]

Good Morning, so back again.
I´ve seen something special to your failure.
You got an CODE 1006 in the Shell, that´s not the Bug we had at ime ago.
But I´ve take a little search in web, do you have a firewall on your PC like Norton, ESET or something else?
https://forum.proxmox.com/threads/undefined-code-1006.93079/
If yes please take a try to deactivate the firewall.

Otherwise for testing, use NPM instead of zoraxy, so you can see, what happend.
If NPM is ok, then the bug is in zoraxy and @tobychui must take a look.

[tobychui]

Not sure if that is related, but the "Proxmox compatibility mode" only activate when your browser have cookies related to proxmox. That asides, even if the compatibility mode is disabled, you shd be seeing error 501 instead of 1006.

[morvane98]

error 1006 only occurs when I want to access proxmox by the subdomain name proxmox .xxxx.ovh
when I access proxmox locally with 192.168.2.250:8006 no problem everything works well
i don't understand why?

[tugsi]

That's exactly the point, internally it's easier for traffic, but from the outside there may be a firewall, virus scanner or something else in between. The cookies are also set differently.
You can't compare internally and externally 1:1, but of course it's good if it works internally ;-)
That's why I asked whether a firewall or something similar was used, as described in the link I posted (example to ESET Scanner).
Alternatively, you can test NPM (NginxProxyManager), which is quickly set up in Proxmox and redirected to the IP in the router. The entry that has to be entered in NPM for Proxmox is posted above. Because if Code 1006 also comes up, it is a problem on the PC and not from the reverse proxy.

[morvane98]

So on my windows pc I only have windows defender which is the original firewall.
yesterday following your message I deactivated it and it's the same thing.
what can prevent the connection on the pc?

[morvane98]

the same thing on my samsung S24.....

[tugsi]

Sorry, but I don't know what to do now.
Please try NPM to see if it also has the problem.
Is quickly set up as LXC with the script from TTeck.
https://tteck.github.io/Proxmox/#nginx-proxy-manager-lxc

Start by creating a proxy host, checking “Block Common Exploits” and “Websockets Support”.
Under Advanced then:

location / {
proxy_pass https://IP-to-Proxmox:8006;
proxy_set_header Upgrade $http_upgrade;
}

If it works fine, be happy, for the first you have a Reverse Proxy, but if it fails too, then there is more, but @tobychui is could say more.

One more thing it could be, tested only with one browser or also with Firefox, Chrome, Safari... ?

[morvane98]

ok but I don't know nginx at all and I have a bit of trouble with the command line, because I don't always understand what I'm doing
I prefer like zoraxy when there is a gui

[tugsi]

The command line is only to simple create the LXC with NPM in it, you also can create an LXC and install NPM by hand in it.
NPM has also a GUI, only a little bit less informations like zoraxy.
My only intention was, to find a way to know, if your homelab has a failure coming from outside or zoraxy has it.
In the bug mentioned here, we can´t get the shell without getting the Code1006, only running in TimeOutError.
Your behavior seems to be another bug.
Unfortunately I can't help you anymore, I'm just a user with advanced knowledge.

[morvane98]

idem with chrome and firefox

[morvane98]

I will try to see with npm

[rimbat]

Skip verification for self signed certificates:

[NitroHype]

Thank you, this fixed the same issue I was having too

[morvane98]

cool all is done!!! thanks