v0.10.0 + decreased max_bpp causes protocol error

[bmagistro]

xrdp version

0.10.0

Detailed xrdp version, build options

xrdp 0.10.0
  A Remote Desktop Protocol Server.
  Copyright (C) 2004-2024 Jay Sorg, Neutrino Labs, and all contributors.
  See https://github.com/neutrinolabs/xrdp for more information.

  Configure options:
      --build=x86_64-redhat-linux-gnu
      --host=x86_64-redhat-linux-gnu
      --program-prefix=
      --disable-dependency-tracking
      --prefix=/usr
      --exec-prefix=/usr
      --bindir=/usr/bin
      --sbindir=/usr/sbin
      --sysconfdir=/etc
      --datadir=/usr/share
      --includedir=/usr/include
      --libdir=/usr/lib64
      --libexecdir=/usr/libexec
      --localstatedir=/var
      --sharedstatedir=/var/lib
      --mandir=/usr/share/man
      --infodir=/usr/share/info
      --enable-fuse
      --enable-pixman
      --enable-painter
      --enable-vsock
      --enable-ipv6
      --with-socketdir=/run/xrdp
      --with-imlib2
      build_alias=x86_64-redhat-linux-gnu
      host_alias=x86_64-redhat-linux-gnu
      CC=gcc
      CFLAGS=-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection 
      LDFLAGS=-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 
      CXX=g++
      CXXFLAGS=-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
      LT_SYS_LIBRARY_PATH=/usr/lib64:
      PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig

  Compiled with OpenSSL 3.0.7 1 Nov 2022

Operating system & version

Almalinux 9

Installation method

dnf / apt / zypper / pkg / etc

Which backend do you use?

xvnc

What desktop environment do you use?

xfce

Environment xrdp running on

vm

What's your client?

No response

Area(s) with issue?

No response

Steps to reproduce

Set max_bpp = 24 and restart. The failure is resolved/worked around by resetting max_bpp to 32.

I forget why we had tuned this on centos 7 + an older xrdp. This setting worked with 0.9.25.

In the MacOS RDP client, the following is what's configured for display

✔️ Expected Behavior

Successful login with reduced color depth. The logs indicated this would occur but the result was a failure.

[2024-06-14T14:15:39.594+0000] [WARN ] Client requested 32 bpp color depth, but the server configuration is limited to 24 bpp. Downgrading the color depth to 24 bits-per-pixel.

❌ Actual Behavior

The above screenshot was from yesterday, the attached logs both working and failure with only the change to ^^ are attached below.

xrdp-failure.txt
xrdp-success.txt

Anything else?

We have also tuned the following settings in addition to the one mentioned above but do not expect them to have any impact on this.

• certificate
• key
• userwindowmanager = .xsession
• defaultwindowmanager = /bin/xfce4-session
• IdleTimeLimit = 86400
• DisconnectTimeLimit = 86400

More than anything this seemed odd to us and wanted to provide the feedback. We have worked around/resolved the issue for now by tuning max_bpp to 32 when using 0.10.0 and things seem to be stable so far.

[matt335672]

@bmagistro - thanks for submitting such a thorough issue report.

I think I can see what's happening here. When the depth is limited to 24bpp we need to ignore the 'client supports GFX' flag, otherwise we start a negotiation that can't possibly succeed. During GFX testing we found clients that would request 24 bpp with GFX (which also doesn't work). We fixed that one, but I forgot to include the server case.

I'm out of time today, but I'll take a more detailed look next week.

[bmagistro]

We are good on our side for now. I'm just glad this is something that you have a lead on from that.

[matt335672]

I've reproduced this with the Microsoft mstsc.exe client.

PR on the way...

[matt335672]

@bmagistro - your setting of max_bpp = 24 may possibly have been related to #1906 / #2307 / #2945. We're pretty sure this wasn't an xrdp problem because of our limited interface to the VNC server. However, we never pinned down exactly what it was.

Some users have reported increased CPU usage running GFX (i.e. 32 bpp) (#3114). It's something else you might want to be aware of.