Severe performance regression in TigerVNC 1.4 relative to 1.2 and 1.3

[dcommander]

Unfortunately, I find that the refactored encoder and decoder have significantly regressed in performance relative to TigerVNC 1.3.x-- by about 20-30% on the encoding side and 30-40% on the decoding side. I consider this to be a bug, as it affects the suitability of TigerVNC as a solution for VirtualGL.

Some notes:

• I notice that the subrectangle mix is a bit different in the new encoder, and this may be partly responsible for the difference in performance. Also note that the compression ratio isn't as good with the new encoder.
• All results were visually verified-- that is, before the benchmark run, I ran compare-encodings with the -o option to generate encoded session files, and I played those back through vncviewer to verify that they were correct. This was done both with the CUT active and inactive, and for both the TigerVNC 1.2/1.3 encoder and the new 1.4 encoder.
• The same encoded datasets were used to test both versions of the viewer, so as to eliminate any differences in the encoder that might have affected those tests.

If you want to reproduce my results, you will need:

Encoder tests:

• The TurboVNC Benchmark Tools: https://sourceforge.net/p/turbovnc/code/HEAD/tree/vncbenchtools/trunk/ (these now contain a mockup of both the TigerVNC 1.2/1.3 encoder and the next-gen TigerVNC 1.4 encoder.)
• The datasets: http://www.turbovnc.org/rfbsessions.tar.bz2
• My convenience scripts for running the tests and parsing the output into a spreadsheet-friendly form:
  http://www.turbovnc.org/files/TurboVNC-Benchmark-Tools-Scripts.tar.bz2

Decoder tests:

• Turbo-encoded versions of the datasets: http://www.turbovnc.org/rfbsessions.turbo.hiqual.tar.bz2 (NOTE: these were all generated using the TurboVNC Benchmark Tools with the -o option.)
• Patches to the TigerVNC 1.3.x and 1.4.x viewers to enable them to play back these processed datasets and generate performance profiling information: http://www.turbovnc.org/files/TigerVNC-Viewer-Benchmark-Patches.tar.bz2
• My convenience scripts for running the tests and parsing the output into a spreadsheet-friendly form: http://www.turbovnc.org/files/TigerVNC-Viewer-Benchmark-Scripts.tar.bz2

[dcommander]

Also, the spreadsheet of results:
http://www.virtualgl.org/pmwiki/uploads/About/tigervnc14.ods

[dcommander]

Further information-- this issue does not just exist at the low level. It is reproducible in end-to-end tests using GLXspheres and TCBench as well. The basic problem is that both the encoder and decoder are now taking 20-40% more CPU time than they did previously. This limits the performance that is achievable in a LAN environment, and in a WAN environment, it increases server load, which affects the number of users that can share the system.

[dcommander]

For the 24-bit datasets, I can confirm that PR #119 fixes the performance regression, and with my hacked-together benchmarks, the performance is now within +/- 5% of TigerVNC 1.3.x. I do still see a regression in the 8-bit and 16-bit datasets on the encoding side-- the 16-bit datasets take generally about 8-10% longer to encode under TigerVNC 1.4.x than TigerVNC 1.3.x, and the 8-bit dataset takes generally about 20% longer. This is not really of much concern to me, since VirtualGL never uses anything but a 24-bit screen depth.

The decoder is now similarly within +/- 5% of TigerVNC 1.2.

I am also evaluating your new encperf/decperf utilities and will comment on those in the PR.

[dcommander]

Also, please let me know when this fix lands in a release (I assume TigerVNC 1.4.3), so I can document to the VirtualGL community that we recommend upgrading to that release rather than using TigerVNC 1.4.0 - 1.4.2. There is at least one ThinLinc customer who reported this to me as well, so they will be interested to know when the fix lands in a ThinLinc update.

[CendioOssman]

I was hoping we would do a 1.5.0 next.

[dcommander]

What new features are there that would necessitate a major release?

[CendioOssman]

There's been a lot of non-bug fix changes, but I haven't made a list yet. IPv6 in the server would be one change.

[dcommander]

That's up to you, but please be aware that VirtualGL users have been warned away from TigerVNC 1.4.x until this issue is resolved. As I've mentioned before, I consider performance to be part and parcel of quality, so a performance regression is a bug. As soon as this bug is fixed, I will inform the VirtualGL community that, if they are using TigerVNC, they should upgrade to the new version.

Again, this issue has popped up in commercial deployments-- specifically from VirtualGL users at Volvo, who reported subpar 3D performance with the latest ThinLinc release, and I strongly suspect that it's due to this. So unless you're talking about putting out TigerVNC 1.5 like next week, I would strongly recommend putting out a 1.4.3 release.

[bphinz]

On Mon, Feb 23, 2015 at 3:50 PM, DRC notifications@github.com wrote:

That's up to you, but please be aware that VirtualGL users have been
warned away from TigerVNC 1.4.x until this issue is resolved. As I've
mentioned before, I consider performance to be part and parcel of quality,
so a performance regression is a bug. As soon as this bug is fixed, I will
inform the VirtualGL community that, if they are using TigerVNC, they
should upgrade to the new version.

Again, this issue has popped up in commercial deployments-- specifically
from VirtualGL users at Volvo, who reported subpar 3D performance with the
latest ThinLinc release, and I strongly suspect that it's due to this. So
unless you're talking about putting out TigerVNC 1.5 like next week, I
would strongly recommend putting out a 1.4.3 release.

I will try to put out a 1.4.3 release by the end of the week which
includes this and the patches for CVE-2015-0255.

Tim,

Let me know if you need any of your recent commits back-ported to 1.4 as
well.

Thanks,
-brian

[bphinz]

On Wed, 2015-02-25 at 08:55 -0500, Brian Hinz wrote:

Tim,

Let me know if you need any of your recent commits back-ported to 1.4
as well.

I'm happy to wait for them. It isn't causing a big problem to carry them
separately at the moment.

Tim.
*/