wishlist: Support multiple clients connected at the same time

[totaam]

Issue migrated from trac ticket # 41

component: server | priority: minor | resolution: fixed | keywords: wishlist

2011-11-15 15:26:04: Norman Rasmussen created the issue

---

aka: session shadowing?

ref: http://code.google.com/p/partiwm/issues/detail?id=42

It would be useful to support multiple clients connected at the same time. This could allow window sharing between multiple desktops (perhaps with sync'ed location? - so that synergy can 'move' windows between desktops)

[totaam]

I don't understand the link with synergy...

Cloning would be possible I think, but probably quite a lot of work to get right. (concurrent multi-user is even harder - read-only clone would be the first step)

For example: handling constantly moving windows around whenever the controlling client decides to move a window would be tricky (think different screen sizes, different positions forced by each client's window manager, etc)

Then there's the problem of managing redraws and requests for window pixel data: probably would have to send all pixel data to both clients irrespective of which client requested it (since this process is asynchronous)

[totaam]

2011-11-15 16:27:22: Norman Rasmussen commented

---

read-only clone would be a great start, read-write (maybe only one copy can write at a time), would be nice too.

Basically I'd like to simulate a multi-monitor set up, by being able to drag a window from one X server to another, at the moment I can xpra attach on the 2nd server, and all windows disappear on the first.

[totaam]

Another useful feature discussed on IRC would be the ability to have different screen update rates for each client so that a slow client does not slow down everyone: it would drop most screen updates and just get a full screen refresh at the appropriate rate.

[totaam]

2012-05-26 23:43:57: http://norman.rasmussen.co.za/ commented

---

I spent a few hours on this today, and the code in XpraServer is really tangled. Some of it is for the single server instance, and some is for the currently connected client. The first step is to untangle these two sets of logic. Do we need a new ServerConnection class, that would have ._protocol and ._server_source? or can we move this per-connection code into ServerSource?

As a crazy work-around, could two copies of xpra start be run connecting to the save xvfb?

[totaam]

Totally agree on the need to entangle the server from the connection stuff. Even if we don't support multiple connections, this will make the code more maintainable and easier to read.

The crazy workaround would not work: xpra is the compositing window manager and there can only be one at a time...

[totaam]

Closed duplicate ticket #169 (my bad), there has been a lot of progress in this area (especially since r1407).

I have managed to connect multiple clients and see something on screen... it's a start.

What is left to do:

• when one client moves the window... it should also move for the other clients (or should it? I can't see how to make it work without moving). When the configure doesn't match... we could end up going back and forth forever, careful!
• command line option to enable session sharing (both server and client)
• keyboard support: ouch, this is going to be hard. It should probably be solved with the other keyboard bugs by using the newer API and defining a new keyboard for each client (new xorg api which I don't know how to use..) - until then, we can probably just live with the first active connection setting the keymap.
• get the different window sources to co-operate so that the window.acknowledge_changes happens only once per damage event (the first to get there)
• set_xsettings_format is problematic since it is hardwired low down, so I will probably make this the default, and if you cannot handle it then you cannot do multi-user (and maybe backport this fix to the stable branches)
• The clipboard is bound to cause problems: the anti-token tells us who owns the keyboard... there can only be one! (I might just disable it for now: only the first user gets it)
• png_window_icons feature is quite useful... but we should not assume that PNG is supported in all cases..
• pulseaudio: first come first serve?
• set_screen_size: which one do we use? The biggest one.

[totaam]

• r1410 fixes a bug where we would end up with duplicate copies of the same window (on the same client!)
• r1411 adds the "--enable-sharing" option and helpful disconnection messages (tells you where to enable sharing: all parties involve must set it, at least for now)
• r1412 selects the best screens resolution to suit all the clients (randr mode)
• r1413 moves png_window_icons to ServerSource, so we only use png format if the client supports it and fallback to premult_argb32 otherwise.

[totaam]

I've posted some pretty graphs comparing performance before and after this patch to make sure it does not adversely affect it, you can find them here

[totaam]

r1434 fixes clipboard: only the first client to request it will get it (doing more than this will be quite difficult and race prone)

[totaam]

2012-08-29 20:31:19: norman commented

---

This is looking great!

For moving windows (and clipboard), why not keep track of 'last client to send keystroke/mouse click' and use that as the primary client. I think you use this already for tracking mouse movements (I did a quick test with xeyes). This would allow someone to use two clients at almost the same time, and generally "the right thing" would just happen.

btw, when you're syncing window position, it would be nice to be able to specify an offset on the client side. So I can run client on workstation & laptop, and window position is sync'ed but windows only appear on one display at a time, eg workstation at +0+0, and laptop at -1440+0.

[totaam]

the clipboard would be very difficult to manage with more than one client (it is already difficult enough as it is!).

I'm not sure I understand the concept of "last client to send keystroke/mouse click": what would I use this for? What is the current behaviour that needs fixing?

As for syncing window position: this still needs doing... and we should not allow windows to be placed in different locations on each client, as this will lead to bugs: many applications and toolkits rely on knowing where the windows are for placing new windows/widgets/etc, if we lie about it, the results won't be good!
That said, since each client may well decide to place windows in slightly different locations (window snap to grid, custom rules, etc..), we may already have a problem on our hands! (and finding a solution that is not racy will be difficult!)

[totaam]

2012-08-30 15:04:05: norman commented

---

My idea of "last client to send keystroke/mouse click" is to help out with cases where multiple clients send conflicting information like: who 'owns' the clipboard, where do I position this window, etc.

The current clipboard implementation means that only one client will have clipboard interaction, where-as if we implement a dynamic primary client (using last user-interacting event), then we have a easy way to tell which client the user is currently interacting with.

Doing this dynamically would allow the user to physically migrate between clients and have a first class experience at all of them.

btw, re: window positioning: I meant a client-side option that would be applied to all windows the same. Windows would still be in the same place relative to each other.

[totaam]

Hah, gotcha, both very good ideas!
I'll see what I can do, but this is all lower priority than fixing the obvious bugs, like: try to minimize the window on one client... still shown on the other (but unusable), etc. Feel free to post changes. Note: the way the clipboard stuff is hooked at present is a bit nasty (just wanted to get something working).

[totaam]

2012-09-25 00:31:23: jpg35 commented

---

I've been using this since you first released it, and it's working really nicely for me: thank you.

Before you did (or announced anyway) any more towards the window placement synchronisation issue, I thought I'd say that for my use case, I hardly ever notice any issue from it. Furthermore, the behaviour as it exists now: allowing different clients to position windows individually, is very useful. My clients have hugely varying screen sizes and layouts, and it'd be unfortunate if every client had to scrunch all the shared xpra windows in to the top-left corner or such to respect the smallest connected client.

In fact, without the window position syncing, all the GTK apps (and some old xlib only ones) I've used have been fine and have responded to mouse clicks in the right places etc. If I move a window on one client, it keeps responding properly on all the others (the issue you described with minimising aside).

The only problem I've seen is with initial window placement. One of my clients has a gnome top panel that extends to the top-left corner of the display, and all the rest don't (because they have multiple screens, and the panels happen to not be on the left-most screens). Initial placement of new windows is always in the top-left of each client, but obeying window manager rules to not overlap the panel. Every new window on the clients without the panel respond to mouse clicks in a position offset by the depth of the panel on the one client that has it, until they're moved for the first time. Presumably after the initial placement in the top-left, the panel'd client's window manager moves the window to avoid the panel, and that's what creates the issue?

I can elaborate further if you'd like to consider fixing this particular case separately to a more general solution; I'm sorry I'm not able to find my way around the code well enough to address it myself (I tried somewhat, and I haven't entirely given up yet). I wondered if each client simulating moving a window to its client-defined current position each time it wants to send any keyboard/mouse events would be enough to fix it? Maybe that's exactly the same as Norman was suggesting?

Anyway, mostly I was just wondering if you'd consider making the window position syncing optional if and when you find time to implement it please?

[totaam]

New issue: maybe system tray forwarding should be disabled for secondary clients.
r2690 fixes system-tray compatibility bugs when clients not supporting the feature, or those that disable it with --no-system-tray, connect to a server which has trays. Problem is that this does not take multi-client into account at all...

In dealing with hidden windows, we can deal with #47

[totaam]

I was looking at this code for unrelated problems, and thought:
instead of keeping a "shown" flag in the DesktopManager, this could be a WeakSet (Python 2.7 only, sigh) of all the ServerSources on which the window is shown. Or maybe we should have a manager per source which could keep track of window state independently: iconic, actual workspace (see #774), etc..
When we get the "contents-changed" signal, we can call damage only on the active sources. Same when deciding if the window is shown or not: take into account who is asking. More difficult is the ownership_election code which parks the window on the root window when not visible? Could create some focus problems I think.. See also r8307

You are right about what needs to be done to handle client events properly: we need to keep where the window got mapped to on each client (almost like having a DesktopManager per client), and force this before processing events. This will not play well with #212..

Another relatively easy fix is for transient-for windows: instead of sending plain coordinates for the new window, we can send them relative to the parent window.

New problems: found a race in acknowledge_changes, we call it multiple times (one for each source), and the later ones may cause the earlier ones to miss some screen updates. This should be a one-off function only, fired by whichever WindowSource happens to process the damage data first.

Note: some recent changes may help a bit here (testing needed), see #765

[totaam]

See also #943 and #990.

[totaam]

2015-11-06 00:29:18: maxmylyn commented

---

Have done some playing around with Sharing on a r11145 Fedora Client(1080p screen), a r11122 Win8.1 Client(1440p screen), and a r11145 Fedora 21 Server(...no physical screen):

• Managed to pin down some behavior with regards to different sized and location windows.

• Also took a few screenshots

---

I've found the server will move and resize the windows according to whichever client moved the window last. (I've also found switching desktops on the fedora client triggers this behavior for all windows) This does some weird things, as the server and client do not track mouse properly. Meaning, mouse clicks can go in all the wrong spots. I've found that the client will send the server mouse inputs (clicks, dragging, etc) from where the mouse is located on that client, without knowing that the window (on the server) is actually somewhere else; as the inactive client doesn't bother to move or adjust the window frames, just the paints within each frame. I chronicled this with a series of screenshots.

• In all screenshots the Windows Machine is the active machine(the one where I moved the window, and has the 'proper' location and size).

Firstly, I opened Firefox, and then minimized it on the Fedora machine while keeping it up on the Windows machine. Even while minimized, it covered up an Xterm I had open(on the server), meaning that right clicking on the xterm sent the right click to Firefox

Secondly, I closed Firefox and did some playing around with Gedit, leaving the Windows client with the 'proper' window size and location. To sum up my experiments, I've found that Xpra doesn't offset the mouse location per client and as such, the clicks on the machine without the 'proper' window size and location will not be able to use the mouse correctly as it'll always be slightly offset.

There'll probably need to be some logic figured out as to when each client gets the 'proper' window location and size....and if we'll want to move windows involuntarily. (might be easier than trying to keep track of window locations and offsetting mouse inputs and outputs; or it might not be)

[totaam]

I've found the server will move and resize the windows according to whichever client moved the window last.

That's a known issue, #990 did some preparatory work on this: we will need to keep track of who "owns" the window, and either:

• translate all event coordinates for clients which have mapped the window at a different location (all mouse hover, clicks, etc)
• move the window every time a different client interacts with a window (so it is mapped in the same place as this specific client and the event coordinates will match)

Or maybe a mixture of the two: until a client focuses a window or clicks in it, maybe we should not bother sending mouse hover events?

The main difficulty is that the code only has a single "desktop manager" for all clients, and we will need to have one per client connection... (and deal with minimizing, etc)

[totaam]

See also r13669: we now have an offset client-side for OR windows in some cases..

[totaam]

2016-09-15 17:14:43: norman commented

---

Cool. Will that also eventually fix the issue with two clients connected, and "pointer events don't line up until the window is moved". Currently the last client to move the window is the canonical source of window location, so pointer events on the other window arrive at the wrong location.

[totaam]

• r14368 generalizes some existing window offset code so that pointer events will still line up for all users when sharing is enabled
• r14370 adjusts the position of the popup menus (same as the parent window they are for)
• r14373 shows the mouse cursor to secondary users
• r14380 (see changeset for details): adds the concept of the "ui-driver" user

Still TODO, some or all of:

• move the window (if needed) before processing keyboard and clicks from the client?
• mouse echo should use the cursor sprite (could also be implemented for non-opengl backend - meh)
• seen some damage updates go MIA

[totaam]

@norman: does that work for you?

[totaam]

2016-11-02 16:56:03: norman commented

---

I'm running Trusty still :-(, so can't test immediately, but that's looks like it addresses my concerns from #comment:24

I think the last would-be-nice-to-have (from #comment:10) would be a way to sync window position between clients, and be able to specify an offset at the client. This would make it possible to simulate having two monitors connected to the same computer; when in fact they're two different machines, with xpra syncing the windows. (i.e. When you move a window on the one monitor towards the other, it would appear to move between the two monitors.)

[totaam]

Minor fixes in r14400.

@norman: there are beta packages with those changes. (those packages will be called "release candidate" within a few days)

[totaam]

Follow up in #1368.
Related fix in r14460 for #1358 - which looks like it could have been caused by these changes.
r14458 + r14461 makes it possible to toggle some settings via dbus or "xpra control", ie: to enable sharing after starting the server:

xpra control :100 toggle sharing on

@afarr: not heard back from @norman, so feel free to just close this one. (though, I won't complain if you do test it!)

[totaam]

2016-11-19 13:58:51: norman commented

---

I tried the beta package, but not with two clients at the same time yet. The only pending enhancement from my side is position sync (#comment:28), which could be tracked as a new ticket if you want.

[totaam]

@norman: yes please, "position sync" definitely won't make the cut for v1.0: this will require a more changes to actually work (ie: the virtual screen size needs to be big enough for both screens, etc).

[totaam]

Split out to #1369

[totaam]

2016-11-21 22:46:56: norman commented

---

Brief testing shows that the new position sync code works well. Tried moving windows around independently and there were no issues. ui-driver works okay, having an extra click before interact means initial mouse-over doesn't work, but I saw no issues with that.

I have seen a regression with gvim where the window tries to size 1 line high. I assume that there's an issue with the height feedback.

Also one of my client's didn't detect window resize events correctly (the one using loopback & has opengl enabled). I just used an xeyes window to test.

[totaam]

Also one of my client's didn't detect window resize events correctly (the one using loopback & has opengl enabled). I just used an xeyes window to test.

What is loopback?
xeyes is notoriously difficult to handle - it doesn't really behave like a normal application. (so not a huge priority for us)

---

The 'remote cursor' cross-hair was only visible on one client (loopback w/opengl).

I've added the 'remote cursor' to the non-opengl backend in r14468. You're probably running on an Intel chipset (greylisted) or a GPU that doesn't support enough opengl at all.

Taking this ticket back as I am seeing an issue with resized windows: we should forward the resizing to the other clients... but it seems to go MIA.
Please file the gvim issue separately unless it is triggered by sharing. I'm not seeing anything wrong with it (apart from the resizing..)

PS: it seems that Firefox intercepts the mouse motion events by placing another input-only window on top of its window... (fixing that would be very very very hard - the lost-motion patch above tried all sorts of things, no luck so far)

[totaam]

The resizing is fixed in r14486.

It's not perfect by any means, but it is definitely a lot more usable than it has ever been. So this will do for this release.

@afarr: feel free to test or just close.

[totaam]

2016-11-28 21:04:22: maxmylyn commented

---

Alright, doing some testing with a Fedora 24 trunk r14493 server, and a Fedora 24 trunk r14493 client, and a Windows trunk r14492 client:

• nothing is outright broken - no crashes

• Sound seems to be behaving properly - almost perfectly in sync between the two client machines.

• alt-menus can sometimes get out of sync, causing them to appear in the wrong place (usually the Firefox URL bar autocomplete dropdown)

• The arrow keys on the Fedora client don't work, and match to seemingly random keys. Same thing for page up/down, end/home/ and delete. (Page down maps to the "Context menu button" aka the button between right alt and right control) However, the keyboard on the Windows client works fine.

• Alt-menus can ignore the proper z-index, and end up on top of windows that they shouldn't be above

• Example: Open Firefox on the remote machine, then open it on the local, and right click on Firefox using the other connected device - causing the alt-menu to pop up on both computers...and above the local Firefox on the first machine. Doing so causes Firefox to get stuck on top - using the tray menu to raise the windows resets and fixes it.

• The workaround for the Win32 locking issue causes the open windows to minimize on a non-Windows system.

[totaam]

• important fix for OR windows in r14496, easily seen in the server log
• moving the keyboard issues to shared mode keyboard conflicts #1373 (too late for this milestone)
• alt-menus can sometimes get out of sync .. ignore the proper z-index - some of those issues may not be fully solvable (largely depends on how well the application behaves wrt window transient hints)

This will do for this release, will follow up in #1373, #1368, #972, #1716