Discussion: mobile and web Window creation APIs

[Osspial]

Right now, Winit exposes window creation and modification functionality the same way on all platforms. Is that something that makes sense? On mobile, only one function in WindowBuilder (with_multitouch) actually does anything, and only three functions in Window (get_hidpi_factor, get_inner_size, and get_outer_size) do anything useful. A few more functions are implemented for Emscripten, but it has much more in common with mobile platforms than it does with desktop platforms.

Should we expose an entirely different window structure for these platforms that better reflects the underlying capabilities? I don't have much experience with those platforms, but I've written up an example of what I think that sort of API would look like:

• Bikeshedding aside, I'm calling it View.
• For creation, View exposes list_views(&EventsLoop) -> impl Iterator<View>, which gets an iterator over all windows the OS provides.
• View exposes the following methods:
  • set_multitouch(&self, bool): enables or disables multitouch support.
  • get_hidpi_factor(&self) -> f32: gets the HiDPI factor.
  • get_size(&self) -> LogicalSize: gets the window's size.
• Winit adds two new public platform modules:
  • mobile
    • Exposes extension trait ViewExt with the following methods:
      • set_valid_orientations(ValidScreenOrientations): used to set the valid screen orientations.
    • Adds ValidScreenOrientations struct, defined as followed:

      pub struct ValidScreenOrientations {
          pub portrait: bool,
          pub landscape: bool
      }

  • emscripten
    • Exposes extension trait ViewExt with the following methods:
      • set_fullscreen(bool): sets if the window is fullscreen or not
      • grab_cursor(bool): grabs or ungrabs the cursor
      • hide_cursor(bool): hides or shows the cursor

That design may be infeasible one way or another, so any API corrections are appreciated.

Relevant to #33
cc @francesca64 @seivan @mtak- @willglynn

EDIT: Renamed WindowBorrowed to View since that's a better name for it, and added a reference to EventsLoop in list_views.

[mtak-]

I completely understand the motivation for this, and I'll provide more feedback on this issue once I've cobbled together an EventLoop2.0 implementation for iOS.

Some early feedback:
As a user of winit, I've been enjoying the unified API. I'm able to benefit from quicker iteration cycles on macos, but not lose the ability to run on iOS. I haven't had to write a large amount of platform specific code (yet).

[Osspial]

@mtak- Yeah, that's my biggest concern with this. Something we could do is have View be the absolute lowest common denominator across platforms, and on desktops have it create a persistent fullscreen window that only gets destroyed when the application quits.

[willglynn]

@Osspial I'm 100% on board with platform-specific extension traits.

By analogy: std::fs::File is the lowest common denominator. I can seek(&mut self) and read(&mut self) all day long, and it works everywhere std works. Great.

If I get annoyed having to carry around &mut File to do reads, I can use std::os::unix::fs::FileExt and File sprouts a read_at(&self) function. Now I can read without mutating! Making me write use … ::unix:: tells me that this will cause problems if I want to target more platforms, but it's at my option. I can opt-out of platform independence and opt-in to UNIX-specific File behaviors with one line of code.

Sometime later, hmm, I guess this needs to work on Windows too. Okay, std::os::windows::fs::FileExt! Well… seek_read(&self) on Windows has different semantics than read_at(&self) on UNIX. The API confronts me with the reality that platforms have differences which can't always be hidden. I'm forced to conditionally compile both the use …::FileExt line and my use of the platform-specific read_at() and seek_read() functions, thus acknowledging that my code intends to do two subtly different things on those two different platforms. This result seems ideal.

[Osspial]

@willglynn Thanks for the feedback!

Also, you seem to have a decent amount of experience with how Emscripten works, from the Event Loop 2.0 discussion thread - does that backend align with how this API works? The impression I've gotten from skimming the source code/Emscripten documentation is that the web browser creates a canvas and then Emscripten users request access to that canvas to draw on, which seems to be pretty similar to how mobile platforms work. If that's the case it would make sense to expose mobile and web canvases through the same API, but I'm not experienced enough with either to be 100% certain that that's how stuff works.

[mtak-]

I left some feedback related to EventLoop2.0 here.

Some additional feedback:

• iOS can support multiple MonitorHandle's (+[UIScreen screens]'s).
• set_decorations could be used for hiding the statusbar, tho it should probably have it's own name
• PortraitUpsideDown tends to be a special orientation that's usually disabled on phones except in rare circumstances.
• There are a couple more app lifecycle events on iOS (didBecomeActive, willResignActive) than we currently have in Event, they are being sent as WindowEvent:Focus right now, which is a lie. Android has more too, but they are tied to the activity, so they would be in WindowEvent.
• I think View would be a UIWindow on iOS which is confusing because UIView is also a thing.

What about having a WindowExtDesktop and WindowExtMobile?

[Osspial]

@mtak-
I agree that showing and hiding the statusbar should probably be in its own function.

I don't think PortraitUpsideDown is supported on Android, so support for it should probably be in an iOS-specific extensions module.

Should didBecomeActive and willResignActive be implemented as Suspended? It seems like that may align more with what the iOS events intend, but I don't know all the semantics about when those events are dispatched so that may or may not make sense.

Hmm, so maybe View wouldn't be the best name. Since it's a Window that's managed by Winit/the OS, perhaps it should be called WindowManaged?

My issue with implementing this as WindowExtDesktop and WindowExtMobile is that the window operations that are exclusive to one platform or the other are... well, all of them. The idea of having a window that the application can control, resize, and destroy is an idea that fundamentally isn't portable to mobile devices.

[mtak-]

@Osspial

didEnterBackground/willEnterForeground are implemented as Event::Supsended. The current setup works decent enough because you will get a Focused(false) before a Suspended(true), and a Suspended(false) before a Focused(true).

//!  - applicationDidBecomeActive is WindowEvent::Focused(true)
//!  - applicationWillResignActive is WindowEvent::Focused(false)
//!  - applicationDidEnterBackground is Suspended(true)
//!  - applicationWillEnterForeground is Suspended(false)

Looking at our other shipped (c++) games, we actually don't use didEnterBackground and willEnterForeground. Atleast for my needs, it makes sense to do what you suggested and have
didBecomeActive, willResignActive be sent as Suspended, and the keyWIndow events be sent as Focused(true/false). That would make the iOS implementation a bit more honest.

WindowManaged is fine for iOS :).

iOS supports multiple UIWindow's, and the application can destroy, show/hide/create, and even resize and reposition them - though I've always had fullscreen UIWindows, and used UIViews for everything else.

I think I'm leaning towards wanting the an API split, mostly because an application designed for desktop that just "happens" to work on mobile, probably won't have good results.

[Osspial]

@mtak- On iOS, what happens if you do have multiple UIWindows? What does it look like? What happens if the window doesn't cover the entire screen?

The main reason I can see for a mobile application to create a separate window is to have a picture-in-picture window. Is there any other reason I'm missing? If there is, fully adopting this API in its current form may not be the best move.

[mtak-]

@Osspial Having some sort of Alert window is also a good use case. In our other games we'll sometimes call out to apis to have users pick a custom backdrop, which I imagine is another UIWindow, but it could just be a UIView. UIWindow/UIView have a lot of overlap.

Here's a photo of winit with two windows. One on the apple tv and one, not fullscreen window, on the phone. It's definitely not a common use case, but not something we should make impossible.

In the topright corner theres a screenshot of both windows on the same Monitor (i.e. the phone itself).

EDIT: the brown square is one window, the pink background thingy with the servo logo is another.

[Osspial]

@mtak- Thanks! So there are three windows in that picture (The Apple TV one, the brown box, and the pink background with the servo logo)? How does multi-monitor work on iOS? Does the Window on the Apple TV share the same window coordinate space as the ones on the iPhone screen, or do they have different coordinate spaces?

We should also look into how Android handles multi-window applications/multiple displays, as that may also align more closely with how desktop windows work than may be immediately apparent.

[mtak-]

@Osspial I could have explained that image better. There's two images merged into one (I thought it'd look less spammy that way). One is a photo, and one is a screenshot (top right).

Screenshot (topright corner):

• Has two windows.
• Both are on the iphone.
• One fullscreen window with pink background and servo logo.
• One brown non-fullscreen window in the topleft displayed overtop the servo logo window.

Photo:

• Has two windows.
• One fullscreen window on the apple tv with the servo logo and pink background.
• One brown not fullscreen window displayed on the iphone.
• If there's only one window on a monitor, and it's not fullscreen, it looks like this particular os/device gives a black background.

The monitors have different coordinate spaces and AFAICT there's no way to convert between them. Actually, there is a way, it just gives back nan, nan for x/y position and 0,0 for wiidth/height.

[ryanisaacg]

I don't mean to revive an old thread, but I'm currently working on #589 again now that eventloop 2.0 is in a more ready state. I would agree that a View API would probably make sense, because a lot of the existing Window APIs don't translate well to the web.

[madsmtm]

So, this issue is several years old, and not really actionable any more, so I'm going to close it.

Multiple windows should be possible on both iOS and the web, on Android each window is run in its own process, and as such not really possible for Winit to support (CC #3505).

We might want to add some sort of subsurface / view support, if you want that feel free to open a new issue, though honestly, I think it can and should be kept out of Winit.

[MarijnS95]

on Android each window is run in its own process

Not at all, anything related to the same app (services, multiple Activities which are probably the closest synonym to Windows) should all run in the same process.

This is actually an open issue, where certain Android system interactions can cause it (the Android system) to launch another Activity one is already open via a callback. Both android-activity and winit are assuming there is only one Activity singleton with (related) state stored across various static globals, and everything falls apart.

It is however unlikely to have multiple windows open at the same time, bar tricks with multiple task stacks, split-window, popout windows and perhaps desktop mode.