proposal: golang.org/x/exp/shiny: an experimental GUI library.

[nigeltao]

Purpose.

First and foremost, this is an experiment. This is not 'the official Go GUI (Graphical User Interface) library' or the only blessed way to do things. Unlike a well-understood problem like decoding a JPEG image into an in-memory buffer, there are many possible approaches to GUIs, including existing code both under golang.org/x/mobile and elsewhere. This is simply an exploration. I've reached the point where I have an interesting if minimal demo, and I'd like to share and discuss the code, design and ideas.

Name.

There's nothing particularly deep about the "shiny" name. I just like shiny things.

The code would live at golang.org/x/exp/shiny. To emphasize what I've said above, this is an experiment, and this is a GUI library, not the GUI library, so it's not e.g. golang.org/x/gui.

It is under golang.org/x because I want to use the same code review process as everything else under golang.org/x, a process I use every working day.

Overview.

There are two layers: a lower-level window layer, and a higher-level widget layer.

For the lower layer, the primary concept is a Window. On the input side, a shiny.Window looks similar to what the golang.org/x/mobile/app package currently calls an App: you get a channel of events, such as key and mouse events.

The output side is where things diverge. x/mobile/app assumes that you paint pixels with OpenGL, either directly via the x/mobile/gl package, or indirectly via e.g. x/mobile/exp/sprite uses x/mobile/exp/gl/glutil uses x/mobile/gl. Instead, shiny has twin concepts of Buffers and Textures which map naturally to an OpenGL implementation, but also allow other implementations such as one that speaks a pure X11 protocol, without the need for cgo and OpenGL shared libraries. I assume that you could do somewhat similar things on Windows and Mac.

A Buffer is an in-memory, in-process bucket of bytes. A Texture is an opaque handle to something 'out-of-process'. A Buffer's contents can be Uploaded to a Texture, and Textures can be Drawn to the Window. As an optimization, it may be possible to Upload a Buffer directly to a Window.

For OpenGL, you can think of a Buffer as the result of a simple make([]byte, etc) call, and a Texture as an OpenGL texture. For X11, you can think of a Buffer as MIT-SHM shared memory (if applicable) and a Texture as a (server-side) Pixmap.

Uploads are 1:1 between source (Buffer) and destination (Texture) pixels. Draws can be subject to an arbitrary affine transformation (including simpler copies and scales) between source (Texture) and destination (Window) pixels. For OpenGL, affine transforms are simple shader programs. For X11, they are provided by the XRender extension.

For the higher layer, you construct a tree of widgets like buttons, text areas and containers. This shouldn't be overly surprising, except for the fact that the Go language doesn't have traditional object-oriented inheritance. The details of how that works is best explained with actual code, so I'll save it for a CL, if this proposal is approved.

The widget set is pure Go, like Swing instead of AWT or SWT. Sweating the details of a production-quality widget set is an enormous amount of work, and hitting that level of quality is not a short term goal. The primary goal is to explore what's possible in the GUI space with Go, and not necessarily to crank out a production-ready classic Win32 or Cocoa app any time soon.

A secondary goal is to drive development of the Freetype Go port - it has some basic functionality but also some long-standing known deficiencies. Writing a quality GUI text editor is again a substantial amount of work, and I don't intend to build one, but I intend that working on shiny will push the Freetype Go library to be good enough to let someone or some team do exactly that, if they had the time and inclination.

Relationship to golang.org/x/mobile.

There is a lot of overlap between shiny and x/mobile, and obviously I have contributed to both. The key difference, I think, is that the x/mobile code is rightly focused on mobile apps, where the dominant model is each app has only one (full-screen) window, whereas shiny starts more from a desktop perspective. To mix some metaphors, triangulating from different starting points (mobile vs desktop GUIs) can help avoid overfitting to local optima. In the medium term, the two should end up sharing a lot of code, but it's not obvious to me right now whether x/mobile should depend on shiny or vice versa, and it would be premature to pick a winner now. As I keep saying, there are many possible approaches, and I think it's valid to explore more than one of them concurrently.

There is also the operational concern that I would like the freedom for shiny to depend on third party libraries, such as github.com/golang/freetype and github.com/BurntSushi/xgb, but I'd also like to keep the golang.org/x/mobile tree free of such dependencies, at least for now.

Relationship to github.com/google/gxui.

I'm obviously aware of their work, but the two projects aren't strongly linked. As I've said, there are many possible approaches that are worth trying here. Longer term, we might discuss what we can take from each other's codebase, but it's premature to do that now.

Closing Remarks.

I'll repeat that this is experimental and exploratory. I just have a demo and I'd like to share it. I know that there could be a lot of excitement in the community for a 'Go GUI library', but please don't set your expectations too high at this stage, and the existance of this project should not invalidate any other Go GUI projects.

[robpike]

SGTM

[taruti]

What is the license policy of the x/ tree? Probably the FTL (or GPL) requirement is fine for most uses, but to avoid clients accidentally breaking the attribution requirement it should be clearly documented if there is a freetype dependency.

[ajstarks]

Interesting proposal. Can you sketch what a simple client program would look like?

[eaburns]

Looks very interesting. I also prototyped a UI library in Go. It shares many of the ideas that you propose.

It has two halves: the ui package (described below) and the gui package (not written) for simple widgets.

The ui package deals with Windows and Events. It's currently implemented on top of SDL2 using cgo. With ui, one can draw Buffers to Textures and Textures to Windows. However, I call Buffers image.Images and drawing them to Textures happens via draw.Draw. It also has an interesting (I think) solution to the annoying OpenGL thread context issue. In short, one calls Window.Draw with a drawing function that accepts a ui.Canvas. The Canvas has methods for actually drawing to the Window. Window.Draw ships the draw function and Canvas to the thread with the OpenGL context in order to perform all the drawing. This obviates the need for channel operations and context switching for each call to a draw primitive.

Perhaps you'd like to take a peek: http://godoc.org/github.com/eaburns/T/ui.

Also, I'd be interested in ditching my package in order to help out with Shiny. Particularly because the motivation for my ui package is to write a text editor, and you explicitly mention as a target use for Shiny.

[andrewchambers]

My vote for first demo app is an acme clone :)

[crawshaw]

SGTM

Note that x/mobile/app's OpenGL dependency will eventually be generalized to ES2, ES3, Metal, and Vulkan. But all of these share a similar GL style and shiny will need different machinery.

I would like to see the code in x/mobile/app converge in the medium term. There's a lot of subtly in cross-platform window and event management I would like to combine in one place.

[mattn]

Sounds good to me. Are you talking about mobile limited?

[aclements]

In the medium term, the two should end up sharing a lot of code, but it's not obvious to me right now whether x/mobile should depend on shiny or vice versa, and it would be premature to pick a winner now.

Rather than either depending on the other, would it make sense for them to share a common third dependency? I don't have a strong sense of exactly what would be shared between them, but, as an example, the OpenGL bindings seem like a common dependency that is neither mobile- nor GUI-specific, but is required by both.

Put another way, x/mobile is focused on mobile; it needs things like windows and OpenGL, but doesn't (itself) need a GUI toolkit. shiny is focused on GUIs; it needs things like windows and OpenGL, but isn't specific to mobile.

[owenthereal]

👍 Is there any more info on what it is like?

[theGeekPirate]

I've reached the point where I have an interesting if minimal demo, and I'd like to share and discuss the code, design and ideas.

Mind sharing?

[crawshaw]

@aclements I believe that is what Nigel and I are both trying to say, there are just enough moving parts that's it is difficult to describe. Another attempt:

There should ultimately be one shared piece of code for creating windows and delivering events. It may have two separate interfaces, one for mobile and one for desktop, or maybe not. To some extent, it will have to know about the underlying drawing technology the user wishes to use: GL (probably all varieties can be collapsed at this level), Metal, Vulkan, or a *image.RGBA that is fed into the underlying system some other way (the X11 protocol). This is the work mostly being done in the x/mobile/app package right now, and needs to be generalized a little bit and live somewhere other than x/mobile. Both mobile and shiny will use it. In some ways this window+event package is like glfw with two differences: firstly that it's written in Go, and secondly that it doesn't necessarily require GL.

Then there are the packages for accessing the graphics system. Right now we have x/mobile/gl for GL ES 2. It has no mobile dependencies, and should live somewhere else, and can be optionally used by shiny. There are other GL packages out there that expose more features (and a fork of x/mobile/gl that works on GopherJS.) The x/mobile/app and its future generalizations will depend, probably in a "pluggable" way, on these graphics system packages.

[skelterjohn]

Consider treating the input/output (keyboard and mouse/pixels on the screen) as a completely separate issue from GUI widgets.

As an example, I created github.com/skelterjohn/go.wde to do the input/output, and github.com/skelterjohn/go.uik to do GUI widgets using go.wde.

[phonkee]

Sounds interesting, is there anything we can help with?

[andlabs]

Oh boy, this is quite a long thread with a lot to talk about. here goes!

---

@nigeltao original post

Your design is in good company: this is pretty much how GTK+ works; the name of its lower layer is GDK.

What I learned from my own UI package attempt is that the way operating systems give you events and the way Go cahnnels work don't line up fully. With a Go channel, a transaction is considered complete once the message is received, and the code that produces the message can continue operating regardless of what the receiver does. But with a traditional GUI system, the system expeccts that you handle a message fully before asking for another one, and will generate input, painting, and timer events according to that design. In practice, things may seem to work at first, but unless you have an "okay, done with this one" channel for everything, you're eventually going to fall into the trap I fell in with the first version of my package ui: needing to introduce such a channel for painting events and everything deadlocking as a result. I also don't know how x/mobile handles it.

Of course, I could be wrong and this could be a non-issue if done right; only experimentation will tell. (Until then, I've decided to separate my own package ui from Go as much as possible, with the intention of avoiding conflicts between the Go runtime and what the C world expects.) There is definitely room for experimentation here (experimentation I am very interested in).

Next, here is a very very important question. Or rather, two questions whose importance is mutual. Using "shiny.low" to refer to the lower layer and "shiny.high" to refer to the higher layer:

1. Will a shiny.low.Window map solely to a top-level window (such as the one your web browser runs in) or to an arbitrary region of the screen that has its own independent event handling?
2. Will each shiny.high.Widget have its own shiny.low.Window?

If you answer "the latter" to 1 and "yes" to 2, the GTK+ project has done this in the past and found that it doesn't really work well on X11. The link is to a video that shows a window being resized in GTK+ 2 and the massive flicker that resulted as the widgets were redrawn. (Windows and OS X don't have this problem as much because of heavy optimization and API design tricks in the former's case and a similar design to what I'm about to say in the latter.) You may find it more performant to chop up a shiny.low.Window into each shiny.high.Widget's parts and mux out events that way. GTK+ 3 has switched to this model.

Another thing is that for the widget library, you would need more than just a text rendering library; you would also need a full vector graphics library. I know the old code.google.com freetype package had one built in, but I'm not sure how comprehensive it is for the purposes of a GUI package. That also leaves open the question of theming.

Threading is another issue. Windows and OS X (I don't know about the other Unix display servers) assume that all your UI operations exist on the same thread; OS X goes even further by saying that it must be the very first thread the OS creates (though there is an undocumented function to change it; I have yet to list the full side effects). I'm aware of runtime.LockOSThread() and that might be the solution you'll want to go for. (The future of my package ui will lie in having the GUI thread be a C thread that Go knows nothing about, and that uses other methods of communicating with other threads. In my early experiments this seems to work fine as well.)

That being said, I would be happy to contribute to this project if it did exist; I could even possiby (no promises, just a thought) prototype the low level part this weekend using code that I already have in my ui and libui projects. skelterjohn has also tried something like this with his go.wde (the low layer) and go.uik (the high layer) projects; possibly using some of that code might also help.

One big question that looms in my mind is how cgo, moving garbage collection, and arrays and slices will work, since it isn't very well defined (as far as I'm aware, anyway). At some point we're going to have to feed Go data blobs to C. I should have stated this in that one issue; I'm really not against moving garbage collectors so long as the ability to safely feed large amounts of data to the C world in a well-defined manner is provided (or the rules for doing so defined).

---

@crawshaw

See what I said above about how to map a low-level window to a high-level widget. I'm not sure how this will conflict with golang.org/x/mobile's needs... And in fact, wouldn't having a single end-all-be-all OpenGL package be the logical conclusion of this?

[rsc]

SGTM.

[ajstarks]

I agree with @andlabs on having a robust 2D vector API. In fact this is more important to me than standard "widgets" like buttons; I'd like to have the ability to build my own interaction objects without being constrained to one set. Would the vector stuff go in shiny.Widget or shiny.Window?

[nigeltao]

@taruti. The golang.org/x code is licensed the same as Go itself. For example, see https://go.googlesource.com/exp/+/master/LICENSE

IANAL, but I think you're right that shiny packages should document that an (implicit) Freetype dependency, or other third party dependency, could have license implications. Such a dependency would be in the high-level layer, not the low-level one.

[nigeltao]

@ajstarks, it looks like the consensus is to go ahead with this, so the best place to show example code will be in what we call a CL or change list (a git-codereview change).

[nigeltao]

@eaburns thanks for the link. I'll check it out.

[nigeltao]

@mattn, I'm sorry but I don't understand the question. What does "mobile limited" mean?

[nigeltao]

@aclements, yes, there might be a common third dependency in the future, and packages might move, but I don't know what the ideal package 'org chart' should be yet, and part of landing code now and iterating will be to learn what that should be.

[nigeltao]

@jingweno, @theGeekPirate, yes, I will start sending out some code, as it looks like the consensus is to go ahead with this. Code reviews will be CC'ed to the https://groups.google.com/forum/#!forum/golang-codereviews mailing list. Grep for "shiny".

[nigeltao]

@skelterjohn, yes, I think we're in agreement here. There will be separate shiny packages for the lower level (key/mouse/pixels) and higher level (widgets) as I originally described.

[nigeltao]

@phonkee, not anything specific right now, but constructive feedback is always welcome.

One general issue is that I'd like shiny to work on Windows, whether via OpenGL or otherwise, but I don't really run Windows any more, and certainly haven't written any Go code for Windows. So, after I submit what I've got so far, I'd appreciate any Windows expertise, from you or any other gopher.

[nigeltao]

@andlabs, the x/mobile code tries hard not deadlock around two-way channel communication. For example, see the pump function in https://go.googlesource.com/mobile/+/master/app/app.go.

A lowshiny.Window is always a top-level window, and highshiny.Widgets will not have their own 'window'. Client-side widgets, a la GTK+3.

Re vector graphics, that would indeed be nice-to-have, and the Go Freetype port does have a Bezier curve rasterizer, but I'm not convinced that it is a must-have yet. I know that @ajstarks is keen on SVG, but e.g. I'm not aware of a lot of web-app that use SVG for their GUIs, and to echo @andrewchambers' comment, we could possibly write an acme clone without the Go equivalent of a Java2D. I'll repeat that a Go2D would be nice-to-have, but I think it's a separate package than either lowshiny (which gives you an *image.RGBA that you can draw vectors on) or highshiny (which would be irrelevant; affine transformations of buttons and text areas are out of its scope).

Re threading, we already have this concern with golang.org/x/mobile/app and OpenGL, and I think the same approach can apply here.

Re cgo, blobs of bytes and a potentially moving GC, yes, this is an open question, but not one I think limited to shiny.

[andlabs]

Fair points. I'm not familiar with how x/mobile deals with threading and two-way communication; apart from pump(), where else can I look to learn what it's doing?

What exactly do you have ready now, since you seem to be implying that you do have something? (Not asking to upload anything; just wondering.)

[nigeltao]

I don't have a better suggestion than reading the x/mobile/app source code, although it's not the simplest Go package, as it has a lot of OS-specific parts.

As for what I have ready now, I'll start sending out code reviews shortly.

[mattn]

@nigeltao I expect to use this GUI library on Windows, OSX, Linux.

[adg]

SGTM

[tv42]

Actually, damage in Wayland seems to go the other way around: client tells the compositor what parts of its buffers it changed. Because of the compositing nature, no buffer is damaged by the compositor. http://wayland.freedesktop.org/architecture.html

[andlabs]

Good to know.

Last question before I upload Fill: do the other drivers send a size.Event before the first paint.Event? The basic example seems to imply so.

[nigeltao]

I would expect them to, but I don't think it's mandatory, and I think that any correct program should be able to handle seeing a paint.Event before any size.Event. The example/basic works here because filling zero-width zero-height rectangle is a perfectly fine no-op.

[andlabs]

All right; the Windows code will need to synthesize a fake size.Event in this case since Windows doesn't have an initial 0x0 state (and doesn't generate a sizing message for the initial size). I've done this already; just need to implement the draw.Src filling and then I can start the C->Go conversion. :)

[adg]

I think we can close this issue now that the proposed change is well underway.

[andlabs]

Do you have a minimum OS requirement in mind? If we're willing to drop anything before Vista SP2 with Platform Update, I'm wondering if using Direct2D instead of GDI/GDI+ would help, especially since it does support retaining window contents in the same way x11driver does. The only problem is recreating COM, though I think package syscall's Syscall* functions might help here...

[nigeltao]

My instinct is that I'd like shiny to work on whatever version of Windows that regular Go does, but I'm not overly familiar with Windows programming these days.

OTOH, if WinXP is no longer officially supported by Microsoft, then I suppose that assuming version >= Vista is fine, if it makes things significantly easier.

So really, I don't have a firm opinion, one way or the other. Whatever's easiest, I guess. I would have thought that wrestling with GDI+ would be easier than wrestling with COM, but I don't know for sure.

[minux]

Go still supports XP and 2003, so I'd expect shiny to support those platforms as well.

[egonelbre]

I probably missed some discussion - what's the problem "retaining window contents"? Do you simply mean that you don't want to draw directly to the screen? In that case it's possible to implement the back-buffer yourself and blit it to the screen.

[nigeltao]

Yeah, we decided to always double-buffer, so the windriver (and x11driver) backends will have to implement the back-buffer themselves.

[bonly]

good work!

[sqweek]

I see I am very late to the party, but I wanted to add one thing:

@andlabs wrote:

you're eventually going to fall into the trap I fell in with the first version of my package ui:
needing to introduce such a channel for painting events and everything deadlocking as a
result.

@skelterjohn's go.wde package avoids this particular pitfall by taking care of the paint event itself, rather than passing it on to the application. Here "taking care of" means "blitting the most recent buffer the application has marked as a completed frame to the screen". From an application perspective I feel it works very nicely, allowing painting and input event handling to be handled independently.

[yalay]

SGTM TOO!

[ghost]

@nigeltao

MDL components
https://github.com/material-components

Google have put iOS, web, android MDL all together now.

If you be cool if shiny was at this point too.

[tildeleb]

Status update? It's been a year since the author commented (here) on this project and any major commits are months old. Is this dead?

[nigeltao]

It's moving slowly, as shiny is not the only thing that I'm working on.

Some recent progress might not be so obvious, as it involves some font work in golang.org/x/image/font, which will certainly be used by shiny, but that code is not under golang.org/x/exp/shiny per se.

[ghost]

@nigeltao
Thanks for the update.

It's definitely a tough / complex thing to get right. Also i saw the work on the vector graphics which looks like a really good base. i can see how you want to make the foundation right before letting loose on the easy stuff higher up.

If you dont mind i want to say a few things below and scratch my itch :) ; my intention is to highlight architectural opportunities, not increase pressure...

I have to ask though why not let others contribute in some form ? I stress "in some form". In the above thread allot of people for also itching to contribute, but i also see that the foundation was not ready and so there was too much flux in the direction to take.
If a game plan of the architecture tasks is done, i think you would have many contributors that would sign the CLA and use gerrit. Maybe some work can be done on non foundation areas; the MDL widgets can be done in parallel, whilst the layout engine catches up in the background.
Also the way the Flutter team solved the cascade layout / update issues is now well known where they gave very detailed video discussions on the matter.

Personally I played with different approaches and ended up resorting to use QT with golang to get what Shiny offers for now. It works very well and QT now has very decent MDL support. Their licensing is now open for Desktop and Mobile, but you pay for Embedded (automative, etc)
https://doc-snapshots.qt.io/qt5-5.8/qtquickcontrols2-material.html
But i would switch back to Shiny instantly if i had the chance because its far easier to innovate in pure golang rather than a QML, c++, golang environment like QT.

As we all know Ubuntu embraced it as their Official GUi engine in the end. https://developer.ubuntu.com/en/phone/apps/qml/tutorials/

Another later idea for Shiny is to use QML. I imagine about using it as the basis of golang code gen for both golang Web and Shiny. After all a QML file is at Design time a serialised form of an DAG, and at Runtime the object representation of that DAG in memory. So, golang code generation would work nicely at Build time to produce the code.

I have a rough golang based DAG for web GUI now using gopherjs, and plan to build a QML code gen approach with it. In terms of data binding, QML enforces very simple techniques (slots and signals) which is stupid simple but works. Its very early days though, and the next phase is to hook up the new MDL components from google.

Itch scratched .....

[andlabs]

What is this Flutter discussion (and what is Flutter)? And you can already contribute by signing the agreement and sending contributions in the same way you would to Go itself.

[nigeltao]

@andlabs Flutter is https://flutter.io/. I'm assuming that the discussions are the video'd talks such as https://www.youtube.com/watch?v=UUfXWzp0-DU about Flutter in general. I am not aware of any discussions of applying Flutter specifically to shiny.

@gedw99 as for contributions, yes, I'm aware that there are people itching to contribute, but I think that it's still a little early and the foundations are still not ready. Inviting people to write Material Design components isn't free from my point of view, as I'd still have to review code, answer questions, and write docs. Sure, I'd have to do that anyway in the long term, but my shiny time is already scarce enough in the short term. Yes, I know that progress is slow. Sorry.

[chenzhekl]

Is this project still alive? There seems to be few commits for a long time.

[nigeltao]

It's certainly not very active right now. I work part time, for uninteresting reasons, and shiny is unfortunately not my primary project.

I'd like to work on it again at some point, but I can't promise when that will happen.

[zacps]

@nigeltao I'd suggest handing this off to another maintainer. I'm sure there's someone around who'd have the time and experience to keep this going; it's obviously wanted.

[joeblew99]

Well to do that requires a roadmap first.

I doubt a person with the right skills to do this is available also. Would be better to delegate and supervise. But its really down to @nigeltao :)

i have noticed in the lat 6 mots there are about 30 projects using shiny now out here in github.
Its getting to the point where the framework is usable.
The packages of Node, widgets and flex helped allot to allow people to compose things.

For example:
https://github.com/hajimehoshi/ebiten

https://github.com/aarzilli/nucular

There are a few others that are written higher level layout, reflow and animation packages on top of shiny too. Reflow is a hard one.

Then there are people writing key native functionality for Mobile and Desktop too:

Virtual keyboard;

• E.G On mobile you are in an openGL TextBox, and the virtual keyboard should come up.

Cut and paste;

• E.G Keyboards based & Native OS overlay based.

Audio and Video capture and playback;

Intimate gesture control and keyboard mapping;

Printing via pdf templates;

• E.G in Form archetypes you will need to bind the data to a PDF template.
• Even printing via opengl even.

Sharing;

• E.G. You are in an app page and click “Share”, and the OS brings up all apps that handle that intent.

Notifications;

• E.G From within the app you wish to raise a native notification
• E.G You raised a notification from the Server via the mobile push gateways and the user has pressed on it in their Notifications bar, and it routes down into the right part of the App.

Linking, so that web links passed in route to the correct location within the app.

• E.G you send a http link to a friend and he has the app for it and so it opens the app, and navigates into the app to the right location. Google maps does this for instance.

---

There is no common GUI Material Design toolkit yet.
Because shiny is a simple framebuffer with immediate mode its not super hard to made these.
https://github.com/TheGrum/renderview

• Allows making image based controls, but whats needed is ones driven by shaders..

[nigeltao]

Handing this off to another maintainer requires there being a candidate to be that maintainer. For example, Rob Pike and Russ Cox are prominent gophers who have worked on the image/* packages in the Go standard library in the past, but these days, Rob is working on https://upspin.io/ and Russ is working on https://blog.golang.org/toward-go2 and they don't have much spare time for shiny. David Crawshaw has also worked on shiny in the past but he is similarly busy with other projects.

If somebody or bodies from the community wants to fork the source code and run with it, they have my blessing (but I won't have enough spare time to supervise). In hindsight, shiny would probably have been better as a vanilla github-hosted project instead of a golang.org/x project, although at the time, the github code review UI wasn't as good as it is now.

FWIW, there was some recent discussions of a new, multi-person Go GUI effort at https://groups.google.com/d/msg/golang-nuts/JyrN8cxWCrU/H6oZpv02BQAJ