nannou v0.9 - Vulkan graphics, simpler event API and much more

[mitchmindtree]

This is easily the biggest update for nannou so far!

OpenGL -> Vulkan

The switch from OpenGL to Vulkan for cross-platform graphics is the main story here. You can find more details at #208 and #216, but the primary reasons for switching to Vulkan are:

• Gaining support for compiling and running languages other than GLSL (e.g. RLSL, HLSL) on the GPU via Vulkan's low-level SPIR-V representation.
• Access to finer-grained control over the GPU for more thorough optimisation if necessary.
• No longer having to deal with OpenGL bugs resulting from inconsistent implementations across different devices and versions.
• Switching to a more modern, future-proof, cross-platform graphics standard.

Both High-level and Low-level Access

While the existing high-level graphics APIs should behave almost exactly the same (apart from smoother graphics and slightly better performance), this PR also allows for much lower level access to the graphics stack than was previously available. In other words, nannou now allows for fully custom graphics pipeline creation, access to each window's unique swapchain, custom vulkan instance creation, custom device selection and loads more. To demonstrate this low level access, @JoshuaBatty and I have started a new series of vk_*.rs examples in the examples/vulkan/ directory. These demonstrate how to work with basic geometry, images, 3D, first-person cameras, compute shaders and more. We'll continue adding to these over time - requests are very welcome!

GLSL

Note that GLSL is still well supported so you can still reach into your existing bag of shader tricks! Note however that currently only the most recent versions of GLSL are supported, so you may need to update older versions. Funnily enough, this might make it easier to run GLSL 450 code on macOS as I believe support for more recent versions of OpenGL and GLSL from Apple have been lacking.

MacOS

For folks wondering about how macOS is supported, nannou uses vulkano under the hood which links to MoltenVK on macOS for translating Vulkan to Apple's Metal API. We're hoping that by the time this is merged, @freesig's work on automating the MoltenVK installation process at build time will have landed. In theory this means that you should be able to run cargo build like normal and everything should Just Work™ (you might have to restart your terminal), though don't be surprised if there are hiccups as this is all still very new.

Tradeoffs

All that said, this change will mean that nannou may no longer be compatible on some older systems that don't yet have Vulkan support. Nannou aims to foster the next generation of creative coding and with our small team's limited capacity this unfortunately means we must draw a line in the sand at some point. If you're curious about the level of support for Vulkan on your system's GPU you can search for it here to find detailed reports. If you're not sure what GPU you have or it's not listed in that database, the site will allow you to download the tool for generating your own report.

If nannou was working for you in 0.8 but stopped in 0.9 we would love for you to leave an issue and let us know. We'll do our best to address each issue, though keep in mind that our ability will be limited in cases where Vulkan driver support is lacking.

Simpler Event API

One glaring issue with nannou's current event API is the way that user's are expected to handle events via the event function. While the event type is thorough and allows for accessing loads of hardware, application and window events, handling the Event type itself could be quite verbose and confronting for new users who might be unfamiliar with the fancy pattern matching required.

This update allows for using functions to handle events in a way that will be much more familiar to users coming from Processing, OF and Cinder and reduces the amount of language-learning users have to go through before they can start being creative.

For a closer look at these changes check out #221 and in particular the changes to the examples in the diff. The new all_functions.rs example demonstrates all of the new event functions that may be registered with apps and windows in one place.

---

For details on the other small tweaks, patches, fixes and features check out the CHANGELOG diff and the commit messages below.

Merging is currently pending:

• Mac install vulkano-rs/vulkano#1145 / moltenvk_deps integration #233
• Change LoadOp of swapchain image color attachment from DontCare to Load for Ui and Draw renderpasses #241.
• Wait for next release of MoltenVK with patch.
• Update nannou codebase to 2018 proper.
• Issues with Vulkan image layout transitions #271 - pending Fix image layout transitions vulkano-rs/vulkano#1171 and a following published version of vulkano.
• Address @JoshuaBatty 's slow frame-rate on 2017 MBP.
• Fix MoltenVK segfault on exit (This is being worked out here Vulkan SDK 1.1.106.0 - Segmentation fault from vkDestroySwapchainKHR KhronosGroup/MoltenVK#584)
• Beta testing period. Test on as many machines as possible (windows!)
• Do some profiling to check for any easy wins (use new GPA on linux and Nsight).
• Blog post.
• Press kit for publishing.

[freesig]

Awesome stuff!!
I might suggest waiting for the next release of the lunarg SDK because otherwise we might have to distribute a build of MoltenVK for this to actually work. Looks like they usually release monthly and we are already over due for one. Hopefully there's a release while we are waiting for the moltenvk_deps to merge.

[Walther]

Awesome work here, I'm excited for this to land! Is there any way in which i (or others) could help?

[freesig]

I've bumped both shade_runner and moltenvk_deps so that should be all ready to go.
I've run nannou on windows and it works fine but I still want to run all the examples.
The only exception is it has an additional dependency of ninja which is not super simple to install. I might add to the readme on how to do that. Also add the cmake and python are required for all os.

I'd like to get to the bottom of this slow framerate on @JoshuaBatty laptop. I'm not seeing this unless the application is in the background #307 Can you please confirm josh if you are seeing a slow framerate with the application in the foreground?

MoltenVK segfault is fixed and just waiting on the next vulkansdk to be released

I'm going to run some profiling on windows and ubuntu. See what I can find.

Hey @Walther welcome. I'm excited to we are very close now. The only blockers are the website and the sdk.
I think the most helpful thing right now would be to checkout this branch and run the examples. Let us know if anything goes wrong or you get stuck on anything.

With a change this big and a team this small we are no doubt going run into some issues initially after release into the wild but the more people playing with it, the quicker we can find and fix them.

[Walther]

A couple of hours of testing later:

nannou v0.9 branch testing

System details:

• macOS Mojave 10.14.5 (18F132)
• MacBook Pro (15-inch, 2018)
• 2.6 GHz Intel Core i7
• 16 GB 2400 MHz DDR4
• Radeon Pro 560X 4 GB
• Intel UHD Graphics 630 1536 MB

Key takeaways:

• Exits can be unclean: segfaults, illegal hardware instruction
• Resizing the window(s) has an extremely high likelihood for crashing immediately (>50% on an unscientific guesstimate)
• All the examples that I tried seemed to run, possibly even correctly
  • Except for the Laser examples: couldn't figure those out on a quick test through
  • I skipped the examples/nature_of_code/ directories, because there's just so many examples there. Perhaps someone else can verify them?
• One of the vulkan examples (vk_image_sequence) took disproportionately long time to load / even start the window
  • Would it be better to draw an empty window first or a loading screen?
• vk_hotload example is magical, i'm in awe of both the hot reloading and the debug prints if one uses invalid values. Wish i had heard of this earlier! <3 This will make demoscene stuff so much easier to work with

Let me know if there's something you'd like to get more details on - happy to help!

Below, more verbose & individual comments on the test runs, in the order I ran the examples.

all_functions

• Creates a window with a blue background
• Interestingly enough, if I have Amethyst window manager on and it wants to resize the window, the example segfaults
• On further testing, even manual repeated resizing can segafult the example

basics/1_nannou_events

• Creates a window with a grey background
• Prints out key events in the console
• Pressing escape quits the example
  • Sometimes cleanly
  • Sometimes zsh: illegal hardware instruction
  • Sometimes zsh: segmentation fault

basics/2_variables_window_console

• Creates a window with a grey background
• Prints out a couple of variables in the console
• Similar unclean quit issues

basics/3_variable_scope

• Similar behaviors as above

basics/4_conditionals

• Creates a window with a colored background and a colored dot in the middle
• Colors change depending on mouse position within the window
• Similar unclean quit issues

basics/5_loops

• Creates a window with a grey background
• Prints out a bunch of variables from loops to the console
• Similar unclean quit issues

basics/6_functions

• Creates a window with a grey background
• On keypresses, various functions are called & their results are printed in the console
• Similar unclean quit issues

loop_mode

• Creates a window with a grey background
• Events are printed on the console, e.g. mouse movement
• Keypresses cause the loop mode to change, resulting in e.g. continuous logging
• Really easy to get to crash via segfault! Press e.g. different arrow keys a couple of times

multi_window

• Creates three windows, titles a b c, with different background colors
• Mouse and keyboard events are printed in the console, including which window they targeted
• Didn't crash on key and mouse events! (at least based on a limited time of attempts)
• Crashes on resizes

osc_receiver && osc_sender

• Requires two examples to be run
• Receiver creates a blue window with white text
• Sender creates a red window with white text
• Mouse events in the sender window get transmitted to the receiver
• The events are printed within the window itself, instead of the console log
• Resizing windows segfaults again

simple_audio

• Creates a window with a grey background
• A sine wave sound plays
• Controls: space stops/starts the sound, up arrow raises pitch, down arrow lowers pitch
• Yup, still getting segfaults on resize

simple_draw

• Creates a window with lots of simple graphical elements
• Time and mouse position affect the elements
• Resize = segfault

simple_mesh

• Creates a window with a black background and a moving blue sine wave -esque filled path
• Mouse position affects the frequency and amplitude of the "signal"
• Resize caused zsh: illegal hardware instruction this time

simple_polygon

• Creates a window with a black background and two pentagons, one in white and one with color gradient
• Resize segfault

simple_polyline

• Creates a window with a black background and a colored line consisting of blocks, animated to approximate a sine wave
• Mouse movement again affects frequency
• Resize crash

simple_ui

• Creates a window with a simple ui for controlling a polygon
• Resolution, Scale, Rotation, and Random Color controls are fast
• Position control is super slow slideshow
• Resize crash

simple_window

• Creates a simple window with a blue background
• Mouse and key events are printed to the console
• Resize crash

template_app

• Creates a window with a background color and a circle in the middle
• Does not react to any key events
• Resize crash

template_sketch

• Effectively a 1:1 copy of the above

generative_design/p_1_0_01

• Creates a large window with a colored background and a colored square in the middle
• Mouse position affects the colors and the size of the square
• Could not crash with resizing!

generative_design/p_1_1_01

• Creates a small window with a grid of color hues and saturations
• Mouse position affects the amount of grid elements and hence visible color choices
• Could not crash with resizing!

generative_design/p_1_2_3_01

• Creates a small window with a grid of various colors
• Mouse position affects the amount of grid elements and colors
• Crashes with resizing

generative_design/p_1_2_3_02

• Creates a window with continuously appearing rectangles in various colors, painted on top of each other
• Crashes with resizing

laser/laser_frame_stream

• Creates a window with a grey background
• Does not appear to do anything else?

laser/laser_frame_stream_gui

• Creates a GUI window for controlling parameters
• Doesn't seem to affect anything, even with the other laser_frame_stream windows open

laser/laser_raw_stream

• Creates a window with a grey background and a white dot at the middle
  • On a second run, the white dot follows the mouse
• Doesn't appear to do anything else?
• Stacktrace from a crash (possibly related to having another of the examples running at the same time?)

    Finished dev [unoptimized + debuginfo] target(s) in 0.38s
     Running `target/debug/examples/laser_raw_stream`
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "SendError(..)"', src/libcore/result.rs:997:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
   1: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at src/libstd/sys_common/backtrace.rs:59
             at src/libstd/panicking.rs:197
   3: std::panicking::default_hook
             at src/libstd/panicking.rs:211
   4: <std::panicking::begin_panic::PanicPayload<A> as core::panic::BoxMeUp>::get
             at src/libstd/panicking.rs:474
   5: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:381
   6: std::panicking::try::do_call
             at src/libstd/panicking.rs:308
   7: <T as core::any::Any>::type_id
             at src/libcore/panicking.rs:85
   8: core::result::unwrap_failed
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libcore/macros.rs:18
   9: core::result::Result<T,E>::unwrap
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libcore/result.rs:798
  10: laser_raw_stream::mouse_moved
             at examples/laser/laser_raw_stream.rs:70
  11: nannou::app::process_and_emit_winit_event
             at ./src/app.rs:1948
  12: nannou::app::poll_and_process_events
             at ./src/app.rs:1676
  13: nannou::app::run_loop_mode_refresh_sync
             at ./src/app.rs:1485
  14: nannou::app::run_loop
             at ./src/app.rs:1246
  15: nannou::app::Builder<M,E>::run
             at ./src/app.rs:636
  16: laser_raw_stream::main
             at examples/laser/laser_raw_stream.rs:7
  17: std::rt::lang_start::{{closure}}
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/rt.rs:64
  18: std::panicking::try::do_call
             at src/libstd/rt.rs:49
             at src/libstd/panicking.rs:293
  19: panic_unwind::dwarf::eh::read_encoded_pointer
             at src/libpanic_unwind/lib.rs:87
  20: std::panicking::update_count_then_panic
             at src/libstd/panicking.rs:272
             at src/libstd/panic.rs:388
             at src/libstd/rt.rs:48
  21: std::rt::lang_start
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/rt.rs:64
  22: laser_raw_stream::view

vulkan/vk_quad_warp

• Creates two windows: one with a rotating Utah teapot, another with a GUI for controlling the corners
• Moving the control dots causes the teapot to warp
• Somehow the control dots are sometimes connected with a white line, sometimes not, depending on positions?
• Resize crashes

vulkan/vk_shader_include

• Creates a window with a grey background and a moving circle
• Circle's color changes periodically
• Resize crash

vulkan/vk_compute_shader

• Creates a window with a black background and a blue-to-purple gradient in the shape of a pencil (sharp tip, then fills rest of rectangle)
• Doesn't change over time
• Starts with a wide window and the color gradient has gaps
• Resizing to a smaller size makes the gradient full
• Resizing can again cause crashes

vulkan/vk_debug

• Creates a window with a blue background
• Lots of debug prints in the console
• Resizing crashes

vulkan/vk_hotload

• Creates a window with a blue background and a white triangle
• Editing the hotload shader files hot reloads the window
• Adding invalid values in the shader file gives debug prints in the terminal
• Hot dang this is magic, and would be so useful in demoscene stuff, i love this <3
• Yup, same resize crash

vulkan/vk_image_sequence

• Loads for quite a while compared to other examples (about 5 ish seconds without a window)
• Colorful clock icon with super-fast spinning handles
• Yup, same resize crash

vulkan/vk_image

• Creates a window with a colorful background gradient that moves and nannou logo in white
• Resize crash

vulkan/vk_images

• Creates a window with image tiles
• Resize crash

vulkan/vk_teapot_camera

• Creates a window with a black background and a white teapot
• WASD + mouse controls for moving around
• Captures mouse, esc terminates example
• Space unlocks / toggles mouse control
• Resize crashes

vulkan/vk_teapot

• Creates a window with a black background and a spinning teapot
• No controls
• Resize crashes

vulkan/vk_triangle_raw_frame

• Creates a window with a blue background and a red triangle
• Resize crashes

vulkan/vk_triangle

• Effectively 1:1 copy of above

[freesig]

Hey @Walther
Wow thanks for doing such a detailed test! This is very useful.
The segfault on close and resize is actually the Moltenvk bug I was talking about and is fixed in the lastest MoltenVK code but we need them to release the next vulkan sdk so that we can use this fix. This should be happening any day now.

The load time on the images example is interesting. I'll look into that.

I'm not sure about the laser examples perhaps @mitchmindtree or @JoshuaBatty might be of more help as they wrote it.

I'm glad you like the hot loader!

[freesig]

Ok I can confirm all examples are working on Windows in release and debug. Nothing errors printed.

[freesig]

Also @Walther did you run the vk_image_sequence like cargo run --release --example vk_image_sequence or cargo run --example vk_image_sequence?

[Walther]

I ran the examples in default aka debug mode. Running it in release mode launches the window pretty quickly. I still found it interesting how big the difference was compared to the other examples - others launch the window under a second, vk_image_sequence takes 5+. Additionally, there's no indication that anything is loading, as the window gets drawn only after the whole thing is ready. (Should nannou have some setting/mode/default of drawing a base window immediately, while still loading/computing the content?)

These were not intended as a complaint or anything severe, just an interesting observation :)

[freesig]

Ah yep so debug is way slower then release mode for rust.
I see what you mean. It's basically happening because the images are loaded (from disk) in the model function but the screen won't draw till until after that function returns.

The way you'd handle this in a proper application is to load the images on another thread but this might be adding a bit too much complexity to this example.
It might be worth making an example showing multi-threading in a Rust / nannou as some point though.

These were not intended as a complaint or anything severe, just an interesting observation :)

No worries at all. All feedback is welcome and valuable.

[mitchmindtree]

Alright, I think it might be finally time to land this behemoth!

[JoshuaBatty]

Epic work everyone, amazing to have this merged now! Onwards to 0.10 :)