RxJS-like DOM library

[OddCoincidence]

Summary

RxJS is "a library for composing asynchronous and event-based programs by using observable sequences." Its core abstraction is the "observable", which is a collection of future values that has many commonalities with rust streams. I propose to create a similar API for gloo based on streams.

Motivation

gloo needs a library for interacting with DOM events that works well within the idioms of rust. The patterns in RxJS might provide a good fit. Basing it off of streams takes advantage of rust's strong async story, and will only become more ergonomic in the future with features such as async/await and language-level support for streams.

Detailed Explanation

A wrapper type would be created for each element, with a Stream-returning method for each event that the element can emit.

For example, the RxJS smart counter example (which implements a simple odometer effect) might look something like this:

fn smart_counter() -> impl Future<Item = (), Error = ()> {
    let input = gloo_dom::Input::get_by_id("range")
        .expect("expected input element with id #range to exist");
    let update_btn = gloo_dom::Button::get_by_id("update")
        .expect("expected button element with id #update to exist");

    update_btn
        .clicks()
        .map(move |_| input.value().parse::<i32>().unwrap())
        .for_each(move |end| {
            IntervalStream::new(20)
                .zip(stream::iter_ok(0..=end))
                .for_each(|(_, num)| {
                    gloo_dom::Element::get_by_id("display")
                        .expect("expected element with id #display to exist")
                        .set_inner_text(&num.to_string());
                    Ok(())
                })
        })
}

(I prototyped just enough to get this working here)

Drawbacks, Rationale, and Alternatives

Since this would just be another crate in gloo, I think the main alternative is to just not do this.

The major drawback is that this API is not as ergonomic as it could be with rust as it is today, but future language features might change that.

Unresolved Questions

• Are streams and observables really the same thing? (This has already been discussed here)
• There's already an RxRust project (that does not use streams IIRC) - does this overlap?

[Pauan]

I agree, and I wrote some libraries that do this:

https://crates.io/crates/futures-signals
https://crates.io/crates/dominator

We are discussing how deeply Gloo should integrate with Signals (personally, I think we should have Signal-based APIs where it makes sense to do so).

Are streams and observables really the same thing?

No, they are completely different in their API and behavior. Elm originally tried to conflate them together, but ran into a lot of problems, so ended up splitting them into two types, which is correct.

[fitzgen]

(More thoughts on this proposal in general coming soon...)

No, they are completely different in their API and behavior. Elm originally tried to conflate them together, but ran into a lot of problems, so ended up splitting them into two types, which is correct.

Do you have any links to elm's discussion of this, the problems they ran into, and how they decided to solve it, etc?

[Pauan]

Do you have any links to elm's discussion of this, the problems they ran into, and how they decided to solve it, etc?

Not off-hand, it was many years ago. I'll see if I can find anything, but it's possible the links are now lost to the sands of time.

On the other hand, I do remember the specific issues they ran into, so I can always explain that.

[Pauan]

After doing a lot of searching, I'm pretty sure the discussions don't exist anymore. They were all the way back in 2014 - 2015.

However, I can summarize (some of) the reasons.

First, to explain a bit of background: Elm used to have a Signal, which was an FRP value which changes over time.

However, Elm tried to use these Signals in two contradictory ways: to represent a value, and also to represent events.

There were some APIs which were intended for only one or the other: for example the foldp function only makes sense for events, and the merge function only makes sense for values.

If two Signals updated at the same time, then merge would end up dropping one of the updates. That's perfectly fine if Signals are values, but it's not acceptable to drop events.

Conversely, there was unexpected behavior when using foldp, because foldp ignores the current value (because it was intended only for events, not values).

So even though Elm is statically typed, and even though these APIs both accepted the Signal type, they actually had very different behavior and expectations! This ended up causing bugs in real applications.

There are also some performance implications: because Signals were used for events, that means they must never drop updates. And it also means that updates must always occur in the correct order.

That's all well and good for events, but values do not need those restrictions, and so it's possible to implement values more efficiently (by dropping intermediate updates, by delaying updates, by re-ordering updates, etc.)

So by forcing Signals to act as both events and values, that caused the values to have an unnecessary performance cost. As far as Rust is concerned, that basically equates to having an extra VecDeque per Signal to hold unpushed events.

In addition, there were some fundamental API restrictions placed upon Signals, and those restrictions existed precisely because Elm was trying to treat Signals as both values and events at the same time:

https://youtu.be/Agu6jipKfYw?t=804 (it gives a good overview of FRP in general, I recommend watching the full video)

Basically, because foldp allows for state (which is necessary when dealing with events), it doesn't (sanely) allow for dynamically changing the Signal graph.

But dynamically changing the Signal graph is a completely reasonable (and useful!) thing to do, so this restriction isn't great.

All of these issues were solved by splitting Signals into two types: Stream and Varying. Stream is used for events, and Varying is used for values.

Now it's straightforward to only define foldp for Stream (not for Varying). And now because foldp no longer exists on Varying, it's now possible to have dynamic signal graphs.

In addition, the merge function now has different behavior: for Varying it will drop the second update, but for Stream it will keep both updates. So it now behaves sanely in all situations, no more bugs!

To use a Rust analogy, Elm's Varying is similar to futures_signals::Signal, and Elm's Stream is similar to futures::Stream. The first is intended for a value which changes over time, whereas the second is intended for events.

I took a look at the most recent version of Elm, and as far as I can see they've completely abandoned FRP, now they essentially use some sort of event subscription + effect system. So everything I mentioned is just for historical reference.

[Pauan]

Here are some relevant (very old) discussions from Elm that I found:

https://groups.google.com/d/msg/elm-discuss/w2Rmim4IUn4/pVOZlvZGTqoJ
https://groups.google.com/d/msg/elm-discuss/w2Rmim4IUn4/IydCXb1tDc4J
https://groups.google.com/d/msg/elm-discuss/w2Rmim4IUn4/EgWWQ2XR1SQJ
https://groups.google.com/d/msg/elm-discuss/Okh03gltc2c/zAUHcdCWCwAJ
https://groups.google.com/d/msg/elm-discuss/xq6iKNACsxw/Sq_UMcJ6xpcJ

Keep in mind the above posts are generally discussing the old Signal system, before the split.

None of them really go into why Elm made the split, or what exactly the split was, but they do provide some interesting background.

The post by John Mayer is particularly good, and clearly explains what the difference is between a Stream and a Varying (e.g. mouse click vs mouse position).

I also wrote a detailed GitHub comment which explains some of the design decisions I made with my futures-signals library:

Pauan/rust-signals#1 (comment)

I've spent the past several years researching (and using in practice) various different FRP systems, so I have a good amount of knowledge about the different designs and trade-offs. I'm happy to discuss more, if anybody is interested or has any questions.

[OddCoincidence]

Thank you for all the links / info, I have quite a bit of reading to do. :)

There are also some performance implications: because Signals were used for events, that means they must never drop updates. And it also means that updates must always occur in the correct order.

That's all well and good for events, but values do not need those restrictions, and so it's possible to implement values more efficiently (by dropping intermediate updates, by delaying updates, by re-ordering updates, etc.)

Since Stream is a trait in rust, couldn't these performance concerns / behavioral differences be handled by a particular Stream implementation? Also - the Signal trait in your futures-signals crate looks extremely similar to Stream to me. What if instead of making it a separate trait, futures-signals provided Stream extensions for combinators that only make sense on signals?

(You are probably correct here as you are much more knowledgable about this than I am, but since it seems that streams are destined to become a core rust language abstraction for any asynchronous series of values, I feel that we should be absolutely certain that these differences are irreconcilable with the Stream trait before committing to a different abstraction.)

I took a look at the most recent version of Elm, and as far as I can see they've completely abandoned FRP

I had not realized this, that's disappointing.

[iamcodemaker]

Blog post explaining why Elm abandoned FRP: https://elm-lang.org/blog/farewell-to-frp

In short, they found a simpler, easier to learn design that did what was needed.

[fitzgen]

Thanks for all the context, everyone. I enjoyed reading all that and watching Evan's talk.

I think it eventually makes sense to have a signals submodule in our utility crates that exposes Web APIs as Signals (where that makes sense) similar to the callback and future submodules design we've decided upon.

I have a few concerns that I think we should address before we actually do this:

• The futures-signals crate currently uses the new futures design, and wasm-bindgen-futures uses the current stable futures. We plan on switching wasm-bindgen-futures over to the new design as soon as it becomes stable. In the meantime, either the futures-signals crate temporarily ports to the old, stable futures, or we delay movement here until the new futures are stable.

• How stable will the futures-signals interface and semver version be? When we are taking on a public dependency (e.g. implementing the Signal trait), that means that our stability and version bumps get tied to that public dependency's stability and version bumps. The futures-signals crate doesn't have a ton of usage yet (chicken and egg problem) and therefore it isn't clear to me how stable the traits and interface will be in the face of more usage experience.

• Semi-relatedly, the futures-signals crate currently has a single owner/maintainer. For private dependencies, this isn't that big of a deal, since we can always fork and swap out an internal implementation if need be. For public dependencies, it means that in a worst-case scenario, if futures-signals becomes unmaintained and @Pauan moves on (for whatever reason), then we are forced into a fork and a major version bump. I would like to see more active maintainers in general, and at minimum I think if we build support into Gloo, then more Gloo folks should have publish rights on crates.io.

Put all these things together and I think this is a reasonable plan for the near term: build an FRP/signals-based framework/library on top of Gloo outside of this repository, before we attempt adding signals submodules to the utility crates. Potentially this is "just" porting Dominator over to wasm-bindgen and Gloo. This FRP-based framework/library is done with the idea of making implementing the eventual signals modules inside Gloo easier (easy to separate bits for inclusion into Gloo crates eventually). This would give us some Real World experience not just with writing FRP/signals-based APIs for our Web APIs in Rust, but also would help validate that Gloo can be an enabler for writing Web frameworks in Rust. This would also make room for "useful" work to be done while waiting for the new futures to stabilize, rather than just backporting something that we will throw away in a couple months anyways.

What do y'all think?

[Pauan]

@OddCoincidence Since Stream is a trait in rust, couldn't these performance concerns / behavioral differences be handled by a particular Stream implementation?

Unfortunately, they cannot.

The actual methods are defined on StreamExt, and StreamExt is defined for all Streams.

A lot of APIs behave completely differently between Signal and Stream:

• chain makes no sense for Signals.
• flatten behaves completely differently with Signals.
• zip behaves completely differently with Signals.
• fuse makes no sense for Signals.
• filter has different behavior and a different type with Signals.
• fold makes no sense for Signals, and unleashes the issues Elm ran into.
• etc.

It's an API issue, not an implementation issue, so merely changing the implementation doesn't solve the problem.

Also - the Signal trait in your futures-signals crate looks extremely similar to Stream to me.

If you only look at the type of poll_change, then yes they look very similar. But there are some differences:

A Signal has a contract which requires it to always have a current value. Thus when a Signal is first polled, it must return Poll::Ready(Some(current_value)) (I need to document this contract).

This requirement doesn't exist for Streams, which might not have any values at all (i.e. they might be currently empty).

There is also the fact that Streams are generally expected to contain all values in the correct order (unless you explicitly use an API which drops values), whereas Signals are expected to drop values whenever they want (as long as they don't drop the current value). This has huge implications for performance and preventing bugs.

Also, the primary reason why Signal is a new trait is not because the underlying types are different; the primary reason is to exclude Stream APIs (like fold), and to add APIs which only make sense for Signal (like map2).

In other words, it's the same reason people use newtypes in Haskell/Rust: even though the underlying type might be the same, the API/behavior is different.

As an example, you might have two types: Fahrenheit and Celcius. They might both be an f64 internally, but they certainly have different behavior, and should not be mixed up!

So the fact that the internal type of Stream/Signal is similar does not mean that they can be unified (just like how Fahrenheit and Celcius both using f64 does not mean they can be unified!)

it seems that streams are destined to become a core rust language abstraction for any asynchronous series of values

Indeed, Streams work great for that! But a Signal is not an asynchronous series of values.

The way to think of it is that a Signal is a mutable variable that notifies you when it changes, whereas a Stream is an asynchronous Vec.

Naturally their behavior is rather different, even if the underlying types might seem similar.

I feel that we should be absolutely certain that these differences are irreconcilable with the Stream trait before committing to a different abstraction.

I'm never 100% sure of anything, but I'm 99.999% sure that they are incompatible in their implementation, in their API, and conceptually/mathematically.

There's a reason why FRP splits things into Behavior and Event (despite academics considering this inelegant, and spending a lot of effort in trying to unify them).

Also, I don't see it as "committing" to anything. I have always advocated for using Stream where it makes sense, and using Signal where it makes sense. It's not an either-or decision.

Some web APIs might even make sense for both Stream and Signal, in which case we should provide both!

In the same way that some APIs might make sense for both Future and Stream, so we should provide both.

What if instead of making it a separate trait, futures-signals provided Stream extensions for combinators that only make sense on signals?

If I tried to do that, then:

• Signals would automatically get all of the Stream methods (including the Stream methods which don't make sense for Signals).

• Streams would automatically get all of the Signal methods (including the Signal methods which don't make sense for Streams).

This would break the Signals contract, and unleash all the issues that Elm ran into.

There's no way to "remove" extension methods, you can only add extension methods. But correct behavior of Stream/Signal requires the removal of APIs.

P.S. Thank you for asking all these questions! It will help me to write up a good explanation of why Signals are useful (and why we can't just use Streams).

[Pauan]

@fitzgen In the meantime, either the futures-signals crate temporarily ports to the old, stable futures, or we delay movement here until the new futures are stable.

I'm really not excited about forking Signals to use Futures 0.1 (I already spent a lot of effort moving from Futures 0.2 to Futures 0.3)

Because Futures 0.3 provides some compat shims which allows it to work with Futures 0.1, I believe it should be possible to use those shims to use futures-signals with Futures 0.1 (though I haven't tested it).

If those compat shims don't work, I'm okay with waiting for Gloo to move to Futures 0.3

How stable will the futures-signals interface and semver version be?

This is a good question! futures-signals has been around since March 2018, and it originally had a lot of churn (since it was hard to figure out what API works best with Rust's memory model), but things have stabilized a lot since then.

The most recent breaking change was in November 2018, and it was an extremely small breaking change that only affected one very specific API (the commit looks big, but the breaking change itself is very small). The core has not changed in a long time.

I'm pretty confident that the API and implementation of futures-signals is correct, and is unlikely to change. I've used futures-signals extensively in combination with dominator in real (non-toy) programs, and I haven't run into any problems in a long time.

Naturally if a breaking change occurs, it will use semver (as usual). And once futures becomes 1.0 I also plan to move futures-signals to 1.0 as well.

Semi-relatedly, the futures-signals crate currently has a single owner/maintainer.

Yes, this bothers me a lot as well! I would like to have more contributors/maintainers.

I've also considered moving futures-signals into rust-lang-nursery (along with futures-rs), since I think it fits in very nicely with the overall Futures ecosystem.

Put all these things together and I think this is a reasonable plan for the near term [...]

That all sounds great to me!

[limira]

Hi,

build an FRP/signals-based framework/library on top of Gloo outside of this repository, before we attempt adding signals submodules to the utility crates

I think it's worth to mention that I am doing an experiment with @Pauan's futures-signals. It is an Elm-like framework that tries to help users separate application logic away from the view renderer.