Not all frameworks are created equal. Are we comparing apples to oranges?

[jorgebucaran]

I would like to revisit and understand the goals of this project. If anyone knows, please let me know 🙏 @krausest @leeoniya

I use js-framework-benchmark all the time to see how my frameworks fare against other VirtualDOM-based libraries, e.g., Inferno, React, domvm, etc., as well as the vanilla JavaScript implementation.

But not all frameworks are created equal and some of the results derived from this benchmark test can be rather uninformative. 😅

I certainly want to compare my results against a vanilla JavaScript implementation of the same example. That's how I know how I fare against the "raw metal", but I am disappointed every time I see a new framework claiming to be blazing / lightning fast & turbo-boosted when they are basically a glorified wrapper around createElement, insertBefore and removeChild.

Clearly a wrapper library around the CRUD DOM API will score just as well as the vanilla implementation! But certainly their authors would like to fine tune their work too.

Would it be possible to compare only VirtualDOM-based libraries together and those that are not based on a VirtualDOM visible under a different filter?

Disclaimer: I am not trying to criticize any of the frameworks included in this benchmark, whether I like using some of them or not is my personal opinion and it's definitely not what I am trying to say here.

[jorgebucaran]

@krausest BTW you did a great job updating the documentation since the first time I installed this module. I reinstalled and tried again from scratch recently following the instructions and I have had
no problems ever since. 👏👍

[leeoniya]

i could have sworn this was in the readme somewhere (maybe it was removed). the goal is to compare the performance characteristics of data-driven javascript view layers (or frameworks). it compares the efficiency of dom operations and the ovethead incurred by each lib & state management solution.

simple dom wrappers dont fit this category because they have no generic reconciliation algorithms that enable declarative, data-driven implementations. also see #126.

hope that helps!

[jorgebucaran]

@leeoniya In that case what about oddballs like svelte? Or non-VirtualDOM-based libraries like RE:DOM?

[leeoniya]

svelte is still declarative; it's just compiled. RE:DOM and surplus sometimes reach a bit low into the DOM in their impls for my taste, but they're still both very much data-driven.

[localvoid]

But not all frameworks are created equal and some of the results derived from this benchmark test can be rather uninformative.

Especially some small ones that ignore many features, edge cases, browser quirks, etc :)

[jorgebucaran]

@localvoid That too definitely. 👍

[adamhaile]

If you're talking about surplus, I'm more than happy to discuss the design decisions there. I'd only ask that people be clear. I can't decipher all the innuendo being thrown around ("oddballs," "some small ones," what?).

Vdom libraries are great, and there are lots of smart ones out there (e.g. present company), but it's not the only way to be declarative or data-driven. Like all things, it's a design decision with tradeoffs.

One of the costs of vdom is that it makes interop with other libraries and code difficult, because the data structure they share in common, the DOM, is walled off and "owned" by the vdom library. There are certain access ports, like React refs, and also a "lifecycle" that must be learned, all of which enable but also complicate the interop situation.

Surplus intentionally takes a different path. It's just a compiler+runtime for creating declaratively-defined and automatically-updated DOM nodes. What you do with them is your call, and you can do stuff with them, because it's just the DOM API. Want to do some of the node configuration in your own code? Sure. Want to stick a jQuery DateTimePicker in there somewhere? Go ahead. Did your boss buy a Telerik license and wants you to put some Telerik widgets in the app? I'm sorry for you, but surplus won't stand in your way. This might not have been possible 5 years ago, when each browsers' DOM API was quite different, but we're (thankfully) not in that world anymore.
-- Adam

[leeoniya]

@adamhaile

there are really 2 types of virtual dom - one where the left side of the diff is an old vtree that lives in memory and one where the left side is the real dom. in both versions, the declarative structure creates a new vtree, but in one case it becomes the new truth and in the other it's thrown away after the real dom is reconciled.

your examples of interop are all isolated widgets, not necessarily jquery code that will mutate the declarative dom already generated by Surplus, correct? there are multiple vdom libs that support the "widget" type of interop, where the vdom lib just avoids touching elements (or attrs) it did not create.

also, "learning" of lifecycle hooks, etc. is replaced by learning how to deal properly with state management in Surplus, since it depends on tightly-coupled data structures, no?

[localvoid]

@adamhaile

I'd only ask that people be clear.

I've talked about small vdom libraries that @jbucaran is probably trying to compare against React, etc :)

[leeoniya]

if anyone's ever confused about @localvoid's vague comments, he's usually trolling :D

[adamhaile]

@leeoniya I'd heard of the diff-to-dom vs diff-to-last-version distinction, but thought it was just about optimization. I.e. diff-to-last-version is faster, as it's a pure JS operation. I never thought about it, but I guess there's a behavioral difference in that if the DOM changed underneath the last version, it wouldn't show up in the diff, so therefore diff-to-last-version is more permissive?

I know that there are ways to get to the DOM in most vdom libraries, particularly for the "widget" scenario. The point is that each vdom library has its own terms and its own time in which you can do that. Vdom libraries are "DOM on the inside, with restricted access"; surplus tries to be "DOM on the outside." It's an intentional experiment (meaning, encouraging but still not 100% proven out) in transparency and easy interop.

For the surplus implementation of this benchmark, I'm intentionally using that capability in a couple places, like setting the className of rows. I'm uglily (if that's a word?) using to set the aria- attributes right now. Won't excuse that bit -- a better fix is coming.

also, "learning" of lifecycle hooks, etc. is replaced by learning how to deal properly with state management in Surplus, since it depends on tightly-coupled data structures, no?

Those are different parts of the pipeline, aren't they? I.e., we all need to get data into our library and DOM out. All the libraries have some way to get data in, whether it's calling setState() or update() or modifying S data signals. The vdom libraries impose a lifecycle on the DOM out process, which surplus doesn't have. You just get the node immediately.

You can use surplus without S, but the DOM you get will be static, not updating, so not so useful.

Huh, that makes me wonder how hard it would be to port surplus to other reactive systems, like mobx or RxJS. Might try that for a weekend hack :).

[localvoid]

@leeoniya I am quite serious, it is impossible to get apples-to-apples comparison even with just virtual dom libraries.

For example, here are some differences between React and ivi that has a noticeable impact on performance:

React

• Immutable virtual dom nodes.
• Key namespaces (there won't be any key collisions when there are several nested arrays)
• It normalizes way much more browser differences for all public APIs (all other vdom libraries aren't even close to React)
• It will work when there are several different React instances on the page.
• Abstraction that allow to easily experiment with different rendering engines (Native, Canvas, Terminal, etc).

ivi

• Working context updates
• updated lifecycle is always invoked for all parents when some descendant were updated
• attached lifecycle is invoked from top to bottom (requires to go through entire vdom tree twice when it is created)

And this list of differences can go on and on. Some of the React features aren't important for me, but I am pretty sure that they are important for the React team. And many small vdom libraries can't even rearrange nodes in a predictable way, Vue2 is probably one of the worst cases when it comes to rearranging nodes, so there won't be any apples-to-apples comparison.

[leeoniya]

you're right, of course. but you can say this about any non-thorough benchmark that covers libraries not adhering to a single, uniform specification. for instance, it's easy to say that different implementations of a hashing algorithm can be compared purely on their cpu/mem metrics because they otherwise are only defined by correct output. that would be green apples to green apples. (even then, you can say one impl prioritizes security/safety over speed.)

view libs and frameworks will cover different use-cases, because what would be the point of re-creating an identical framework? of course there's 75% common overlap in what they can do, and this bench tries to compare at least a decent portion of that overlap. i'd say this benchmark is red apples to green apples comparison. it's definitely not like comparing clean room API-matched libs like react/preact/react-lite/inferno.

[jorgebucaran]

what would be the point of re-creating an identical framework?

😏

[localvoid]

of course there's 75% common overlap in what they can do, and this bench tries to compare at least a decent portion of that overlap.

Yes, but "I am disappointed every time I see a new vdom framework claiming to be blazing / lightning fast & turbo-boosted when they are basically a glorified wrapper around createElement, insertBefore and removeChild." :)

[jorgebucaran]

@localvoid Exactly my thoughts! 👍

[localvoid]

@jbucaran I understand that it comes from inexperience and I've made similar mistakes when I've started to work on a virtual dom library.

Btw, what do you think when looking at things like this:

• noNormalize madness
• ref=sCU

Also, there are libraries in this benchmark that has good results in the benchmark, but in real applications they will trigger deopts in all hot code paths and their performance will be even worse :)

You just can't take seriously many of this numbers when you know all this details.

[jorgebucaran]

@localvoid Btw, what do you think when looking at things like this:

I am not too familiar with Inferno's custom API options, but I think it looks a bit awkward. 🤔

I'd be willing to pay the penalty for lacking some of those "power features" just so that I don't have to deal with the added complexity.

[pakastin]

RE:DOM sure is data-driven and for example makes it so much easier to work with 3rd party code than virtual dom libraries. Having more control over what's happening and when also makes it different. Sometimes when you go too far abstracting things, you give up a lot of control.

[pakastin]

When "JS framework benchmark" got renamed to "virtual dom framework benchmark"? I didn't get the memo. There's not a single mention of "virtual dom" in the README.

[krausest]

Maybe I also should add my 2 ct.
@jbucaran > I would like to revisit and understand the goals of this project.
I just wanted to know how angular 2, angularjs or react compared performance wise. Then suddenly all went out of control and ~60 framework implementations were added. At the same time it turned out that Angular 2 and React actually were almost indistinguishably fast. 😀

Seriously: I'm not fond of restricting the benchmark to virtual dom frameworks - as a developer I'm just asking for a framework maybe combined with a language with a nice API and good performance.
Personally I appreciate the honesty of idiomatic implementations close to the framework's docs (if such exist...).

[jorgebucaran]

@krausest Thanks! I had no idea how things went and it's funny to know you also think it went out of control. 😄

[eavichay]

I have found out that manually testing on latest firefox turns the tables. React is much slower compared to other libraries. I am not sure why, but it looks like the rendering engine is way improved and perhaps even have it's own "diffing" mechanism for dom changes. Also the developer tools have been upgraded. Probably there is a way to figure out how to extract perf. data out of firefox.

[pakastin]

I need to try out RE:DOM in Firefox, that sounds promising! 👌