[v1.5.1] Usability improvements and content update

This is an impromptu patch that came to be because of some experimentation I did with a solution to make it easier to interact with sockets using the mouse to establish new connections. I call this solution socket proximity detection. It consists in detecting nearby sockets whenever:

• clicking near them in order to establish a new connection
• releasing the mouse dragging the new connection near the other socket

In other words, now doing both things is easier, because the mouse doesn't need to be exactly over the sockets for any of those actions to take place.

Additionally, special graphics/shapes are drawn on the screen when dragging a new connection to another socket in order to signal the detection to the user.

For instance, here's what it looks like when the socket detection graphics are "reaching hands":

In the image, the mouse cursor becomes a hand that extends from the first socket. The second socket is detected due to its proximity, so a hand also extends from it to meet the hand cursor. Granted, a static image doesn't do justice to the usefulness of the feature nor to how smooth the movement is.

There's also no risk of the wrong socket connecting by mistake by moving the mouse accidentally close to another socket because the connection is only established when the mouse button is released, so you can move the cursor freely and only release it when there's visual confirmation that the socket you intend to connect to was properly detected and is in range (for this particular graphics used, we know the socket is in range when the hands hold each other at the cursor).

The user can even customize the experience by:

• picking one of the following graphics to be used:
  • assisting line
  • reaching hands
  • hands and eyes
  • baseball elements
  • baseball elements and eyes
• setting the distance at which a target socket is detected when the mouse is dragging a new connection
• setting the distance at which a connection can be established with the detected socket

A few key content pieces were update as well. The first chapter of the user manual now has a new subsection on sharing resources among nodes. A very useful and long needed resource. In addition to that, a few other parts of the chapter were improved for clarity.

The README file of the repository is much more informative now and makes usage of more images and animated GIFs to showcase Nodezator features and capabilities. Like the first chapter of the manual, the README file also had several parts of its text improved for clarity.

We also made sure to include important information and warnings to avoid common misconceptions people tend to have when learning about Nodezator for the first time. For instance, some people seem to have a hard time telling whether Nodezator is an application or a framework/GUI library at first, so in the text we made sure to emphasize what Nodezator is and what it is not.

Finally, made a few small changes to the menu subpackage to make it easier to use widgets like checkbuttons and radiobuttons on the menus. The menu subpackage always had such widgets available, but we never needed to use them before. Now we use radiobuttons to allow users to choose their preferred socket detection graphics, so it was a great opportunity to improve their back end a bit. The appearance of the radiobuttons was improved as well. They are now rendered from SVG text.

[v1.5.0] System testing, topological sorting and QOL features

The scheduled work for this release consisted of 02 features, but we delivered additional QOL (quality-of-life) features and other changes as well. The scheduled features were:

• Finishing and releasing the automated system testing service
• Replacing the makeshift graph execution algorithm by topological sorting

Among the additional QOL features, we highlight:

• Birds eye view
• The ability to jump to any node in the graph by providing its id

The remaining changes will be presented futher ahead, in a dedicated section.

In addition to the information presented in this release text, you can find much of the development process documented in this GitHub discussion: #72. It was used as a development log (devlog) detailing changes and related decisions.

New automated system testing service

This release was focused on providing a new key feature to aid in development of Nodezator: an automated system testing service.

Nodezator, being a node editor, is a software with many features, tools and a varied workflow. There are innumerable actions that users can take and underlying systems supporting it. Additionally, being an open-source software project, it is subject to external development work being merged into it. Even if there was only my own work being merged into it, the use-cases/systems are just too many to be manually tested by a single person.

In the past, despite my careful testing hundreds of times, I still couldn't manually test every possible action and it resulted in some nasty bugs being introduced in some past releases (bugs that were already solved). Although we also have unit tests, the coverage is still low and, even if it was high, the tests would only be able to ensure the isolated units work. This means we needed a tool that could test the entire system, ensuring every part works in tandem to perform each and every task the user can perform in Nodezator.

In other words, we needed a tool that ensured that, given the same actions performed by a user like clicking some element, typing something, pressing specific keys and key combinations, in order to perform a specific task, the app would be able to successfully perform that task, resulting in specific outputs and/or state.

This tool is called System testing. At the beginning, because it relies on GUI automation, we were mistakenly calling it automated GUI testing. However, because we are interested in testing the entire system, not only the graphical user interface, system testing is indeed the right term.

Performing system tests

Since version 1.4, Nodezator featured a Playback submenu on the menubar, although none of its options were available to users. In 1.5, this submenu was renamed to GUI automation and its Automated GUI tests option was renamed to System testing. This System testing submenu has all the options needed to set and perform an automated system testing session, with some keybindings assigned to the most common ones. Here's an image depicting the GUI automation submenu as well as the options available in the System testing submenu inside it:

Among the options within the System testing submenu, there's one that presents a form where a custom testing session can be set and triggered. The form is presented in the image below:

The form can be used to pick which test cases must be executed and the speed used. There's an speed option because although most people will usually want the tests to run in top speed, it might be useful to run specific tests in normal or slower speed in order to careful observe the execution of the task to help spot possible errors/mistakes.

Once an automated system testing session begins, the app starts moving by itself, executing each task. During the session, there are several controls that can be used to perform tasks like pausing/resuming the session or aborting it. You can check such controls in the general controls page (that you can access in the Help > General controls menubar option).

Once a system testing session ends, a report is presented. Here's what the initial portion of the report looks like:

The actual report is much larger since it has details about the system and each test case executed.

On the bottom of the report there are also buttons to export it to other useful formats like .html, .png and .pyl (a Python file containing only literals).

Remaining work for system testing and Nodezator in general

All that remains is to populate the system testing service with all the needed test cases, which is expected to be done in the next patch or so.

In order to aid in development of not only of the system testing feature, but the whole app, a holistic view of its design is needed. Each and every task and feature must be listed. This is specially important for system testing because we need such information in order to define and implement all the needed test cases.

Because of that, since the beginning of our scheduled work for this version, we also started producing a Software Design Document (SDD) for Nodezator, which you can find in this dedicated repository: https://github.com/IndiePython/nodezator-sdd

A system testing document is also being produced. It is kept in a separate repository although it is considered part of Nodezator's SDD: https://github.com/IndiePython/nodezator-system-testing. It is used to help manage system testing for Nodezator and to document its test cases. It is kept as a separate repository so changes to system testing can be managed and tracked separately from the rest of the SDD.

Finishing those documents is important not only for system testing, but also for Nodezator in general. For instance, once these documents are ready, we'll be able to organize the missing features in a sensible order for scheduling and implementation. We'll also be able to present a proper roadmap to users of the app and followers of the project.

Breadth-first-search-based topological sorting for graph execution

Replacing our makeshift graph execution solution by one based on the topological sorting algorithm/technique was a feature request made by Mr. Alexander and accepted after considering the merits of the request.

Although other materials on the topic were researched, the most concise and straightforward definition for the topological sorting technique/algorithm was found on Quora (A Questions & Answers platform):

Topological sort is a fundamental technique in graph theory that organizes the nodes of a directed acyclic graph (DAG) while maintaining the edge direction. The significance of topological sort stems from its ability to offer a linear ordering of nodes, which may be used to address a range of issues such as scheduling, job sequencing, and dependency resolution. (source)

In summary, topological sorting has many applications in computer science and software engineering and is an established key technique to solve many different kinds of problems. The specific algorithm picked was one based on a breadth-first search algorithm and also recommended by Mr. Alexander. This means nodes which don't require inputs are listed first, gathered in groups called generations, then the nodes immediately fed by their outputs are also listed as their own generation and so on until all the existing nodes are listed.

It should be particularly useful to us in the future when we implement parallel execution, since the algorithm groups together nodes which don't depend on each other and can thus be executed in parallel.

We implemented a custom solution adapted from this pseudocode and with insight from other researched materials. Our solution presented the same output but superior speed to an alternative solution using an established 3rd-party library for manipulating graphs, the NetworkX library. Specifically, this NetworkX tool was used.

For the record, we don't fault NetworkX for its inferior speed in this specific case, since, being a library, we assume its offers a general solution designed to meet the requirements of many environments, whereas our custom solution is free to take advantage of local services and existing data structures to achieve top performance. NetworkX actually helped us validate our custom solution, that is, because NetworkX is a renowned library and our custom solution produced the same output as its solution.

In the distant future we intend to research NetworkX further, as well as other graph manipulation libraries like igraph. This would be done in order to learn even more about our problem ...

[v1.4.6] More additions and a bug fix

Among the many changes/additions for this patch release, we have:

• code contributed by Mr. @sefgit via pull request, that consists of:
  • mousewheel pan feature
  • the addition of a .gitattributes file
• code contributed by me (@KennedyRichard):
  • the ability to confirm/dismiss/navigate the dialog using the keyboard
  • a bug fix for a dialog that was growing past the screen

The mousewheel pan feature is the ability to pan/move the graph with the mousewheel. The .gitattributes file added is a simple text file that is used to define attributes to paths. Certain operations by Git can be influenced by assigning particular attributes to paths, which is what this file is used for. Some additional changes already discussed with the contributor were also made by me.

I also added/set the ability to confirm/dismiss dialogs with the keyboard using the Enter and Escape keys, respectively (though, often, confirming will also dismiss the dialog when it is the only action available). I use the terms "added/set" because part of the feature was already implemented and just needed to be properly set in the existing dialogs defined throughout the source. Naturally, some dialogs may not offer the option to be so casually dismissed like that when the consequences of the action cannot be reversed. Currently, though, we don't have any dialog like that.

The user can now also navigate the available dialog buttons/options using arrow keys or the Tab and Shift+Tab keys.

The bug fix consisted in preventing a specific dialog from growing past the screen.

In this specific bug, we were allowing text from error reports to be displayed on the dialog. The right practice, however, would be to just use known predefined text and record the error report in a more appropriate place for the user to check. Text from error reports can grow a lot when there are too many errors, so we must not use such text in dialogs. That's what was causing the problem.

Now, instead of reporting the error(s) on the dialog, we just display an informative small message and log the errors in both the default logger and our custom user logger. This way the user can check the error report using the log files or the user log that can be accessed on the app's GUI. The user log can be accessed by pressing Ctrl+Shift+j when you are on the graph/canvas or by selecting the Help > Show user log option on the menubar.

[v1.4.5] Additions, improvements and fixes to tests

New unit tests for the function used to load nodes were provided by Mr. @BMaxV via pull request. I @KennedyRichard also refactored and complemented them.

I also used the opportunity to make some small improvements in the doctests and to fix one of them.

In order to make it more convenient to store test data, .zip files found in Nodezator's source are now also included in the source distribution. That is, they are downloaded in conjunction with the Python modules when someone installs Nodezator.

[v1.4.4] Fixed bugs in SVG exporting and change checking

The bug in SVG exporting was fixed by removing unused code. The unused code was causing a name error because it was made from repurposed pre-existing code and was trying to access a variable that didn't exist in the new spot. Since HTML exporting relies on SVG exporting, HTML exporting was also being affected.

Some other lines that weren't causing errors but were not needed either were also removed to avoid confusion and needless maintenance.

The other bug fixed is related to change checking, that is, checking whether a change in a widget's value took place. Change checking was causing errors due to missing type checks. Such type checks were missing in different modules in the source and are now implemented.

More specifically, when a number changed, if the value was the same, even though the type was different, that change wasn't being treated properly, because the app was only checking the value and reaching the conclusion that nothing changed, because the value was the same in the end. However, this is not the right thing to do. That is, we must also take the type into account, because numeric values may have the same value and still have different types.

For instance, 0, 0.0 and 0j all have the same value, although they are an integer, a float and a complex number respectively. Because of that, wherever a value was being checked to determine whether a change took place, we also added a type check so this kind of change is also taken into account.

[v1.4.3] Fixed error when executing operation nodes

The error was caused by using the wrong attribute when retrieving the existing sockets from an operation node. Using the correct attribute name, by including an if/else clause that picked the appropriate attribute when needed, fixed the error.

The error was causing execution of the graph to fail altogether whenever operation nodes were present.

[v1.4.2] Fixed error when executing collapsed nodes

The error was caused by using the wrong attribute when retrieving the existing sockets from a collapsed node. Using the correct attribute name fixed the error.

The error was causing the output of collapsed nodes to be handled in the wrong way because it wasn't able to accurately determine the number of existing output sockets. This problem could cause all sorts of other problems, like the graph failing to execute or the wrong data being passed along to other nodes.

[v1.4.1] Tiny packaging fix

This tiny fix consists of the addition of a missing empty __init__.py file for packaging purposes. The absence of this file was causing some modules to be ommited in the pip installation, making the app crash on launch due to the missing modules.

[v1.4.0] Many new features and improvements

Adoption of pygame-ce for the graphical user interface

As previously announced on GitHub with appropriate arguments/evidence in favor of the change (and after giving users enough time to voice their opinions), we now use pygame-ce (pygame community edition) for Nodezator's GUI, instead of the original pygame library instance.

Since pygame-ce is, for the most part, a drop-in replacement for the original pygame, the change was rather swift.

The only additional measure users need to take is to make sure the regular pygame library is not installed anymore in the Python instance they use to run Nodezator and that pygame-ce is installed. This can be done with these commands:

pip uninstall pygame
pip install pygame-ce --upgrade

Since both libraries use the same name for import statements (the pygame name), if users don't uninstall the original pygame library, it may shadow the pygame-ce library. If this happens, Nodezator will only show a dialog informing the need to uninstall regular pygame and make sure pygame-ce is installed. This is needed because Nodezator is only useable with pygame-ce due to exclusive imports.

Different modes for nodes

All nodes that represent callables now exist in 01 of 03 modes: expanded, collapsed and callable mode. Together, all of these modes contribute for your graph to be more powerful, versatile and compact.

The expanded mode is the default one, in which the node represents a call to its underlying callable, and thus receives inputs and returns outputs, leaving all of them visible so users can easily interact with them. In collapsed mode the node also represents a call, but it is more compact, only showing inputs/outputs that are connected to other nodes. In callable mode, however, the node represents a reference to its callable, rather than a call, and can thus pass this reference to other nodes, enabling users to perform seemingly complex tasks with simplicity, power and versatility.

Such modes are presented in more detail in this section of the manual.

Shadowed widgets are hidden

As we know, input sockets that have a connection use the incoming data from the connection instead of the widget's value when executed. In other words, the widget is not relevant anymore is just adds visual noise to the graph. Because of that, in order to make the graph more readable and free of visual noise, widgets that are shadowed by connections are now hidden, regardless of whether the node is in expanded or collapsed mode. If the connection is severed, the widget becomes visible again.

New viewer node capabilities

Viewer nodes can now display visualizations on the graph/canvas (for instance, from their outputs).

Original strawberry basket image by NickyPe can be found here.

The new viewer nodes supersede the old ones. The old viewer nodes still work fine, but the new ones have more features.

The matplotlib example node pack has a new viewer node called view_figure2 that uses such new features. The Pillow example node pack has several new viewer nodes that use the new features as well, all in the visualization category. Nodezator now also has some general viewer nodes available by default (we'll talk more about this a bit further).

You can learn all you need to know about all viewer node functionality in the manual. There is a simple and some complex ways to define viewer nodes (you should probably start with the simple one). Also, complex viewer nodes aren't better than the simpler ones. The complex ones exist to meet specific visualization needs. The way to define simpler viewer nodes is described in a single chapter. The way to define complex nodes is described across several chapters towards the end of the manual, since they are a more advanced topic. All these new chapters on viewer nodes are listed below:

• Introductory chapter on viewer nodes and the simpler way to define them
  • Defining viewer nodes
• Chapters on more complex viewer nodes
  • Defining a custom visualization loop for viewer nodes
  • Defining a more complex viewer node
  • More advanced viewer node features - Part 1
  • More advanced viewer node features - Part 2

New chapters/appendix in the manual

New chapters were added to the manual. Besides the ones related to viewer nodes mentioned in the previous section, there were also 02 new chapters. One on conditional execution and another on looping in Nodezator. There is also a new appendix with recipes for common tasks.

All these new chapters/appendix are listed below:

• Conditional execution (branching) in Nodezator
• Looping/iterating in Nodezator
• Appendix: Recipes

New categories of app-defined nodes

There are 02 new categories of app-defined nodes (nodes available by default in Nodezator):

• General viewer nodes
• pygame-ce nodes

Because this version introduces new capabilities for viewer nodes, specially the ability to display visualizations on the graph itself beside the viewer nodes, we also introduced new categories of app-defined/default nodes to help in such tasks and other related ones. This way users won't need to write their own nodes for the most basic visualization tasks. As a consequence this also prevents a lot of code repetition that would exist across node packs defined by users if they had to write nodes for such basic actions.

The vast majority of the new nodes are from the pygame-ce nodes category. The image below represents a tiny sample of the many nodes introduced with both categories.

Original pomegranate tree image by AselvadaAna can be found here.

You can learn more about general viewer nodes and pygame-ce nodes in this section and the subsequent one from the manual.

New app-defined nodes in preexisting categories

Besides the many nodes in the new app-defined categories, there are also some new app-defined nodes in preexisting categories:

• New standard library nodes added:
  • operator.attrgetter
  • operator.itemgetter
  • operator.methodcaller
  • 02 different signatures of pathlib.Path
• New snippet nodes (under the Encapsulations submenu of the popup menu):
  • for_item_in_obj_pass
  • get_constant_returner
  • perform_attr_call
  • return_untouched
• The ternary operator a if c else b

Changes in appearance

Inputs closer to the bottom, outputs closer to the top

Although all the changes in this version are backward-compatible, your graphs will look a bit different when opened in the new version. This is so because the position of inputs and outputs in the nodes were changed. Before, the inputs were closer to the top of the node and the outputs were closer to the bottom. Now this was inverted and the inputs are closer to the bottom and the outputs closer to the top, as can be seen in the image below:

This change was made because the empty space that sometimes remains below the outputs of a node (between the last output socket and the bottom of the node) is a proper/nice spot for placing the visualizations shown beside viewer nodes on the graph. This positioning of the sockets is also adopted by other existing node-editing applications, like Blender3D (by which Nodezator is heavily inspired).

The image below shows the same graph opened in the previous version (at the top) and in the new one (at the bottom):

Though they are the same graph with the same nodes and same connections, they appear slightly different due to the position of the sockets (and, as a consequence, the position of the lines that represent their connections). There's not much difference in this small graph, but in larger graphs the differences may be even more pronounced.

Operation nodes are more compact

Operation nodes had their size reduced, speci...

[v1.3.14] Performance fix

It has an important performance fix provided by Mr. Richards (@nrichards) via pull request. In summary, specific pygame.font.Font objects defined after many calculations were not being stored back for later use in the dictionary used for font caching due to a missing dictionary item asignment. The font caching system can be found in the nodezator/fontsman/cache.py module.

This was causing the calculations to be needlessly carried out over and over, whenever text had to be rendered, rather than only once. This was harming the performance of any service that relies in text rendering, specially on startup where most of the interface is rendered and when displaying text in the text viewer or displaying hypertext.

Despite the apparent simplicity of the fix, spotting the performance problem required careful profiling, investigation and testing, all of which were diligently carried out by Mr. Richards. He even wrote unit tests in order to better understand the system.

As a result of the fix, Nodezator is even faster and the speed improvement can be perceived in many common tasks, like launching the app, loading a file, rendering text to be displayed, rendering hypertext like in the in-app manual and many other tasks. For instance, check this video demonstrating the simple task of launching Nodezator and opening the manual in the chapter about viewer nodes, executed in version 1.3.13 and in this new 1.3.14 version:

performance_fix_demonstration.mp4

Other commits consist of slight changes to his original fix and tests. I also updated obsolete docstrings in the font caching module.