Automated editor interaction for documentation screenshots and integration tests

[nathanfranke]

Describe the project you are working on

Godot documentation

See also:

• NathanLovato's comprehensive documentation proposals, including a proposal to automate screenshots: Proposal: redesigning the docs for Godot 4 godot-docs#3390
• Specific comment with some third-party tool suggestions: Proposal: redesigning the docs for Godot 4 godot-docs#3390 (comment)
• Add accessibility/screen reader integration to the base UI components #1964

Describe the problem or limitation you are having in your project

As Godot 4 approaches release, we need to rewrite a great deal of the documentation. One of the most tedious parts of this is the screenshots.

• Contributors may have different editor themes, and may forget about their minor theme changes.
• We need some way to highlight sections of an image, and I can guarantee that no matter how much planning, contributors will still create them in slightly different ways.
  
• The engine changes (wow!). Having this automated would 1) throw an error if something is wrong (e.g. if a setting is removed) and 2) stay consistent with the engine (font changes, theme changes, layout changes, etc.)

Also integration tests

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Some notes:

• I do not have any experience with this sort of automation. I am making a proposal to facilitate discussion on how this should be done properly.
• This feature should prioritize ease of use for documentation contributors. The easier screenshots are to create and manage, the more likely contributors will include them.
• Performance is not a huge deal, since this code should be disabled in all official (and most unofficial) editor builds.

Proposal 1: EditorAutomation singleton (not part of official builds). Vote: 🎉

Add an engine singleton (e.g. EditorAutomation) with methods that make this automation straight-forward. This singleton should be disabled by default and not provided in official builds. It can be enabled in scons with editor_automation=yes.

(Note: If modules can be kept disabled by default, then it should use a module instead to not overcomplicate the root SConstruct. I'm not sure if this is possible).

Pros:

• Would work on every platform that the editor already runs on.
  Cons:
• Adds a lot more code to maintain.

Proposal 2: Fully integrate accessibility and use a third-party tool to interact with the editor. Vote: 🚀

The majority of work will be implementing accessibility.

Pros:

• Resolve two issues with one stone.
  Cons:
• Could be more difficult for cross platform development (e.g. contributors making screenshots), since accessibility is different on every platform.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Proposal 1 cont.

Internally, the singleton would have the basic methods we need.

void click(String name); // Click a button, popup item, etc.
void focus(String name); // Focus something (e.g. focus `Project Settings...` to show users how to open it). Should not be used for highlighting.
void write(String name, String text); // Write text into a LineEdit or related.

void box(Rect2 area); // Queue a box to be drawn in the screenshot.
void arrow(Vector2 from, Vector2 to); // Queue an arrow.
void screenshot(Rect2 bounds); // Output file?

• Note the use of "name". We should make it possible to access most UI elements using some readable name (semantics). This work will not go to waste, as it can be re-used when accessibility is needed. Labels, Buttons, etc. already have text values, but more work may be needed for other inputs such as LineEdit.
• We should agree on some arbitrary size to base our measurements on. For example, 1280x720. When actually generating screenshots, however, we should scale up the editor. For example, screen-shooting 200x200+200,200 should result in a 400x400 image if the editor is at 200% scale (1440p). Years from now we should be able to change this without much issue.
• Not sure what the implications of this are, but we should save screenshots in a format such that subsequent screenshots, when unchanged, do not create unnecessary version diffs. This should be possible using oxipng. @aaronfranke
• We should assume that after clicking/focusing/etc., the engine takes a maximum of 1 frame to render and prepare for the next action. Otherwise, this is probably a bug but we could always add a "delay" script method.

When it comes to running these methods:

• It could be useful to have some text version. This way, contributors can "script" their screenshots with relative ease.

  click("Project")
  click("Project Settings...")
  box(200, 200, 200, 200) # project name
  screenshot("changing_project_name.png")

• For tests, I think it's best to run them within C++, as additional checks may be needed.

  EditorAutomation::get_singleton()->click("Project"); // How do we wait a frame?
  EditorAutomation::get_singleton()->click("Project Settings...");
  EditorAutomation::get_singleton()->write("Project Name", "hello world");
  // save somehow
  assert_somehow(ProjectSettings::get_singleton()->get("application/config/name") == "hello world")

Proposal 2 cont.

This will of course vary greatly based on what third-party tool we choose, but the semantics portion will stay the same; we need some way to identify screen elements with a readable name.

We will also need another way to draw on the screen. This may be possible with the tool we use, otherwise we will probably use something like ImageMagick. As for interfacing that with our tool suite, not sure.

If this enhancement will not be used often, can it be worked around with a few lines of script?

Not without tool script hacking.

Is there a reason why this should be core and not an add-on in the asset library?

Documentation+integration tests are core

[Calinou]

Related to #1760. See my proof of concept linked there 🙂