Stretch Mode 2D, Mouse Position scaled incorrectly

[nathanfranke]

Godot version:
Release 3.1.1 (Latest)

OS/device including version:
Windows 10

Issue description:
My project is configured at the resolution 1600x900.
When the project is scaled up (I.E Fullscreen) the Viewport size is increased to 1920x1080. However, if the mouse is moved to the bottom right, the value is about 1600,900 (Actually around half a pixel less)

Note: If this is not a bug:
Documentation of Viewport states that the mouse position is relative to the viewport...

Steps to reproduce:
Make a brand new project, change the stretch mode to 2D.
Write a script that prints the mouse position.
Scale the window and check what the mouse position is claimed to be.

Minimal reproduction project:
Test.zip

[chippmann]

I have the same issue with scale to viewport as well. Not only 2d.
OS is Kubuntu 19.04 though.

[Calinou]

There is a way to get an "unscaled" coordinate by dividing it with some method with stretch in the name. (I'm on mobile right now, I don't remember the name.)

[nathanfranke]

get_viewport().warp_mouse() has different behaviour than get_mouse_position(); the former is actually relative to the Viewport, but the latter is not

[chippmann]

There is a way to get an "unscaled" coordinate by dividing it with some method with stretch in the name. (I'm on mobile right now, I don't remember the name.)

@Calinou could you please tell the method name and what you did? I'm curious if that would lead to a usable workaround in the meantime.

[Calinou]

@SGMcejo This seems to work:

var unscaled_mouse_position = get_viewport().get_mouse_position() / get_viewport().get_size_override() * OS.window_size

Test project (displays scaled and unscaled coordinates for comparison): test_mouse_coords.zip

[Grandro]

Hey, this issue has been bugging me for quite some time now, but I believe I
found a solution, or at least a problem with the current source code regarding viewports.
For my solution I am using a custom build of the latest (v.3.4.3 stable) Godot version,
because I implemented it with GDScript, but you cannot access all needed functions there by default.
This made debugging easier, as I didn't have to recompile the engine all the time, but it could
easily be implemented in C++ instead.

I am neither experienced with the Engine's Source Code nor with the concept of all the transformations
you have to do in order to convert coordinates into another space, so let me talk about what I found
out and what I did and correct me if you think I misunderstood something:
(Note: I also only tested this in 2D, I don't know if there are differences)

1. How are Viewport-Coordinates (viewport.cpp) currently implemented?
   Looking at the source code, you can find following function:

Vector2 Viewport::get_mouse_position() const {
	return (get_final_transform().affine_inverse() * _get_input_pre_xform()).xform(Input::get_singleton()->get_mouse_position() - _get_window_offset());
}

1. get_final_transform()

Transform2D Viewport::get_final_transform() const {
	return stretch_transform * global_canvas_transform;
}

stretch_transform:
Updated in _update_stretch_transform()
This takes size_override_size + size_override_margin into account and stretches the coordinates
by a scale.

global_canvas_transform:
Identity Matrix until set manually in set_global_canvas_transform.
I am confused about it being named global_canvas_transform, as it doesn't
seem to take any parent transforms into account.

2. affine_inverse()
   Inverse Transformation Matrix because in this case we have screen coordinates
   but want CanvasLayer mouse coordinates.

3. _get_input_pre_xform()
   This is the Identity Matrix unless set_attach_to_screen_rect has been called.
   This defines a rectangle for the viewport to render in. For the root viewport it erases
   the black borders caused by window resizing.

4. _get_window_offset()
   Irritating name imo, this function gets the global viewport offset by getting the global_position
   of the direct parent if it exists.

5. Input::get_singleton()->get_mouse_position()
   For Windows this leads to OS_Windows::get_mouse_position(), which returns window coordinates,
   meaning the black bars also get coordinates. This is an issue, because while this works on the root
   viewport (Where set_attach_to_screen_rect has been called), it doesn't work properly with viewports
   that don't have a to_screen_rect set. Because of this, _get_input_pre_xform() returns the Identity Matrix
   and the black bars offset the mouse coordinates in any sub-viewport.

My solution to that was to get Viewport-Coordinates recursively: The Viewport-Coordinates of any sub-viewport
should be relative to the Viewport-Coordinates of all parent viewports:

func get_vp_mouse_pos(p_vp):
	var root = get_tree().get_root()
	var final_transform = p_vp.get_final_transform()
	var affine_inverse = final_transform.affine_inverse()
	var pre_xform = affine_inverse * get_vp_input_pre_xform(p_vp)
	
	# The only change:
	# Only the root viewport uses Input.get_mouse_position(),
	# because there we know to_screen_rect is set.
	var window_mouse_pos = Vector2.ZERO
	if p_vp == root:
		window_mouse_pos = Input.get_mouse_position()
	else:
		var parent_vp = get_closest_parent_vp(p_vp)
		window_mouse_pos = get_vp_mouse_pos(parent_vp)
	
	var window_offset = get_vp_window_offset(p_vp)
	var real_pos = window_mouse_pos - window_offset
	var mouse_pos = pre_xform.xform(real_pos)
	
	return mouse_pos

I had to create a new function to fix the issue of a viewport returning itself with node.get_viewport():

func get_closest_parent_vp(p_instance):
	if p_instance == get_tree().get_root():
		return p_instance
	
	if p_instance is Viewport:
		return p_instance.get_parent().get_viewport()
	else:
		return p_instance.get_viewport()

This works already even for nested sub-viewports if there is no transform set for any parent CanvasItem of a Viewport, and if
there is no current Camera2D (Meaning the canvas_transform of the Viewport is not being manipulated by the Camera2D).
So my next step was to take the canvas_transform of the viewports into account.
(Note: This is only necessary because the global_canvas_transform isn't global at all and can therefore only be used on the viewport you want the position from.)
I ended up with:

func get_vp_mouse_pos(p_vp, p_global):
	var root = get_tree().get_root()
	
	var final_transform = Transform2D.IDENTITY
	if p_global:
		# This is stretch_transform * global_canvas_transform
		final_transform = p_vp.get_final_transform()
	else:
		# This is stretch_transform * canvas_transform
		final_transform = p_vp.get_canvas_transform() * p_vp.get_stretch_transform()
	var affine_inverse = final_transform.affine_inverse()
	var pre_xform = affine_inverse * get_vp_input_pre_xform(p_vp)
	
	var window_mouse_pos = Vector2.ZERO
	if p_vp == root:
		window_mouse_pos = Input.get_mouse_position()
	else:
		var parent_vp = get_closest_parent_vp(p_vp)
		window_mouse_pos = get_vp_mouse_pos(parent_vp, false)
	var window_offset = get_vp_window_offset(p_vp)
	var real_pos = window_mouse_pos - window_offset
	var mouse_pos = pre_xform.xform(real_pos)
	
	return mouse_pos

Now you can have a current camera in all Viewports and move it around without the coordinates being off.
Remember that this function is recursive, so I basically create the global_canvas_transform myself.

Note: All this time I only calculated viewport coordinates and not the global mouse coordinates.
Until this point I used the default CanvasItem.get_global_mouse_position() for that.
(Only difference being that I translated it to GDScript and made a parameter for the viewport coordinates I calculated)

Only the transforms of parent CanvasItem's are not taken into account now. The issue is, Viewports don't have
information about those transforms even tho they actively change where the Viewport renders.
Therefore my idea was to loop through the parents of all viewports and get their global_transform and multiply them all
together to get the final global_transform. This does only work for one sub-viewport tho, because the second sub-viewport
contains the global_transform of the first one too. In my opinion it should work if I got the transform of all nodes between two
viewports and multiply them together, but this is where workarounds and probably the performance ends.

func get_global_mouse_pos(p_instance, p_vp_mouse_pos):
	var root = get_tree().get_root()
	
	# CanvasLayer
	var transform = p_instance.get_canvas_transform()
	var affine_inverse = transform.affine_inverse()
	
	# CanvasItem
	# This doesn't work properly for nested viewports.
	var closest_vp = get_closest_parent_vp(p_instance)
	var global_transform = Transform2D.IDENTITY
	while closest_vp != root:
		global_transform *= get_vp_global_transform(closest_vp)
		closest_vp = get_closest_parent_vp(closest_vp)
	
	affine_inverse *= global_transform
	var mouse_pos = affine_inverse.xform(p_vp_mouse_pos) 

	return mouse_pos

func get_vp_global_transform(p_vp):
	var parent = p_vp.get_parent()
	if parent && parent.has_method("get_global_transform"):
		var transform = parent.get_global_transform().affine_inverse()
		# I believe I have to set the origin to zero because this is in the canvas_transform already.
		transform.origin = Vector2.ZERO
		return transform
	else:
		return Transform2D.IDENTITY

To me it feels like this was never properly considered when the user got the ability to instance sub-viewports.
Maybe someone experienced could mercy themselves to implement this properly with the info I provided? I can try to help of course.
I have uploaded my test project with the custom build so you can try it out yourself and play around with any transforms and resize the window to see what happens.
You can right-click drag inside a viewport to change the camera position, the labels display the global mouse position.

https://drive.google.com/drive/folders/1uT-fp5XQuWakTOv8WjsQnRd3g70J9Ybs?usp=sharing

[Calinou]

@Sauermann has been looking into fixing various Viewport input issues in master, but I'm not sure if their fixes can translate to the 3.x branch.

[Sauermann]

This is very difficult to tell for me, because I have not looked at the 3.x code base and currently I focus on fixing the transform-problems in master. But from what you write Grandro, there are noticeable differences, so I have to assume, that my changes can not be cherry-picked to 3.x but would have to be individually adopted.