Provide a hook for frame delta smoothing

[lawnjelly]

OS/device including version:
All

Issue description:
In the past few years some of the problems of relying on OS timings in order to decide how far to advance a game on each frame have become clear:

https://medium.com/@alen.ladavac/the-elusive-frame-timing-168f899aec92
http://bitsquid.blogspot.com/2010/10/time-step-smoothing.html
http://frankforce.com/?p=2636

In Godot currently the frame delta is calculated by simply a call to OS::get_singleton()->get_ticks_usec() at the start of Main::iteration(). This does a call to QueryPerformanceCounter on windows (and I presume equivalent funcs on other platforms).

This tells us the OS time at which Godot begins to simulate the game world (i.e. run the physics ticks, then run the rendering code). But as far as producing a smooth simulation to the viewer, this is not the time that is required. The relevant time for viewing is the time at which that frame is shown on the screen. There is (barring some attempts with newer extensions in Vulkan I believe) afaik no easy way of getting this information.

http://themaister.net/blog/2018/11/09/experimenting-with-vk_google_display_timing-taking-control-over-the-swap-chain/

More so, it is currently almost impossible to know in advance when a frame will be shown on screen afaik, because this will depend on things like whether the GPU tasks went over a boundary required to hit a vsync on that particular frame.

So currently the deltas we get from the OS often bear only a very loose relationship with the actual frame display timings. This is one of the major causes of jitter.

Possible solutions
While waiting for some of the possible solutions to this problem to be implemented in the APIs / hardware, many developers have had some success at alleviating these problems simply by applying a filtering layer between the deltas from the OS and something sensible that can be used for stepping the game.

For instance, a very simple technique is to calculate a weighted average of deltas over a series of frames. This can help alleviate random fluctuations in timing due to things like OS scheduling etc. Of course this does not solve all situations, particularly when there are dropped frames.

I have already had success adding such a simple delta smoothing algorithm to Godot, but given the techniques used in this area are still being worked out by the wider community and are subject to change, I was wondering whether it might be a good idea to add some kind of hook or callback in Godot, in order to let gdscript addons, GDNative or modules to apply custom smoothing to this delta value. An alternative might be to hard code one or several algorithms into the engine, but the idea of leaving this to the user does seem more future proof.

The function I am currently using is something as simple as this:

int SmoothDelta(int delta);

Where it passes in the delta (in usecs) from the OS and the function returns a smoothed delta (in usecs). The default could be to just use the delta from the OS (this is what currently happens) but it would be good to allow games the ability to have their own behaviour for smoothing.

My question would be to the guys who have been working on the engine longer than me:

Can you think of a reasonable way of exposing a function such as this to the user?

As it is only done once per frame, it could even be done in gdscript if this was possible. The caveat is that it would have to be called outside the standard _process() _physics_process() system as it would need to be called before the main iteration began.

[lawnjelly]

Thanks to some help from Akien and IronicallySerious I now have a proof of concept version of this user defined path working, calling an optional gdscript function that can do delta smoothing. It is very interesting seeing the measured delta values, here is a selection:

	16578
	16615
	16972
	16537
	16589
	16779
	16640
	16754
	16692
	16590
	16609
	16782
	16710
	16658
	16573
	16787
	16573
	16775
	16678
	17116
	16422
	16868
	16503
	16751
	31464
	2482
	15312
	16609
	16756
	16965

This is in windowed mode running in the editor. Note two things:

1. The usual variance
2. Near the end there is a long delta, followed by an unfeasibly short delta

The usual variance will be very easy to smooth. The combination of the long and short delta is interesting .. as far as a naive interpretation is concerned this has been a dropped frame, however what is probably happening is there has been a CPU side delay so it has filled that frame late, but still meets the timing for vsync (remember there is probably triple buffering) and as a result the next frame can be piped off early, because everything is ultimately waiting on either the double buffer / triple buffer or queued up commands.

Thus according to the current naive timing you would get a jitter on these 2 frames, whereas in fact they both render at the correct vsync time.

This is described in the articles above, but it is fairly easy to solve this case in a gdscript smoother.

Frame delta measurement

I will also double check that the frame delta is measured at the best possible point in the code. The best point is probably the most consistent in terms of which OpenGL command 'stalls' the main Godot thread. This may well be immediately after a swapbuffers, or it could be any GL command if it is to do with the command queue (in which case there might be no 'best place'). It may vary on different platforms. It is worth a look though.

The current place where delta time is measured is at the beginning of Main::iteration(), which sounds sensible, however, the timing of this is determined by what goes before, and what goes before may be what is at the end of iteration: an audio server update (which may not be constant time), and a bunch of profiling stuff etc. So there may well be more consistent timings at another point, I will investigate.

[zmanuel]

There currently is a way to get quite accurate timings on the question of 'when did the frame X frames ago get shown to the user' (or something very similar). The key is to use fences. Right after the glFinish call, insert a performance counter and a fence; X frames later, you wait on the fence and read the performance counter (or just take the regular system time). It's not perfect, there are no guarantees the times taken that way actually mean anything. Experiments with a sample size of 1 (my machine, nvidia+linux) got much more stable results than the current method.

I had proof of concept code here:
eb97679
But it turns out the methods used aren't in GLES and just coincidentally worked on the desktop, so I quietly swept the code under the rug. But Vulkan has similar functions.

[lawnjelly]

Closing as we have gone for fixed method approach in #48390.