Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Precise input timing information #9087

Open
cryslith opened this issue Jul 9, 2023 · 3 comments · May be fixed by #9034
Open

Precise input timing information #9087

cryslith opened this issue Jul 9, 2023 · 3 comments · May be fixed by #9034
Labels
A-Input Player input via keyboard, mouse, gamepad, and more C-Feature A new feature, making something new possible

Comments

@cryslith
Copy link

cryslith commented Jul 9, 2023

What problem does this solve or what need does it fill?

The application I'm concerned with is rhythm games. Currently it is not possible to determine the time at which an input event (such as a keyboard press) occurred with precision better than 1 frame. However, rhythm games require input timing information which is more precise than this. For example, the ECFA Fantastic window is 11ms long, and the osu! 300 window can be as short as 13.33ms. On the other hand, a frame at 144fps is 6.94 ms long, while a frame at 60fps is 16.66 ms long. This means that the error incurred by measuring the time of an input precise to the next frame could be 50% or more of the actual timing window, making it impossible for players to reliably input in the window.

Besides rhythm games, precise input timing information is also applicable to other issues, such as #5984 and #6183.

What solution would you like?

Add Instant::now() to events processed in the winit event loop and the gilrs event loop. Additionally rewrite the gilrs integration so that gamepad events are polled continuously (like winit events) rather than once per frame so that these timestamps are accurate.

What alternative(s) have you considered?

Use timing information from the OS passed through by winit and gilrs, instead of measuring it ourselves using Instant::now() in the event loop. This has several issues:

  • OS timing information may be unreliable. For instance, some OS timing information is precise only to the millsecond, which might be suitable for rhythm games but not for other uses of timestamps. They might also be nonmonotonic etc.
  • winit doesn't currently expose timing information from the OS (Input event timestamps rust-windowing/winit#1194)
  • gilrs timestamps are expressed as SystemTime, which could lead to issues. For instance these timestamps aren't monotonic and are affected by system time changes.

Additional context

@cryslith cryslith added C-Feature A new feature, making something new possible S-Needs-Triage This issue needs to be labelled labels Jul 9, 2023
@alice-i-cecile alice-i-cecile added A-Input Player input via keyboard, mouse, gamepad, and more and removed S-Needs-Triage This issue needs to be labelled labels Jul 9, 2023
@B-head B-head linked a pull request Jul 10, 2023 that will close this issue
8 tasks
@maniwani
Copy link
Contributor

maniwani commented Jul 22, 2023

Just a couple notes because we've actually been building towards a solution to this for a while.

Add Instant::now() to events processed in the winit event loop and the gilrs event loop.

This is the approach we're going for, but (currently) any timestamps taken in the winit event loop are likely to be aliased to the Instant a frame ends because the app updates inside the event loop. The loop can't timestamp events while the app is updating. Instant::now() seen by the loop is likely much later than the true Instant an event arrived.

#9122 will remove this obstruction, but until then these timestamps won't be much better than time.last_update().

It'd be best if winit propagated OS timing since #9122 can't help wasm32, but someone has to investigate if the timestamps come from monotonic clocks (or can be configured to come from them) and if they have 1ms accuracy (not just resolution) or better. That does seem to be the case based on this comment and my own reading, but I'm not sure. (Getting 1ms accuracy from Windows is just a little complicated1.)

Additionally rewrite the gilrs integration so that gamepad events are polled continuously (like winit events) rather than once per frame so that these timestamps are accurate.

gilrs internally spawns a thread to run its event loop (except on wasm32) and that event loop is collecting events constantly and forwarding them through an mpsc. It's just the app only drains the mpsc receiver once a frame. Draining the receiver once a frame is the correct thing to do though. There are special cases2, but rhythm games typically aren't exceptions.

I think we should upstream changes to gilrs so that either its loop captures monotonic timestamps or the API exposes a way to construct the event loop thread manually.

IIUC gilrs events report SystemTime because its evdev dependency on Linux does. libevdev can set its clock source to CLOCK_MONOTONIC, so the crate probably did it because there's no Instant epoch or constructor.

Footnotes

  1. The clock source of GetMessageTime is usually only accurate to ~16ms, but it can be changed to 1ms using timeBeginPeriod. But timeBeginPeriod has significant caveats on builds older than Windows 10, version 2004.

  2. XR apps want the renderer to read events that occurred in the current frame to do things like "late latching" and "reprojection".

@cryslith
Copy link
Author

cryslith commented Jul 22, 2023

@maniwani Thanks very much for the detailed note! I'd be happy to help out with this effort somehow - what would be useful for me to work on?

@maniwani
Copy link
Contributor

maniwani commented Jul 22, 2023

@cryslith, I think we need help adding timestamp support to events in general, only the "accurate" part is blocked on rust-windowing/winit#1194 or #9122. For rhythm games, I assume there's probably more to it than just having timestamps? Do we also need to translate between "audio time" (e.g. DSP clock time, track time) and "app time" (time.elapsed())? Stuff like that.

For gilrs, I'd like to ask them to make either one of those changes I described (or both), but if we need something sooner, we could go with your plan and funnel their events through another thread that calls Instant::now(). It'd just be extra overhead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-Input Player input via keyboard, mouse, gamepad, and more C-Feature A new feature, making something new possible
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants