input delay

[Former user]

Original issue 274 created by etank2 on 2012-09-11T14:49:08.000Z:

What steps will reproduce the problem?
1. using genplus via libretro and retroarch, load any game that polls input from the system during vblank (which afaict seems to be most games). i'll be using sonic 1 as an example.
2. get to a point where you're able to control an onscreen character.
3. press the button retroarch has assigned to frame_advance once.
4. press and hold an input button that will produce a visual reaction on screen, such as a/b/c to make sonic jump while on the ground.
5. continue pressing the frame advance button until you see a reaction in the game, note the number of presses/frames needed to respond.

What is the expected output? What do you see instead?
for sonic 1 it should only require 2 frames to see a response in the game, with genplus via libretro and retroarch it takes 3. (mednafen for example only takes 2 frames to respond)

What version of the emulator are you using (official, SVN revision,...)?
latest git versions of both retroarch and libretro-genplus as of this date

Please provide any additional information below (Emulator settings, Console
setting,...)

sonic 1, like many genesis games, polls input during vblank and uses the results when it steps the game logic for the next frame. the system_frame step functions in genplus look like they're setup to emulate the active display portion of the frame followed by the vblank portion before returning. unless it would adversely affect other areas of the program, to avoid what i feel is an unnecessary input delay (which could be up to 1 frame) perhaps these functions should be restructured in a way so that they return right after the active display portion of a frame is complete, then the next call to system_frame_* would begin by polling input (osd_input_update) and emulating the vblank portion of the frame it left off on.

if i'm not mistaken, many other emulators are setup like this. gambatte will return from its main loop as soon as a frame is complete, and fceu is simply structured in the opposite way of genplus (ie vblank first (even on the first call to its step frame function), then the active display portion of the frame)

[Former user]

Comment # 1 originally posted by ekeeke31 on 2012-09-11T18:38:33.000Z:

genesis plus gx already implements what you are describing: emulated inputs are updated (through osd\_input\_update, which is platform dependent) DURING the system\_frame function, just BEFORE triggering the emulated vblank interrupt, so it is read immediaty by the emulated game polling routine and used in the next frame without any delay.

I am not 100% sure if the method you are using is valid (the core does not provide any frame\_advance functionality) but what you see is most likely due to the way osd\_inputs\_update is implemented in the retroarch implementation you are using, how system\_frame is synced with the host hardware controllers refresh and if specific GUI keys are read at the same time as normal buttons or not. Anyway, it is related to libretro or retroarch, not genesis plus gx itself, so you might better report this on their side.

[Former user]

Comment # 2 originally posted by libretro on 2012-09-12T16:22:24.000Z:

If you look at our implementation of osd\_input\_update, you can see that we do it almost exactly the same as you do in the standalone GX version - first we poll the frontend for input state, then this data is passed to the core and does something with it.

https://github.com/libretro/Genesis-Plus-GX/blob/master/libretro/libretro.c

Anyway, since RetroArch has been ported to so many systems, it might help if the OP would report on which specific system/console he is testing this against -ie. PC, Wii, PS3, Xbox 1, Xbox 360, or whatever. As it is, all these systems can produce varying amounts of input latency.

[Former user]

Comment # 3 originally posted by ekeeke31 on 2012-09-12T19:40:49.000Z:

Well, i don't the think the method used in the first place can accurately claim if an emulator has more input latency than an other, for the main reason that it is tied to when retroarch polls and detect the frame\_advance key as well. Since this is most likely done just before emulating the frame (system\_frame call in our case), if emulated inputs are not updated at the same time (i.e before system\_frame), there will always be one frame delay between the two events, even if vblank is emulated just after updating emulated inputs like i'm doing.

However, that does not necessarily mean the emulator has one frame latency compared to others because what matters is the delay between when host system inputs are being updated from hardware direct access (by the system itself, OS, driver, etc) and when the effect is seen on screen. Since most of the time this is done during host system VSYNC period and frame emulation is generally synced with beginning of VSYNC, this would generally happen DURING frame emulation, and, in the case emulated inputs are updated before frame emulation, AFTER inputs have been polled for that frame. So in any case, input change will not be reflected in upcoming frame but the next one.

Similarely, if you use the frame\_advance function, key press will be detected with one frame delay and system\_frame not executed while it would have likely detected button change if it had been run, since input\_update function is processed later in the frame.

That's said, after some thoughts, i agree it makes more sense to start frame emulation with VBLANK emulation and emulate the corresponding display in the same frame, not the next one, in the case where frame emulation is synced to host system so that hardware inputs have updated betwen the end of previous emulated frame and the beginning of the next one. I know this is actually not the case on Wii so it does not mattered (and requires some heavy core design changes) but it could be done differently on other systems...

I don't know if I was very clear, this is not necessarily easy-to-understand concepts, but what i am sure is that using advance\_frame functionality, which will obviously be synced to key inputs itself and will affects the function where emulator inputs are being updated, in order to measure input latency, is not rigourous.

[Former user]

Comment # 4 originally posted by libretro on 2012-09-13T03:10:46.000Z:

ekeeke - On a somewhat related note to this -

[quote] That's said, after some thoughts, i agree it makes more sense to start frame emulation with VBLANK emulation and emulate the corresponding display in the same frame, not the next one, in the case where frame emulation is synced to host system so that hardware inputs have updated betwen the end of previous emulated frame and the beginning of the next one. I know this is actually not the case on Wii so it does not mattered (and requires some heavy core design changes) but it could be done differently on other systems... [/quote]

Perhaps the audio/video sync we arrived at for RetroArch Wii would appeal to you - we sync on both video and audio by way of a trick we call 'variable audio input rate control'. It has worked very well across nearly all emulators/games we've ported to RetroArch and it's audio pop-free/video-glitch free.

Genesis Plus GX on libretro RetroArch Wii BTW runs very well using this syncing method - I can't profess to know how exactly it performs on the standalone Wii port, but Virtua Racing runs in all three attract mode intros at 60fps - even at the bridge which you previously mentioned was a pretty intensive scene. This is all without any frameskipping.

[Former user]

Comment # 5 originally posted by ekeeke31 on 2012-09-13T09:21:10.000Z:

As soon as you sync with audio, yes, it will be audio pop free. The same actually happens if you disable VSYNC in Genesis Plus GX and you would hardly notice any skipped or repeated frame either because the rates are pretty close enough. BUT that does not mean it does not happen once in a while, which is why i am asking how you can be 100% sure it NEVER happens with your implementation. Audio pops are easy to figure, a single repeated or skipped frame once in while (every 30s sometime if rates precision is good), much less. When you start syncing with both, from what i've observed, it will always derive and ends with a missing frame, depending on which one you synchronize first and how much precise your timings are.

I couldn't actually understand how audio rate control exactly worked from looking at retroarch code, seemed like to me it was something related to resampling but couldn't figure how it would work in our case since the number of samples rendered per frame is fixed by the emulator and that wii output samplerate is also fixed.

Anyway, a good way to check if frames are never dropped (case where emulation renders two frames before video backend had time to display one) or repeated (case where emulation was too late at rendering next frame and the previous one ends being displayed twice) is with Sonic 2 during splitted screen VS mode. If you turn ON original render mode and VSYNC OFF in genplus-gx here, you will notice that every 10-15s, the screen got glitched. What happens is that this part of the game switches into interlaced video mode which doubles the resolution of the screen to 448 lines but halfs the refresh rate (odd field is refreshed at 60hz then even field and the whole frame is refreshed at 30hz). In original mode, this is emulated very accurately just like a real Mega Drive would do: the VDP still generate 224 lines on each system\_frame call like usual so we still output a 224 lines-high frame buffer (GX single-field framebuffer mode) as in non-interlaced original mode but each field must be rendered sequentially and feeded to video backend in good time. If a SINGLE field is skipped or repeated because of brief desync, ALL the consecutive screens will have odd lines in place of even lines, until the next desync where they got replaced correctly.

Off course, this does not happen if you use Wii double-field framebuffer interlaced mode since in this case, the framebuffer is 448 lines and odd/even lines are always rendered then displayed to their proper location on screen. The result is less accurate however and very different from how a real Mega Drive would display (and flicker) in interlaced mode.

Since I think retroarch-wii only uses double-field framebuffer mode, it would be interesting to know how it handle this game part if you ever implement single-field rendering mode support (also referred as 240p or "original scanline" mode, though here it is more 240i in fact and there is no visible scanlines because screen is interlaced). This is actually the only way to be 100% sure emulation is perfectly synced with video as well as audio, since the effect is quite visible and amnoying. I have personally tried many techniques (including syncing with both video and audio or adjusing input rates) but never succeeded in getting this one right unless emulation is synced with video only.

[Former user]

Comment # 6 originally posted by etank2 on 2012-09-13T09:48:24.000Z:

i didn't explain myself very well either. i understand the method i used isn't conclusive, the way system\_frame\_\* is currently structured \_may\_ not be causing any added input delay. however, afaict restructuring these functions couldn't hurt and may actually help, particularly on other platforms where the hardware might update the input faster / at a higher rate.

what i was trying to describe in the op i think is best explained with a figure/drawing, so i scribbled one out quickly showing what i believe the difference in approaches would look like:

http://i48.tinypic.com/2lvi984.png

in the middle would be the host system, small boxes being vblank period, larger ones being the active display. the left and right columns would be the emulated system frames, left with system\_frame\_\* the way it's currently structured and right with the way i was suggesting it be changed to.

[Former user]

Comment # 7 originally posted by ekeeke31 on 2012-09-13T12:03:40.000Z:

Yes, as i said before, you would ideally start the frame with vblank so that end of frame immediately reflects input changes detected during vblank by the game but the problem is that the design of the emulator does not easily permits that. The main problem is that overscan areas, which normally appears at the start (bottom border) and end (top border) of vblank are emulated and their height will change if active area height is modified during active area. This is actually why i made the frame starts with active area so i know the borders height to render in current frame when it ends (not sure if this is very clear).

However, i am still not convinced it would really reduce input latency since it pretty much depends on how system\_frame updates are synced with the video output anyway and having inputs polled then vblank near the end of the current frame is very similar to having them at the beginning of the next frame.

Another solution could be to get ride of the traditionnal use of system\_frame call and use an infinite loop that call audio/video/inputs update function at specific points of the frame... but again , i am not sure if there is much to gain. What is sure is that i never got complaints from users experimenting input latency with this emulator (as opposed to other ones on wii) and we are speaking about AT MAX (probably much less) one frame (16.67 ms) delay that no human could realistically be able to detect.

[Former user]

Comment # 9 originally posted by ekeeke31 on 2014-12-14T23:27:47.000Z:

Guess it's never too late to change his mind so it's now 2 years later and I ended up implementing this in r927.

Also removed one frame of delay when vsync is used in r899.

Two frames delay might not be much and probably not noticeable by anyone on their own but it's still two additional frames of delay that would add themselves to any existing hardware latency and could make it become noticeable.