I managed to track down that nasty flickering system cursor glitch in OpenGL letterbox modes, where it shows over black borders outside of the viewport area of our fullscreen window.
A short story is that the Allegro version (4.4.2-agspatch) we use with AGS Engine, forces Windows operating system (OS) to draw a system arrow cursor if the mouse cursor position is outside of the window's viewport area defined with the Allegro void set_mouse_range(int x1, int y1, int x2, int y2) function. And that arrow cursor showing is pretty hardcoded inside Allegro internals, I have to say. WinAPI's ::ShowCursor(FALSE) won't help, because it relies on OS' window message handling, and Allegro totally controls it with its own window procedure, simply doing it not the way ShowCursor(FALSE) wants...
At the moment, we pass our letterbox inner area coords into Allegro's set_mouse_range(). If OS' mouse cursor is inside that area, Allegro draws a customizable OS system cursor, which we have defined as show_os_cursor(MOUSE_CURSOR_NONE). But if it's outside, Allegro uses OS' arrow cursor, no matter what...
A quick solution would be to set the Allegro's mouse range to the whole screen/fullscreen-window even in letterbox modes. The player still won't be able to move their game-mouse-cursor over that black area, as it's restricted elsewhere in the engine. The only question remains -- if is it ok with the AGS engine to expand Allegro mouse range to the entire screen/window? I've tested it on Windows OS and it seemed to work fine. But I'm not sure about other ports of the engine... In theory, if it gets pushed back inside game viewport area by some other engine code (who let you out of the box?!), it shouldn't cause any trouble, should it?
And this is the suggested quick fix to consider:
(original source code: ags/Engine/device/mousew32.cpp)
Code: cpp
If the above fix or its variations won't be suitable, then the other methods would involve either directly patching Allegro source codes to replace the hardcoded arrow cursor with no_cursor, or trying to take remote control over its window message loop with void win_set_window(HWND wnd) and/or void win_set_msg_pre_proc(int (*proc)(HWND, UINT, WPARAM, LPARAM, int *retval)) functions, which is also tricky and would add more bulk into the AGS engine source code, because we then may have to copy Allegro window message handling function code over into the engine and mock up our own version of it, fixing the hardcoded cursor but also replicating everything else Allegro does with handling window messages.
To clarify the original issue, the Allegro version we use in AGS, basically does the following:
When we call allegro void set_mouse_range(int x1, int y1, int x2, int y2), it among other things stores the passed constraints in its internal global variables:
(source code: allegro-4.4.2-agspatch/src/win/wmouse.c)
Code: c
Next, Allegro tries to repeatedly update its mouse position variables (globals _mouse_x and _mouse_y) asking for a new cursor position values from Windows OS, and checking if the new position is within the bounds set by set_mouse_range, with the following macros/function.
Code: c
Allegro's window message handling procedure to invoke the queued call into mouse_set_syscursor():
(source code: allegro-4.4.2-agspatch/src/win/wwnd.c)
Code: bf
And finally, the cause of all the trouble -- mouse_set_syscursor():
(source code: allegro-4.4.2-agspatch/src/win/wmouse.c)
Code: c
The _mouse_on==FALSE condition gets us into the ARROW CURSOR branch. And !gfx_driver->windowed can't help us much, since technically we are in windowed mode where our window has no borders and is being stretched all over the screen.
The arrow cursor is only a fraction of a second outside of the inner viewport area before it gets pushed back by the AGS engine code.
Still that time is enough to get it noticed over a black portion of the screen...
Sorry for "a short story"... :p
p.s.
It keeps me amused reading Allegro manuals and source code. Lots of fun ^^
Code: c
A short story is that the Allegro version (4.4.2-agspatch) we use with AGS Engine, forces Windows operating system (OS) to draw a system arrow cursor if the mouse cursor position is outside of the window's viewport area defined with the Allegro void set_mouse_range(int x1, int y1, int x2, int y2) function. And that arrow cursor showing is pretty hardcoded inside Allegro internals, I have to say. WinAPI's ::ShowCursor(FALSE) won't help, because it relies on OS' window message handling, and Allegro totally controls it with its own window procedure, simply doing it not the way ShowCursor(FALSE) wants...
At the moment, we pass our letterbox inner area coords into Allegro's set_mouse_range(). If OS' mouse cursor is inside that area, Allegro draws a customizable OS system cursor, which we have defined as show_os_cursor(MOUSE_CURSOR_NONE). But if it's outside, Allegro uses OS' arrow cursor, no matter what...
A quick solution would be to set the Allegro's mouse range to the whole screen/fullscreen-window even in letterbox modes. The player still won't be able to move their game-mouse-cursor over that black area, as it's restricted elsewhere in the engine. The only question remains -- if is it ok with the AGS engine to expand Allegro mouse range to the entire screen/window? I've tested it on Windows OS and it seemed to work fine. But I'm not sure about other ports of the engine... In theory, if it gets pushed back inside game viewport area by some other engine code (who let you out of the box?!), it shouldn't cause any trouble, should it?
And this is the suggested quick fix to consider:
(original source code: ags/Engine/device/mousew32.cpp)
#include "gfx/graphicsdriver.h"
extern IGraphicsDriver *gfxDriver;
void Mouse::SetGraphicArea()
{
//Rect dst_r = GameScaling.ScaleRange(play.viewport);
//mgraphconfine(dst_r.Left, dst_r.Top, dst_r.Right, dst_r.Bottom);
DisplayMode dm = gfxDriver->GetDisplayMode();
mgraphconfine(0, 0, dm.Width-1, dm.Height-1); // <-- mgraphconfine() in its turn calls Allegro's set_mouse_range()
}If the above fix or its variations won't be suitable, then the other methods would involve either directly patching Allegro source codes to replace the hardcoded arrow cursor with no_cursor, or trying to take remote control over its window message loop with void win_set_window(HWND wnd) and/or void win_set_msg_pre_proc(int (*proc)(HWND, UINT, WPARAM, LPARAM, int *retval)) functions, which is also tricky and would add more bulk into the AGS engine source code, because we then may have to copy Allegro window message handling function code over into the engine and mock up our own version of it, fixing the hardcoded cursor but also replicating everything else Allegro does with handling window messages.
To clarify the original issue, the Allegro version we use in AGS, basically does the following:
When we call allegro void set_mouse_range(int x1, int y1, int x2, int y2), it among other things stores the passed constraints in its internal global variables:
(source code: allegro-4.4.2-agspatch/src/win/wmouse.c)
static int mouse_minx = 0; /* mouse range */
static int mouse_miny = 0;
static int mouse_maxx = 319;
static int mouse_maxy = 199;Next, Allegro tries to repeatedly update its mouse position variables (globals _mouse_x and _mouse_y) asking for a new cursor position values from Windows OS, and checking if the new position is within the bounds set by set_mouse_range, with the following macros/function.
#define READ_CURSOR_POS(p) // p - out param that's passed in to get Windows OS cursor position send back to the macros/function invoker
{
GetCursorPos(&p); // use WinAPI to retrieve mouse cursor position into a temporary
p.x -= wnd_x;
p.y -= wnd_y;
if ((p.x < mouse_minx) || (p.x > mouse_maxx) || // if mouse cursor position is outside of the area earlier defined with set_mouse_range()...
(p.y < mouse_miny) || (p.y > mouse_maxy)) {
if (_mouse_on) { // then...
_mouse_on = FALSE; // mark the mouse as OUTSIDE; _mouse_on will be used within mouse_set_syscursor()
wnd_schedule_proc(mouse_set_syscursor); // queue up a background call into mouse_set_syscursor() via window message procedure
}
}
else {
if (!_mouse_on) {
_mouse_on = TRUE;
wnd_schedule_proc(mouse_set_syscursor);
}
_mouse_x = p.x;
_mouse_y = p.y;
}
}
Allegro's window message handling procedure to invoke the queued call into mouse_set_syscursor():
(source code: allegro-4.4.2-agspatch/src/win/wwnd.c)
// Window procedure for the Allegro window class.
static LRESULT CALLBACK directx_wnd_proc(HWND wnd, UINT message, WPARAM wparam, LPARAM lparam)
{
....
if (message == msg_call_proc)
return ((int (*)(void))wparam) (); // <---this is what really calls mouse_set_syscursor()
// (NOTE: see the "Code:" tag to get the idea of how this works :PpPpPppp)
if (message == msg_suicide) {
DestroyWindow(wnd);
return 0;
}
....
....
....
switch (message) {
case WM_CREATE:
case WM_DESTROY:
case WM_SETCURSOR: // <---WinAPI's ShowCursor() works with this message (probably trying to disable it which is useless)
if (!user_wnd_proc || _mouse_installed) {
mouse_set_syscursor(); // <---another (direct) call into mouse_set_syscursor()
return 1; /* not TRUE */
}
break;
case WM_ACTIVATE: ....
case WM_TIMER: ....
case WM_ENTERSIZEMOVE: ....
case WM_EXITSIZEMOVE: ....
case WM_MOVE: ....
case WM_SIZE: ....
....
case WM_INITMENUPOPUP: ....
case WM_MENUSELECT: ....
case WM_MENUCHAR : ....
case WM_CLOSE: ....
}
....
}And finally, the cause of all the trouble -- mouse_set_syscursor():
(source code: allegro-4.4.2-agspatch/src/win/wmouse.c)
// mouse_set_syscursor: [window thread]
// Selects whatever system cursor we should display.
int mouse_set_syscursor(void)
{
HWND allegro_wnd = win_get_window();
if ((mouse_dinput_device && _mouse_on) || (gfx_driver && !gfx_driver->windowed)) {
SetCursor(_win_hcursor); // <---this is what sets our NO_CURSOR thing when the cursor is inside the letterbox viewport area...
// ...because _win_hcursor == NULL, as we ordered with "show_os_cursor(MOUSE_CURSOR_NONE)"
/* Make sure the cursor is removed by the system. */
PostMessage(allegro_wnd, WM_MOUSEMOVE, 0, 0);
}
else
SetCursor(LoadCursor(NULL, IDC_ARROW)); // <---the hardcoded ARROW CURSOR when outside! (SetCursor() is a WinAPI function)
return 0;
}The _mouse_on==FALSE condition gets us into the ARROW CURSOR branch. And !gfx_driver->windowed can't help us much, since technically we are in windowed mode where our window has no borders and is being stretched all over the screen.
The arrow cursor is only a fraction of a second outside of the inner viewport area before it gets pushed back by the AGS engine code.
Still that time is enough to get it noticed over a black portion of the screen...
Sorry for "a short story"... :p
p.s.
It keeps me amused reading Allegro manuals and source code. Lots of fun ^^
Spoiler
int gfx_directx_show_mouse(struct BITMAP *bmp, int x, int y)
{
if (hcursor) {
_win_hcursor = hcursor;
}
if (_win_hcursor) {
POINT p;
SetCursor(_win_hcursor);
/* Windows is too stupid to actually display the mouse pointer when we
* change it and waits until it is moved, so we have to generate a fake
* mouse move to actually show the cursor.
*/
GetCursorPos(&p);
SetCursorPos(p.x, p.y);
return 0;
}
return -1;
}[close]
