Rendering artifacts with oversized/HiDPI cursors | seeking contribution advice

[jaceee-gh]

Select the version

25.1.2.X

Describe your issue

(Seemingly) When using buffered window managers (xlibre xfce/freedesktop xorg gnome) with both nvidia proprietary or intel mesa, x applications have the ability to create custom cursors via XcursorImageCreate that may exceed hardware cursor max dimensions which then in turn causes bizzare visual artifacting of unknown cause. Either a type of overflow or unstable hw <-> sw cursor switching with more complex compositors. I refrain from conviction as I'm not an expert in X. Compositors like i3 seems to be working just fine... I assume it's something to do with i3 being not-buffered or something or the compositor syncs the screen differently? I'm really not sure.

So, from what I gather, x protocol partially helps the users interface with the phase transition point of hardware vs software cursor rendering is XQueryBestCursor, where I assume this is one of the snippets that relate to that functionality (hw/xfree86/ramdac/xfCursorRD.c):

static void
xf86CursorQueryBestSize(int class,
                        unsigned short *width,
                        unsigned short *height, ScreenPtr pScreen)
{
    xf86CursorScreenPtr ScreenPriv =
        (xf86CursorScreenPtr) dixLookupPrivate(&pScreen->devPrivates,
                                               &xf86CursorScreenKeyRec);

    if (class == CursorShape) {
        if (*width > ScreenPriv->CursorInfoPtr->MaxWidth)
            *width = ScreenPriv->CursorInfoPtr->MaxWidth;
        if (*height > ScreenPriv->CursorInfoPtr->MaxHeight)
            *height = ScreenPriv->CursorInfoPtr->MaxHeight;
    }
    else
        (*ScreenPriv->QueryBestSize) (class, width, height, pScreen);
}

In libX11 the relevant part I think is (./src/QuCurShp.c):

Status XQueryBestCursor(
    register Display *dpy,
    Drawable drawable,
    unsigned int width,
    unsigned int height,
    unsigned int *ret_width,
    unsigned int *ret_height)
{
    xQueryBestSizeReply rep;
    register xQueryBestSizeReq *req;

    LockDisplay(dpy);
    GetReq(QueryBestSize, req);
    req->class = CursorShape;
    req->drawable = drawable;
    req->width = width;
    req->height = height;
    if (_XReply (dpy, (xReply *)&rep, 0, xTrue) == 0) {
        UnlockDisplay(dpy);
        SyncHandle();
        return 0;
        }
    *ret_width = rep.width;
    *ret_height = rep.height;
    UnlockDisplay(dpy);
    SyncHandle();
    return 1;
}

Steps to reproduce

In some compositors with cursor hw acceleration and limited dimensions we can basically break/glitch rendering with something like this. The program basically creates a cursor twice the 'best' size:

#include <X11/Xlib.h>
#include <X11/Xcursor/Xcursor.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(void)
{
    Display *dpy = XOpenDisplay(NULL);
    if (!dpy) { fprintf(stderr, "Can't open display\n"); return 1; }

    Window root = DefaultRootWindow(dpy);
    int screen  = DefaultScreen(dpy);

    unsigned int best_w = 0, best_h = 0;
    XQueryBestCursor(dpy, root, 65535, 65535, &best_w, &best_h);
    printf("XQueryBestCursor: best=%ux%u\n", best_w, best_h);

    /* pass large cursor to XcursorImageCreate > XQueryBestCursor */
    int req_w = best_w * 2, req_h = best_w * 2;
    XcursorImage *image = XcursorImageCreate(req_w, req_h);
    image->xhot = 0;
    image->yhot = 0;

    /* fill with a checkerboard so artifacts are obvious */
    for (int y = 0; y < req_h; y++) {
        for (int x = 0; x < req_w; x++) {
            int checker = ((x / 32) + (y / 32)) % 2;
            image->pixels[y * req_w + x] = checker ? 0xFFFF0000  /* red */
                                                    : 0xFF0000FF; /* blue */
        }
    }

    printf("Sending large cursor to X server (limit is %ux%u)...\n",
           best_w, best_h);

    /* X server receives a cursor bigger than it supports */
    Cursor cursor = XcursorImageLoadCursor(dpy, image);

    Window win = XCreateSimpleWindow(dpy, root,
                                     100, 100, 500, 500, 1,
                                     BlackPixel(dpy, screen),
                                     WhitePixel(dpy, screen));
    XStoreName(dpy, win, "Wine cursor artifact reproducer");
    XDefineCursor(dpy, win, cursor);
    XMapWindow(dpy, win);
    XSelectInput(dpy, win, ExposureMask | KeyPressMask);

    printf("Move your cursor over the window - look for rendering artifacts.\n");
    printf("Press any key in the window to quit.\n");

    XEvent ev;
    while (1) {
        XNextEvent(dpy, &ev);
        // if (ev.type == KeyPress) break;
    }

    XFreeCursor(dpy, cursor);
    XcursorImageDestroy(image);
    XCloseDisplay(dpy);
    return 0;
}

gcc -o xcursor_visual_glitch xcursor_visual_glitch.c -lX11 -lXcursor

I know the code I provided is a toy example but it basically happens in real world scenarios where some windows apps under wine create absurdly large cursor buffers which manifest this type of issue. Here is an example of usage: https://gitlab.winehq.org/wine/wine/-/blob/master/dlls/winex11.drv/mouse.c?ref_type=heads#L591. (Code does not query BestCursor at all and just passes through whatever, in this case winapi wants)

What did you expect?

No visual artifacting during display of oversized cursor.

Additional Information

I'm happy to contribute or help but some guidance in terms of the best place to address or start with this type of issue would help.

I can tell there are more assumptions, conventions and protocol rigor or intentional lack there of is something that I'm not fully aware of.

So yeah, any pointers would help.

I tried to get a footage of the bug but for some reason in obs it's perfectly fine, I assume obs is being probably safer or more graceful about syncs or what have you, not sure. But I did manage to get screenshot in a game that clearly shows the artifact.

But yeah, I do think this bug is a bit of a gray zone, because at first I would think maybe it's a matter of one of the compositors being weird. but yeah, it happened to gnome/mutter/xorg, and also now xfce/xlibre so... And like it's not just an intermittent glitch when alt-tabbing, it's a persistent glitching rectangle thing even when you move the cursor inside the window.

And yes, I do think from protocol perspective there is some degree of responsibility for the applications to implement safe guards and I'm planning to contributing to the wine project, where the bug originates from, but at the same time even there, it's not obvious in terms of how to solve this. in context of wine you are trying to 1:1 a different environment with limited cursor buffer and you end up having to either scale down or crop the buffer which is not a good solution in either case.

P.S:

I think at the end of the day there are kind of two issues.

1. there is probably something weird with the rendering pipeline (likely) of software cursors where compositors when they flush/sync display frames these frames somehow desync/leak.
   I tried hwcursor off with nvidia proprietary driver and that's also is a whole mess, i get similar, not exactly the same artifact, but the crucial detail is that then it starts happening even with smaller system cursors

2. consumer apps being able to just blindly get to a broken state with no guardrails

Extra fields

• I have checked the existing issues
• I have read the Contributing Guidelines
• I'd like to work on this issue