Popup border issues (Win11 VM-specific)

[ololololo93809]

We use Flatlaf at my workplace for the software we are developing. Our product runs on different versions of Windows, MacOS, and Linux. Popups (that is, Popup, PopupMenu, Combobox dropdown, and ToolTip) work "as they should" and we have done no customizing beyond what Flatlaf defaults to for those so far.

However, our internal development is entirely done on VMs, accessed through remote desktop and similar. Most of these VMs are currently running Win11. We have an issue where specifically on Win11 VMs, the borders for all these popup-like components are broken. "Broken" meaning that there is no border visible, no corner radius, and no drop shadow around the popup. The behaviour is perfectly fine in the release versions (I made sure to check in with QA), and this behaviour only happens in Win11 VMs specifically. I can attest to that as my VM was upgraded from Win10 to Win11 not too long ago, which is precisely when this issue started happening.

Changing the (global) style keys for borderCornerRadius on Popup, PopupMenu, ToolTip, and ComboBox to 0 "fixes" it in the sense that the border and drop shadow return, but of course without the default corner radius. Any other values of borderCornerRadius have no effect.

Now, checking the Win11 documentation on customizing those border corners shows that the feature for rounded border corners is specifically disabled when running in a VM. My thinking here is that FlatLaf tries to paint the border, gets some crap value from Win11 running in a VM when trying to access the Win11 border corner radius properties, and then is unable to draw the border (and drop shadow). I was not able to investigate deeper than this yet because there's more pressing issues to take care of at work right now as this is "just annoying in development".

[DevCharly]

duplicate of #968

Now, checking the Win11 documentation on customizing those border corners shows that the feature for rounded border corners is specifically disabled when running in a VM.

Interesting, didn't know that...

Do other applications (e.g. Windows Explorer) have rounded corners when running in the VM?

[ololololo93809]

Searched up and down if this was already submitted in the issue tracker, and apparently did not use the proper keywords to find that post you linked..

Can confirm that they do not - set up a fresh Win11 VM yesterday using the ISO on Microsoft's website, installed IntelliJ and the Flatlaf demo project, and the issue persists. There are no rounded borders in the Win11 VM for popups, with some exceptions - for example, the Win11 start menu (the thing that appears when the Windows key is pressed) has rounded corners. "Normal" windows though - explorer, for example - do not have rounded borders. Even IntelliJ itself has this issue, where some popups are painted without borders and shadows, making them similarly difficult to read at times.

For this reason, a hypothetical fix also wouldn't (IMO) restore the rounded border corners in Win11 VMs, because that would probably be some truly insane work. I think a fix would be to have a corner radius of 0 on popups, but at least still have the borders and drop shadows be there.

Question is now, how would one do that. Quick reasearch shows me there is no perfectly reliable way to detect if we are running in a VM, but maybe that information is outdated. Maybe Win11 has some internal bit (similar to that enum on which border corner radius is active) that exposes if running virtualized or not? No idea. If this is possible, then this could become a new entry in SystemInfo and could then be used (together with isWindows_11_orLater) to fix the four borderCornerRadius properties to 0.

If that isn't feasible, then another possibility is to investigate where exactly in the border painting process on a Win11 VM it breaks down, then apply some fix there.

If I find some time, I'll see about investigating this a little more deeply, maybe I can find the exact problem spot. I also don't know which of these two ways would be preferred as "the Flatlaf way", if you will.

[badbadbadbadbad]

So I cloned the project on my Win11 VM and investigated a little further. I can tell you that commenting out this line at least restores the "basic" Swing border, like the comment says.

FlatLaf/flatlaf-core/src/main/java/com/formdev/flatlaf/ui/FlatPopupFactory.java

Line 392 in 8339a60

((JComponent)contents).putClientProperty( KEY_POPUP_USES_NATIVE_BORDER, true );

I didn't get much further due to lack of time, but I'm fearing the ultimate issue may be something in FlatNativeWindowsLibrary, which seems to be private? I'll investigate further when I find some time.