Moonlight/Sunshine for Remote Rendering?

[frankNiessen]

Hi,

this idea might be out there and not feasible for very simple reasons that I might not be aware of, but I had a thought yesterday when I was looking into remote desktop solutions for a Mac host and Linux client. I came across Moonlight / Sunshine as a new low-latency way of remote access that was developed for gaming. It appears that everything, even the mouse movement, are rendered remotely and the media is compressed with modern video-stream standards, leading to very low latency. Individual programs or the entire desktop can be controlled this way remotely. Of course my mind immediately built the bridge to VTK-Trame remote rendering, which can be quite sluggish depending on the displayed data and the general environmental conditions. Do you think the Moonlight protocol could be used to provide a superior remote rendering experience in VTK-Trame or are their technical limitations that make this unfeasible?

Best wishes
Frank

[jourdain]

Thanks for the pointers. After looking at the libraries, it seems that they are GPLv3 license which means we can't really use them.
Also they seem to rely on UDP with some network hop in the middle. While such setup could be possible, it won't be practical for many. Also it was unclear to me if the client was working within a browser or via a native application on each device. Browser have limited network connection options.

In general, the concept is the same as the one available in trame-rca with vtkStreaming when using nvenc. The main issue with vtkStreaming is that we run out of funding to properly support it across hardwares and simplify its usage with general VTK.

But for general usage, trame-rca with turbo-jpeg is reliable and can be quite good.

[jourdain]

From their documentation

Is there a Moonlight web client?

No, while there is a ChromeOS app that runs on Chromebooks, there is not a pure web-based Moonlight client. The GameStream protocol requires us to use raw TCP and UDP sockets which is not currently supported in web browsers. Fortunately, there is a proposed Direct Sockets web standard in development that could make it possible to implement a web-based Moonlight client in the future.

[bourdaisj]

there is also the WebTransport web API that could maybe help us improve the performances.
couple of remarks:

• I have no idea whether the (hypothetic) performance gain would be worth the implementation effort
• for rendering I'm not sure the bottleneck is usually the transport - for slow network it is indeed another story
• for the interaction event streams to the server the bottleneck is indeed the transport, so UDP-like datagram could help reduce latency?

I guess we would have to play around, benchmark and quickly prototype to see if there are "quick" wins or feasible solutions