WVNC: web based protocol and API for accessing VNC server from browser via websocket

This post is deprecated, WVNC now is a part of AntOS eco-system. The easiest way to setup a Web base VNC client is to used the AntOS docker image that is presented in this post.

As with AntOS and its applications, one can remotely access and edit server resources from browser in a desktop-like manner. However, sometime these web based applications are not enough for some specific tasks. For a long time, I've thinking of a web based API for controlling remote desktop right from the browser ( or AntOS application). The VNC protocol is a good starting point. After playing around with libvncserver/libvncclient, i came up with wvnc a web based protocol and API for accessing VNC servers using websocket.

• Demo: demo shown in the animated gif is available at: https://apps.lxsang.me/webVNC
• Project pages:
  • Client API wvnc.js: https://github.com/lxsang/wvnc.js
  • Server bridge antd-wvnc-plugin: https://github.com/lxsang/antd-wvnc-plugin

Overview

Since VNC is not a web friendly protocol, accessing a VNC server using only Javascript is not feasible. Therefore, we need something in-between the browser and the VNC sever to bridge this gap. This "middle-ware" should be able, on the one hand, (1) to comply with the web standard and, on the other hand, (2) to communicate with the VNC server. For the first requirement, websocket is a good choice for web-based data streaming. For the second one, the middle-ware could rely on libraries dedicated to the VNC protocol such as libvncclient. The figure below shows an overview of WVNC

As shown, WVNC consists of two parts: the server bridge and the client API:

WVNC Server bridge

The server bridge, which is implemented in C, is not a standalone application but is an Antd (my HTTP/HTTPS web server) plugin. This allows me to simplify the development since Antd handles everything related to the HTTP/HTTPs as well as the websocket protocol. I only need to implement the VNC-to-websocket bridge. To communicate with VNC servers, the plugin relies on the well-known libvncclient library. In addition to this, it defines the base protocol for communicating with web application via web socket.

To speed up data streaming between web applications and the plugin, some data compression options are available based on Zlib and libjpeg. Using the provided protocol, the web based client have 4 choices for data compression:

1. No compression, only send raw frame buffer segments: This is not recommended, since sending raw pixels will eat up a lot of bandwidth.
2. JPEG compression of frame buffer segments: The client can configure the JPEG quality of the compression, this trades transmission speed with image quality.
3. Zlib compression of raw frame buffer segments: Using zlib, it is able to speedup the data transmission without loosing the image quality.
4. Using both JPEG and Zlib for data compression

For the last option, one may ask what is the point of using both JPEG and Zlib for compression since compressing an already compressed data doesn't really produce any improvement. Well, this indeed does improve the data transmission if we look closely at how WVNC compresses and streams data. Let's take an example, suppose that the size of the raw frame buffer segment is 300 bytes and the client chooses JPEG for data compression, the plugin will try to compress the raw data to JPEG format. It then compares the size of the compressed data with the original one, if the size is smaller, it will stream the compressed data, otherwise, the raw data is sent instead. In this case, let's say the compressed data is around 600 bytes due to the added JPEG header, obviously, the plugin will send the raw data (300 bytes). Now if the client chooses both JPEG and Zlib as compression option, this 300 bytes raw data will then be compressed using Zlib producing the final data to be streamed of around 75 bytes. This does a lot of improvement in term of data transmission.

WVNC Client API

The client API, called wvnc.js, is a Javascript based API implementing the protocol defined by the WVNC plugin. It allows to communicate with the plugin, receive, decode data and render the remote desktop on HTML canvas. In additional to this, the library also listens to user events (mouse, keyboard), and send them to the plugin for remote desktop controlling using the VNC protocol.

Decoding compressed data from server is a heavy task on the client side. To speed up this process, wvnc.js uses web worker to provide background data decoding support. Further more, the decoding algorithm is implemented using Web assembly (with the help of libjpeg and zLib) to achieve fast decoding and rendering.

Using WVNC

Please refer to the two project pages mentioned earlier to find out how to build and use WVNC