The maize harvest is in full swing. The harvester runs nearly 24/7. The driver starts hearing the occasional strange noise from the engine. The area cut per hour drops significantly. The driver calls tech support and starts sharing the screen of the terminal in the harvester. The support guy remotely changes some engine settings. All is fine again. My Italian business partner, Ispirata, and I developed a VNC server for the Freescale i.MX53 terminal of Krone’s BiGX forage harvester – to make this scenario possible.
The Challenge
Without the screen sharing solution through VNC, the driver would have had to stop harvesting and wait for the technician to arrive in 1-2 days time. Often, the technician would not have the right spare parts: another 1-2 days would be gone. It might even take longer. Sending technicians to remote fields somewhere on the globe is pretty expensive both for Krone and the customer.
Krone wanted to minimise the downtime of their harvesters and the time technicians spend on travelling to remote fields. A quick and easy way was to install a VNC server on the terminals of their harvesters. The VNC server mirrors the terminal’s screen over a mobile Internet connection to remote VNC clients like PCs, laptops, phones and tablets. For example, the harvester could be on a field in the US and the support technician in an office in Germany.
The technician in the office sees exactly the same screen contents as the driver on the field. He can even remotely control the HMI on the terminal by using the mouse or keyboard – as if sitting on the harvester himself. The technician can unlock additional diagnostics functionality with a special password.
On the fields, the harvester typically has an EDGE (2.5G) or UMTS (3G) connection to the mobile Internet. The maximum data rate for EDGE is 500 kbit/s. Our screen sharing solution via VNC must still work at rates around 300 kbit/s. We are lucky in that the HMI screens for problem diagnosis and fixing are mostly static. The sparse information on these screens changes infrequently. The home screen with lots of frequently changing information is not really interesting during diagnosis.
Krone’s harvester terminal has a Freescale i.MX53 system on chip (SoC) inside. The i.MX53 is a single-core ARM Cortex-A8 with OpenGL acceleration – running at 800 MHz. It’s a pretty decent medium-range SoC.
Unfortunately, Freescale or now NXP have pretty much abandoned it in favour of the i.MX6 and i.MX7. Getting a VNC server running on the i.MX53 SoC would have been pretty easy, if the i.MX53 came with Wayland drivers. When we asked Freescale (now, NXP) about these drivers, they suggested that we upgrade to the i.MX6: There is no Wayland support for the i.MX53 and there never will be.
I’ll explain in the rest of this post how we solved these challenges. Here is a short video of the VNC solution that hopefully keeps you reading on.
The video shows the VNC server running alongside the HMI on the harvester terminal and three VNC clients connected to the VNC server. The terminal, the laptop and the tablet are connected to the same private network. The phone is connected over 3G with the terminal and hence is outside the private network. My router has a public and static IP address and forwards requests on its VNC port to the terminal.
Structure of the VNC Solution
The following diagram shows the structure of the VNC solution. The arrows show how data flows between the participants and which means are used to transport the data.
When the driver turns on the ignition of the harvester, the driver terminal – a Linux system – starts up. Eventually, one of the Linux init scripts starts the terminal software, called QML Application in the diagram. When the driver requires remote support, he starts screen sharing via VNC from the QML Application. The QML Application starts the VNC Server in a separate process. At this point, remote VNC Clients can connect with the VNC Server via a TCP/IP connection.
We built the VNC Server using TigerVNC. TigerVNC provides the building blocks to assemble a bespoke VNC Server. As TigerVNC is under GPLv2, we could modify and extend the source code to our exact needs.
For testing the VNC solution, I used RealVNC Viewer as my VNC client. RealVNC Viewer runs on all relevant platforms – Windows, Mac OS X, Linux, iOS and Android – and is free. When connecting VNC Clients to the VNC Server, I only had to enter the IP address of the driver terminal. If the terminal has a public and static IP address, this is all we need. If the terminal has a private IP address (e.g., 192.168.1.24 or 10.10.3.37) and is connected via LAN to a 3G/LTE router with a public IP address, we must configure port forwarding from the router to the terminal.
The QML Application and the VNC Server run in their own processes. Both can access the Framebuffer through shared memory. The QML Application renders the latest frame into the Framebuffer, that is, it writes to the Framebuffer. The VNC Server reads the latest frame from the Framebuffer and sends a copy of the Framebuffer contents to the VNC Clients.
The QML Application and the VNC Server communicate via Unix socket with each other. The QtToRemote library inside the QML Application notifies the VNC Server whenever a new frame is available. The VNC Server forwards key and mouse events from the VNC Clients to the QML Application. This socket connection is established once the VNC Server sends a connection request to the listening QML Application.
QQuickWindow is an object created by the QML Application. The Qt rendering engine displays the QML HMI (QML scene) in the QQuickWindow. QQuickWindow emits the signal frameSwapped() whenever a new frame is available for display.
QtToRemote is a shared library loaded by the QML Application. It handles the communication with the VNC Server. QtToRemote communicates with QQuickWindow by Qt signals and slots, that is, by function calls.
Rendering a Frame
Now, let’s see the participants of our VNC solution in action. The following diagram shows what happens when the QML Application has rendered a new frame into the Framebuffer.
- Step 1. When the HMI shown in the QQuickWindow changes, Qt renders the changed contents into the framebuffer.
- Step 2. When the back buffer of the double-buffered Framebuffer becomes the front (visible) buffer, that is, when the Qt rendering engine thinks its time to display the changed contents of the Framebuffer, QQuickWindow emits the signal
frameSwapped(). - Step 3. When QtToRemote receives the signal
frameSwapped(), it notifies the VNC Server that a new frame is available in the Framebuffer. - Step 4. When the VNC Server learns about this new frame, it copies the new frame from the Framebuffer into its own data structure.
- Step 5. The VNC Server sends the copy of the new frame to all the listening VNC Clients.
The five steps above are repeated whenever a new frame becomes available.
Handling a Mouse Press
When the support guy remotely controls the harvester HMI through a VNC Client using the mouse or keyboard, this user interaction triggers the following sequence of messages in the VNC solution.
- The user presses the mouse on an HMI control displayed in VNC Client 2.
- Step 1. VNC Client 2 sends the mouse-pressed event with event type, position and other properties to the VNC Server.
- Step 2. The VNC Server sends the mouse-pressed event to the QtToRemote part of the QML Application.
- Step 3. QtToRemote converts the mouse-pressed event from the VNC format into the Qt format and injects it into the Qt event loop. The Qt event loop eventually delivers the event to QQuickWindow, which forwards it to the correct HMI control.
- Steps 4 – 8. A mouse press typically implies a change in the displayed HMI screen. For example, a button shows the pressed background or the HMI changes to different screen in response to the mouse press. Hence, the Qt rendering engine produces a new frame. The steps from the section “Rendering a Frame” are executed and the screens of the VNC Clients are updated accordingly.
Key events are handled in the same way.
Results
When the QML Application is on the home screen with quite a lot of frequently changing information and there are no VNC Clients connected, it uses roughly 30% of the CPU. The CPU load of the VNC server is negligible (less than 0.1%). If we connect one VNC Client to the VNC Server, the load of the QML Application goes up to 38% and that of the VNC Server to 28%. If we connect a second VNC Client, the loads are 48% for the QML Application and 30% for the VNC server.
With one client connected, we have a total average CPU load of 66%. With two clients connected, it’s 78%. This leaves sufficient reserves when the QML Application or the Linux system need an additional CPU boost. We can reduce the CPU load of the VNC Server by passing the option MaxProcessorUsage=NN to the VNC Server on startup. This limits the CPU time consumed by the VNC server to at most NN percent. Obviously, this may lead to frames being dropped.
As mentioned earlier, the home screen with its many frequent changes is mostly irrelevant to diagnosing and fixing a problem. Problem diagnosis and fixing is done on mostly static screens – with few infrequent changes. If we are on a completely static screen like most Settings pages or the Main page, the total CPU load is well below 2%. Typical diagnostics screens would have a total average CPU load between 20 and 30%.
All is fine on the server side. The i.MX53 is more than powerful enough to handle the additional load of the VNC server. So, let’s turn our focus to the client side.
When we have a fast Internet connection like LTE and (W)LAN from the VNC Server to the VNC Clients (“lab conditions”), we saw rates of up to 25 frames per second (fps). This is more than enough to watch the home screen “almost live” through a VNC Client. The needles of the five round instruments are not quite as smooth as on the terminal but they are pretty close.
When we have a good 3G connection (> 60% signal strength), we can still watch the home screen but the needles are a bit jerky (10-15 fps). Working on the mostly static screens (basically all except the home screen) is not a problem at all. When we have a bad 3G connection (< 20% signal strength), the VNC client starts struggling to keep up with the frequent updates of the home screen. The most noticeable sign is that the response to pressing a button on the home screen can easily take 5 seconds or more. Navigating the other screens is still good but the response to mouse presses become noticeable (1-2 seconds).
With an EDGE connection (< 5 fps), we can forget the home screen. We better restrict ourselves to the Main screen and its subscreens, which are anyways much more relevant to problem analysis, diagnosis and fixing than the home screen. The response to mouse events is about 2-3 seconds. So, we need some patience but remote support is still working.
Resources
As TigerVNC is licensed under GPLv2, we made the complete TigerVNC repository with our changes and extensions available on GitHub. Our shared-memory VNC Server can be found in the subdirectory tigervnc/unix/shmvncserver. If you are familiar with cmake and have set up a toolchain for cross-compilation, the short description “Building TigerVNC shmvncserver” will be helpful.
The QtToRemote library is proprietary and is licensed by Ispirata to interested parties. Although we cannot provide the source code of QtToRemote, it shouldn’t be too difficult to implement its functionality.
Whodunnit?
Davide Bettio (Ispirata) did an excellent job in implementing the QtToRemote library and the shared-memory VNC server. Louis Kröger (Krone) did an equally brilliant job in field testing the VNC solution and providing us with great feedback on the performance of our solution. The project was kept on track by a triumvirate of managers: Valeri Rein (Krone), Dario Freddi (Ispirata) and yours truly.