Responsive QML HMIs with Scaling

The HMIs of in-vehicle infotainment systems, TVs, phones and many other systems must adapt to different screen resolutions and formats. This adaptation should happen with as little duplicate effort as possible. The simplest way of doing this for QML HMIs is to scale the values of all x, y, width, height, margin and border properties in proportion to a reference resolution. Based on the HMI of a music player, I’ll show you how to do this by changing only the screen width and height.

Motivation

A manufacturer like Volkswagen Group sells cars under several brands like VW, Seat, Skoda, Audi, Porsche and Lamborghini, where each brand comes with many models. For example, the VW brand currently has more than a dozen models like Up!, Polo, Golf, Phaeton, Sharan and many more. Most of these models nowadays come with an in-vehicle infotainment system (IVI system) on board. Car owners can typically choose between two or more IVI systems depending on how much features they want and how much money they are willing to spend.

Given this variety, it is not surprising that the IVI systems come with many different screen resolutions and formats – not unlike the fragmentation of Android phones. As developers, we should strive to minimise the additional effort to support yet another screen size. Our customers – car manufacturers and suppliers – should be happy as well, as they would pay considerably less as well.

A very simple approach to tackle the problem of different screen sizes is to use scaling. We develop the HMI for a reference screen size, say 1280×800 (16:10 format). One of our target screen sizes is 640×384 (5:3 format). If a button has a width of 200 pixels and a height of 120 pixels in the reference HMI, it will have a width of (640 / 1280) * 200 = 100 pixels and a height of roughly (384 / 800) * 120 = 58 pixels in the target HMI. With the scaling approach, we will

  • multiply every horizontal size property like x, width and anchors.rightMargin by the ratio of the target and reference screen width, and
  • multiply every vertical size property like y, height, anchors.topMargin and font.pixelSize by the ratio of the target and reference screen height.
Different screen sizes: left - 800x600 (4:3), top right - 640x386 (5:3), bottom right - 400x225 (16:9)

Different screen sizes for music player: left – 800×600 (4:3), top right – 640×386 (5:3), bottom right – 400×225 (16:9). Click on the image to see the actual screen sizes.

The image above shows the result of the scaling approach for three different screen sizes. The left variant has a screen size of 800×600 (4:3), the top-right variant has 640×386 (5:3) and the bottom-right variant has 400×225 (16:9). This simple approach works well, as long as the structure (layout) of the HMI does not change and the HMI need not be pixel-perfect. The small variant (400×225) shows the limits of the scaling approach. It is too small for the car and would need a different structure.

Solution

The QML component MusicCategorySwitcher implements the vertical group of tab buttons “Now Playing”, “Artists”, “Albums” and so on on the left side of the music player. When we write the code for MusicCategorySwitcher without having different screen sizes in mind, the essential parts of this component could look as follows (note that I left out parts irrelevant for the scaling approach):

// Extract from MusicCategorySwitcher.qml
Column {
    Button {
        text: buttonText
        style: ButtonStyle {
            background: BorderImage {
                source: "qrc:/img/bgTabButton.png"
                border.left: 2
                border.top: 2
                border.right: 2
                border.bottom: 2
            }
            label: Text {
                x: 16
                text: control.text
                font.pixelSize: 42
            }
        }
        width: 319
        height: 99
    }

    Divider {
        height: 1
        width: 319
    }
}

All the magic size numbers in the code above are written for the reference screen size of 1280×800 pixels. These numbers are also those numbers that need to be scaled to the target screen size, say, 800×600. Magic numbers are very bad coding, as they make changing the code very difficult and make HMIs inconsistent.

For example, the normal font size for buttons in the code above is 42 pixels (see the label property of ButtonStyle). In a bigger system with multiple apps, we can be sure that the normal font size for buttons ranges from 36 to 48. The font size is inconsistent, which looks bad in the HMI. If we want to change the font size from 42 to 36, we’ll have to find all the occurrences of 42 and its “approximations” in the code of all apps. We would definitely miss some of the occurrences – creating even more inconsistencies.

We eliminate these magic numbers by replacing them with read-only properties defined and initialised in a QML singleton component AppTheme. For example, we replace the magic number 42 – the normal font size of a button text – by the read-only property or constant AppTheme.textSizeNormal. We do the same with all other magic size numbers. The abridged code for MusicCategorySwitcher looks as follows after the replacement:

// Extract from MusicCategorySwitcher.qml
Column {
    Button {
        text: buttonText
        style: ButtonStyle {
            background: BorderImage {
                source: "qrc:/img/bgTabButton.png"
                border.left: AppTheme.buttonBorderWidth
                border.top: AppTheme.buttonBorderWidth
                border.right: AppTheme.buttonBorderWidth
                border.bottom: AppTheme.buttonBorderWidth
            }
            label: Text {
                x: AppTheme.screenLeftMargin
                text: control.text
                font.pixelSize: AppTheme.textSizeNormal
            }
        }
        width: AppTheme.leftTabButtonWidth
        height: AppTheme.leftTabButtonHeight
    }

    Divider {
        height: AppTheme.dividerSize
        width: AppTheme.leftTabButtonWidth
    }
}

We must at least replace the values of the properties x, y, width, height, font.pixelSize, spacing, anchors.leftMargin, anchors.rightMargin, anchors.topMargin, anchors.bottomMargin, border.left, border.right, border.top and border.bottom by constants in AppTheme. In general, we must replace all values that influence the horizontal or vertical size of a QML component.

A positive effect of introducing these size constants and using them consistently everywhere in our code basis is that we must call the scale functions only once when initialising a size constant in AppTheme. Otherwise, we would have had to call the scale functions on every occurrence of a magic size number. In real life, we should not have written the code with the magic numbers at all, but should have used the size constants right from the beginning.

Size constants really force us to be consistent with sizes. If the normal text size is 42 pixels and we are about to introduce another normal text size of 40 pixels, we must define a new constant for another normal text size. At this point, we should stop and think twice whether this second normal text size makes sense. It is almost always a sign of sloppy and inconsistent UI design. We should not go down this road, because users will notice it as poor UI design. Working with size constants right from the first line of code is a best practise.

One final piece is missing: the code of AppTheme.qml. Here is an abridged version of AppTheme.qml, which contains the relevant parts of the scaling approach and the size constants for the MusicCategorySwitcher.

// Extract from AppTheme.qml
pragma Singleton
import QtQuick 2.4

QtObject
{
    readonly property real refScreenWidth: 1280
    readonly property real refScreenHeight: 800

    readonly property real screenWidth: 800
    readonly property real screenHeight: 600

    function hscale(size) {
        return Math.round(size * (screenWidth / refScreenWidth))
    }

    function vscale(size) {
        return Math.round(size * (screenHeight / refScreenHeight))
    }

    readonly property real screenLeftMargin: hscale(16)
    readonly property real screenRightMargin: screenLeftMargin
    readonly property real dividerSize: 1

    readonly property int textSizeNormal: vscale(42)
    readonly property int textSizeSmall: vscale(32)

    readonly property real leftTabButtonWidth: hscale(319)
    readonly property real leftTabButtonHeight: vscale(99)
    readonly property int buttonBorderWidth: 2
}

The first two properties refScreenWidth and refScreenHeight define the reference screen size as 1280×800 pixels. The next two properties screenWidth and screenHeight define the target screen size as 800×600. The ratio of screenWidth by refScreenWidth is the horizontal scaling factor (here: 800 / 1280 = 0.625). The ratio of screenHeight by refScreenHeight is the vertical scaling factor (here: 600 / 800 = 0.75). As the horizontal and vertical scaling factor may differ, we need a function hscale() and vscale() for horizontal and vertical scaling, respectively.

Given, for example, the reference width of a tab button in the MusicCategorySwitcher as 319 pixels, the target width is hscale(319) = 319 * 0.625 = 199.375. The read-only property leftTabButtonWidth is assigned the rounded value 199 as the target width. Similarly, leftTabButtonHeight is assigned the value vscale(99) = 99 * 0.75 = 74.25, which is rounded down to 74. For each property, we must decide whether it is horizontally or vertically scaled and call hscale() and vscale(), respectively. We use hscale() on the values of properties like x, width, spacing (in Row), anchors.leftMargin and anchors.rightMargin. We use vscale() for their vertical counterparts y, height, spacing (in Column), anchors.topMargin, anchors.bottomMargin and font.pixelSize.

There are two properties in AppTheme.qml that are not scaled: dividerSize and buttonBorderWidth. The property dividerSize specifies the thickness of the vertical and horizontal lines surrounding the buttons and panels of the music player. These lines look best with a thickness of 1 for all target screen sizes. The property buttonBorderWidth defines the thickness of the four borders of a BorderImage. As we use the same image file for the BorderImage in all target resolutions, we must also use the same border thickness. Otherwise, the border of the green selection frame would be scaled and look ugly.

We may even use different scaling functions for some properties. Properties of font sizes like textSizeNormal and textSizeSmall are a good example. The 800×600 variant of the music player (see the variant on the left in the image above) shows us why. The text “Now Playing” barely fits into the top button. The reason is that texts are vertically scaled and that the button is scaled much more horizontally than vertically. We could introduce a special function tscale(), which takes the mean of hscale() and vscale():

    function tscale(size) {
        return Math.round((hscale(size) + vscale(size)) / 2)
    }

    readonly property int textSizeNormal: tscale(42)
    readonly property int textSizeSmall: tscale(32)

This crude heuristic makes the text fit much better on the button, as it reduces the font size of the button text by 10%. We can custom-tailor the scaling of each size property by more or less clever heuristics. Having all the size properties in one place makes it easy to apply the right scaling heuristic to each size property.

There is one thing left to do in AppTheme. At the moment, we must change the target screen size manually. Ideally we want our program to pick up the screen size automatically. It could do so from a configuration file provided with our program. On embedded systems, programs normally use the full screen size. Hence, they can use the attached properties Screen.width and Screen.height, which pick up the screen size from the underlying graphics driver. The beginning of AppTheme.qml would look as follows:

// Extract from AppTheme.qml
pragma Singleton
import QtQuick 2.4
import QtQuick.Window 2.2

QtObject
{
    readonly property real refScreenWidth: 1280
    readonly property real refScreenHeight: 800

    readonly property real screenWidth: Screen.width
    readonly property real screenHeight: Screen.height

With this change, the music player will always use the full screen size, which is the right thing on embedded devices most of the time. If we don’t want this, we change the screen size back to some concrete values or retrieve it from a configuration file, where AppTheme.qml could double up as the configuration file.

Conclusion

We have shown a simple approach how to adapt a QML HMI to different screen sizes and formats. We scale the HMI written for a reference screen size to the target screen size by using the ratio of the reference and target screen size. We can fine-tune this scaling approach by using heuristics in the scaling functions as we have shown for scaling the font size of button texts. If the system provides the screen size like QML does through the attached properties Screen.width and Screen.height, the QML HMI adapts automatically to the actually screen size. This is almost always the case for embedded systems.

We only have to write the QML code once against the reference screen size and extract all size properties into a QML singleton (here: AppTheme.qml). Then, we assign the scaled initial values to the read-only properties. The initial values are scaled by calling the appropriate scaling functions on the reference initial values. We showed the normal functions for horizontal and vertical scaling and two custom scaling functions, one for scaling font sizes and one for not scaling values at all.

We can see two problems with the scaling approach when when we look at the smallest variant with a screen size of 400×225 in the image above. First, scaling is not pixel-perfect. The bottom border of the “Now Playing” button does not align with the bottom border of the song progress bar and the bottom border of the “Songs” button does not align with the top border of the music tool bar. The reason are rounding errors introduced by the scaling functions. These rounding errors can sum up and cause the misalignment of elements. The simplest way to avoid these rounding errors is to correct the values of the size properties by hand. But then we need to provide one implementation of AppTheme.qml for each screen size.

Second, the 400×225 variant could never be used in a car, because the HMI elements are too small to see easily, let alone to interact with easily. A good solution could be to move the MusicCategorySwitcher (the tab button group on the left-hand side) into a dialog, which is only shown when the user presses a menu button. This allows us to use the screen real estate of the MusicCategorySwitcher for the song info and music controls. However, this solution changes the structure of the HMI. We would have to implement parts of the small-sized HMI differently to the larger-sized HMIs.

With both problems, the way out is to provide a different implementation for each screen size and then switch between these implementations on startup. Such switching based on device characteristics is made easy by using Qt’s file selectors. But this will be the topic of my next post on responsive QML HMIs.

Getting the Source Code

You can download the source code of the scaling example from here. The archive unpacks into the directory scaling, which contains the Qt project file scaling.pro. Load the project file into QtCreator, configure, build and run the project. I am working with Qt 5.5, but nothing should prevent the project from working with Qt 5.1 or newer.

3 thoughts on “Responsive QML HMIs with Scaling

  1. Thanks for sharing the insights on scaling. The use of read-only constants is cleverly programmed to avoid magic numbers and the use of font-heuristics is also interesting. This would be helpful. I was quite curious to know that even though the Qml components like BorderImage comes with inbuilt scalablility, the same image still doesnt looks the same across different screen resolutions. Do we have to land up with different images?

    • Hi Jaweriya,

      No, we do not need different images.

      The “problematic” BorderImage is the one with the source image “bgTabButtonChecked.png”, which shows a green frame of 2 pixels around the selected tab button. If we wrote something like

      border.left: hscale(2)
      border.top: hscale(2)
      

      the value of border.left could be 1 and that of border.top be 2 – depending on the horizontal and vertical scaling factors. The left and top border would have different thicknesses. Moreover, half of the left border would be scaled and the other half not. All in all, the result wouldn’t look that great.

      These problems are the reason why the border properties are not scaled but have the same value for all target sizes. Border properties depend on the image provided and not on the target screen size.

      – Burkhard

      • Hi Burkhard,

        Thank you for the reply. I got it right.
        Loooking forward for your next blog.

Comments are closed.