Responsive QML HMIs with File Selectors

In my previous post, I have shown how to use scaling to adapt QML HMIs to different screen sizes and formats. We reach the limits of scaling if we must change the structure of the HMI or if the HMI must be pixel-perfect. The solution to these problems is to provide a different implementation for each screen size. Switching between these different implementations is done with QML file selectors.

Motivation

The HMIs of in-vehicle infotainment (IVI) systems, in-flight entertainment (IFE) systems, TVs, set-top-boxes and phones must run on different devices with different screen sizes and formats. For example, an IVI system could be built in three different variants with an extra large (XL) screen size of 1280×800 (16:10), a medium (M) screen size of 800×600 (4:3) and an extra small (XS) screen size of 400×225 (16:9).

In my previous post “Responsive QML HMIs with Scaling”, I have explained how to use scaling to adapt to different screen sizes. Scaling is very simple and works pretty well, if the reference and target screen sizes are not too far apart (rule of thumb: scaling factors less than 2 work OK) and if we can live with slight misalignments between HMI elements due to aggregating rounding errors.

The image of the scaled 400×225 variant shows that the HMI is far too small to be used in a car and that the “Now Playing” and “Songs” buttons are not properly aligned with the song progress bar and the music tool bar, respectively.

The pretty obvious solution to the first problem (HMI too small) is to provide a different implementation of the HMI with a different structure for the XS variant. The solution for the second problem (HMI not pixel-perfect) is to provide different implementations of AppTheme.qml – one for each screen size. AppTheme.qml defines the size properties and initialises them with the scaled values. Instead of assigning scaled and rounded values to the size properties, we assign manually calculated and corrected values. This way we can avoid aggregating rounding errors with their resulting misalignments.

The classical way to switch between different implementations would be to use conditional compilation with #ifdef‘s in C++ or conditional instantiation with the QML Loader component. Such code quickly becomes hard to understand and even harder to change and maintain. This is where QML file selectors come to rescue. As stated in the documentation, they provide a convenient way to select different file variants (here: implementations) based on platform and device characteristics (here: screen sizes).

For example, the file main.qml is the same for all screen sizes. It instantiates MainMusicPage, which is different for screen size XS – as we can see in the screenshots below. A QML file selector picks up the correct variant of MainMusicPage for the given screen size. It picks up the file qml/MainMusicPage.qml for screen sizes XL and M and the file qml/+sizeXS/MainMusicPage.qml for screen size XS. We do not see anything of this size-dependent selection in the QML code.

Left: Variant M (800x600, 4:3). Right: Variant XS (400x225, 16:9).

Left: Variant M with screen size 800×600 (4:3), which looks the same as variant XL with screen size 1280×800 (16:10). Top right: Main music page of variant XS with screen size 400×225 (16:9). Bottom right: Screen for selecting music categories reached when user presses menu button in bottom left corner of main music page.

Solution

We begin with eliminating the misalignments between QML widgets as evident in the screenshots of the scaling solution. This allows us to introduce QML file selectors in a simple context. We will then use this knowledge about file selectors to implement a different structure for the XS variant.

The fileSelector application takes a command-line argument screenSize with the values sizeXL (1280×800), sizeM (800×600) and sizeXS (400×225). For example, if we pass the argument --screenSize=sizeXS to the application, the application shows the HMI for the screen size 400×225. We use the value of the option screenSize to determine the file selector. The default screen size is XL. The corresponding file selector is the empty string “”. The file selector for the other screen sizes M and XS is just the value of the command-line option, sizeM and sizeXS. The conversion from option values to file selectors is done by the function screenSizeSelector() in main.cpp. The relevant code for setting the file selector in main() looks as follows

int main(int argc, char *argv[])
{
    QApplication app(argc, argv);
    // [...] Code for parsing the command line for option screenSize
    QQmlApplicationEngine engine;
    engine.addImportPath("qrc:/qml");
    QQmlFileSelector *selector = new QQmlFileSelector(&engine);
    selector->setExtraSelectors(QStringList() << screenSizeSelector(screenSize));
    engine.load(QUrl(QStringLiteral("qrc:/qml/main.qml")));
    return app.exec();
}

The QML file selector must be a child of the QML engine, because the QML engine takes ownership of the file selector and will eventually delete the file selector. As our screenSize file selectors are not built-in file selectors, we must set the selected screen size as a custom or extra selector. Platform names like “android”, “ios”, “mac”, “linux” and “windows” and locale names like “en_GB” and “de_DE” are built-in file selectors.

We will now provide three variants of the file AppTheme.qml for the three different screen sizes XL, M and XS such that the file selector has something to select from. Actually, we will provide three variants of the file SizeTheme.qml, which is the new and more telling name for AppTheme.qml. SizeTheme.qml contains the definitions of all the size properties needed by our sample music app.

The starting point for producing the three SizeTheme.qml variants is the file AppTheme.qml from the scaling approach. Let us look at the size property leftTabButtonHeight plus some related properties in AppTheme.qml.

    // Extract from AppTheme.qml taken from scaling approach
    // screenHeight = 225, refScreenHeight = 800
    function vscale(size) {
        return Math.round(size * (screenHeight / refScreenHeight))
    }

    readonly property real mainPageHeight: vscale(599)
    readonly property real dividerSize: 1
    readonly property real leftTabButtonHeight: vscale(99)

The property leftTabButtonHeight speficies the height of the tab buttons “Now Playing”, “Artists” and so on on the left-hand side of the HMI. The height for the reference screen size XL is 99. Every tab button is followed by a border of size 1 as defined by property dividerSize. The height of all six buttons is 600, which is equal to the 599 pixels of mainPageHeight plus a 1-pixel border at the bottom. The scaling method would yield

    leftTabButtonHeight = Math.round(99 * (225 / 800)) = Math.round(27.84) = 28
    mainPageHeight = Math.round(599 * (225 / 800)) = Math.round(168.47) = 168

The height of all buttons would be (28 + 1) * 6 = 174, which is greater than the 169 pixels of mainPageHeight plus border. This explains why the bottom border of the tab button “Songs” does not align with the top border of the music tool bar. If we used 27 instead of 28 for leftTabButtonHeight, we would get (27 + 1) * 6 = 168 and 167 for mainPageHeight, the alignment would be perfect. Such corrections of a pixel up and down are typical when calculating pixel-perfect values of size properties.

Our sample size properties would look as follows in the SizeTheme.qml file for the size XS.

    // Extract from SizeTheme.qml for screen size XS (400x225)
    readonly property real mainPageHeight: 167
    readonly property real dividerSize: 1
    readonly property real leftTabButtonHeight: 27

In the final version, we will not use leftTabButtonHeight and set it to 0, because we will provide an implementation for MusicCategorySwitcher.qml specific to screen size XS. But, the property leftTabButtonHeight was a good example where the misalignments in the scaling approach come from and how to fix them.

The same properties for size M look as follows.

    // Extract from SizeTheme.qml for screen size M (800x600)
    readonly property real mainPageHeight: 450
    readonly property real dividerSize: 1
    readonly property real leftTabButtonHeight: 75

And here are the properties for size XL.

    // Extract from SizeTheme.qml for screen size XL (1280x800)
    readonly property real mainPageHeight: 600
    readonly property real dividerSize: 1
    readonly property real leftTabButtonHeight: 100

We calculate the other size properties for all screen sizes in a similar way. This gives us three variants of SizeTheme.qml, one for each screen size XS, M and XL. We now have to put these three variants in the right directories such that QML’s file selection mechanism picks them up according to the size file selector set in the main() function.

The reference variant (XL) of SizeTheme.qml lived in the QML module EmbeddedAuto.Themes and hence in the directory qml/EmbeddedAuto/Themes. The variants of SizeTheme.qml for XS and M go into the subdirectories +sizeXS and +sizeM of qml/EmbeddedAuto/Themes, respectively. The directory layout looks as follows.

    qml/EmbeddedAuto/Themes/SizeTheme.qml
    qml/EmbeddedAuto/Themes/+sizeM/SizeTheme.qml
    qml/EmbeddedAuto/Themes/+sizeXS/SizeTheme.qml

It is important that the variant subdirectories start with a plus sign followed by the name of the file selector. The plus sign is the indicator for the file selection mechanism to check for variants. The reference variant for XL in the base directory must be available. If not, the file selection mechanism fails with an error.

Let us understand how QML resolves the following reference to the singleton SizeTheme, if the file selector sizeXS has been set in main():

    // Extract from SongInfo.qml
    import EmbeddedAuto.Themes 1.0
    // ...
    SongInfo {
        anchors.margins: SizeTheme.songInfoMargin
        // ...
    }

The QML engine finds SizeTheme in the QML module EmbeddedAuto.Themes, which is mapped to the directory qml/EmbeddedAuto/Themes – according to the added import path in main(). The QML engine has found the reference or base variant qml/EmbeddedAuto/Themes/SizeTheme.qml. As the file selector sizeXS is active, the QML engine will look for the file SizeTheme.qml in the subdirectory +sizeXS of the directory with the reference variant. So, it will look for qml/EmbeddedAuto/Themes/+sizeXS/SizeTheme.qml. The XS variant of SizeTheme.qml exists. Hence, the QML engine will look up the value of the property songInfoMargin in the variant file +sizeXS/SizeTheme.qml.

Now that we have a good idea of how file selectors work, we can apply our knowledge to provide a different implementation of the main music page. Our side objective is to write as little additional code as possible, that is, to avoid code duplication.

From the screenshots above, it is obvious that the main music page (MainMusicPage.qml) for XS differs quite a bit from the main music page for M and XL. MainMusicPage for XS has neither a status bar nor an application tool bar. It does not have the tab button group (MusicCategorySwitcher.qml) for switching between music categories on the left-hand side either. It actually provides MusicCategorySwitcher on a separate screen, some kind of dialog. The song progress bar (SongProgressBar.qml) is a one-line affair showing only the progress of a song and omitting the shuffle and repeat buttons. Finally, the music tool bar (MusicToolBar.qml) has only four tool buttons instead of five.

We will provide XS-specific implementations for the QML components MainMusicPage, MusicCategorySwitcher and SongProgressBar, because their structure is very different to their M and XL counterparts. These XS variants will be handled by file selectors. We can deal with the other differences – status bar, app tool bar and music tool bar – through some clever use of the size properties. Let us work on the XS variants first.

The reference implementations of MainMusicPage.qml, MusicCategorySwitcher.qml and SongProgressBar.qml are located in the directory qml. The XS variants will be put into the directory qml/+sizeXS. This leaves us with the following directory layout.

    qml/MainMusicPage.qml
    qml/MusicCategorySwitcher.qml
    qml/SongProgressBar.qml
    qml/+sizeXS/MainMusicPage.qml
    qml/+sizeXS/MusicCategorySwitcher.qml
    qml/+sizeXS/SongProgressBar.qml

If we select XL in main(), the QML engine will select the reference variant from the qml directory. It will also fall back to this reference variant if we select M in main(), because there is no directory qml/+sizeM. In general, if there is no subdirectory for a given file selector, the QML engine falls back to the reference variant. If we select XS in main(), the QML engine picks up the variants from the directory qml/+sizeXS – as we would have expected from the discussion about selecting the right variant of SizeTheme.qml.

The best thing about Qt’s file selection mechanism is that we do not have to write any if-then-else cascades in our source code to handle variants. The relevant code for main.qml looks as follows.

// Extract from main.qml
Window {
    Column {
        AppStatusBar { /* ... */ }

        MainMusicPage { /* ... */ }

        AppToolBar { /* ... */ }
    }
}

The code for main.qml is the same for all variants. The instantiation of MainMusicPage does not reveal that there are two variants. The selection mechanism is fully transparent to the source code. If the file selector for XS is active, the QML engine will select the variant qml/+sizeXS/MainMusicPage.qml. Otherwise, it will select the variant qml/MainMusicPage.qml.

It gets even better. The XS variant of MainMusicPage instantiates the QML component NowPlayingView. As NowPlayingView is the same for all variants, we do not have to provide an extra variant for XS. The QML engine is clever enough to figure out that qml/NowPlayingView.qml is the right file to choose. NowPlayingView instantiates SongProgressBar, which comes in two variants again. If XS is the active variant, NowPlayingView will load qml/+sizeXS/SongProgressBar.qml. Otherwise, it will load qml/SongProgressBar.qml. So, it is perfectly OK to switch back and forth between the reference variant and a specific variant. Qt’s file selectors will just make it work seamlessly!

Now we know how to deal with the big structural differences. But how can we deal with the smaller differences in the status bar, app tool bar and music tool bar – without using file selectors?

Getting rid of the status bar and app tool bar is pretty simple. We make StatusBar and AppToolBar invisible when their height is 0. The relevant code in main.qml looks as follows.

Window {
    Column {
        AppStatusBar {
            height: SizeTheme.statusBarHeight
            visible: height !== 0
        }

        MainMusicPage {
            height: SizeTheme.mainPageHeight
        }

        AppToolBar {
            height: SizeTheme.appToolBarHeight
            visible: height !== 0
        }
    }
}

This leaves us with MusicToolBar, which has three buttons for XS and four buttons for the other screen sizes – not counting the More button. We introduce the size property musicToolBarButtonHeight, which defines the number of visible buttons in the tool bar and which is set to 3 for XS and 4 for the other sizes. The code for Button in MusicToolBar.qml looks as follows.

Button {
    visible: index < SizeTheme.musicToolBarButtonCount
}

In a full version of the music app, the app would show the next musicToolBarButtonHeight number of buttons when the user pressed the More button (with the three dots). The More button would act like a PageNext button. Of course, the expression assigned to the visible property would have to be adapted.

There is another little detail that differs between XS and the other sizes. The MusicToolBar in XS has no border at the bottom, whereas the other sizes have a bottom border delimiting the music tool bar from the app tool bar. Again, we can solve this variation by using a size property. The style property of the Button is defined as follows.

    style: ToolButtonStyle {
        backgroundNormal: SizeTheme.appToolBarHeight !== 0 ? BrandTheme.bgToolButtonNormalRTB 
                                                           : BrandTheme.bgToolButtonNormalRT
                    backgroundSelected: BrandTheme.bgToolButtonSelected
    }

If the height of the app tool bar is not 0 as for M and XL, there is a bottom border. Otherwise, there is no bottom border as for XS.

We could have made our lives easy and have simply reimplemented main.qml and MusicToolBar.qml for size XS. These two new files would have been put into the directory qml/+sizeXS. But that would have resulted in code duplication and violated our side objective of writing as little additional code as possible.

In general, we should use file selection only when the HMI structure and hence the code of its implementation differ considerably. This is obviously the case for MainMusicPage, MusicCategorySwitcher and SongProgressBar. We should avoid file selectors if we can get away with a few, simple conditional statements. We need two conditional statements in each of main.qml and MusicToolBar.qml. If we find ourselves writing too many conditional statements, we should start thinking about writing a variant and using file selection.

Conclusion

QML's file selectors are a powerful, non-intrusive and simple way to adapt an HMI to different screen sizes and formats. File selectors remedy the shortcomings of the scaling approach, which cannot be used to realise HMI variants with considerably different structures like the XS variant and which suffers from misaligned elements because of aggregating rounding errors.

The implementation of the scaling approach is a very good starting point for the file-selector approach. Having extracted all the size properties into a single file, SizeTheme.qml, is essential for both approaches. We could actually use the scaling functions to print out the SizeTheme.qml files for the different sizes: just add console.log() calls for every property in the Component.onCompleted handler of SizeTheme.qml and print the size properties in the correct QML syntax. The scaled values are a good starting point to reach pixel-perfect values. Most of the scaled values are correct. Only a few must be corrected. At this point, we have a SizeTheme.qml file for each screen size and a pixel-perfect scaled HMI - without any misalignments. This solves the first problem of the scaling approach.

The second problem of the scaling approach is that small variants like for screen size XS (400x225) become unusable. File selectors allow us to provide different implementations (variants) for only the relevant parts of the HMI. Providing variants always means some sort of code duplication. So, we must strive to keep the number of variants to a minimum. For the music app, we provided an XS-specific implementation for three QML components: MainMusicPage, MusicCategorySwitcher and SongProgressBar. We avoided two more such implementations for main.qml and MusicToolBar.qml by using simple conditional statements. As a rule of thumb, we should provide size-specific implementations if we find ourselves writing too many conditional statements.

The best thing about file selectors is that they do all of the above without any if-then-else statements for selecting variants in the code. The selection of the variants happens completely behind the scenes. It is not visible in the source code.

Getting the Source Code

You can download the source code of the scaling example from here. The archive unpacks into the directory fileSelector, which contains the Qt project file fileSelector.pro. Load the project file into QtCreator, configure, build and run the project. Pass the command-line option --screenSize=sizeXS, --screenSize=sizeM or --screenSize=sizeXL to the executable fileSelector to select the screen size XS, M or XL, respectively. I am working with Qt 5.5, but nothing should prevent the project from working with Qt 5.2 or newer.