An Introduction to Docker for Embedded Developers – Part 4 Reducing Docker Image Size

Posted on December 7, 2017 by Niall Cooling

In Part 3  we managed to build a Docker image containing the tools required to compile and link C/C++ code destined for our embedded Arm target system. However, we’ve paid little attention to the size of the image. Doing a quick Docker image listing we can see its grown to a whopping 2.14GB:

$ docker image ls
REPOSITORY              TAG     IMAGE ID       CREATED             SIZE
feabhas/gcc-arm-scons   1.0     6187455a4bfe   8 days ago          2.14GB
gcc                     7.2     7d9419e269c3   2 months ago        1.64GB

In your day-to-day work the size of a Docker image may not bother you as Docker caches images locally on your machine. But after a while you’ll certainly need to prune them.

Apart from freeing up disk space, why else look to reduce the size of an image?

Continuous-Integration (CI)

As previously mentioned, the overriding benefit of using a Docker-based build is consistency and repeatability of the build. But, for modern CI to be effective, we want the build/test cycle to be as quick as possible.

A local provisioned build server (such as a Linux server running Jenkins) will also cache images after the first build, so less of an issue.

Cloud-based CI services (such as the previously-mentioned Travis-CI, Bitbucket, Gitlabs, etc.) are typically costed on build-minutes for a given period,  e.g. build-minutes per month.  In these cases pulling or building larger images naturally takes longer; and costs more.

Generating Smaller Images

There is plenty of good guidance around, and I’m sure what I’m doing here can be improved up on. Our basic approach to minimise our image consists of these steps:

  1. Start with a minimal Base image
  2. Only install what we need
  3. Remove anything we only needed to help install what we needed!
  4. Reduce the number of Docker layers

Minimal Base Image

Our base image is gcc:7.2 which comes in at 1.64GB. To that we added Scons and the gcc-arm cross compiler, but for cross-compilation we don’t require the host GCC (x86) compiler.

The most widely used minimal image is call Alpine. Alpine Linux is a security-oriented, lightweight Linux distribution based on musl libc and Busybox.

Continue reading →

Posted in Agile, ARM, C/C++ Programming, Cortex, Testing | Tagged , , , | 4 Comments

An Introduction to Docker for Embedded Developers – Part 3 Cross-Compiling for Cortex-M

Posted on November 24, 2017 by Niall Cooling

In the previous posting we looked at defining a custom Dockerfile where we can add specific tools (and their dependencies). From that we created a Docker image and this allowed us to build C/C++ code in a Docker container, ensuring a consistent build environment.

So far we have to build all our code using the native GCC toolchain which is part of the base Docker image (gcc:7.2). However, I want to be able to build an image I can download and run on a target system (in our case an ARMv7-M, Cortex-M4, STM32F4-based system).

There are three stages required:

  1. Create a Docker container with the gcc-arm-embedded compiler installed
  2. Have a base “hello world” project for the board
  3. A custom Scons file for the cross-build (of course make or CMake could be used here)

Pre-built GNU toolchain for Arm Cortex-M

The latest version of the pre-built GNU toolchain for Arm Cortex-M (and Cortex-R) is found here.

Normally we’d go through the process of selecting the correct package for our development machine (Windows/Mac/Linux), downloading it and installing it (setting up appropriate paths, etc.).

But with Docker we can create a container for this current version and use this to cross-compile our code.

Building on the Dockerfile from the previous post (gcc:7.2 + scons) we need too:

  1. Download the Linux tarball (tar archive) for the gcc arm cross-complier
  2. Extract the code from the tarball
  3. Remove the tarball file from the Docker image
  4. Set up the PATH to compiler /bin

Continue reading →

Posted in Agile, ARM, C/C++ Programming, Cortex, Testing | Tagged , , | 3 Comments

An Introduction to Docker for Embedded Developers – Part 2 Building Images

Posted on October 12, 2017 by Niall Cooling

In the initial post, we covered the basics of getting Docker setup and using an official base image for compilation.
But let’s suppose the base image doesn’t include all the facilities our company uses for development. For example, we have migrated from make files to CMake, but more lately we have taken to using the python-based Scons build system for C/C++ projects.
The official gcc base image supports make but not Scons or CMake. As before, we can search for a Scons docker image, but will see no official image exists. We now have two choices:

  1. Use an unofficial (user) image
  2. Build our own

Unofficial Images

Using an unofficial image is a bit of a hit-or-miss affair. A good unofficial image will have a decent set of instructions about its use and how it was built. Unfortunately, most don’t have any information and you can waste a substantial amount of time trawling through the repositories looking for a suitable image. In general, it may be better to build you own image from a base image.
It is worth noting there are some gems out there in the user community so it still can be worth a cursory browse.

Build your own image

How easy build your own image will depend on you experience with using a Linux package manager. If you are familiar with adding packages to a running Linux distribution (e.g. apt-get install), then you’ll see creating an image is straightforward. If you’re not very experienced with this aspect of Linux, then it’s worth spending a little bit of time getting a foundation understanding of the Advanced Packaging Tool.
It is important to understand that different Linux distributions (Ubuntu, Fedora, etc.) each have their own package manager. I still find this one of the most frustrating aspects when working with Linux (Ubuntu uses APT, Fedora uses yum). You will need to know the base Linux distribution and thus the associated package manager to build Docker images.

Basic workflow

There are two methods for building a Docker image:

  1. Build an image locally on your working machine
  2. Use a github/bitbucket repository to automatically build in the cloud

Of course, the choice depends on the intended use, maturity of the image, and how widely it is going to be used. Initially, though, while understanding Docker it is better to use a local build.
For a local build, the steps couldn’t be simpler:

  1. Create a “Dockerfile” file that specifies to image contents and behaviour
  2. Build the image (docker build)
  3. Use the image (docker run)

Dockerfile

Our Dockerfile specifies the following items:

  1. The base image we want to work from
  2. Updating the base image
  3. The new packages/files that need installing
  4. Setting up working directories/path information
  5. Default behaviour when run

Continue reading →

Posted in Agile, General | Tagged , | 5 Comments

An Introduction to Docker for Embedded Developers – Part 1 Getting Started

Posted on September 7, 2017 by Niall Cooling

Docker is a relatively new technology, only appearing just over four years ago. The core building blocks have always been part of Unix; but the significant support, Linux containers (LCX), first appeared back in 2008.

Initially Docker was only supported on Linux, but more recently native support for OSX (my development OS of choice) and Windows (albeit Windows 10 Pro) suddenly opens up some interesting workflow choices.

The “What”

So, first, what is Docker? I’m always trying to find the right words here that does Docker justice but doesn’t over simplify the technology. A one-liner is:

“A lightweight Virtual Machine”

The danger of this over-simplified statement is the natural follow-on questions trying to compare Docker to, say, a hypervisor technology such as VirtualBox.

Another one-liner I try is:

“It’s like a Linux process with its own file system and network connections”

However, a fuller description is:

“Linux containers are self-contained execution environments—with their own, isolated CPU, memory, block I/O, and network resources—that share the kernel of the host operating system. The result is something that feels like a virtual machine, but sheds all the weight and startup overhead of a guest operating system.”

So, Docker allows me to wrap up a program and all its dependences (e.g. python tools, libraries, etc.) into a single, isolated executable environment. The wrapping up of the program and its dependences is called a Docker image; when image is executed it runs as a Docker container.

The “Why”

Probably more importantly is “Why would I use Docker as an Embedded developer?”. Most of the current Docker development is in the field of DevOps called “microservices” where Docker is being used to deploy applications. This is, currently, far removed from most embedded systems and I won’t address here. [ See Getting started with Docker on your Raspberry Pi if that floats your boat]

How, then, can Docker help an Embedded developer?

Some of the key benefits are:

  • For anyone not developing on Linux it opens up the world of Free Open Source Software (FOSS) that quite often are not available on other platforms (or difficult to install)
  • It allows developers to use tools in their local development environment without having to install them (even if there is a build available).
  • It allows code to be checked against variants of toolchains without the struggle of tools co-existing
  • It ensures all team members are using exactly the same tools and build environment
  • It ensures my build server (e.g. Jenkins on Linux) is building against the same tools used in development and vice-versa.
  • It allows me to create a virtual TCP/IP network of separate applications on a single machine
  • It allows me to experiment with support technology without having pollute my development machine (e.g. run a local nginx HTTP server as a target for my IoT development in a Docker container)

There are plenty more good reasons to look at Docker which I’m sure you’ll start to see as you get use to it.

Continue reading →

Posted in Agile, Design Issues, Testing | 6 Comments

Making things do stuff – Part 9

Posted on August 3, 2017 by Glennan Carnie

As a final instalment in this series on hardware manipulation I thought I’d revisit read-only and write-only register types.

Using tag dispatch is not the only way to solve the read- or write-only Register problem.  For completeness let’s explore two other alternatives – SFINAE and constexpr if.

For these examples I’m going to use a simplified version of our Register class.  I’m ignoring the bit proxy class and using a reduced API.  Once understood, the techniques below can be applied to these elements in the same way we applied tag dispatch.

As before, if you’ve jumped into this article I’d highly recommend going back and reading the previous articles to get an idea of the context of the problem.

Continue reading →

Posted in C/C++ Programming, Design Issues | Tagged , , , , , , , , , , | 1 Comment

Making things do stuff – Part 8

Posted on July 20, 2017 by Glennan Carnie

We’ve been using templates to provide a hardware register abstraction for use with hardware manipulation problems, typical of what you’d find in a deeply-embedded (“bare-metal”) system.

Previously, we looked at using trait classes to establish pointers and tag-dispatch to handle special-case registers, such as read-only or write-only register.

In this article we’re going to add some syntactic sugar to our Register class, to allow the developer to access individual bits in the register using array-index ([]) notation.  This will allow us to explore the concept of proxy objects with a concrete and practical use case.

If you’re new to hardware manipulation in C++, I’d recommend going back to the basics article for a refresher of what we’re trying to achieve.

If you’re familiar with hardware manipulation, I’d highly recommend going back to article 2 to familiarise yourself with our problem domain.

Continue reading →

Posted in General | Tagged , , , | 2 Comments

Making things do stuff – Part 7

Posted on July 6, 2017 by Glennan Carnie

In our previous article we explored using templates to build a generic ‘register’ type to allow programmers to access hardware without all the nasty syntax of integer-to-pointer casting, etc.

At the moment, this class gives us little extra functionality beyond cleaning up the syntax (although, in its favour, it also doesn’t incur any additional run-time cost/performance).

In this article we’re going to extend our design to consider special hardware register types – notably read-only and write-only registers – and see how we can add extra functionality to our Register type.

Although there is more than one way to solve this problem we’re going to use a simple function-overload selection mechanism called ‘tag dispatch’.

Continue reading →

Posted in C/C++ Programming | Tagged , , , , , , | 2 Comments

Making things do stuff – Part 6

Posted on June 22, 2017 by Glennan Carnie

As code designers we tend to eschew specific ‘stove-pipe’ code in favour of reusable code elements.  Up until now we’ve been coding some very specific examples so it’s probably worth looking at some more generic solutions.

In this article we’ll look at building generic register manipulation classes (or, as one commenter referred to them, ‘register proxy’ classes).  Here, we’re really exploring code design rather than coding ‘mechanics’.  I’m using this to explore some factors like the balance between efficiency, performance and flexibility.

I’ll be making some design choices that you may not agree with.  That’s fine.  Leave your discussion points in the comments below.

If you’re new to hardware manipulation in C++, I’d recommend going back to the basics article for a refresher of what we’re trying to achieve.

Continue reading →

Posted in C/C++ Programming | Tagged , , , , , , , | 3 Comments

Making things do stuff – Part 5

Posted on May 18, 2017 by Glennan Carnie

We’ve been looking at using C++ to manipulate I/O hardware.   Previously, we’ve looked at the fundamentals of hardware manipulation; and how to encapsulate these mechanisms into classes.  If you’ve not been following along I’d recommend reading the previous articles first before continuing.

This time we’ll explore a lesser-known feature of C++ and its application in hardware manipulation – placement new.

Continue reading →

Posted in C/C++ Programming | Tagged , , , , , | 3 Comments

Making things do stuff – Part 4

Posted on May 4, 2017 by Glennan Carnie

In the last article we explored the design of a class to encapsulate a physical hardware device.  In that article I deliberately ignored how the class would actually interact with the hardware.

In this article we explore the options available to us for accessing hardware and the consequences of those choices.

Continue reading →

Posted in ARM, C/C++ Programming, Cortex, General | Tagged , , , , , , | Leave a comment