Building Linux Images For Use With Openstack Heat


This Article is published on with My Permission.



The Openstack image is cloud-init enabled image. Instances created from the image run the cloud-init script on early boot to do some configurations. The directory “/var/lib/cloud” is the directory which contains the cloud-init specific subdirectories.

# ls /var/lib/cloud/
data handlers instance instances scripts seed sem


In these subdirectories you can find the current instance id, previous instance id (it could be different) , current and previous datasources, start and end time , errors if exists, custom scripts (scripts/) downloaded/created by the corresponding handlers (handlers/ subdirectory), user data downloaded from metadata service (/instance/scripts/user-data.txt as RAW data and user-data.txt.i as post-processed data),….etc.

But what if we want some configurations/actions on different life cycles and not only on early boot (cloud-init). For example if you are using Openstack autoscaling service, this means your servers will be added and deleted automatically without human intervention. When any of these servers is determined to be killed, it needs some time to clean itself and it needs some configuration to be applied on this stage (e.g drains its active sessions, its active connections to other services if exist,..). Here we come to what is called Heat Software Deployment which is a communication between the Openstack Heat unit and the instance to configure the instance on different life cycles and not only on boot. This feature is available since Openstack Icehouse release. For this we need some agents to be installed in the image and some definitions in the Heat template (mainly based on “OS::Heat::SoftwareDeployment” and “OS::Heat::SoftwareConfig” resource types).

The agents can be installed on boot (script that installs the agents on boot or definitions in Heat template to install the agents on boot). Here I will talk about how to install the agents in the image. The agents are: heat-config, os-collect-config, os-refresh-config, os-apply-config, heat-config-cfn-init, heat-config-puppet, heat-config-salt and heat-config-script.

Preparations For Image Building

Here i am assuming that you have your own Openstack image and you want to add the Heat agents to your image. I will use the DiskImage-Builder to build the image. Diskimage-Builder (DIB) is driven by HP, RedHat, and Cisco and licensed under the Apache License, Version 2.0. My operating system is Fedora 21 and i will build Centos7 custom image with Heat support.

If your image is already on the cloud, you can download it using glance client after you source your credentials (keystonerc).

To install the glance client on Fedora:

# yum install python-devel libffi-devel

# pip install python-glanceclient

To download the image, execute:

# glance image-download –file outputImageFilename imageID

Ex: # glance image-download –file my_image.qcow2 155ae0f9-56ee-4dad-a3a3-cc15a835c035

By doing this, you will get the file my_image.qcow2 on the local disk which is in my case Centos7 image.

The DiskImage-Builder needs enough RAM for caching. If your image is big and your RAM is not enough , you can use /tmp for caching on disk (See how to use /tmp further bellow -Notes Section).

Now we need to clone the repositories: diskimage-builder, image elements, and heat-templates:

# git clone

# git clone

# git clone

Some environment variables need to be exported:

DIB_LOCAL_IMAGE variable: This variable contains the path to the local image (in my case my_image.qcow2). If this variable is not set, the diskimage-builder will go to internet and bring a generic image (e.g. in case centos7, it will go to If you don’t have any image just don’t export this variable to keep the default.

DIB_CLOUD_IMAGES variable (HTTP address): If your image is public on internet, you can export this variable.

ARCH variable: This variable contains the architecture. In my case “amd64”: # export ARCH=”amd64″. This will be converted to x86_64.

My image is Centos7 so the source file which use this variable is “…/diskimage-builder/elements/centos7/root.d/10-centos7-cloud-image”. I recommend you to check the corresponding file for your choice (Go to the root.d directory in your OS choice directory (centos7 in my case) and check what is supported for ARCH). Check this file also for DIB_CLOUD_IMAGES variable and others.

Then we need to install some packages:

# yum install qemu or # yum install qemu-img

# yum install python-pip git

# pip install git+git://

Image Building

Now i want to add the Heat agents to my image using diskimage-builder. So the instances created from my image can interact with Heat on different life cycles. Write the following in bash script and execute it or take it line by line:

Note: You need to be in the directory where you have cloned the repositories or use the absolute paths.

export ELEMENTS_PATH=tripleo-image-elements/elements:heat-templates/hot/software-config/elements

# customize this line. Possible values: fedora centos7, debian, opensuse, rhel, rhel7, or ubuntu

export BASE_ELEMENTS=”centos7 selinux-permissive”

export AGENT_ELEMENTS=”os-collect-config os-refresh-config os-apply-config”

export DEPLOYMENT_BASE_ELEMENTS=”heat-config heat-config-script”

# For any other chosen configuration tool(s). e.g. heat-config-cfn-init, heat-config-puppet, or heat-config-salt. NOT IN MY CASE.


export IMAGE_NAME=software-deployment-image-gold


Note the variable IMAGE_NAME determines the image name.

Image Uploading

After building the image , you will get the file software-deployment-image-gold.qcow2. If your image is big, you can compress it before uploading it to Openstack (Glance unit):

# tar -zcvf gw-software-deployment-gold.qcow2.tar.gz gw-software-deployment-gold.qcow2

You can also pass the option “-t tar” to the command diskimage-builder/bin/disk-image-create to have the image compressed by the diskimage-builder. Also use this option “-a i386|amd64|armhf” for the ARCH mentioned above. See the full documentation Here.

To upload the compressed image, execute:

# glance image-create –name “gw-software-deployment-gold” –is-public false  –disk-format qcow2 –file gw-software-deployment-gold.qcow2.tar.gz

As i found, Firefox cannot be used to upload large files like images whereas Chrome can. So you can upload your image from the Openstack control panel (Horizon) using Chrome browser.

You may also need to tune the performance parameters of the operating system (Update system files) and the application itself so the servers created from the image are already tuned. You can do the tuning on the live server before taking a snapshot or you can update the system files of the local image using guestfish before uploading.

Now you can create an instance from the new uploaded image where you can start working with software deployment on different lifecycle in your Heat template.

Using “/tmp” For Caching On Disk

If your RAM is not enough for the image building process, you need to increase the /tmp. For example 15 GB:

# dd if=/dev/zero of=/usr/tmp-dir bs=1024M count=15

# mke2fs -j /usr/tmp-dir; chmod 777 /usr/tmp-dir

# mount /usr/tmp-dir /tmp


  • During image building, if you find some problems related to packages update, you can temporarily disable repositories that cause the problems by using yum-config-manager –disable repositoryName in the live server before creating the image (The image that you will download from the cloud to work on).

  • Commenting the line “yum -y update” in the file “…./heat-templates/hot/software-config/heat-container-agent/scripts/”, could solve the problems that arise on update (conflict between different versions of packages).

  • You could do the process many time with different updates till the image creation process completed successfully. If you are using the default, you are not supposed to have any problems.

  • Package dependencies file in my case Centos7 is “…/diskimage-builder/elements/centos7/element-deps”: If you like you can add more here so it will be installed.

More Information