Building Linux Images For Use With Openstack Heat


This Article is published on with My Permission.



The Openstack image is cloud-init enabled image. Instances created from the image run the cloud-init script on early boot to do some configurations. The directory “/var/lib/cloud” is the directory which contains the cloud-init specific subdirectories.

# ls /var/lib/cloud/
data handlers instance instances scripts seed sem


In these subdirectories you can find the current instance id, previous instance id (it could be different) , current and previous datasources, start and end time , errors if exists, custom scripts (scripts/) downloaded/created by the corresponding handlers (handlers/ subdirectory), user data downloaded from metadata service (/instance/scripts/user-data.txt as RAW data and user-data.txt.i as post-processed data),….etc.

But what if we want some configurations/actions on different life cycles and not only on early boot (cloud-init). For example if you are using Openstack autoscaling service, this means your servers will be added and deleted automatically without human intervention. When any of these servers is determined to be killed, it needs some time to clean itself and it needs some configuration to be applied on this stage (e.g drains its active sessions, its active connections to other services if exist,..). Here we come to what is called Heat Software Deployment which is a communication between the Openstack Heat unit and the instance to configure the instance on different life cycles and not only on boot. This feature is available since Openstack Icehouse release. For this we need some agents to be installed in the image and some definitions in the Heat template (mainly based on “OS::Heat::SoftwareDeployment” and “OS::Heat::SoftwareConfig” resource types).

The agents can be installed on boot (script that installs the agents on boot or definitions in Heat template to install the agents on boot). Here I will talk about how to install the agents in the image. The agents are: heat-config, os-collect-config, os-refresh-config, os-apply-config, heat-config-cfn-init, heat-config-puppet, heat-config-salt and heat-config-script.

Preparations For Image Building

Here i am assuming that you have your own Openstack image and you want to add the Heat agents to your image. I will use the DiskImage-Builder to build the image. Diskimage-Builder (DIB) is driven by HP, RedHat, and Cisco and licensed under the Apache License, Version 2.0. My operating system is Fedora 21 and i will build Centos7 custom image with Heat support.

If your image is already on the cloud, you can download it using glance client after you source your credentials (keystonerc).

To install the glance client on Fedora:

# yum install python-devel libffi-devel

# pip install python-glanceclient

To download the image, execute:

# glance image-download –file outputImageFilename imageID

Ex: # glance image-download –file my_image.qcow2 155ae0f9-56ee-4dad-a3a3-cc15a835c035

By doing this, you will get the file my_image.qcow2 on the local disk which is in my case Centos7 image.

The DiskImage-Builder needs enough RAM for caching. If your image is big and your RAM is not enough , you can use /tmp for caching on disk (See how to use /tmp further bellow -Notes Section).

Now we need to clone the repositories: diskimage-builder, image elements, and heat-templates:

# git clone

# git clone

# git clone

Some environment variables need to be exported:

DIB_LOCAL_IMAGE variable: This variable contains the path to the local image (in my case my_image.qcow2). If this variable is not set, the diskimage-builder will go to internet and bring a generic image (e.g. in case centos7, it will go to If you don’t have any image just don’t export this variable to keep the default.

DIB_CLOUD_IMAGES variable (HTTP address): If your image is public on internet, you can export this variable.

ARCH variable: This variable contains the architecture. In my case “amd64”: # export ARCH=”amd64″. This will be converted to x86_64.

My image is Centos7 so the source file which use this variable is “…/diskimage-builder/elements/centos7/root.d/10-centos7-cloud-image”. I recommend you to check the corresponding file for your choice (Go to the root.d directory in your OS choice directory (centos7 in my case) and check what is supported for ARCH). Check this file also for DIB_CLOUD_IMAGES variable and others.

Then we need to install some packages:

# yum install qemu or # yum install qemu-img

# yum install python-pip git

# pip install git+git://

Image Building

Now i want to add the Heat agents to my image using diskimage-builder. So the instances created from my image can interact with Heat on different life cycles. Write the following in bash script and execute it or take it line by line:

Note: You need to be in the directory where you have cloned the repositories or use the absolute paths.

export ELEMENTS_PATH=tripleo-image-elements/elements:heat-templates/hot/software-config/elements

# customize this line. Possible values: fedora centos7, debian, opensuse, rhel, rhel7, or ubuntu

export BASE_ELEMENTS=”centos7 selinux-permissive”

export AGENT_ELEMENTS=”os-collect-config os-refresh-config os-apply-config”

export DEPLOYMENT_BASE_ELEMENTS=”heat-config heat-config-script”

# For any other chosen configuration tool(s). e.g. heat-config-cfn-init, heat-config-puppet, or heat-config-salt. NOT IN MY CASE.


export IMAGE_NAME=software-deployment-image-gold


Note the variable IMAGE_NAME determines the image name.

Image Uploading

After building the image , you will get the file software-deployment-image-gold.qcow2. If your image is big, you can compress it before uploading it to Openstack (Glance unit):

# tar -zcvf gw-software-deployment-gold.qcow2.tar.gz gw-software-deployment-gold.qcow2

You can also pass the option “-t tar” to the command diskimage-builder/bin/disk-image-create to have the image compressed by the diskimage-builder. Also use this option “-a i386|amd64|armhf” for the ARCH mentioned above. See the full documentation Here.

To upload the compressed image, execute:

# glance image-create –name “gw-software-deployment-gold” –is-public false  –disk-format qcow2 –file gw-software-deployment-gold.qcow2.tar.gz

As i found, Firefox cannot be used to upload large files like images whereas Chrome can. So you can upload your image from the Openstack control panel (Horizon) using Chrome browser.

You may also need to tune the performance parameters of the operating system (Update system files) and the application itself so the servers created from the image are already tuned. You can do the tuning on the live server before taking a snapshot or you can update the system files of the local image using guestfish before uploading.

Now you can create an instance from the new uploaded image where you can start working with software deployment on different lifecycle in your Heat template.

Using “/tmp” For Caching On Disk

If your RAM is not enough for the image building process, you need to increase the /tmp. For example 15 GB:

# dd if=/dev/zero of=/usr/tmp-dir bs=1024M count=15

# mke2fs -j /usr/tmp-dir; chmod 777 /usr/tmp-dir

# mount /usr/tmp-dir /tmp


  • During image building, if you find some problems related to packages update, you can temporarily disable repositories that cause the problems by using yum-config-manager –disable repositoryName in the live server before creating the image (The image that you will download from the cloud to work on).

  • Commenting the line “yum -y update” in the file “…./heat-templates/hot/software-config/heat-container-agent/scripts/”, could solve the problems that arise on update (conflict between different versions of packages).

  • You could do the process many time with different updates till the image creation process completed successfully. If you are using the default, you are not supposed to have any problems.

  • Package dependencies file in my case Centos7 is “…/diskimage-builder/elements/centos7/element-deps”: If you like you can add more here so it will be installed.

More Information

Permit Root Login Into Instance Launched From Centos Official Image


If you tried to access your instance using SSH as root:

# ssh -i PrivateKey.pem  root@Floating-IP-Address Where Flaoting-IP-Address is the flaoting IP address associated with your instance. You might get this message: “Please login as the user “centos” rather than the user “root”.” Then the connection will close.

Steps To Allow Root Access Through SSH

  • Login as the user “centos”: # ssh -i PrivateKey.pem  centos@Floating-IP-Address
  • Change to root: # sudo -s
  • Edit the file “/root/.ssh/authorized_keys” (# vi /root/.ssh/authorized_keys) and keep it only contains the key  (starts with “sh-rsa”) without the restrictions that exist at the begining of the file.
  • Open the file “/etc/ssh/sshd_config” and uncomment the line “PermitRootLogin yes”
  • Restart SSH daemon: # systemctl restart sshd

Now you can SSH as root to your instance.

Note: You need to automate the above if your servers are created/ deleted on demand and SSH public keys are inserted in servers on boot by Openstack.

Datagram Transport Layer Security (DTLS)



The DTLS protocol ([RFC 6347]) is based on TLS protocol to provide similar security for the network traffic transported on datagram transport protocols (e.g. UDP). Usually the real time applications like media streaming and internet telephony are delay sensitive for the transported data so they use datagram transport to carry their data. DTLS  runs on UDP to secure the data in a transparent way (inserted between the application layer and transport layer). DTLS runs in application space without any kernel modifications. The DTLS preserves the in-order delivery of data which is not provided by the datagram transport. Current version of DTLS is 1.2

Why DTLS and NOT TLS for Datagram Transport

The answer is simply because using datagram transport like UDP means the packets could be lost or reordered and TLS cannot handle this (this is handled by TCP when it is used). So we take the TLS and add minimal changes to fix the unreliability problem and we call the result DTLS.

WhyDTLSMore specifically, the problems that are in TLS if datagram transport are used:

  • In TLS there is what is called integrity check which depends on the sequence number. For example record N is lost –> then the integrity check on record N+1 will fail because the wrong sequence number. The sequence numbers are implicit in the records. The record could also reach but in a wrong order. For example record N+1 reached before the record N.
  • The record could reach many times (replayed).
  • The TLS handshake will break if the handshake messages are lost.
  • Handshake message size is big (many kilobytes): as we know in UDP, datagrams are limited to 1500 bytes.

So the goal is changing TLS to solve the above problems and then we get DTLS. Briefly DTLS solves the problems by:

  • Banning the stream ciphers to make the records independent (don’t have the same cryptographic context – cipher key).
  • Adding explicit sequence numbers in the records.
  • Using retransmission timer for packet loss handling.
  • Handshake message fragmentation –> Each DTLS handshake message must contain fragment offset and fragment length.
  • Maintaining a bitmap window of received records so if a record is previously received it will be discarded.

The client automatically generates self-signed certificates for each peer. This means there is no certificate chain verification. The certificates themselves cannot be used to authenticate the peer because they are self-signed. So the DTLS provides encryption and integrity, but let the authentication to be done by the application.

Library Support For DTLS 1.2

Botan, GnuTLS, MatrixSSL, OpenSSL, wolfSSL