Building Linux Images For Use With Openstack Heat

.

This Article is published on opensource.com with My Permission.

Introduction

548426__fiery-blaze_p

The Openstack image is cloud-init enabled image. Instances created from the image run the cloud-init script on early boot to do some configurations. The directory “/var/lib/cloud” is the directory which contains the cloud-init specific subdirectories.

# ls /var/lib/cloud/
data handlers instance instances scripts seed sem

.

In these subdirectories you can find the current instance id, previous instance id (it could be different) , current and previous datasources, start and end time , errors if exists, custom scripts (scripts/) downloaded/created by the corresponding handlers (handlers/ subdirectory), user data downloaded from metadata service (/instance/scripts/user-data.txt as RAW data and user-data.txt.i as post-processed data),….etc.

But what if we want some configurations/actions on different life cycles and not only on early boot (cloud-init). For example if you are using Openstack autoscaling service, this means your servers will be added and deleted automatically without human intervention. When any of these servers is determined to be killed, it needs some time to clean itself and it needs some configuration to be applied on this stage (e.g drains its active sessions, its active connections to other services if exist,..). Here we come to what is called Heat Software Deployment which is a communication between the Openstack Heat unit and the instance to configure the instance on different life cycles and not only on boot. This feature is available since Openstack Icehouse release. For this we need some agents to be installed in the image and some definitions in the Heat template (mainly based on “OS::Heat::SoftwareDeployment” and “OS::Heat::SoftwareConfig” resource types).

The agents can be installed on boot (script that installs the agents on boot or definitions in Heat template to install the agents on boot). Here I will talk about how to install the agents in the image. The agents are: heat-config, os-collect-config, os-refresh-config, os-apply-config, heat-config-cfn-init, heat-config-puppet, heat-config-salt and heat-config-script.

Preparations For Image Building

Here i am assuming that you have your own Openstack image and you want to add the Heat agents to your image. I will use the DiskImage-Builder to build the image. Diskimage-Builder (DIB) is driven by HP, RedHat, and Cisco and licensed under the Apache License, Version 2.0. My operating system is Fedora 21 and i will build Centos7 custom image with Heat support.

If your image is already on the cloud, you can download it using glance client after you source your credentials (keystonerc).

To install the glance client on Fedora:

# yum install python-devel libffi-devel

# pip install python-glanceclient

To download the image, execute:

# glance image-download –file outputImageFilename imageID

Ex: # glance image-download –file my_image.qcow2 155ae0f9-56ee-4dad-a3a3-cc15a835c035

By doing this, you will get the file my_image.qcow2 on the local disk which is in my case Centos7 image.

The DiskImage-Builder needs enough RAM for caching. If your image is big and your RAM is not enough , you can use /tmp for caching on disk (See how to use /tmp further bellow -Notes Section).

Now we need to clone the repositories: diskimage-builder, image elements, and heat-templates:

# git clone https://git.openstack.org/openstack/diskimage-builder.git

# git clone https://git.openstack.org/openstack/tripleo-image-elements.git

# git clone https://git.openstack.org/openstack/heat-templates.git

Some environment variables need to be exported:

DIB_LOCAL_IMAGE variable: This variable contains the path to the local image (in my case my_image.qcow2). If this variable is not set, the diskimage-builder will go to internet and bring a generic image (e.g. in case centos7, it will go to http://cloud.centos.org/centos/7/images). If you don’t have any image just don’t export this variable to keep the default.

DIB_CLOUD_IMAGES variable (HTTP address): If your image is public on internet, you can export this variable.

ARCH variable: This variable contains the architecture. In my case “amd64”: # export ARCH=”amd64″. This will be converted to x86_64.

My image is Centos7 so the source file which use this variable is “…/diskimage-builder/elements/centos7/root.d/10-centos7-cloud-image”. I recommend you to check the corresponding file for your choice (Go to the root.d directory in your OS choice directory (centos7 in my case) and check what is supported for ARCH). Check this file also for DIB_CLOUD_IMAGES variable and others.

Then we need to install some packages:

# yum install qemu or # yum install qemu-img

# yum install python-pip git

# pip install git+git://git.openstack.org/openstack/dib-utils.git

Image Building

Now i want to add the Heat agents to my image using diskimage-builder. So the instances created from my image can interact with Heat on different life cycles. Write the following in bash script and execute it or take it line by line:

Note: You need to be in the directory where you have cloned the repositories or use the absolute paths.

export ELEMENTS_PATH=tripleo-image-elements/elements:heat-templates/hot/software-config/elements

# customize this line. Possible values: fedora centos7, debian, opensuse, rhel, rhel7, or ubuntu

export BASE_ELEMENTS=”centos7 selinux-permissive”

export AGENT_ELEMENTS=”os-collect-config os-refresh-config os-apply-config”

export DEPLOYMENT_BASE_ELEMENTS=”heat-config heat-config-script”

# For any other chosen configuration tool(s). e.g. heat-config-cfn-init, heat-config-puppet, or heat-config-salt. NOT IN MY CASE.

export DEPLOYMENT_TOOL=””

export IMAGE_NAME=software-deployment-image-gold

diskimage-builder/bin/disk-image-create vm $BASE_ELEMENTS $AGENT_ELEMENTS $DEPLOYMENT_BASE_ELEMENTS $DEPLOYMENT_TOOL -o $IMAGE_NAME.qcow2

Note the variable IMAGE_NAME determines the image name.

Image Uploading

After building the image , you will get the file software-deployment-image-gold.qcow2. If your image is big, you can compress it before uploading it to Openstack (Glance unit):

# tar -zcvf gw-software-deployment-gold.qcow2.tar.gz gw-software-deployment-gold.qcow2

You can also pass the option “-t tar” to the command diskimage-builder/bin/disk-image-create to have the image compressed by the diskimage-builder. Also use this option “-a i386|amd64|armhf” for the ARCH mentioned above. See the full documentation Here.

To upload the compressed image, execute:

# glance image-create –name “gw-software-deployment-gold” –is-public false  –disk-format qcow2 –file gw-software-deployment-gold.qcow2.tar.gz

As i found, Firefox cannot be used to upload large files like images whereas Chrome can. So you can upload your image from the Openstack control panel (Horizon) using Chrome browser.

You may also need to tune the performance parameters of the operating system (Update system files) and the application itself so the servers created from the image are already tuned. You can do the tuning on the live server before taking a snapshot or you can update the system files of the local image using guestfish before uploading.

Now you can create an instance from the new uploaded image where you can start working with software deployment on different lifecycle in your Heat template.


Using “/tmp” For Caching On Disk

If your RAM is not enough for the image building process, you need to increase the /tmp. For example 15 GB:

# dd if=/dev/zero of=/usr/tmp-dir bs=1024M count=15

# mke2fs -j /usr/tmp-dir; chmod 777 /usr/tmp-dir

# mount /usr/tmp-dir /tmp

Notes

  • During image building, if you find some problems related to packages update, you can temporarily disable repositories that cause the problems by using yum-config-manager –disable repositoryName in the live server before creating the image (The image that you will download from the cloud to work on).

  • Commenting the line “yum -y update” in the file “…./heat-templates/hot/software-config/heat-container-agent/scripts/configure_container_agent.sh”, could solve the problems that arise on update (conflict between different versions of packages).

  • You could do the process many time with different updates till the image creation process completed successfully. If you are using the default, you are not supposed to have any problems.

  • Package dependencies file in my case Centos7 is “…/diskimage-builder/elements/centos7/element-deps”: If you like you can add more here so it will be installed.

More Information


Permit Root Login Into Instance Launched From Centos Official Image

Introduction

If you tried to access your instance using SSH as root:

# ssh -i PrivateKey.pem  root@Floating-IP-Address Where Flaoting-IP-Address is the flaoting IP address associated with your instance. You might get this message: “Please login as the user “centos” rather than the user “root”.” Then the connection will close.

Steps To Allow Root Access Through SSH

  • Login as the user “centos”: # ssh -i PrivateKey.pem  centos@Floating-IP-Address
  • Change to root: # sudo -s
  • Edit the file “/root/.ssh/authorized_keys” (# vi /root/.ssh/authorized_keys) and keep it only contains the key  (starts with “sh-rsa”) without the restrictions that exist at the begining of the file.
  • Open the file “/etc/ssh/sshd_config” and uncomment the line “PermitRootLogin yes”
  • Restart SSH daemon: # systemctl restart sshd

Now you can SSH as root to your instance.

Note: You need to automate the above if your servers are created/ deleted on demand and SSH public keys are inserted in servers on boot by Openstack.


Monitoring Agent For Rackspace’s Auto scaled Servers

Introduction

Ceilometer is used to collect measurements of different utilizations (memory, CPU, Hard, ….) from OpenStack components. It is designed originally for billing. It is not a complete monitoring solution for metering because it does not allow service/application level monitoring and it ignores the detailed metrics of the guest system.

Rackspace’ cloud which is Openstack based cloud solution has its own monitoring service which allow the tenants to keep their measured data whether standard (e.g. CPU, Memory, …) or custom (application/service specific metrics) on the cloud and create the notification plans they want.

In this article, i will show you how to automate the setup of Rackspace monitoring agent on the virtual machine. So when your auto scale policy is triggered, you will have a new server with the monitoring agent installed and connected to the cloud. I have Centos-7 for my virtual machine which i will use later to create the image. The image will be used by the auto scaling service to create new servers. You need to have an account with Rackspace cloud provider.

Rackspace Monitoring Agent Installation on Centos 7

Install the package signing key

# curl https://monitoring.api.rackspacecloud.com/pki/agent/centos-7.asc > /tmp/signing-key.asc
# rpm –import /tmp/signing-key.asc

Add the agent repository to yum

  • Create and edit the file  “/etc/yum.repos.d/rackspace-cloud-monitoring.repo”

# vi /etc/yum.repos.d/rackspace-cloud-monitoring.repo

  •  Add the configuration of the repository. In my case i have centos7:

[rackspace]
name=Rackspace Monitoring
baseurl=http://stable.packages.cloudmonitoring.rackspace.com/centos-7-x86_64
enabled=1

Install the agent

# yum install rackspace-monitoring-agent

Now we have the agent installed on the current virtual machine.

Create oneshot systemd or init service for the agent setup

The setup process is needed to configure the monitoring agent for the specific server, verifies the connectivity with the cloud, and association with the monitoring entity of the server. The script that you will write does the setup of the agent as following:

     # rackspace-monitoring-agent –setup -U username -K apikey

Replace the username and apikey with yours. You can take the API key from your account settings when you access the web control panel.

The script needs also to start the agent if it is not started:

      # systemctl start rackspace-monitoring-agent

As this service will be executed on boot, you need to be sure that it is executed only when the server is created (only once). So you need to write a check which examine if rackspace-monitoring-agent service is started or not. If it is started so do NOT set it up again.

Clean after preparation

If you test the setup on the current virtual machine, you need to clean it up so the new servers created from the image will not have the old configuration of the server that is used to create the image. Simply stop the service, uninstall the agent. Then install the agent again without the setup. If you want to have your image independent on the account information you need to make the installation and the setup of the monitoring agent as YAML template executed by the cloud. See the last link in the section “More Information” further down.

Server-Side Agent Configuration YAML File

For example a YAML configuration file that creates a CPU check with alarm. Bind the check with the auto scaling policy notification plans. Create the file “cpu.yaml” in the folder “/etc/rackspace-monitoring-agent.conf.d” with this content:

type : agent.cpu

label: CPU

period: 60

timeout: 10

alarms:

cpu-usage-up:

label: CPU Usage Up

notification_plan_id: scale-up-notification-plan-id-here

criteria: |

     if (metric[‘usage_average’]> 80){

return new AlarmStatus(CRITICAL);

}

cpu-usage-down:

label: CPU Usage Down

notification_plan_id: scale-down-notification-plan-id-here

criteria: |

     if (metric[‘usage_average’]< 50){

return new AlarmStatus(WARNING);

}

To get the ids of your created notifications, execute this:

# curl -s -X GET https://monitoring.api.rackspacecloud.com/v1.0/$tenantID/notifications  -H “X-Auth-Token: $token”   -H “Accept: application/json” | python -m json.tool

Create a new image

Now you can go to the web control panel and create a new image that will be used in auto scaling process.


Next

The next article will be about how to send custom measured data (custom metrics) to your cloud using the monitoring agent. this is called creating custom plugin. I will show you how to create a custom check.

More Information


Root Password’s Hash Injection Into Linux Image File

.

Resetting-a-Forgotten-Root-Password-2Here i will show you how to set the root password permanently in the image. As an example i will use the following image: http://download.fedoraproject.org/pub/fedora/linux/releases/21/Cloud/Images/i386/Fedora-Cloud-Base-20141203-21.i386.qcow2. This image is cloud-aware image and it is in qcow2 format. You need to install the following: “guestfish” and “libguestfs-tools”:

# yum install guestfish libguestfs-tools

To generate an encrypted password:  # openssl passwd -1 Your-Password

I will set the root password as “binan” but you need to choose a strong password:

# openssl passwd -1 binan
$1$PNq4EoLe$EFwgE1BVdVG3uXqv05Pb5/

Now i will set the generated hash value in the file “/etc/shadow” in the image file. This is done by executing (guestfish –rw -a <image-name>):

# guestfish –rw -a /home/binan/Downloads/Fedora-Cloud-Base-20141203-21.i386.qcow2

><fs> run

><fs> list-filesystems

/dev/sda1: ext4

><fs> mount /dev/sda1 /

><fs> vi /etc/shadow

Now i will write the hash value of the password ($1$PNq4EoLe$EFwgE1BVdVG3uXqv05Pb5/) in its place:

root:$1$PNq4EoLe$EFwgE1BVdVG3uXqv05Pb5/::0:99999:7:::

If the root password in the image file is locked, replace the word “locked” with the generated hash. Now each instance created from this image will have “binan” as root password.

Note: After mounting the file system you can do whatever you want. This is not restricted to the “/etc/shadow” file.

To set different root passwords for different instances, use “cloud-init”.

 


Transferring Data Using cURL

.

Introduction

cURL is free and open source tool used to transfer data. I will show you as an example how to use it to transfer data with the cloud.  Here i will send the commands from Linux console to Openstack. The following are the steps:

Taking An Authentication Token

You need to send an authentication request to the cloud identity web service and get a token in the response upon successful authentication:

# curl -s https://YOUR-Cloud-Identity-Service-URL -X ‘POST’ -d ‘{“auth”:{“passwordCredentials”:{“username”:”YOUR-UserName”, “password”:”YOUR-Password”}}}’  -H “Content-Type: application/json” | python -m json.tool

Upon successful authentication, you will get the response in JSON format. To make the output easy to read, you can direct the output to the “json.tool” python tool as above.

One of the important information in the response is the authentication token which will be passed in the header “X-Auth-Token” in each request you send. You need to extract the token from the JSON respone. You can use the “jsawk” tool.

“jsawk” Installation

Follow these steps to install “jsawk”:

# yum install js, js-devel

# curl -L http://github.com/micha/jsawk/raw/master/jsawk > jsawk

# chmod 755 jsawk && mv jsawk /usr/bin/

Note curl is used here also.

 How To Use “jsawk”

You direct the response to “jsawk” and ask for a specific property in the JSON response. You can also do some modifications. See the examples in “jsawk” github page.


 More Information


Root Password Injection Into Openstack Instance

.

To inject a root password into Openstack instance on Nova boot, you need to enable it. Follow these simple steps:

  • Install “libguestfs” on Nova compute node:

# yum install libguestfs python-libguestfs libguestfs-tools-c

  • Open the file “/etc/nova/nova.conf ” and update these lines:

inject_password=true
inject_key=true
inject_partition=-1

  • Restart nove-compute: # service openstack-nova-compute restart
  • Open the file “/usr/share/openstack-dashboard/openstack_dashboard/loal/local_settings.py” and enable setting root password:

OPENSTACK_HYPERVISOR_FEATURES = {
…..
‘can_set_password’: True,
}

Now you can set root password on dashboard when you launch your instance under “Access & Security” tab.

Screenshot from 2015-03-16 16:20:45


How To Change OpenStack’s Dashboard Root URL

.

Horizon is the canonical implementation of OpenStack’s Dashboard which provides web interface to OpenStack services like Nova, Swift, …. To access Horizon, you browse to “http://Your-IP-Address/dashboard/” and you get this page:

But what if you have multiple applications and you want to have the URL like this “http://Your-IP-Address/Something/dashboard“.  Follow these steps:

Usually the Apache main configuration file is “/etc/httpd/conf/httpd.conf“. In this file you have this statement: IncludeOptional “/etc/httpd/conf.d/*.conf”. This means the configuration files in this folder “/etc/httpd/conf.d/” are included. Open the file “/etc/httpd/conf.d/15-horizon_vhost.conf” :

# vi /etc/httpd/conf.d/15-horizon_vhost.conf

Lets say you want to add /openstack to the original URL. So you want to browse the dashboard as following “http://Your-IP-Address/openstack/dashboard“. Change this line:

WSGIScriptAlias /dashboard “/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi”

To

WSGIScriptAlias /openstack/dashboard “/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi”

Now open the file “/usr/share/openstack-dashboard/openstack_dashboard/local/local_settings.py” and customize the following lines to use “/openstack/dashboard” instead of just “/dashboard”:

LOGIN_REDIRECT_URL=”Your-IP-Address/dashboard”

LOGIN_URL = ‘/dashboard/auth/login/’
LOGOUT_URL = ‘/dashboard/auth/logout/’
LOGIN_REDIRECT_URL = ‘/dashboard’

Now you need to restart Apache:

# systemctl restart httpd.service

Now browse to “http://Your-IP-Address/openstack/dashboard” and you will get this page again. Congratulation !

Click on the picture below and look at the URL. It is changed.

And this is the page you get after successful login. Click on the picture and look at the URL:

Screenshot from 2015-03-10 21:29:51You can also make it like this “http://Your-IP-Address/openstack/horizon-dashboard“. In this case the WSGIScriptAlias will be /openstack/horizon-dashboard. The same for other parameters customization.