Building a Custom Linux kernel for your VOIP System – Part1



Is your VOIP system based on userspace applications ? What about building kernel based VOIP system ?. This is not new but we are going to go through it here. I am assuming Linux as an operating system. The Linux kernel is about core and modules (micro kernel + loadable modules). You might not touch the core but you might need to build a module that support your VOIP system in the kernel for performance purposes. The new module supposed to do VOIP jobs (filtering, QoS, protection, media relay). There are already some useful kernel modules that can support your system like NATing, IPv6ing, and so on . Use them!

In this article i will just introduce what you need to know to follow this series of articles: how to compile and install a new, fresh and clean Linux kernel and what you need to know to compile and install a new kernel module.

We will not touch the current running kernel but building a new one, install it, boot from it, write a new module, add it to the kernel and test it. You need to be very careful when you work in the kernel.

Compilation and Installation Steps

This is a traditional common way to all Linux distributions to compile and install a Linux kernel:

Download the source:

I will work on the kernel 3.19.5 as an example. For other versions, go to and see the list there.

# wget -c

  Decompress the gz file and go to the directory “linux-3.19.5”:

# tar -xzvf linux-3.19.5.tar.gz

# cd linux-3.19.5

Configure the kernel:

Fresh configuration: traditional way: # make menuconfig or the modern way: # make nconfig

Note: To detect memory leaks in the kernel modules, you can use the tool ‘kmemleak’. To use it, you need to compile the kernel with ‘CONFIG_DEBUG_KMEMLEAK’ option enabled. Enable it as following: make menuconfig –> ‘kernel Hacking’ –> ‘Memory Debugging’ –> ‘Kernel Memory Leak Detector’. Enable this option and do save. You can check this file “/sys/kernel/debug/kmemleak” while testing.

You can use the config file of the current running kernel. In Fedora this file exists as /boot/config-$(KernelName). For example: /boot/config-3.19.5-100.fc20.x86_64. Then update it with ‘make menuconfig’ and save.


Prepare for the compilation: # make mrproper

Compile the kernel: # make

Install the compiled modules:

# make modules_install

The modules will be installed in /lib/modules/3.19.5/

Install the kernel: Now we want to copy and rename the image bzImage (arch/x86_64/boot/bzImage) to /boot + creating the initial RAM disk + creating the map/kernel symbols , and update the grub config file). This done by: # make install

After compiling and installing the kernel, we can boot from it (appear in the GRUB startup List).

Header Files needed to compile a new module:

If you want to build your own kernel module, you need to have the header files.  Check the folder /lib/modules/3.19.5/build/ where 3.19.5 is the new kernel version.

The module is header and source files with Makefile file. When it is built we got .ko file. In the Makefile you need to specify the kernel source and the obj-m.

After building the module, it will be ready to be installed. You need to boot from the new kernel and do ‘modprob $ModuleName’

Again this article is just an introduction to this series. More information is coming next.

Installation of MySQL Database Server


These simple steps install and start MySQL database server on Fedora 20:

  • Install the package “community-mysql-server”: # yum -y install community-mysql-server
  • Start the daemon service: # systemctl start mysqld.service
  • Enable the daemon service: # systemctl enable mysqld.service
  • Connect to MySQL: # mysql -u root. Type Enter
  • Set the root password: mysql> set password for root@localhost=password(‘password’);

You can replace the localhost with your domain or IP address.

  • Delete anonymous users: mysql> delete from mysql.user where user=”;
  • mysql> exit

If you forgot the root password, do these:

  • # mysqld_safe –skip-grant-tables &
  • mysql> use mysql;
    mysql> update user set password=PASSWORD(“New-Password”) where User=’root’;
    mysql> flush privileges;
    mysql> quit
  • # systemctl stop mysqld
  • # systemctl start mysqld
  • # mysql -u root -p

SIP Stress Testing -Part 3 (SIPp)



This is Part 3 of “SIP Stress Testing” topic. In Part 1, I have talked about the definition of the stress, opensipsctl (command line tool), and OpenSIPS-CP (web tool) and how they are used in testing. In Part 2 i have talked about SIPSAK (command line tool) which is used as SIP traffic generator. In this part i will talk about SIPp which is an emulation tool which generates SIP traffic and gives some statistics. The current version (3.4) have these features:

  • Establishes and releases multiple calls with the INVITE and BYE methods.
  • Read custom XML scenarios files which described simple to complex call flows.
  • Dynamic display of statistics.
  • Support of IPv6, TLS, SCTP.
  • Support of SIPauthentication, conditional scenarios, UDP retransmissions, error robustness.
  • Call specific variable.
  • Posix regular expression to extract and re-inject any protocol fields.
  • Custom actions (log, system command exec, call stop).
  • Emulate live users using external CV file .
  • Send RTP traffic (audio/video).

SIPp Installation

Pre-requisites to compile SIPp on Fedora Red Hat:

  • Install C++ compiler: # yum install gcc-c++
  • Install ncurses-devel: # yum install  ncurses-devel
  • For TLS support,  install OpenSSL (version > 0.9.8): #yum install openssl-devel
  • For pcap play support, install libpcap and libnet : #yum install  libnet libnet-devel  libpcap libpcap-devel
  • For SCTP support, install lksctp: #yum install lksctp-tools lksctp-tools-devel
  • For distributed pauses, Install GNU Scientific Libraries (gsl): #yum install gsl gsl-devel

Please check your system for the corresponding libraries names.

Download the source code of SIPp:

To compile SIPp, we have four options:

  • Without TLS, SCTP, or PCAP support.
  • With TLS support.
  • With PCAP support
  • With SCTP support

I will combine these options to compile SIPp which supports TLS, SCTP, and PCAP:

  • # tar -xvzf sipp-3.3.990.tar.gz
  • # cd sipp-3.3.990/
  • # ./configure –with-sctp –with-pcap –with-openssl
  • # make
  • # make install

Controlling SIPp Interactively

You can control SIPp through the keyboard. For example pressing on ‘1’, will display the scenario screen, and pressing ‘2’ will display the statistics screen and so on.

SIPp can alse be controlled via a UDP socket. The default base port is 8888 for the first instance and it will increase by 1 for each following SIPp instance (8889, 8890,…). Use the option -cp to change the base port. For example to put SIPp in pause state, we execute this shell command: # echo p > /dev/udp/x.y.z.t/8888 where x.y.z.t is the IP address of the SIPp instance.

Check SIPs Network State

If we have SIPp instance running on the machine (UAS instance :# sipp -sn uas -i and we want to check the network state of that instance, we execute “netstat” command as following:

# netstat -lpn  |grep sipp

The output will be like this

udp        0      0  *                           14144/sipp
udp        0      0 *                           14144/sipp
udp        0      0 *                           14144/sipp
udp        0      0 *                           14144/sipp


  • 8888 is the port for SIPp control. Use the option -cp in the command line to change the base number.
  • 5067 is the port for SIP signaling. Use the option -p in the command line to change it.
  • 6000 is  the default local RTP echo port.Use the option -mp in the command line to change it. RTP/UDP packets received on this port are echoed to their sender
  • 6002 is the RTP echo port +2. RTP/UDP packets coming on this port are also echoed to their sender.

SIP User Agent Definition

  • User Agent Client (UAC): It is a logical entity that creates a new SIP request. This role (the role of being UAC) lasts only for the duration of the transaction (SIP transaction).
  • User Agent Server (UAS): It is a logical entity that generates a response to a SIP request. This role (the role of being UAS) lasts  only for the duration of the transaction (SIP transaction).

UACUAS Example

SIP traffic between two SIP user agents on the same host (IP: The traffic is passed directly from the user agent client (UAC) to the user agent server (UAS) without passing a SIP proxy.

Scenario Definition

We want to generate SIP traffic with this specification:

  • Number of calls per second (call rate) = 100 cps. This can be specified as a parameter in the “sipp” command line after the option -r. This rate can be changed interactively (i.e. during the test) by pressing keys on the keyboard. For example pressing ‘+’ will increase the rate by 1 (‘-‘ decrease by 1, ‘*’ increase by 10, ‘/’ decrease by 10).
  • Rate Period in milliseconds for the call rate=10000 ms. This can be specified in the “sipp” command line after the option -rp.

To have n calls every m milliseconds, we mix the previous command line parameters together. So it will be like this –r n -rp m. Here in this scenario, i want to generate 100 calls per 10 seconds so the parameters will be -r 100 -rp 10000.

  • The UAC and UAS are on the same machine. For UAC, i will choose the port 5061 for SIP signaling. For UAS, i will choose the port 5067 for SIp signaling.
  • I will use IPv4 addresses.
  • Call Flow between UAC and UAS: The default behaviour of the UAC (what UAC sends and what it expects as response for each request it sends) is defined in the scenario file (uac.xml). As it is the default scenario so i don’t need to pass the name of the xml file as a parameter in the command line “sipp”. This is the default call flow of the UAC and UAS :


In the default scenario, authentication is not used and there is no record routing.

Here is the XML scenario file of the UAS  uas.xml.You can create your own custom XML scenarios as i will do in the next article. In this case you have to pass the names of the XML files which describes the call flows as parameters to “sipp” command (after the option -sf).

Run the UAC and UAS

SIPp will generate SIP traffic according to scenario defined above. SIPp has an embedded SIP user agent client (UAC) which will generate this traffic. Open one terminal and execute this command “# sipp -sn uac  -r 100 -rp 10000  -i -p 5061” to run the SIP client (uac). According to this, the UAC ( will send 100 calls per 10 seconds to the UAS (

SIPp has an embedded user agent server (UAS). Open another terminal on the same host and execute this command “# sipp -sn uas -i -p 5067“. This will run the UAS on

Be sure that the ports you want to use are free (not used by another application). This article is like a tutorial so run your UAC and UAS on different machine. Note that we are using the same command “sipp” to run the UAC and UAS. Remember SIPp is an emulation tool for testing the SIP protocol.

Scenario Screenshots

The scenario screen display the call flow as well as some information. This is the scenario screen of the UAC:

Here is the scenario screen of the UAS:

Several screens are available to monitor SIP traffic. You can change the screen by pressing 1 to 9 keys on the keyboard. For example click key ‘2’ to display the statistics screen. Here is the statistics screen of the UAC:

Here is the statistics screen of the UAS:

As you can see the number of successful calls is 150 and the number of failed calls is 0.

UACSIP ProxyUAS Example

SIP traffic between two SIP user agents. The traffic is passing the proxy from the user agent client (UAC) to the user agent server (UAS).  We will take the same previous scenario but with proxy. The UAC, UAS, and the proxy are all on the same machine. The proxy will be OpenSIPS. The routing script of the proxy must work properly with UAC and UAS.


Register and Run the UAS

Assume OpenSIPS is running on We will send packets continuously to the UAS through the proxy. So we need UAS to stay registered during the test (OpenSIPs always knows UAS IP address). So we will register UAS manually by adding a static mapping in the OpenSIPS ‘s “Location” table and this entry will be permanent (available at any time).  

Do as following:

  • Go to OpenSIPS’s sbin directory: # cd /usr/local/opensips_1_11/sbin/
  • Add UAS with the user name = 1234Test: # opensipsctl ul add 1234Test@ sip:1234Test@ This will add an entry for this user in the location table.


We have added in OpenSIPS “Domain table” (# opensipsctl domain add

  • Run the UAS:  sippsn uas -rsa  -i -p 5067. The address which comes after the option -rsa  is the remote sending address (Here it is the OpenSIPS proxy).

Now the UAS is ready. You can check its status by executing “# ./opensipsctl ul show”. The output will be as this:

Domain:: location table=512 records=1
AOR:: 1234Test
Contact:: sip:1234Test@ Q=1
Expires:: permanent
Callid:: dfjrewr12386fd6-343@opensips.mi
Cseq:: 1
User-agent:: OpenSIPS MI Server
State:: CS_SYNC
Flags:: 0
Methods:: 4294967295

As you can see this is a static entry and it is permanent (No need for refreshment). The OpenSIPS now always can route all requests destined to UAS (Request-URI=1234Test@ (OpenSIPS URI)).

Avoid Authentication of SIP Requests and Loose Routing for Sequential Requests:

Here i am taking the default routing script of OpenSIPS which is generated for residential service. Do the following changes:

  • Avoid loose routing of SIPp packets by skipping the loose routing section where the client is sipp ($fU is “sipp”):

 if (has_totag()  && ($fU!=“sipp”)) {

// Do the loose routing


  • Avoid Authentication for SIPp packets: Comment these lines in your routing script:

  # if (!proxy_authorize(“”, “subscriber”)) {
#                   proxy_challenge(“”, “0”);
#                   exit;
# }
# if (!db_check_from()) {
#                sl_send_reply(“403″,”Forbidden auth ID”);
#               exit;
# }

# consume_credentials();
# caller authenticated

  • Restart OpenSIPS (# systemctl restart opensips.service)

Run The UAC

Run this command : #sipp  -sn uac -s 1234Test -i -p 5061 -d 20000 -r 100 -rp 10000

The option ‘-s’ sets the username part of the request URI. The option -d controls the length of calls (duration of pause).


The following is the scenario screens of UAS:

Server-Scenario-Ex2and the statistics screen of the UAS:



If you got some errors, enable logging by setting the option -trace_err in command line. For example enable logging at UAC side:

#sipp -trace_err -sn uac -s 1234Test -i -p 5061

Press the key ‘q’ to quit and check the log file. The scenarios of the UAC and UAS and the routing script of the proxy, should be compatible with each other regarding what have been received and what should be sent otherwise you will get unexpected messages.

What does play a role in the results of this test ?

  • The operating system (CPU, Memory, Scheduler,….etc): Fedora Red Hat
  • The Proxy (internal design, routing script, Private and shared memory) : OpenSIPS
  • The traffic generator Performance (internal design): SIPp
  • The network design: UAS, Proxy, and UAS are on the same machine (This article is like a tutorial).

Now you can do what is called tuning (change the parameters that affect the performance). Best is using tools that help you to do the tuning and give you some statistics.

The Part 4 of “SIP Stress Testing” topic will also be about SIPp.  It will be writing custom scenario for the example UAC-SIP Proxy-UAS above.

More Information

OpenSIPS Compilation and Installation Troubleshooting On Fedora


OpenSIPS Installation

Here i will install OpenSIPS 1.11. To get more control over the installation process, I will compile the OpenSIPS from the source.

# cd /usr/local/src/

Get the latest release of OpenSIPS source from OpenSIPS’s GitHub repository:

Go to the place where “” is remotely storing OpenSIPS and get the OpenSIPS ‘s repo HTTP-address“. Then clone the repo.

# git clone -b 1.11 opensips_1_11

# cd opensips_1_11

Note: The option -b is used to choose the branch “-b 1.11”

Install the prerequisites needed to configure the compilation:

  • Flex and Bison needed for parsing of configuration file.
  • ncurses-libs, ncurses-devel, and ncurses-static for grafical display (text-based user interfaces).

Some prerequisites must be installed for GCC like Flex, Bison, TCl and others. If you could not compile, check what is missed in your system. Have a look here.

Open the configuration interface:

# make menuconfig

If you couldn’t open the interface and have an error like “Make error *255*“, this means  needed prerequisites are not all installed. You could also need  to try these:

  • Remove GCC and install it again:

# yum remove gcc

# yum install gcc

If you are patient, try to do the same thing with some broken installed prerequisites.

  • You could need to change the Makefile (In my case: “/usr/local/src/opensips_1_11/menuconfig/Makefile“). For example adding include (-I) and library (-L) paths.

$(CC) -o $(EXEC_NAME) $(CFLAGS) $^ -I/usr/include -L/usr/lib -lncurses

  • Still have a problem, do system restart.

The text-based user interface to configure and compile will look like this:

Here we can do the following

  • Configure some compile options used at OpenSIPS compilation time.
  • Compile and install OpenSIPS.
  • Cleanup OpenSIPS sources.
  • Generate OpenSIPS script
  • Exit and save all changes.

To get the features you want, go to the first menu and configure the compilation. You can select the modules you want to compile (“Configure Compile Options” –> “Configure Excluded Modules”). The selected modules here (marked with star *) will be included in the compilation of the OpenSIPS. Also you can configure the installation directories (“Configure Compile Options” –> “Configure Install Prefix”). It can be for example “/usr/local/opensips_1_11/”. Note: press q letter or left-arrow to go back to the previous screen.

After the configuration press on “Save Changes” to save the changes you did. After this you will get a message of the required libraries which must be installed according to the modules included in the configuration of the compilation (module dependencies). Go ahead and download them before compilation. Put the graphical interface in the background (press control-Z) or open a new terminal so you can see the names of the required libraries from the graphical interface and install them in the new terminal.

For example the module “db_mysql” needs mysql-devel (mysql client development library) to be installed.

Back to the interface and select “Compile and Install OpenSIPS” , then the compilation will start. It takes few minutes.

After the installation is completed, we get OpenSIPS installed as it is configured. In my case, it will be installed in “/usr/local/opensips_1_11/”. This directory contains these:

In “etc” we have the OpenSIPS configuration files.

In “lib” we have OpenSIPS modules

In “sbin” we have the executable files.

In “share” we have the documentation.

Create OpenSIPS Mysql Database

Install your database server. As an example install MySQL database server.

Now we are in “/usr/local/opensips_1_11/”

  • Configure the database parameters: Edit the configuration file “etc/opensips/opensipsctlrc”. This file contains the configuration of the database connection. # vim opensipsctlrc. Adjust the parameters like the name of the database, the read/write user that the opensips will be using, password, the database root user, and others.
  • Create the OpenSIPS database.Now back to “/usr/local/opensips_1_11/”. In the sub-directory /sbin we have opensipsdbctl tool. Use it to create the database. Here is the command and its output:

# ./opensipsdbctl create  

        MySQL password for root:

INFO: test server charset

INFO: creating database opensips …

INFO: Core OpenSIPS tables succesfully created.

Install presence related tables? (y/n): y

INFO: creating presence tables into opensips …

INFO: Presence tables succesfully created.

Install tables for imc cpl siptrace domainpolicy carrierroute userblacklist b2b registrant call_center? (y/n): y

INFO: creating extra tables into opensips …

INFO: Extra tables succesfully created.

opensipsdbctl tool takes its configuration from the file “etc/opensipsctlrc”


  • Install and configure mysqld as a system service:

# yum -y install community-mysql-server

       # systemctl start mysqld.service

       # systemctl enable mysqld.service

  • Check if its running: # ps aux | grep mysql
  • Default password is empty so only press “Enter” when asking for password. For security, set root password as following:

  mysql> set password for root@localhost=password(‘fedoracore2’);

Generate OpenSIPS Configuration File

The prerequisite is to install m4 package which comes by default with most Linux distribution. M4 is GNU macro processor and it is useful for writing text files which can be logically parsed.

Now we are in “/usr/local/opensips_1_11/”

Go to sbin and execute ./osipsconfig. You will get the text interface to generate the configuration file.


Three kinds of configuration files/scripts can be generated:

  • Residential scripts: If we have customers and we want to offer them certain services.
  • Trunking scripts: where we have PBXs and SIP trunks and we want to route the traffic.
  • Load-balancing scripts: Simple load balancer that can load the traffic between chosen destinations.

Example: Generating residential script (“Generate OpenSIPS Script” –> “Residential Script” –> “Configure Residential Script”) so we come to this page:


Here we can configure how OpenSIPs will work. After this configuration press “q” to go to the previous page then click on “Generate Residential Script”.


The generated script will be placed in the OpenSIPs etc folder (/usr/local/opensips_1_11/etc/opensips) marked with the date of its generation. Example: “opensips_residential_2014-9-25_12:30:52.cfg”. After the generation of the file we need to customize it little bit. For example fix the IP addresses and ports that the OpenSIPs will be working on. For example:

listen=udp: # CUSTOMIZE ME
Also customization of the places where DB connection is required. Example:

modparam(“siptrace”, “db_url”, “mysql://user:passwd@host/dbname”)
modparam(“usrloc”, “db_url”,”mysql://opensips:fedoracore2@localhost/opensips”) # CUSTOMIZE ME

Don’t forget to check the path of the installed modules: mpath=”/usr/local/opensips_proxy/lib/opensips/modules”

OpenSIPS Init Script

To make OpenSIPS works as system service, we need Init script. Follow these steps:

  • Go to the src folder : “/usr/local/src/opensips_1_11/packaging”
  • Here you have to select your system. For example if you have Fedora go to fedora sub-directory.
  • In that folder you will see the script opensips.init which is the actual init script. Copy this script to /etc/init.d/opensips and give it execution right.

# cp opensips.init /etc/init.d/opensips
# chmod +x /etc/init.d/opensips

  • Edit the script
    # vim  /etc/init.d/opensips

Here is a part of the script where there will be changes. The red line replace the line above it. Please see the corresponding to this in your system.









Include New Syslog Entry for OpenSIPS

  • Open the configuration file. In the Global Configuration section you will see the following :




          Change 0 to 1


  • Edit rsyslog.conf: #vim  /etc/rsyslog.conf
    Add this entry:   local1.*                        /var/log/opensips.log
  • Restart rsyslog daemon : #systemctl restart rsyslog OR  #service rsyslog restart
    All OpenSIPs log messages will be stored in this file /var/log/opensips.log

   By adding ““, the logging will be in Asynchronous mode so opensips will not wait until the messages are propagated.

Add Linux User and Group For OpenSIPS

Execute “# system-config-users”. Add user=opensips and create group=opensips for that user

Run OpenSIPS

The database server must be running before starting OpenSIPS. For example if the database server is mysql, check its status (# systemctl status mysqld.service) and run it if it is not running (# systemctl start mysqld.service).

To enable and start opensips as a system service.

# systemctl start opensips.service

# systemctl enable opensips.service

Check the file “/var/log/opensips.log” to see if everything is ok and no error messages (# tail -f  /var/log/opensips.log).

Example of an error message in the opensips.log:

ERROR:uri:db_checks_fixup1: configuration error – no database URL is configured!

uri” is the name of the module which has a problem.

Note: Problems which arise before the loading of the configuration file are still going to “/var/log/messages”.

Now the opensips should be running and we can check this by executing some commands:

♠ Check the status of the OpenSIPS’s service (i.e. Loaded and Running): #systemctl status opensips.service

♠ Check network’s state of the OpenSIPS server: # netstat -lpn |grep opensips

tcp 0 0* LISTEN 32634/opensips

udp 0 0* 32634/opensips

Note: OpenSIPS will not start on database init failure and it will exit immediately (return -1) on module initialization failure (calling the function “mod_init”).

Start/Restart OpenSIPS

To restart OpenSIPS, do the following:

# systemctl restart opensips.service


#  cd /etc/init.d/

# ./opensips restart

Start Working

Now after OpenSIPS has been installed successfully, you can start using its modules in the routing script. What i recommend when using any module:

  • Reading in the module documentation (Last Version). Note the section “Exported MI Functions” in the documentation of each module lists the module’s exported MI functions that can be called from outside of OpenSIPS (External applications). Example:

# opensipsctl fifo debug 1 

OpenSIPS project is growing quickly so please always read in the latest documentation.

  • Sometimes you need to look at the Database Schema. Some fields are pushed to database only when a specific module parameter is set. Example: In multi-domain scenarios, you might need to save the domain part of the user so the domain part and the username part are used to identify the user. So in this case you need to set the “use_domain” parameter of “usrloc” module to non zero value which means true.

modparam(“usrloc”, “use_domain”, 1)

More Information