Linux Tuning For SIP Routers – Part 1 (Interrupts and IRQ Tuning)

.

Introduction

Caring about the performance of any SIP router (real-time application) involves caring about the performance of the operating system and the routing application which runs on the operating system. The hardware performance is also important. I will mention some tips on how you can tune Linux to serve real time application like SIP routing. This article will be in many parts. In this part i will talk about interrupts and IRQ tuning for real time processing.

.

Interrupts and IRQ Tuning

Usually the network devices has 1 to 5 interrupts lines to communicate with the CPU. In multiple CPU system, round robin scheduling algorithm (Fair interrupts distribution) is used to choose the CPU core that will handle the interrupt. The file “/proc/interrupts” records the number of interrupts per CPU core per IO device. Execute this “cat /proc/interrupts” to get the list of interrupts in your system. To get the line associated with Ethernet driver, execute “grep p4p1 /proc/interrupts” where p4p1 is the name of my Ethernet driver. The output will look like this:   

30:      36650      16711      62175      18490   PCI-MSI-edge      p4p1

Where: 30 is IRQ number. 36650 of that interrupt handled by CPU-Core-0. 16711 of that interrupt handled by CPU-Core-1, 62175 of that interrupt handled by CPU-Core-2, 18490 of that interrupt handled by CPU-Core-3. PCI-MSI-edge is the interrupt type. p4p1 is the driver which receives the interrupts (This could be a comma-delimited list of drivers).

It is recommended that all interrupts generated by a specific device to be handled by the same CPU cores. IRQ fair distribution between all CPU cores is not recommended because when the interrupt goes to another fresh CPU core, the new CPU core will load the interrupt handler function from the main memory to the cache (time overhead).

IRQs have a property called interrupt affinity or smp_affinity. This property defines the CPU cores (CPU Set) that will handle the interrupts of a specific device. This property can be used to improve the performance of the SIP router (as any real-time application) by assigning same CPU set to the process/thread and to the interrupt. This minimizes the delays by allowing cache sharing between the process/thread and the interrupt.

Affinity Value for a Specefic IRQ

The affinity value (CPU cores) for a specific IRQ is stored in the file “/proc/irq/IRQ_NUMBER/smp_affinity”. The value in this file is a hexadecimal bit-mask (The lowest order bit corresponding to the first logical CPU). For example to set the affinity value to the IRQ-30 (the example above), do this as root:

  • Display the current affinity value: “cat /proc/irq/30/smp_affinity”.
  • Change the affinity of the IRQ-30 to the first 2 cores (0011). The corresponding hex value is 3. We write this number 3 in the file “/proc/irq/30/smp_affinity” : “echo 3 > /proc/irq/30/smp_affinity”

Setting the smp_affinity can be done at the hardware level (no intervention from the kernel) on systems which support interrupt steering.

“irqbalance” Daemon Configuration

Most of the current distributions comes with “irqbalance” daemon which distributes the interrupts fairly between CPU cores. So disable the irqbalance for those CPU cores that you want to bind them with specific IRQs. This can be done by editing the file “/etc/sysconfi/irqbalance” and changing the parameter IRQBALANCE_BANNED_CPUS which is 64 bit mask which allows you to indicate which CPUs should be skipped when balancing IRQs.

Affinity Value for a Specific Process/Thread

You can use the command “taskset” to set/get the affinity value of a running process or to run a new command with a given affinity. This will be done as following for non-running process (command): “taskset 3 /…./my_process“. This will bind my_process to the CPU-Core-0 and CPU-Core-1. Scheduling will happen between these two cores.

To bind running process to specific cores, do this : “taskset -p 3 PID” where 3 is the CPU-Set hex mask and PID is the process ID.

To check the CPU cores, execute “cat /proc/PID/status“.

I will apply this to OpenSIPS (SIP router):

OpenSIPS 1.X is multi-processes SIP router so when you execute “ps aux |grep opensips“, you will get many lines. You can change the affinity value to all or some of these processes as following:

  • # taskset -p 3 19904 where 19904 is one of these processes
  • Check the status of the process 19904: # cat /proc/19904/status

Cpus_allowed:   3

OpenSIPS 2.X is multi-threads SIP router (under development).

The affinity can also set by the process in its code. For example:

      #include <sched.h>

cpu_set_t mask;

CPU_ZERO(&mask);

unsigned int len = sizeof(mask);

CPU_SET(cpu_nr, &mask);

sched_setaffinity(0, len, &mask);

 Description

  • cpu_set_t is a data structure (set of bits) which represents the set of CPU cores.
  • CPU_ZERO(…) clears the set (no CPUs).
  • CPU_SET(…) adds CPU core number cpu_nr to the CPU set.
  • sched_setaffinity(…) Set the affinity mask. First parameter is PID (here it is zero so it means the calling thread/process).

Note: To scale the performance more, choose the best CPU cores for interrupts affinity and process affinity.


Last Word

If there is a lot of reading/writing from/to the hard disk (e.g. Stateful SIP Router),  Attaching the same CPU set to the processes/threads and the interrupts will be useful.

  • Get the driver name of your hard disk:
    • Execute “udevadm info -a -n /dev/sda” and parse the output. Search for something like DRIVERS==”Something”.  I have got this DRIVERS==”ahci”.
  • Execute “# grep ahci /proc/interrupts” and get the IRQ number.
  • Set the affinity value for the processes/threads and to the interrupts as discussed above.

Next

The Next Part will be about tuning the access to the disk to optimize the performance of the SIP routers.


 

Advertisements

Compilation, Installation, and Configuration of OpenSIPS’s Perl Module

.

If you want to implement a new feature (i.e. not exist in OpenSIPS), you can extend OpenSIPS by either writing new module (in C) or writing Perl script and call it within the routing script using OpenSIPS’s Perl module. In this article i will explain how to compile, install and configure OpenSIP’s Perl module. Then you will be ready to write your own extension.  To do that, follow these steps.

Installation of “ExtUtils::Embed” Perl Module

It is Perl module used for embedding Perl interpreter and extensions in C/C++ (i.e. Cross Compiling). To install this module on Fedora: #yum install perl-ExtUtils-Embed. The Makefile of the application can call functions from this module while building the application. In OpenSIPS, we will call the functions manually and set the results in environment variables accessed by the Makefile. One of these functions is “ldopts” which gives the GCC ‘s arguments for linking the perl library to  C/C++ application (Perl is dynamically linked to OpenSIPS). You can call this function as following:

# perl –MExtUtils::Embed –e ldopts.

Set the option -M to specify the module name which is “ExtUtils::Embed” and the option -e to call the function “ldopts”.

Pre Compilation Procedures

OpenSIPS is C application. C is a compiling language whereas Perl is a scripting language with C++ interpreter. OpenSIPS uses Perl interpreter and extensions in its Perl module to Parse/Load/Execute Perl scripts. So we need  to prepare  GCC (C/C++ compiler) to be able to compile OpenSIPS’ s Perl module. In this tutorial i am using perl (v5.18.4) and GCC (v4.8.3) on Fedora 20.

  • Execute the command #perl -MExtUtils::Embed -e ldopts.

In my case the output is:

-Wl,–enable-new-dtags  -fstack-protector  -L/usr/lib64/perl5/CORE -lperl -lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc

Assign the output to the environment variable PERLLDOPTS:

#export PERLLDOPTS=-Wl,–enable-new-dtags  -fstack-protector  -L/usr/lib64/perl5/CORE -lperl -lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc.

To make this permanently, edit the file “.bash_profile” and write the previous export statement there. Check its value: # echo $PERLLDOPTS

  • Execute the command #perl -MExtUtils::Embed -e ccopts.

In my case the output is:

-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64  -I/usr/lib64/perl5/CORE

Assign the output to the environment variable PERLCCOPTS. Check its value : # echo $PERLCCOPTS

  • Execute the command : #echo “`perl -MConfig -e ‘print $Config{installprivlib}’`/ExtUtils/typemap” and assign the output to environment variable TYPEMAP.
  • OpenSIPS’s Makefile is already prepared for cross compilation. It will recognize these variables. For example “Makefile.defs-> CROSS_COMPILE ?=” will include PERLCCOPTS‘s value (DON’T DO ANYTHING).

Notes:

  • If you write these variables in bash profile (.bash_profile), you must reload the file: # source .bash_profile
  • See the corresponding procedures in your system.

Compilation & Installation of OpenSIPS’s Perl Module

  • If you don’t have OpenSIPS yet, have a look  here. Include Perl in the compilation (i.e. Click on “Configure Compile Options” –> “Configure Excluded Modules”. Then compile the OpenSIPS by selecting “Compile and Install OpenSIPS”. This will not rebuild everything only the new selected modules.

OR

From the OpenSIPS ‘s src  folder (“/usr/local/src/opensips_1_11”), execute:

#make include_modules=”perl”modules

# make install

  • To see if the module is compiled and installed , check the Perl module path “/usr/local/opensips_1_11/lib64/opensips/perl/”. Check also Perl.so in the OpenSIPS’s C modules path: In my case  “ /usr/local/opensips_1_11/lib64/opensips/modules/“.

Configuration of OpenSIPS’s Perl Module

♣ The module “Perl” must be loaded in OpenSIPS’s routing script (i.e. the configuration file) after loading the module “sl”.  To load the module, write this statement:

loadmodule “perl.so.

♣ Set the name of the new extension (i.e. the name of your Perl script):

modparam(“perl”, “filename”, “/usr/local/opensips_1_11/Your-Extensions-Folder/Name-Of-Your-Script.pl”)

♣ Set the module path

modparam(“perl”, “modpath”, “/usr/local/opensips_1_11/lib64/opensips/perl/”)

♣ Restart OpenSIPS: #systemctl restart opensips.service

♣ Check the permission to execute the script: # ls -l  /Path-To-Your-Script/Name-Of-Your-Script.pl. (  use “chmod +x” to give it execute permission).

♣ Use “strace”  to debug the script execution. # strace file Name-Of-Your-Script.

Writing a New OpenSIPS ‘s Perl Extension

Later i will write a Perl script that extend the feature “Advice of Charge (AOC) in SIP”. If you are in hurry you can go here and see an example ( Replace 183 early media reply with 180 (Ringing)). Now you can write your Perl script and call functions from OpenSIPS’s Perl module. For example the package Message  provides access functions for an OpenSIPS sip_msg structure. For example calling the function getRURI() gives the Request-URI of the current message.


More Information


Attribute Value Pair (AVP) Variables in OpenSIPS

.

Here i want to talk about AVPs from routing script’s point of view and the database’s point of view.

AVPs IN The Routing Script

These variables are used in the routing script. They are allocated at the beginning of the transaction and unallocated when the transaction is completed. So they are called transaction-persistent variables (if stateful processing is used). If stateless processing is used, theses variables will be attached to a singular message. When the message is received or the transaction is created, it will initially have an empty list of AVPs. The AVPs will be created directly in the routing script or through functions called from the script. They will automatically attached to the message/transaction.

The AVPs are visible in the routes where the message will be processed (i.e. branch_route , failure_route, onreply_route). If we want the AVPs to be visible in the onreply_route, we need to enable the TM parameter “onreply_avp_mode”.

OpenSIPS ‘s AVPs are Read-Write variables (You can read data from AVP or write data to AVP). AVP can also deleted. When we assign a value to an AVP, the new value is appended to the AVP. So the AVP may contain multiple values. New assignment (write operation) will add a new value to the AVP. The AVP is like a stack (Last in First out).

AVP’s Synatx: The name starts with $ sign. It looks like $avp(name) or $(avp(name)[N]). The index “N”  referes to a specific value. If no index is used, the last added value will be returned. See the figure below

The assignment can be done directly in the script or by calling a function which brings values from database and add them to the AVP. To delete a specific AVP’s value, we do this  $(avp(test)[N])=NULL. To remove the last value, we do this $(avp(test))=NULL. If no values remained, the AVP will be deleted. You can also delete the AVP by avp_delete(“$avp(test)/g”).


AVPs In The Database

The AVPs in the routing script can be loaded by values taken from the database. The default table to store the AVPs is “usr_preferences”. This table is used by the “avpops” module. Each user which uniquely identified by Unique User ID (UUID) or (username and domain), may have several preferences formed in (attribute, value) pairs and stored in the usr_preferences table.  The AVP is defined by this tuple (Attribute, Type , Value). The Attribute is the name of the AVP(The name can be string or number). The type is number (0 (str:str), 1 (str:int), 2 (int:str), 3 (int:int)). For example $avp(123)=2 <=> Type = 3 (integer name and integer value). The value (as a field) is a string.

The figure below explains the structure of the usr_preferences table.

The table “usr_preferences” is the default but you can change it by adding this line in the opensips configuration file with your table’s name:

modparam(“avpops”, “avp_table”, “usr_preferences”)

When a SIP message is received or new SIP transaction is created, the user’s preferences should be taken into account otherwise it will not called preferences, right?.

The function avp_db_load(“$fu”,”$avp(test-name)”) loads the AVP which belongs to $fu and has the name “test_name”.  This function exists in the “avpops” module. So the module must be loaded when using this function or any other AVP ‘s functions. The second parameter of this function can be empty – which means all AVPs identified by the source(first parameter).

Opensipsctl and AVPs

You can use opensipsctl tool to manage AVPs (add, remove, or configure) in the database. To get help info, type #./opensipsctl avp help

For example to add AVP which has the name “trace”, type 1 (string,integer), and the value =1 (e.g. <=> true):

opensipsctl avp add  -T usr_preferences test@youripaddress trace 1 1

This correspond to $(AVP(trace)) in the routing script. The value will be 1.

Note: You can use your favourite database editor to manage  the AVPs  in the database directly (in “usr_preferences” table). For example “phpMyAdmin” (Mysql Administration on the web).


More Information