Tuning The Linux Connection Tracking System



trackerThe connection tracking entry (conntrack entry) looks like this:

udp 17 170 src= dst= sport=137 dport=1025 src= dst= sport=1025 dport=137 [ASSURED] use=1

It contains two elements the original direction (the red) and the reply direction (the blue).

To display all table’s entries, read  “/proc/net/nf_conntrack”.

The conntrack entry is stored into two separate nodes (one for each direction) in different linked lists. Each linked list is called  bucket. The bucket is an element in a hash table. The hash value is calculated based on the received packet and used as index in the hash table. Iteration is done over the linked list of nodes to find the wanted node.

Long list is not recommended (iteration cost): The cost depends on the length of the list and the position of the wanted conntrack node.

Long hash table is recommended (constant time): The cost is the hash calculation.

Linked List Size (Bucket Size)= Maximum Number of nodes / Hash Table Size (Number of Buckets)

The hash table is stored in the kernel memory. We can tune the size of the bucket and the maximum number of nodes. The required memory = conntrack node’s memory size * 2* simultaneous connections your system aim to handle. Example: 304 bytes per conntrack and 1M connections requires 304*2 MB.

It is not recommended to set so big values if you have less than 1G RAM.

Tuning the Values

If your server has a lot of connections to be handled and the conntrack table is full, you will get this error “nf_conntrack: table full, dropping packet“. This will limit the number of simultaneous connections your system can handle.

To get the maximum number of nodes:

# /sbin/sysctl -a|grep -i nf_conntrack_max
net.nf_conntrack_max = 65536

To get the hash table size (number of buckets):

# /sbin/sysctl -a|grep -i nf_conntrack_buckets
net.netfilter.nf_conntrack_buckets = 16384

The bucket size (linked list length)= 4 (65536/16384).

To temporarily change the value of hash table size to 2*16384=32768:

# echo 32768 > /sys/module/nf_conntrack/parameters/hashsize

To permanently change the value:

# echo “net.netfilter.nf_conntrack_buckets = 32768” >> /etc/sysctl.conf
# /sbin/sysct -p

The same way for “nf_conntrack_max”:

Temporarily: # echo 131072 > /proc/sys/net/nf_conntrack_max

To permanently:

# echo “net.netfilter.nf_conntrack_max = 131072” >> /etc/sysctl.conf
# /sbin/sysct -p

This requires 38 MB memory.

“nf_conntrack” Other Values

You can change these values also in the same way:

# /sbin/sysctl -a|grep -i nf_conntrack
net.netfilter.nf_conntrack_acct = 0
net.netfilter.nf_conntrack_buckets = 16384
net.netfilter.nf_conntrack_checksum = 1
net.netfilter.nf_conntrack_count = 817
net.netfilter.nf_conntrack_events = 1
net.netfilter.nf_conntrack_expect_max = 256
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_helper = 1
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_log_invalid = 0
net.netfilter.nf_conntrack_max = 65536
net.netfilter.nf_conntrack_tcp_be_liberal = 0
net.netfilter.nf_conntrack_tcp_loose = 1
net.netfilter.nf_conntrack_tcp_max_retrans = 3
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_timestamp = 0
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 180
net.nf_conntrack_max = 65536

Last word: Disable “nf_conntrack” if it is not necessary.


Regular Expressions For SIP Trunk



The regular expression is a sequence of characters that form a search pattern. See the definition in Wikipedia. It is a huge topic and takes a lot of time to explain. If you didn’t use it, you will get stuck in a nested “If” statements situation. Using regular expressions means professionalism. I am assuming you know about it and i will just apply it to SIP.


Regular Expressions For a SIP Trunk

Usually when a SIP router receives a SIP request which is addressed to a PSTN gateway (i.e. Its Request-URI contains a telephone number), it checks which group the requested user (called party) belongs to. The router also must check if the caller has permission to make this call. So the check can be done on the caller and calle. Any check done by the router is translated into this simple thinking logic:

If (Routing Condition){

  Do Specific Work


This can be in the router’s routing script or internally integrated in the router’s application.


If (The Call Is Addressed To A Local Number){

Do Specific Work


In OpenSIPS SIP router, the previous condition can be treated in the routing scripts in two ways:

  • Using the Script Variables (e.g. the reference to the request’s uri ($ru)) with the regular expression directly:

If ($ru=~sip:[2-9][0-9]{6}@){


The previous regular expression is for any 7 numbers starting by one digit from the range [2-9].

  • Using a function like “pcre_match” in “Regex” module which matches the given string against a given regular expression. The return value is TRUE if it is matched, FALSE otherwise:

If (pcre_match(“$ru”,sip:[2-9][0-9]{6}@){


The variable “$ru” is read/write variable so be aware where in the script you are checking its value. You can use “$ou” which is a reference to the request’s original URI. The module “Regex” is based on the library “PCRE” which is an implementation of regular expression pattern matching where the regular expression parameter will be compiled in PCRE object. So the development libraries of “PCRE” must be installed (“libpcre-dev” or “pcre-devel”).

Note: Use “&&” if you want to concatenate multiple conditions in one “if” statement.

If ((Routing Condition-1) && (Routing Condition-2)){

  Do Specific Work


Examples of Regular Expressions For SIP Trunk

You should test your regular expression before using it to know if you have constructed it correctly. There are many online free services that you can use to do your tests. Use search engines to search for syntax symbols that you can use to construct the regular expression. The following are examples. Note that each one defines a group:

  • Any user on any domain/IP: sip:(.*)@(.*)
  • Any user on certain domain: sip:(.*)@mydomain.com
  • Any user on certain IP (  sip:(.*)@198\.18\.250\.10 In regular expression, the “.” is interpreted as “any character” symbol whereas “\.” is just a period (dot).
  • Any user on IP Range ( – sip:(.*)@198\.18\.250.* The symbol “*” is quantifier which means the preceding character is found 0 or more times. Here i left the last part of IP to be anything but you can restrict it.
  • To group a set of SIP URIs that are within a certain domain or certain subdomain, use .*mydomain\.com.* For example these URIs will be matched: sip:mydomain.com:5060;transport=tcp, and sips:test.mydomain.com:5061 SIP/2.0
  • To group a set of SIP URIs that have certain string in the username part of URI and the ports 5060 and 5061 are accepted, use the regular expression .*group1@198\.18\.250.\10:506<01>.* For example these two URIs will be matched sip:serv1group1@;transport=tcp and sips:serv2group1@;tranport=tcp
  • 8-digit number on any domain: sip:[0-9]{8}@(.*) The [0-9] is a range for one number between 0 and 9 and {8} means repeat the preceding number 8 times.
  • 8-digit number starting optionally by 8 on any domain: sip:8?[0-9]{7}@(.*)  The symbol “?” means the repetition is 0 or 1 to the preceding. For example sip:80986853@mydomain.com, sip:0986853@mydomain.com, 1234567@mydomain.com, and so on.
  • 4-digit number (could be an extension number) starting by 6 on a certain domain: sip:6[0-9]{3}@(.*)
  • 4-digit number which is not starting by 55 on any domain: sip:(?!55)[0-9]{4}@(.*)

 More Information

OpenSIPS Memory Debugging



If you have problem with the memory on your SIP router, do these:

  • Check things related to the memory management in your operating system. This is explained in the article “Linux Tuning For SIP Routers – Part 3“.
  • Check the memory management in the application of your SIP router.

Here i will give you just an example and you will take this as a way of thinking.

OpenSIPS Memory Debugging

If you got an error message related to OpenSIPS memory (e.g out of memory message), you have to debug it. The OpenSIPS Memory manager (NOT standard) supports debugging so what you can do is enabling the debugging:

  • From the OpenSIPS ‘s src  folder (“/usr/local/src/opensips_1_11″), run: “make menuconfig”. You will get this text based interface:


  • Press “ENTER” key to enter in “Configure Compile Flags” item:


  • Use the arrow keys to go to the sub-item “F_MALLOC“. Press the SPACE key to uncheck it. Then move to DBG_QM_MALLOC and check it (press SPACE key). Use the option “DBG_QM_MALLOC” in development and the option “F_MALLOC” in deployment because “F_MALLOC” is much faster.
  • Press ‘q’ to go back to the previous screen.
  • Press on “Save Changes”.
  • Press ‘q’ again and then move to the item “Compile and Install OpenSIPS” to compile and install OpenSIPS.
  • Open the configuration file and set memlog=1
  • Restart OpenSIPS and wait OpenSIPS to be completely loaded.

Now everything is ready to get the memory dumps (at shutdown and at runtime).

If you want to get the dumps at runtime:

  • Run “ps aux |grep opensips” to get OpenSIPS processes with their PIDs.
  • Run “kill -SIGUSR1 PID”. Where PID is the ID of the process and SIGUSR1 is the signal name which triggers the memory dump in OpenSIPS. The signals handlers (file: “main.c”) are loaded at the first stage of OpenSIPS startup.
  • Wait some time so the messages for private and shared memory will be dumped.

At shutdown: Stop OpenSIPS and go to the log and get the memory messages.

More Information


Linux Tuning For SIP Routers – Part 4 (Networking)



This is Part  4 of “Linux Tuning For SIP Routers” topic. In Part 1 i have talked about Interrupts and IRQ Tuning. In Part 2 i have talked about File System Tuning, Journaling File System, and Swappiness . In Part 3 i have talked about OOM killer, Private Momory, and Shared Memory Tuning. In this part i will talk about Network tuning.

Network Adapter Settings

To check the setting of your network card, type: “ethtool X” where X is the interface name. The tool “ethtool” is used to read and write the settings of the network card. To update the settings:

  • Find the name of the network interface: # ifconfig
  • Get the current settings: # ethtool X
  • Do your change using ethtool (See the man page: # man ethtool). For example to change the speed to 1000, and the mode to  full duplex: # ethtool -s p4p2 speed 1000 duplex full autoneg off (where p4p2 is my interface name).

Note: Changes must be supported by the network adapter, otherwise it will give an error message “Cannot set new settings: Invalid argument”

To make the changes permanent for p4p2, set the environment variable  ETHTOOL_OPTS:

  • If you are using bash : # vi /etc/sysconfig/network-scripts/ifcfg-p4p1  (note “ifcfg-p4p1” will be different)
  • Add/update this line: ETHTOOL_OPTS=speed 1000 duplex full
  • Restart the network service: # systemctl restart network.service

“txqueuelen” Parameter

The kernel parameter “txqueuelen” is the size of the transmission queue related to the interface. The default value is 1000 frames. The kernel stores the departing frames in this queue (the frames are not loaded onto the NIC’s buffer yet). Tuning this value is important to avoid loosing frames due to the lack of space in the transmission queue. Use high value (long queue) for high speed interfaces (≈> 10 Gb).

To get the current value of txqueuelen: # ifconfig p4p2 where p4p2 is the interface name. The output will look like:

 p4p2: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500 ether 0c:54:a5:08:45:a7  txqueuelen 1000  (Ethernet) RX packets 0  bytes 0 (0.0 B) RX errors 0  dropped 0  overruns 0  frame 0 TX packets 0  bytes 0 (0.0 B) TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 device interrupt 19

You can also execute “# ip link” to get the txqueuelen value (qlen 1000).

To temporarily change the value to 3000 (for ≈10 Gb card): # ifconfig  p4p2 txqueuelen 3000

To permanently change it, add/change the command “/sbin/ifconfig p4p2 txqueuelen 3000” in the file “/etc/rc.d/rc.local”. If this file is not exist, do the following if you have Fedora Red Hat:

  • # vi /etc/rc.d/rc.local
  • # chmod a+x /etc/rc.d/rc.local
  • At the top of the file, add the bash interpreter: #!/bin/sh
  • At the end of the file add exit 0
  • Between the previous two lines add the command: /sbin/ifconfig p4p2 txqueuelen 3000
  • Save the file and reboot (The command will be executed at system startup).

“netdev_max_backlog” Parameter

The kernel parameter “netdev_max_backlog” is the maximum size of the receive queue. The received frames will be stored in this queue after taking them from the ring buffer on the NIC. Use high value for high speed cards to prevent loosing packets. In real time application like SIP router, long queue must  be assigned with high speed CPU otherwise the data in the queue will be out of date (old).

To get the current value of netdev_max_backlog: #  /proc/sys/net/core/netdev_max_backlog. The default value is 1000 frames.

To temporarily change the size to 300000 frame:  # echo 300000 > /proc/sys/net/core/netdev_max_backlog

To permanently change the value:

  • Edit the file “/etc/sysctl.conf
  • Add the line: net.core.netdev_max_backlog=300000
  • Load the change which you did:  # sysctl -p


  • Do a test before going to the deployment.
  • To change the ring buffer size on the NIC: # ethtool -g p4p2 where p4p2 is the interface name. This operation must be supported by the NIC.

Kernel Parameters For the Socket Buffer

Usually SIP uses TCP or UDP to carry the SIP signaling messages over the internet (<=> TCP/UDP sockets). The receive buffer (socket receive buffer) holds the received data until it is read by the application. The send buffer (socket transmit buffer) holds the data until it is read by the underling protocol in the network stack.

The default (initial) and maximum size (In  bytes) of the receive socket buffer:

# cat /proc/sys/net/core/rmem_default

# cat /proc/sys/net/core/rmem_max

You can manipulate this value in the application by calling the function “setsockopt” and specify the option name “SO_RCVBUF”  with the socket level set to SOL_SOCKET. When you specify a value for this option, the kernel will doubles this value.  So the doubled value should be less than the maximum (rmem_max).

The default (initial) and maximum size (In bytes) of the send socket buffer:

#  cat /proc/sys/net/core/wmem_default #  cat /proc/sys/net/core/wmem_max

You can manipulate this value in the application by calling the function “setsockopt” with the option name “SO_SNDBUF”. Choose the value which is suitable to he speed of the network card. For example, to set the maximum to 10 MB (10 * 1024 * 1024 Bytes)=10,485,760 for the receive buffer and send buffer:


# echo 10485760 > /proc/sys/net/core/rmem_max

# echo 10485760 > /proc/sys/net/core/wmem_max


  • Edit the file “/etc/sysctl.conf
  • Add the lines:



  • Save the file and reload the changes:  # sysctl -p

To change the defaults:


# echo 10485760 > /proc/sys/net/core/rmem_default

# echo 10485760 > /proc/sys/net/core/wmem_default


  • Edit the file “/etc/sysctl.conf
  • Add the lines:



  • Save the file and reload the changes:  # sysctl -p

Also You need to set the minimum size, initial size, and maximum size for the protocols. To read the current values (Assuming IPv4 and TCP):

# cat /proc/sys/net/ipv4/tcp_rmem

The output look like this: 4096    87380   6291456 (min, initial, max). To read write buffer settings: # cat /proc/sys/net/ipv4/tcp_wmem

You can do the same for UDP (udp_rmem and udp_wmem) To change the values: Temporarily:

# echo 10240 87380 10485760 > /proc/sys/net/ipv4/tcp_rmem

# echo 10240 87380 10485760 > /proc/sys/net/ipv4/tcp_rmem


  • Edit the file “/etc/sysctl.conf
  • Add the lines:

net.ipv4.tcp_rmem= 10240 87380 10485760

net.ipv4.tcp_wmem= 10240 87380 10485760

  • Save the file and reload the changes:  # sysctl -p

TCP Kernel Parameters

  • Disable TCP timestamp (RFC 1321): TCP timestamp feature allows round trip time measurement (<=>  Adding 8 bytes to TCP header). To avoid this overhead we disable this feature:  net.ipv4.tcp_timestamps = 0
  • Enable window scaling: net.ipv4.tcp_window_scaling = 1
  • Disable select acknowledgements (SACK): net.ipv4.tcp_sack = 1
  • Disable cache metrics so the initial conditions of the closed connections will not be saved to be used in near future connections:

net.ipv4.tcp_no_metrics_save = 1

‘1’ means disable caching.

  • Tune the value of “backlog” (maximum queue length of pending connections “Waiting Acknowledgment”):

tcp_max_syn_backlog= 300000

  • Set the value of somaxconn. This is the Max value of the backlog. The default value is 128. If the backlog is greater than somaxconn, it will truncated to it.

Temporarily# echo 300000 > /proc/sys/net/core/somaxconn

Permanently: Add the line: net.core.somaxconn= 300000 in the file /etc/sysctl.conf. Reload the change (# sysctl -p).

  • ” TIME_WAIT” TCP socket state is the state where the socket is closed but waiting to handle the packets which are still in the network. The parameter tcp_max_tw_buckets is the maximum number of sockets in “TIME_WAIT” state. After reaching this number the system will start destroying the socket in this state. To get the default value:

# cat /proc/sys/net/ipv4/tcp_max_tw_buckets.

Increase the value to 2000000 (2 million). Increasing this value leads to requirement of more memory.


# echo 2000000 > /proc/sys/net/ipv4/tcp_max_tw_buckets

Permanently: Add the line: net.ipv4.tcp_max_tw_buckets= 2000000 in the file /etc/sysctl.conf. Reload the change (# sysctl -p).

  • Other TCP parameters that you can change as above:

net.ipv4.tcp_keepalive_time (how often the keepalive packets will be sent to keep the connection alive).

net.ipv4.tcp_keepalive_intvl (time to wait for a reply on each keepalive probe).

net.ipv4.tcp_retries2 (how many times to retry before killing an alive TCP connection).

net.ipv4.tcp_syn_retries (how many times to retransmit the initial SYN packet).

 Number of Open Files

When the SIP router is high loaded, this means it has a lot of open sockets (Each socket is a file). To increase the number of sockets we increase the number of open files (<=> number of file handles). To know the maximum number of file handles on the entire system:

# cat /proc/sys/fs/file-max. The output will be a number:  391884

To get the current usage of file handles: # cat /proc/sys/fs/file-nr. The output will look like: 10112   0       391884 First number (10112) is the total number of allocated file handles. Second number (0) is the currently used file handles (2.4 kkernel) or currently unused file handles (2.6 kernel). Third number (391884) is the maximum number of file handles.

To change the maximum number of open files temporarily:  # echo 65536 > /proc/sys/fs/file-max

To change the maximum number of open files permanently: Add the line “fs.file-max=65536”  in the file “/etc/sysctl.conf”.

SHELL Limits

The maximum number of open files “/proc/sys/fs/file-max” is for the whole system. This number is for all users so if you want to allow specific user (lets say opensips) to open X files where X < file-max, you can use “ulimit” to do that:

  • As root, edit the file: # vi /etc/security/limits.conf
  • Add/Change the following lines:

opensips              soft               nofile                   4096

opensips              hard               nofile                  40000

At the start time, the opensips can open till 4096 files (soft limit). If opensips gets an error message about running out of file handles, then opensips user can execute “ulimit -n New_Number” to increase the number of file handles to New_Number. The New_Number must be less than the hard limit (the second line) and the hard limit must be less than the file-max.

To avoid getting the error message (“out of file handles”), let opensips in its start scripts execute “ulimit -n New_Number”. Choosing the New_Number depends on your test to know how much your opensips needs. You can as root set permanently “soft limit=hard limit = New_Number”  in the file ” # vi /etc/security/limits.conf”.

Do not set the hard limit to a value equal to /proc/sys/fs/file-max, then the entire system might run out of file handles. Remember file-max is for the whole system.


  • Don’t use the numbers in this article. Do your tests on your system.
  • Use network performance testing tools like netperf, pktgen, and mpstat.
  • You always have to check how to optimize your hardware:
    • If the Network Interface Card support Flow Control (e.g e1000 network interface card).
    • Some NIC drivers support interrupt coalescing (multiple interrupts can be coalesced into one kernel interrupts). If your NIC support this, you can activate it as following: # ethtool -C p4p2 where p4p2 is the interface.

 More Information

Linux Tuning For SIP Routers – Part 3 (Memory)



This is Part  3 of “Linux Tuning For SIP Routers” topic. In Part 1 i have talked about Interrupts and IRQ Tuning. In Part 2 i have talked about File System Tuning, Journaling File System, and Swappiness . In this part i will talk about OOM killer, Private Momory, and Shared Memory Tuning.

Out-Of-Memory (OOM) Killer

Linux kernel uses Out-Of-Memory (OOM) killing technique to recover the memory in “out-of-memory” situation. The OOM killer will select one or more processes to kill. You can make a specific process less likely to be killed by the OOM killer as following:

# echo -15 /proc/PID/oom_adj  where the PID is the process ID and oom_adj represents how likely is the process to be killed. oom_adj takes values between -16 to +15. Setting oom_adj to -17 exempts the process entirely from the OOM killer.

To know how likely is a process to be killed, view the content of the file “oom_score”:

# cat /proc/PID/oom_score

Private Memory (Per Process Memory) Settings

Check if there is any memory limitation for a user process. The command “ulimit -a” display the limitations on all resources not only the memory. Use the option -m to display the max memory size:

# ulimit -m

The output: unlimited

The command “ulimit” is built into the bash shell and used for read and write the limitations on the resources available to the shell and the processes started by it. The system must allow such control on the resources.

What remained is configuring the private memory of SIP router’s process. See the example below:

Example: Tuning the Private Memory of OpenSIPS Process

OpenSIPS private memory (file: mem/mem.h) is a memory dedicated to each process and used for process own business like building buffer for sending SIP message. The default size of private memory used by each OpenSIPS process is 1 MB. Common cases that need large private memory are loading user location contacts and doing NAT pinging. You can set the size of private memory (in megabytes) at startup by using the command parameter -M:

# ./opensips -M 10

Shared Memory Settings

The shared memory is memory shared between processes. The data that are stored  in the shared memory can be accessed by many processes. You need to care about shared memory if your SIP router is multi process and the processes/threads from different processes share some data. To see the shared memory settings in your system, type: # ipcs -lm

—— Shared Memory Limits ——–
max number of segments = 4096
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1

Usually the SIP router has its own memory manager which allocates and frees the memory offered by the operating system between its processes. So tuning the memory involves tuning the memory at the operating system level and at the router level. Linux as an operating system has some parameters that you can configure:

SHMMAX Parameter

It is the maximum size (in bytes) of a single shared segment that a Linux process can allocate in its virtual address space. The value of this parameter depends on whether you are using 32 bit system or 64 bit system (size of the available address space).

What to do

  • Check your system (# uname -a): I am using Fedora Red Hat SMP kernel on 64 bit platform (X86-64) so the maximum size of a shared memory segment theoretically is 2^64bytes. This is correspond to all physical RAM that you have.
  • As a test case, I will set the value of the SHMMAX parameter to “1/4 of physical RAM”. I have 4 GB of RAM so I will set SHMMAX value to 1*1024*1024*1024=1,073,741,824 (bytes):
    • Temporarily:   echo 1073741824> /proc/sys/kernel/shmmax
    • Permanently: Edit the file “/etc/sysctl.conf” and set the entry: kernel.shmmax = 1073741824. This change can be reflected without restart by executing: “/sbin/sysctl -p“.

SHMALL Parameter

It is the total amount of shared memory pages that can be used system wide. Hence, SHMALL should always be at least SHMMAX / PAGE_SIZE. Execute “getconf PAGE_SIZE” to get the page size. In my case the page size is 4096 So the SHMALL value will be 1073741824/ 4096 = 262,144.

After configuring what the operating system allocate to your SIP router application, we need to configure the application itself. The following is an example:

Example: Tuning the Shared Memory of OpenSIPS SIP Router

OpenSIPS 1.X is pure multi process SIP server (version 2.x is multi threads). It stores the state of the calls (SIP transactions or SIP dialogs) in the shared memory accessed by all its processes (file: mem/shm_mem.h). Lets say the SHMEM_SIZE is the size of the shared memory. The recommended value of SHMEM_SIZE depends on the complexity of the routing script. When you run OpenSIPS, you can specify the entire shared memory (in MB) by using the command parameter -m. Ideally it is preferable to have SHMEM_SIZE equal to SHMMAX kernel parameter. This means OpenSIPS shared memory will fit in one shared memory segment. This leads to less job to memory manager and less memory errors. If  SHMEM_SIZE > SHMMAX, this means OpenSIPS memory manager will take more than one segment and manage access to them between all processes.
Here i will show you how to set OpenSIPS shared memory to 1 GB (SHMEM_SIZE = SHMMAX):

# ./opensips -m 1024 (1GB * 1024= 1024 MB)

  • Test the shared memory: You can get some statistics about the shared memory used and managed by OpenSIPS by sending FIFO commands to OpenSIPS. Have a look here on the SHMEM class. Use also system commands like “free” (MemShared or Shmem value) to get information about the memory.
  • The size of the recommended shared memory depends on the complexity of the routing script. So do the test many times for different values and choose the best for your system. Don’t forget to tune the value of  SHMMAX kernel parameter.

Note: Configure both the private memory (-M parameter) and the shared memory (-m parameter) at OpenSIPS startup. Test for different values and choose the one which suitable to your routing script.

Semaphores Settings

Semaphores are like counters that are used for controlling access by multiple processes to shared resources like shared memories. To see all semaphores settings: # ipcs -ls

—— Semaphore Limits ——–
max number of arrays = 128                      # SEMMNI Parameter
max semaphores per array = 250               # SEMMSL Parameter
max semaphores system wide = 32000      # SEMMNS Parameter
max ops per semop call = 32                     # SEMOPM Parameter
semaphore max value = 32767

or you can do this: # cat /proc/sys/kernel/sem

The output will be in this order (SEMMSL, SEMMNS, SEMOPM, SEMMNI):

250     32000   32      128

Tuning these values also depends on the complexity of the routing script. More complexity –> More loaded modules (shared libraries) –> more sharing –> more need for tuning the synchronization between processes.

Keep these values AS IS  if you don’t have problems with performance. To change these values temporarily:

# echo 250 32000 100 128 > /proc/sys/kernel/sem  


# sysctl -w kernel.sem=”250 32000 100 128″

To permanently change these values, Edit the file “/etc/sysctl.conf” and override the settings: kernel.sem=250 32000 100 128

Check The System Limits using “ulimit” System Property

Follow these steps to monitor the resources offered by the system to specific process:

  • Get the ID of the proess using # ps -A | grep …
  • Execute this: # cat /proc/$ProcessID/limits. Replace the ProcessID with the ID you get using the “ps” above.

The output will look like this:

Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             511830               511830               processes
Max open files            1024                 1024                 files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       511830               511830               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us


After tuning the memory of the operating system, you need to care about the memory of  the SIP application itself. Here is an example (OpenSIPS Memory Debugging) .In the Next Part, I will talk about network tuning to optimize SIP Routers.

Linux Tuning For SIP Routers – Part 2 (Disk Access)



If your SIP router does statefull processing and write the state like transactions information to different backends like syslog and SQL, you have to optimize the access to the disk where the state will be stored. The state can also be the dialog information like call duration, call start time, and setup time.

In Part 1 i have talked about interrupts and IRQ tuning. In this part i will show you how to tune the access to the disk to optimize the performance of your SIP router. Here i have Fedora Red Hat operating system, and Ext4 file system.

File System Tuning

To know the file system of your operating system and the mount options, execute the command “cat /etc/fstab“. The output will be like this:

/dev/mapper/fedora_dhcppc7-root                                  /              ext4    defaults        1 1
UUID=a534d6c5-ec57-4854-9c2b-db3ec3e66286            /boot        ext4    defaults        1 2
/dev/mapper/fedora_dhcppc7-home                               /home       ext4    defaults        1 2
/dev/mapper/fedora_dhcppc7-swap                                swap        swap    defaults       0 0

Each line in this output contains the following fields separated by spaces or tabs: partition or storage device, mounting point, file system type, mount options, dump, and pass.

According to this, the file system type is ext4 and the mount options is the defaults. In my system the default mount options are (rw, suid, dev, exec, auto, nouser, and async).

  • The option async allows asynchronous write/read operations on the file system. If your system has sync instead of async, change it otherwise keep it. You can read more about mount options in the man page “man mount“.
  • There is an option called  “noatime” which means reading accesses to the file system will not cause an update to the atime information (access time information) of the accessed files. In other words no writes to the file system for files which are being read. This is useful in case there is state reading from the disk.

To remount the root partition (i.e. /) with the mount option “noatime”, do this as root:

# mount -o remount, noatime   /

To do that permanently, edit the file fstab “vi  /etc/fstab" and update the corresponding line. For example if you want to add the mount option “noatime”  to the root partition (i.e. /):

This line : /dev/mapper/fedora_dhcppc7-root /                    ext4    defaults        1 1

will be : /dev/mapper/fedora_dhcppc7-root /                       ext4    defaults, noatime        1 1

To check the change in mount options: # mount |grep noatime. You will get this:

/dev/mapper/fedora_dhcppc7-root on  /  type ext4 (rw,noatime,seclabel,data=ordered)

Journaling File System

It is a file system feature which allows to serially log the changes in the file system into a dedicated area before committing them to the main file system. If a crash occurred, Any lost data can be recreated and stored in the locations that would have been stored in if the system had not crashed. In real time application like SIP routing, using non-journaling file system enhances the performance by avoiding the delay caused by jourmaling. In Ext4 file system you can turn off journaling feature. Do this to disable journaling:

  • Boot from your Linux installation DVD. Select Troubleshooting –> Rescue a Fedora system. Here you have your file system unmounted.
    • Execute: sh-4.2# tune2fs -O ^has_journal /dev/sda1
    • sh-4.2# tune2fs -O ^needs_recovery /dev/sda1
    • sh-4.2# e2fsck -f /dev/sda1
    • Remove the DVD and reboot.
  • Check file system features (There should be no “has_journal” or “needs_recovery”): # debugfs -R features /dev/sda1. The output:

Filesystem features: ext_attr resize_inode dir_index filetype extent flex_bg sparse_super huge_file uninit_bg dir_nlink extra_isize


The swap is a special place on the hard disk. Linux kernel stores physical memory pages/chunks that are currently unused on the swap to free up that pages from the physical memory. Using the disk as an extension to RAM is called virtual memory (RAM + swap). In real time applications like SIP routing, these are recommended:

  • Use enough physical memory.
  • Disable the swap or make the swapping rare.

How often the kernel do swapping is controlled by kernel parameter “Swappiness”. This parameter can be set to value between 0 and 100. The default value is 60.  To decrease the response latency, we use low value 0-10.  The value 0 makes the kernel swap only to avoid out of memory condition.

To change the “swappiness” value temporarily to 10, do this as root:

  • # echo 10 > /proc/sys/vm/swappiness
  • To verify the change : # cat /proc/sys/vm/swappines

To change the “swappiness” value permanently to 10, do this as root:

  • Edit the file “sysctl.conf” : # vim /etc/sysctl.conf
  • Add this line to override the default: vm.swappiness = 10


In the Next Part , I will talk about memory tuning to optimize SIP Routers.


OpenSIPS Startup Cost


klok.To measure the cost of OpenSIPS startup, i will use the environment variable LD_DEBUG which is used to instruct Linux dynamic linker to dump debug information. To display all valid options, execute:

# LD_DEBUG=help cat

Here I am using the default routing script of OpenSIPS. I will set the variable LD_DEBUG to the value “statistics” while starting OpenSIPS:

  • # cd /etc/init.d
  • # LD_DEBUG=statistics ./opensips start

I got:

19179:     runtime linker statistics:
19179:       total startup time in dynamic loader: 274776 clock cycles
19179:                 time needed for relocation: 51512 clock cycles (18.7%)

It took 274776 cycles to startup OpenSIPS on my device.

To decrease the latency after startup, we load all dynamic libraries at startup. Dynamic Libraries can be instructed to load at system startup by setting the environment variable  LD_BIND_NOW to 1.

  • # LD_BIND_NOW=1 LD_DEBUG=statistics ./opensips start

I got:

19211:     runtime linker statistics:
19211:       total startup time in dynamic loader: 308553 clock cycles
19211:                 time needed for relocation: 47668 clock cycles (15.4%)


With LD_BIND_NOW=1 it took 308553 cycles on startup. They are nearly the same. This means OpenSIPS already load all dynamic libraries at startup. This eliminate the latency of loading after the startup while processing SIP messages. Good Job !

What actually happens is: When OpenSIPS parses the configuration file, it loads all the modules that are configured in the configuration file. The statement loadmodule (Module_Name) corresponds to loading the shared library named Module_Name.

After parsing, the configuration file is not needed any more.

Linux Tuning For SIP Routers – Part 1 (Interrupts and IRQ Tuning)



Caring about the performance of any SIP router (real-time application) involves caring about the performance of the operating system and the routing application which runs on the operating system. The hardware performance is also important. I will mention some tips on how you can tune Linux to serve real time application like SIP routing. This article will be in many parts. In this part i will talk about interrupts and IRQ tuning for real time processing.


Interrupts and IRQ Tuning

Usually the network devices has 1 to 5 interrupts lines to communicate with the CPU. In multiple CPU system, round robin scheduling algorithm (Fair interrupts distribution) is used to choose the CPU core that will handle the interrupt. The file “/proc/interrupts” records the number of interrupts per CPU core per IO device. Execute this “cat /proc/interrupts” to get the list of interrupts in your system. To get the line associated with Ethernet driver, execute “grep p4p1 /proc/interrupts” where p4p1 is the name of my Ethernet driver. The output will look like this:   

30:      36650      16711      62175      18490   PCI-MSI-edge      p4p1

Where: 30 is IRQ number. 36650 of that interrupt handled by CPU-Core-0. 16711 of that interrupt handled by CPU-Core-1, 62175 of that interrupt handled by CPU-Core-2, 18490 of that interrupt handled by CPU-Core-3. PCI-MSI-edge is the interrupt type. p4p1 is the driver which receives the interrupts (This could be a comma-delimited list of drivers).

It is recommended that all interrupts generated by a specific device to be handled by the same CPU cores. IRQ fair distribution between all CPU cores is not recommended because when the interrupt goes to another fresh CPU core, the new CPU core will load the interrupt handler function from the main memory to the cache (time overhead).

IRQs have a property called interrupt affinity or smp_affinity. This property defines the CPU cores (CPU Set) that will handle the interrupts of a specific device. This property can be used to improve the performance of the SIP router (as any real-time application) by assigning same CPU set to the process/thread and to the interrupt. This minimizes the delays by allowing cache sharing between the process/thread and the interrupt.

Affinity Value for a Specefic IRQ

The affinity value (CPU cores) for a specific IRQ is stored in the file “/proc/irq/IRQ_NUMBER/smp_affinity”. The value in this file is a hexadecimal bit-mask (The lowest order bit corresponding to the first logical CPU). For example to set the affinity value to the IRQ-30 (the example above), do this as root:

  • Display the current affinity value: “cat /proc/irq/30/smp_affinity”.
  • Change the affinity of the IRQ-30 to the first 2 cores (0011). The corresponding hex value is 3. We write this number 3 in the file “/proc/irq/30/smp_affinity” : “echo 3 > /proc/irq/30/smp_affinity”

Setting the smp_affinity can be done at the hardware level (no intervention from the kernel) on systems which support interrupt steering.

“irqbalance” Daemon Configuration

Most of the current distributions comes with “irqbalance” daemon which distributes the interrupts fairly between CPU cores. So disable the irqbalance for those CPU cores that you want to bind them with specific IRQs. This can be done by editing the file “/etc/sysconfi/irqbalance” and changing the parameter IRQBALANCE_BANNED_CPUS which is 64 bit mask which allows you to indicate which CPUs should be skipped when balancing IRQs.

Affinity Value for a Specific Process/Thread

You can use the command “taskset” to set/get the affinity value of a running process or to run a new command with a given affinity. This will be done as following for non-running process (command): “taskset 3 /…./my_process“. This will bind my_process to the CPU-Core-0 and CPU-Core-1. Scheduling will happen between these two cores.

To bind running process to specific cores, do this : “taskset -p 3 PID” where 3 is the CPU-Set hex mask and PID is the process ID.

To check the CPU cores, execute “cat /proc/PID/status“.

I will apply this to OpenSIPS (SIP router):

OpenSIPS 1.X is multi-processes SIP router so when you execute “ps aux |grep opensips“, you will get many lines. You can change the affinity value to all or some of these processes as following:

  • # taskset -p 3 19904 where 19904 is one of these processes
  • Check the status of the process 19904: # cat /proc/19904/status

Cpus_allowed:   3

OpenSIPS 2.X is multi-threads SIP router (under development).

The affinity can also set by the process in its code. For example:

      #include <sched.h>

cpu_set_t mask;


unsigned int len = sizeof(mask);

CPU_SET(cpu_nr, &mask);

sched_setaffinity(0, len, &mask);


  • cpu_set_t is a data structure (set of bits) which represents the set of CPU cores.
  • CPU_ZERO(…) clears the set (no CPUs).
  • CPU_SET(…) adds CPU core number cpu_nr to the CPU set.
  • sched_setaffinity(…) Set the affinity mask. First parameter is PID (here it is zero so it means the calling thread/process).

Note: To scale the performance more, choose the best CPU cores for interrupts affinity and process affinity.

Last Word

If there is a lot of reading/writing from/to the hard disk (e.g. Stateful SIP Router),  Attaching the same CPU set to the processes/threads and the interrupts will be useful.

  • Get the driver name of your hard disk:
    • Execute “udevadm info -a -n /dev/sda” and parse the output. Search for something like DRIVERS==”Something”.  I have got this DRIVERS==”ahci”.
  • Execute “# grep ahci /proc/interrupts” and get the IRQ number.
  • Set the affinity value for the processes/threads and to the interrupts as discussed above.


The Next Part will be about tuning the access to the disk to optimize the performance of the SIP routers.


Homer as a SIP Capturing System – Part 2



In Part 1, i have talked about the definition of Homer. In this article i will talk about these:

  • Compilation and Installation of SipCapture module with OpenSIPS. To have more control over the installation process, i will do the compilation from the source.
  • Preparing Homer database.
  • Installation and configuration of Homer web interface.

Please have a look here OpenSIPS Compilation and Installation.

Some Packages must be Installed

To be able to compile the Sip_Capture module and install it, you need some packages installed on your system. Here i am using Fedora Red Hat.

# yum  install autoconf automake bzip2 cpio curl curl-devel curl-devel expat-devel fileutils make gcc gcc-c++ gettext-devel gnutls-devel ncftp nmap openssl openssl-devel mod_ssl mcrypt perl patch unzip wget zip zlib zlib-devel bison flex mysql mysql-server mysql-devel pcre-devel libxml2-devel sox httpd php php-gd php-mysql php-json git php-mysql php-devel php-gd

# yum install php-pecl-jsonc php-pecl-jsonc-devel

Some of above packages are needed for Homer Web interface.

Compilation and Installation of SipCapture Module

  • Go to the folder where you have the source code of OpenSIPS. SipCapture module is included with OpenSIPS (“/usr/local/src/opensips_1_11/modules/sipcapture”). To compile it, do these:

# cd  /usr/local/src/opensips_1_11/

  • Compile the SipCapture module as following:

# make include_modules=”sipcapture” modules

  • Install it:

# make install

  • Go to “/usr/local/opensips_1_11/lib64/opensips/modules/” and be sure there is a file named sipcapture.so

Downoad Homer Web

 # cd /var/www/

# git clone https://code.google.com/p/homer/


# git clone https://github.com/sipcapture/homer/

# cd homer

Preparing Homer Databases

Now we want to prepare the databases that are used by the capturing server and the web interface:

Run Mysql server: # systemctl start mysqld.service

  • We will login to mysql as root (# mysql -uroot -p password) and add the super user “homeruser” :

MariaDB [(none)]> CREATE USER homeruser@localhost IDENTIFIED BY “homerpass”;

MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO homeruser@localhost WITH GRANT OPTION;

  • Create the database “homer_db” by the user “homeruser”:

# mysql –uhomeruser –phomerpass -e “create database IF NOT EXISTS homer_db”;

  • Create another database “homer_users“:

# mysql -uhomeruser -phomerpass -e create database IF NOT EXISTS homer_users;

  • Create the required tables:

# mysql -uhomeruser -phomerpass -Dhomer_db < sql/create_sipcapture_version_4.sql

# mysql -uhomeruser -phomerpass -Dhomer_db < webhomer/sql/statistics.sql
# mysql -uhomeruser -phomerpass -Dhomer_users < webhomer/sql/homer_users.sql

  • Some nodes are added as default. So  i will delete them:

# mysql -uhomeruser -phomerpass -Dhomer_users -e TRUNCATE TABLE homer_nodes;

  • Now i will add my node:

# mysql -uhomeruser -phomerpass -Dhomer_users -e “INSERT INTO homer_nodes VALUES(1,’′,’homer_db’,’3306′,’homeruser’,’homerpass’,’sip_capture’,’node1′, 1);”

  • Add web user: user level can be: 1 – ADMIN, 2 – Power User, 3 – User

I will delete the default user (test@test.com,test123):

# mysql -uhomeruser -phomerpass -Dhomer_users -e TRUNCATE TABLE homer_logon;

Now i will add my admin:

# mysql -uhomeruser -phomerpass -Dhomer_users -e “INSERT INTO homer_logon VALUES(NULL, ‘Admin’, MD5(‘admin123’), 1);”

Configuration of Homer Web Interface

We are in this folder “/var/www/homer/webhomer/”. The configuration and preferences PHP pages are exist in that folder. Just rename the files “configuration_example.php”  to “configuration.php” and  “preferences_example.php” to “preferences.php”.

# cp configuration_example.php configuration.php
# cp preferences_example.php preferences.php

Give (Read,Write,Execute) permission to Homer tmp folder:

# chmod 777 tmp/

If you have SELinux enabled, please run SELinux troubleshooter and follow what it will suggest to do to give permission to “tmp/”

Set the time zone of the Homer server

Take the time zone of your system (# system-config-date) and set it in the variable “HOMER_TIMEZONE” in the file “preferences.php”.

# vim preferences.php

define(‘HOMER_TIMEZONE’, “Europe/Stockholm”);

Configuration of database access credentials

Adjust the database access credentials in the configuration file “configuration.php”.

/* Access Credentials to “homer_users” Database */
define(‘HOST’, “localhost”);
define(‘PORT’, 3306);
define(‘USER’, “homeruser”);
define(‘PW’, “homerpass”);
define(‘DB’, “homer_users”);

/* Access Credentials to “homer_db” Database */
define(‘HOMER_HOST’, “localhost”);
define(‘HOMER_PORT’, 3306);
define(‘HOMER_USER’, “homeruser”);
define(‘HOMER_PW’, “homerpass”);
define(‘HOMER_DB’, “homer_db”);
define(‘HOMER_TABLE’, “sip_capture”);

Some paths must be adjusted:


Apache Configuration

# vi /etc/httpd/conf/httpd.conf

Add these to httpd configuration file:

<Directory /var/www/homer/webhomer>
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Order allow,deny
allow from all

Alias /homer /var/www/homer/webhomer

Restart the Httpd

# systemctl restart httpd

Open the browser and go to the URL : http://IP_address/homer. You will be here:

Login by this user  (username=”admin”, password=”admin123″). If it is ok with authentication, you will be here:

If you could not login from another computer, check the firewall configuration (Open the port 80 and 443 on your server). 443 is the HTTPS port /HTTP over TLS/SSL). We will see the related configuration in another article.

Till now Homer database is empty and we need to run the capturing server which is responsible for capturing packets and insert them in the database.

OpenSIPS Configuration to be a Homer Capturing Server

OpenSIPS configuration file will be like the following:

# This Configuration file is taken from Homer GIT Page:

####### Global Parameters #########



listen=udp:   # CUSTOMIZE ME


####### Modules Section ########
#set module path

loadmodule "db_mysql.so"
loadmodule "sipcapture.so"

####### Routing Logic ########
modparam("sipcapture", "db_url", "mysql://homeruser:homerpass@localhost/homer_db")
modparam("sipcapture", "capture_on", 1)
modparam("sipcapture", "table_name", "sip_capture")
/* activate HEP capturing */
modparam("sipcapture", "hep_capture_on", 1)

/* configuration for Mirroring PORT */
modparam("sipcapture", "raw_socket_listen", "")
modparam("sipcapture", "raw_interface", "eth1")
/* activate monitoring port capturing */
modparam("sipcapture", "raw_moni_capture_on", 1)
modparam("sipcapture", "raw_sock_children", 4)
/* Promiscious mode */
modparam("sipcapture", "promiscious_on", 1)

####### Routing Logic ########

# main request routing logic

# Main SIP request routing logic
# - processing of any incoming SIP request starts with this route
route {
        #For example, you can capture only needed methods...
        #if (!(method =~ "^(OPTIONS|NOTIFY|SUBSCRIBE)$"))) {

onreply_route {

        #And only needed reply or needed requests method
        #if(status =~ "^(1[0-9][0-9]|[3[0-9][0-9]|4[0-9]|[56][0-9][0-9])") {
        #if(!($rm =~ "^(NOTIFY|SUBSCRIBE|OPTIONS|)$")) {

Save the file and restart OpenSIPS:

# systemctl restart opensips.service


Missed column  “authorization” in the capturing table “sip_capture”

MariaDB [(none)]> alter table sip_capture ADD authorization VARCHAR(125);

Now the capturing server can store the packets in the database homer_db

OpenSIPS as a SIP Capturing Agent

OpenSIPS SIPTRACE module can send the captured packets to an external HEP server (Homer capturing server) instead of storing them in the local database. The following is the configuration to send the captured packets out:

modparam(“siptrace”, “duplicate_uri”, “sip:″) # The capturing server address to send a duplicate of traced message to.

modparam(“siptrace”, “duplicate_with_hep”, 1)  # Enable HEP

modparam(“siptrace”, “trace_to_database”, 0) # Disable tracing to the local database


The next part will be exploring Homer through its web interface and enabling statistics.

More Information