Caring about the performance of any SIP router (real-time application) involves caring about the performance of the operating system and the routing application which runs on the operating system. The hardware performance is also important. I will mention some tips on how you can tune Linux to serve real time application like SIP routing. This article will be in many parts. In this part i will talk about interrupts and IRQ tuning for real time processing.
Interrupts and IRQ Tuning
30: 36650 16711 62175 18490 PCI-MSI-edge p4p1
Where: 30 is IRQ number. 36650 of that interrupt handled by CPU-Core-0. 16711 of that interrupt handled by CPU-Core-1, 62175 of that interrupt handled by CPU-Core-2, 18490 of that interrupt handled by CPU-Core-3. PCI-MSI-edge is the interrupt type. p4p1 is the driver which receives the interrupts (This could be a comma-delimited list of drivers).
It is recommended that all interrupts generated by a specific device to be handled by the same CPU cores. IRQ fair distribution between all CPU cores is not recommended because when the interrupt goes to another fresh CPU core, the new CPU core will load the interrupt handler function from the main memory to the cache (time overhead).
IRQs have a property called interrupt affinity or smp_affinity. This property defines the CPU cores (CPU Set) that will handle the interrupts of a specific device. This property can be used to improve the performance of the SIP router (as any real-time application) by assigning same CPU set to the process/thread and to the interrupt. This minimizes the delays by allowing cache sharing between the process/thread and the interrupt.
Affinity Value for a Specefic IRQ
The affinity value (CPU cores) for a specific IRQ is stored in the file “/proc/irq/IRQ_NUMBER/smp_affinity”. The value in this file is a hexadecimal bit-mask (The lowest order bit corresponding to the first logical CPU). For example to set the affinity value to the IRQ-30 (the example above), do this as root:
- Display the current affinity value: “cat /proc/irq/30/smp_affinity”.
- Change the affinity of the IRQ-30 to the first 2 cores (0011). The corresponding hex value is 3. We write this number 3 in the file “/proc/irq/30/smp_affinity” : “echo 3 > /proc/irq/30/smp_affinity”
Setting the smp_affinity can be done at the hardware level (no intervention from the kernel) on systems which support interrupt steering.
“irqbalance” Daemon Configuration
Affinity Value for a Specific Process/Thread
You can use the command “taskset” to set/get the affinity value of a running process or to run a new command with a given affinity. This will be done as following for non-running process (command): “taskset 3 /…./my_process“. This will bind my_process to the CPU-Core-0 and CPU-Core-1. Scheduling will happen between these two cores.
To bind running process to specific cores, do this : “taskset -p 3 PID” where 3 is the CPU-Set hex mask and PID is the process ID.
To check the CPU cores, execute “cat /proc/PID/status“.
I will apply this to OpenSIPS (SIP router):
OpenSIPS 1.X is multi-processes SIP router so when you execute “ps aux |grep opensips“, you will get many lines. You can change the affinity value to all or some of these processes as following:
- # taskset -p 3 19904 where 19904 is one of these processes
- Check the status of the process 19904: # cat /proc/19904/status
OpenSIPS 2.X is multi-threads SIP router (under development).
The affinity can also set by the process in its code. For example:
unsigned int len = sizeof(mask);
sched_setaffinity(0, len, &mask);
- cpu_set_t is a data structure (set of bits) which represents the set of CPU cores.
- CPU_ZERO(…) clears the set (no CPUs).
- CPU_SET(…) adds CPU core number cpu_nr to the CPU set.
- sched_setaffinity(…) Set the affinity mask. First parameter is PID (here it is zero so it means the calling thread/process).
Note: To scale the performance more, choose the best CPU cores for interrupts affinity and process affinity.
If there is a lot of reading/writing from/to the hard disk (e.g. Stateful SIP Router), Attaching the same CPU set to the processes/threads and the interrupts will be useful.
- Get the driver name of your hard disk:
- Execute “udevadm info -a -n /dev/sda” and parse the output. Search for something like DRIVERS==”Something”. I have got this DRIVERS==”ahci”.
- Execute “# grep ahci /proc/interrupts” and get the IRQ number.
- Set the affinity value for the processes/threads and to the interrupts as discussed above.
The Next Part will be about tuning the access to the disk to optimize the performance of the SIP routers.