Thinking Parallel

Skimming through the activity logs of this blog, I can see that many people come here looking for information about pthread_setaffinity_np. I mentioned it briefly in my article about Opteron NUMA-effects, but barely touched it because I had found a more satisfying solution for my personal use (taskset). And while I do not have in depth knowledge of the function, maybe the test programs I wrote will be of some help to someone to understand the function better. I will also post my test program for sched_setaffinity here while I am at it, simply because the two offer similar functionality.

Problem description

Short recap: The problem both functions are trying to solve is to bind a thread (pthread_setaffinity_np) or process (sched_setaffinity) to one or more user-defined processors. You may want to do this because the scheduler is doing something stupid or because you want to keep your caches hot at all costs in a multithreaded program. More information about affinity on Linux can be found in this article on CPU affinity by Robert Love.

sched_setaffinity example

Here it goes. Not much left for me to explain, as it is documented.

/* Short test program to test sched_setaffinity
* (which sets the affinity of processes to processors).
* Compile: gcc sched_setaffinity_test.c
* -o sched_setaffinity_test -lm
* Usage: ./sched_setaffinity_test
*
* Open a “top”-window at the same time and see all the work
* being done on CPU 0 first and after a short wait on CPU 1.
* Repeat with different numbers to make sure, it is not a
* coincidence.
*/

#include
#include
#include

double waste_time(long n)
{
double res = 0;
long i = 0;
while(i < n * 200000) { i++; res += sqrt (i); } return res; } int main(int argc, char **argv) { unsigned long mask = 1; /* processor 0 */ /* bind process to processor 0 */ if (sched_setaffinity(0, sizeof(mask), &mask) < 0) { perror("sched_setaffinity"); } /* waste some time so the work is visible with "top" */ printf ("result: %f\n", waste_time (2000)); mask = 2; /* process switches to processor 1 now */ if (sched_setaffinity(0, sizeof(mask), &mask) < 0) { perror("sched_setaffinity"); } /* waste some more time to see the processor switch */ printf ("result: %f\n", waste_time (2000)); } [/c] The waste_time function does nothing meaningful at all, but is merely there to crank up the CPU. It returns a result and that result is printed out so the compiler cannot optimize out the whole calculation. You may have to adjust the parameter for the function if the program executes too fast and you don’t see the results with top. Also do not forget to press the “1”-key once while watching the output of top to see the CPU usage split by CPU.

pthread_setaffinity_np example

This is almost the same program, except adapted to use a thread doing all the work. Documentation included again. Beware that you need the NPTL-version of the pthreads library for this to work (if you do not have it installed, the compiler will complain that it does not know the pthread_setaffinity_np function).

/* Short test program to test the pthread_setaffinity_np
* (which sets the affinity of threads to processors).
* Compile: gcc pthread_setaffinity_np_test.c
* -o pthread_setaffinity_np_test -lm -lpthread
* Usage: ./pthread_setaffinity_test
*
* Open a “top”-window at the same time and see all the work
* being done on CPU 0 first and after a short wait on CPU 1.
* Repeat with different numbers to make sure, it is not a
* coincidence.
*/

#include
#include
#include

double waste_time(long n)
{
double res = 0;
long i = 0;
while (i < n * 200000) { i++; res += sqrt(i); } return res; } void *thread_func(void *param) { unsigned long mask = 1; /* processor 0 */ /* bind process to processor 0 */ if (pthread_setaffinity_np(pthread_self(), sizeof(mask), &mask) < 0) { perror("pthread_setaffinity_np"); } /* waste some time so the work is visible with "top" */ printf("result: %f\n", waste_time(2000)); mask = 2; /* process switches to processor 1 now */ if (pthread_setaffinity_np(pthread_self(), sizeof(mask), &mask) < 0) { perror("pthread_setaffinity_np"); } /* waste some more time to see the processor switch */ printf("result: %f\n", waste_time(2000)); } int main(int argc, char *argv[]) { pthread_t my_thread; if (pthread_create(&my_thread, NULL, thread_func, NULL) != 0) { perror("pthread_create"); } pthread_exit(NULL); } [/c] So much for my short trip into the world of CPU-affinity on Linux, hope you enjoyed the ride...