Thinking Parallel

I have blogged about OpenMP and exceptions in the past. I have shown you what the OpenMP-specification has to say about exceptions. I have also shown you some (tiny) workarounds for the limitations regarding exceptions sketched in the spec. Unfortunately, this is only one side of the medal. When you actually try to use exceptions and OpenMP together, you may get a different picture today. This article is about my recent experiences in this regard.

Let’s consider this simple program (with thanks to my colleague Björn, who originally wrote it):

#include
#include

int main(int, char const *[])
{
float ts = 42.0f;

std::cerr << "ZERO, before parallel: " << std::endl; std::cout << "ts " << ts << " &ts " << &ts << std::endl << std::endl; #pragma omp parallel num_threads (1) { #pragma omp critical (std_cerr_critical) { std::cerr << "ONE, before try-catch: " << std::endl; std::cerr << "ts " << ts << " &ts " << &ts << std::endl << std::endl; } try { #pragma omp critical (std_cerr_critical) { std::cerr << "TWO, inside try: " << std::endl; std::cerr << "ts " << ts << " &ts " << &ts << std::endl << std::endl; } } catch(...) { #pragma omp critical (std_cerr_critical) { std::cerr << "THREE, inside catch: " << std::endl; std::cerr << "ts " << ts << " &ts " << &ts << std::endl << std::endl; } } #pragma omp critical (std_cerr_critical) { std::cerr << "FOUR, after catch, inside parallel: " << std::endl; std::cerr << "ts " << ts << " &ts " << &ts << std::endl << std::endl; } } // end #pragma omp parallel std::cerr << "FIVE, after parallel: " << std::endl; std::cerr << "ts " << ts << " &ts " << &ts << std::endl << std::endl; return 0; } [/cpp] It's really quite simple, it just monitors the value and the address of a float before, during and after a parallel region. Which is normally not a problem, except in this case there is a try-catch block in the region. The code is compliant to the OpenMP-specification, as all exceptions are caught from the thread that throws them and no exception leaves a structured block associated with any OpenMP-construct. The critical regions are not even necessary here, because I have cut the number of threads used down to one to make the output clearer. So let's see what happens when I compile and run this code with a couple compilers. The Intel Compiler goes first: [code] $ icpc exceptions_test.cpp -o exceptions -openmp exceptions_test.cpp(11) : (col. 2) remark: OpenMP DEFINED REGION WAS PARALLELIZED. $ ./exceptions ZERO, before parallel: ts 42 &ts 0xbfb1d664 ONE, before try-catch: ts 42 &ts 0xbfb1d664 TWO, inside try: ts 42 &ts 0xbfb1d664 FOUR, after catch, inside parallel: ts -2.85996e-05 &ts 0xbfb1d4e4 FIVE, after parallel: ts 42 &ts 0xbfb1d664 [/code] Uh, oh. The address of the variable changed after the exception was caught, which is something that should never, ever happen. It is a serious bug. It took my colleague Björn quite some time to track this down, because this is something you would never expect to happen. At least I would not. No wonder some of my recent code using OpenMP and exceptions was not able to pass its unit tests on the Intel Compiler. But hey, this is just one compiler and I would not judge from this bug about the whole state of exceptions and OpenMP, right? Sure you are right, the story goes on. So I tried this innocent piece of code on a couple more compilers, e.g. the one from IBM: [code] $ xlC_r exceptions_test.cpp -o exceptions_test -qsmp=omp "" 1586-356 (U) Unconditional branch in or out of a critical section. [/code] Huh? It's complaining that I am jumping into or out of a critical section. But call me stupid, I don't see this in my code. After a little trial and error, it turns out the critical region on line 22 is making problems here, commenting it out makes the compiler do what I want and also produces the correct results. Still, I should not have to do this, as a critical region inside a try-catch block appears to be totally valid to me. At least the compiler is complaining this time and does not leave me with a hard to catch bug. Let's try another compiler, while we are at it (from Sun this time): [code] CC exceptions_test.cpp -o exceptions_test -xopenmp -O3 "exceptions_test.cpp", line 22: Error: OpenMP constructs not permitted in try block. 1 Error(s) detected. [/code] Well, at least the compiler tells me what he does not like this time. Same problem as with the IBM one, same solution - comment out the critical on line 22 and it works. Still, yet again, from my point of view this should work just fine and I should not have to get rid of my beloved critical region. But hey, we are having so much fun, why not try another compiler? It's not like there aren't enough :-). Let's try the Portland Compiler this time around: [code] pgCC exceptions_test.cpp -o exceptions_test -mp "exceptions_test.cpp", line 12: error: expected an identifier #pragma omp parallel num_threads (1) ^ "exceptions_test.cpp", line 21: error: support for exception handling is disabled try { [/code] Oh my, the old problem with the Portland Compiler. For some reason, it never appears to like my num_threads-clause. But this is easily fixed by using omp_set_num_threads() instead. The other error is another story: why is exception handling disabled? Going through the compiler documentation, it appears exception handling can be enabled by compiling with –exceptions. So let’s try that – and jay, it works! And gives the correct results this time, even without changing anything more in the code. And it only took me 4 compilers to find one that liked it.

Last but not least, I have one more compiler up my sleeve: the compiler supplied with Visual Studio. Let me just reboot really quick, so I can get to it…

Ok, done. And here the story comes to a happy ending, as the Microsoft Compiler throws a couple of warnings, but compiles the code just fine. No changes required and output is correct. Finally.

Let me summarize my post here: From the five compilers I have tried, one works as expected, one works after forcing it into using exceptions, two could not compile my code and one does compile, but produces a serious and hard to track down bug. Not exactly a good track record. My resume: be careful when using exceptions and OpenMP together, as obviously this is something that not many people do today. I sometimes felt like I was the first person on earth to try this out ;). And while it can be fun to go where no one has gone before (after all, thats what we are being paid to do at the university), working on the bleeding edge is only fun as long as your don’t have a tight deadline to meet. Also keep in mind that everything I wrote in this post is only valid today. The compilers are getting better at a fast pace, and so what is a problem now and for me may not be a problem at all when you read this.

One more thing I would like to add: this post is by no means meant to bash on the compilers or the fine people that work on them. I have read the analogy somewhere that writing a compiler for C++ is like climbing the Mount Everest. Except that it is harder. When taking a look at the language and some of its dark corners, I can only agree. Mix in a pragma-based approach to parallel programming (OpenMP), and maintaining a compiler that does both must be even more challenging. I have a high respect for the people that cope with these difficulties every day. Still, the story of this article deserves to be told, in my humble opinion…