Thursday, January 30, 2014

Software Quality Ceiling

Over the past 3 years I have worked on a project porting a large legacy C++ code base so that it will run on Mac, Windows and in the future other platforms. In doing so we used a number of third party toolkits including Qt.

Portable libraries are nice to use when they work right. The problem is that sometimes you run up against quality issues with the libraries. Things that work right on one platform and not on another or things that simply don't work right on any platform.

Qt is a nice toolkit, but it has many issues. We worked closely with the support team at Digia to correct many issues, but with every software update a new set of problems are added. This means a long and laborious process of first creating a reproducible bit of code, reporting it, sometimes needing to convince the support people that it really is a bug, getting a patch, trying it out and often iterating several times until the patch is correct. Then we hope that the problem is corrected in the next update, which it often is, but about as often has new issues.

Of course Qt is not the only problem. Yesterday I was investigating the new threading library for C++ 11 in Visual Studio 2013. Everything was going along just fine as I was working on a simple ThreadPool class. It worked great except that it would hang when shutting down. I finally did a search and ran across the following http://connect.microsoft.com/VisualStudio/feedback/details/747145. A bug report against Visual Studio 2012. Marked Closed Deferred.

I'm able to workaround the problem by ensuring that my thread pool shuts down before returning from main, but now I'm faced with how to write a class that logically would be global that will correctly shutdown in Visual Studio. I have yet to test it on Mac or Ubuntu, but I'm fairly confident that I won't have the same problem there.

Over the years I have encountered numerous code generation problems with compilers. The good news is that I haven't run across anything recently. But in the back of my mind I continue to worry that one may show up. This is really bad because unless you actually run the code affected in the manner that would exhibit the bug you may never know you have this problem.

Then there are things like the openGL drivers for video cards. In some cases something works fine on one card but not on another. At that point you are faced with answering the question, did I do something wrong? Sometimes I did. In some cases the fact that it worked on one card was wrong. But in other cases I used the API correctly and there is a bug in the driver.

Then there is the unspecified behavior clause in languages. In C++ there are many places in the standard where the behavior is listed as unspecified or compiler dependent. In these cases I would prefer that it didn't compile or at the very least it should crash when I try to use it. In my recent porting project I ran across several cases where the compiler on the Mac crashed on badly written code, but on Windows it didn't. I was happy to find and fix those issues. But they should have not worked in the first place.

All of these issues result in a quality ceiling. My goal is to write code that is 100% bug free. Occasionally, I achieve that for simple cases. More often I have a few bugs that testing should eventually uncover. But if I'm working with tools, compilers, libraries, operating systems, etc. that are buggy I will always be faces with the fact that my software will be of a lower quality than I'm theoretically capable of writing.

As an industry we need to figure out how to break through the quality ceiling.

With that I close, because I have no answers.

No comments: