Many different software development methodologies have been created over the years. Most of them work fairly well. But they all really kind of derive from the Scientific Method.
The scientific method is roughly as follows:
1) Ask a question.
2) Gather information about the question.
3) Create a hypothesis that explains the observations.
4) Test the hypothesis through experimentation and then refine and retest it.
5) Publish the results.
6) Continue to test the results and refine the hypothesis incorporating external input.
In software we have a method that is often referred to as code and fix.
If you look at the scientific method it would be more or less.
1) Ask a question.
2) Create a hypothesis.
5) Publish the results.
6) Refine the hypothesis incorporating external input.
Which kind of works but is frankly an insult to anyone using the software. In a lot of ways it is a lot like religious dogma. Except that in religion refinement based on external input is almost never done or if it is the step of gathering information and testing the hypothesis are steps that tend to be skipped.
Getting back to the scientific method as applied to software.
All reasonable processes have ideas like stating the problem, gathering requirements, creating a specification, testing, documentation, and finally publication followed by iterative refinement.
Most software usability and other issues are caused by not spending enough time observing and testing. And often publication is done in a hurry, which also occurs in the scientific community as well, but less often due to the stigma of publishing something that is subsequently shown to be incorrect.
In software the rigor that is often associated with the scientific method tends to be fairly rare. There is a pressure to perform. Not many years ago there was the idea of "Internet Time" which led to a plethora of buggy software. Getting your software to market before anyone else has been something that many companies embraced as the best way to succeed.
However, over time we have observed that fast an speedy doesn't always win. The tortoise and hare children's story is a classic example of how this lesson was learned, repeated, and well know. However, there still remains a segment of society that embraces the idea that being fast at the start is the way to win.
Long term we find that software companies that embrace quality and continuous rigorous improvement of quality win in the long term. This is a common theme in successful businesses as well.
In the end we don't need some new software development process. Every one of the good software development methods are really just an application of the scientific method with a lot special words attached that tend to disguise the underlying methodology in an effort to make it seem new and different.
Friday, May 15, 2015
Tuesday, March 3, 2015
Zero Defects is Possible in Software
I make the claim in the title for two reasons. First, it sound's like an arrogant and unreasonable viewpoint. Which is more to draw you in than for any other reason.
More importantly my reason is that if you don't believe that you can write defect free code you will never be able to.
Let us start with understanding some ground rules of my claim.
When you write code you will almost certainly introduce bugs. Occasionally, I write a small amount of code that is bug free. Just as you can type words that are spelled correctly that form grammatically correct sentences it is possible to write multiple lines of code that are bug free without modification.
Take this with an aggressive stance on bug fixing and testing and you should be able to write small amounts of code that are bug free after only a few iterations.
An approach that I find that works is to fix every bug in code that I'm designing from scratch. The quicker I do this the faster the code becomes solid. It can be harder if you are a maintainer of code to do this as it means that you need to understand both what the code is doing and what the writer of the code intended. It is very difficult to determine intent if you don't have written requirements.
Written requirements are sometimes hard because they often change as you write code. This is due to the discovery process inherent in writing code to solve a problem that you don't fully understand at the start. It can be hard to develop the discipline to update requirements as they change over time. Creating a process with independent testing can help to make this happen. Even so some things will likely fall through the cracks.
The good news though is that even though not having a requirement makes determining the intent of the writer of the code hard to determine, it is often possible to determine based on how the code is used and using engineering judgment.
But, our goal is to leave the software we write in a bug free state when it is complete and for whatever reasons we move on to something new and leave the maintenance to someone else. If this happens the maintainers job is an easy one as they should never need to update the software because it just plain works.
More than likely if the software needs to be updated it would be to extend functionality which builds on the already solid core, which makes the maintainers job a lot easier.
At this point I imagine that you are probably still not convinced that it is possible to write bug free software.
If you can't believe that you can write bug free software you never will.
Unfortunately, I cannot lay claim to having zero defects in all the software I have written. But I can claim that some of the software I have written has zero defects and that over the years I have gotten better at writing software with zero defects.
Obviously, some things cannot be fixed quickly, or are inherited from the underlying system. While this is unfortunate, at the very least one can document that the defect exists so that a user of your software can make an informed decision about how to handle the problem.
This probably sounds overly idealistic. But there are things that over the years have led to countless problems in software.
In the end while I don't expect that you or I will deliver all software with zero defects. I do think that we can deliver some software with zero defects and for the software that has defects we can continue to fix the problems as they are discovered and with time iterate it to zero defects.
The problem with a less idealistic approach is that it leads inexorably to a point where you are spending all of your time fixing bugs in old code you wrote and not fixing bugs in new code. Bug debt or technical debt of this sort has a compounding effect which can only be corrected by either paying down the debt or by declaring technical bankruptcy and abandoning the debt laden code.
I hope that this inspires you to at least improve the quality of the code you write.
More importantly my reason is that if you don't believe that you can write defect free code you will never be able to.
Let us start with understanding some ground rules of my claim.
- There is an implicit assumption that the computer you are running on is not subject to an out of spec environment, such as excessive heat or solar flares and that all the hardware is functioning properly.
- Second there is an assumption that the operating system is bug free with respect to the code you are writing.
- Further, it is assumed that the compiler you are using generated correct bug free code with respect to the software you are writing.
When you write code you will almost certainly introduce bugs. Occasionally, I write a small amount of code that is bug free. Just as you can type words that are spelled correctly that form grammatically correct sentences it is possible to write multiple lines of code that are bug free without modification.
Take this with an aggressive stance on bug fixing and testing and you should be able to write small amounts of code that are bug free after only a few iterations.
An approach that I find that works is to fix every bug in code that I'm designing from scratch. The quicker I do this the faster the code becomes solid. It can be harder if you are a maintainer of code to do this as it means that you need to understand both what the code is doing and what the writer of the code intended. It is very difficult to determine intent if you don't have written requirements.
Written requirements are sometimes hard because they often change as you write code. This is due to the discovery process inherent in writing code to solve a problem that you don't fully understand at the start. It can be hard to develop the discipline to update requirements as they change over time. Creating a process with independent testing can help to make this happen. Even so some things will likely fall through the cracks.
The good news though is that even though not having a requirement makes determining the intent of the writer of the code hard to determine, it is often possible to determine based on how the code is used and using engineering judgment.
But, our goal is to leave the software we write in a bug free state when it is complete and for whatever reasons we move on to something new and leave the maintenance to someone else. If this happens the maintainers job is an easy one as they should never need to update the software because it just plain works.
More than likely if the software needs to be updated it would be to extend functionality which builds on the already solid core, which makes the maintainers job a lot easier.
At this point I imagine that you are probably still not convinced that it is possible to write bug free software.
If you can't believe that you can write bug free software you never will.
Unfortunately, I cannot lay claim to having zero defects in all the software I have written. But I can claim that some of the software I have written has zero defects and that over the years I have gotten better at writing software with zero defects.
Obviously, some things cannot be fixed quickly, or are inherited from the underlying system. While this is unfortunate, at the very least one can document that the defect exists so that a user of your software can make an informed decision about how to handle the problem.
This probably sounds overly idealistic. But there are things that over the years have led to countless problems in software.
In the end while I don't expect that you or I will deliver all software with zero defects. I do think that we can deliver some software with zero defects and for the software that has defects we can continue to fix the problems as they are discovered and with time iterate it to zero defects.
The problem with a less idealistic approach is that it leads inexorably to a point where you are spending all of your time fixing bugs in old code you wrote and not fixing bugs in new code. Bug debt or technical debt of this sort has a compounding effect which can only be corrected by either paying down the debt or by declaring technical bankruptcy and abandoning the debt laden code.
I hope that this inspires you to at least improve the quality of the code you write.
Technical Bankruptcy
Technical debt or bug debt is an insidious side effect of writing low quality code.
The technical debt that one engineer can accumulate in a career of programing can easily require 2 or more careers of other engineers to pay down.
A pay as you go policy says that you fix all bugs as they are discovered before writing new code. While this lowers your productivity initially. It helps you to develop better habits that introduce bugs at a lower rate. Plus it makes your software more appealing to customers.
All things being equal customers will pick the higher quality software over lower quality software. Which means higher sales. More sales may allow you to under cut your competition by dropping your price leading to more sales. Or you could say that our software is better so we charge more. This can also potentially have a positive impact on sales because people often do believe that higher prices correlate to better quality.
When you have more money coming in you can hire more and better engineers to write more high quality software.
If you go the other way and cut quality in an effort to reduce the time to market, which only really works when you are near the end of a development cycle. You negatively impact your income, assuming you have the competition of better software. In any case if you are the only producer of some software and it is profitable you will eventually find that someone will start competing based on quality.
At some point bad software can get so bad that one should consider declaring personal technical bankruptcy and move on. This may be a job change or if you are lucky only the cancellation of your project.
The technical debt that one engineer can accumulate in a career of programing can easily require 2 or more careers of other engineers to pay down.
A pay as you go policy says that you fix all bugs as they are discovered before writing new code. While this lowers your productivity initially. It helps you to develop better habits that introduce bugs at a lower rate. Plus it makes your software more appealing to customers.
All things being equal customers will pick the higher quality software over lower quality software. Which means higher sales. More sales may allow you to under cut your competition by dropping your price leading to more sales. Or you could say that our software is better so we charge more. This can also potentially have a positive impact on sales because people often do believe that higher prices correlate to better quality.
When you have more money coming in you can hire more and better engineers to write more high quality software.
If you go the other way and cut quality in an effort to reduce the time to market, which only really works when you are near the end of a development cycle. You negatively impact your income, assuming you have the competition of better software. In any case if you are the only producer of some software and it is profitable you will eventually find that someone will start competing based on quality.
At some point bad software can get so bad that one should consider declaring personal technical bankruptcy and move on. This may be a job change or if you are lucky only the cancellation of your project.
Friday, April 25, 2014
Unicode anyone?
As you may have guessed from my previous post I've been digging into Unicode. One may think it odd that I haven't done this sooner, but when working on legacy applications it is hard to justify supporting more modern things when you spend all your time trying to make legacy software better.
Fortunately, I'm now in a position of writing fresh new code that can be built on a strong basis. As I thought through many of the things I wanted to do I fairly quickly came to the conclusion that moving forward I needed to write things using Unicode.
Should be easy. Unicode has been around for 20 years now so, unlike the mere 3 years since adoption of C++ 11, everything should work and the skids should be smooth and well greased.
Of course not!
Windows seems to like UTF-16. That's fine. Other operating systems seem to like UTF-8 or UTF-16.
Out of the box on Mac and Ubuntu I can do something like the following:
std::cout << u8"Α-Ωα-ω\n";
Of course on Windows not only is the u8" syntax not yet supported, but even if you get a std::string with a proper UTF-8 encoding that won't work either.
Turns out that I can set the cmd.exe console encoding to use UTF-8, and it works great if I use printf for my string, but std::cout doesn't work. To top things off Microsoft decided to explicitly disallow UTF-8 in their std::locale implementation. So I can't tell std::cout to send things to the console as UTF-8. Instead it appears that I will need to use printf or find another obscure way of outputting Unicode in my unit test console based application.
I'm not sure what this means, but it does give hope to those worried that the machines will take over. It will likely take them several decades to figure out the mess we have made with software.
Fortunately, I'm now in a position of writing fresh new code that can be built on a strong basis. As I thought through many of the things I wanted to do I fairly quickly came to the conclusion that moving forward I needed to write things using Unicode.
Should be easy. Unicode has been around for 20 years now so, unlike the mere 3 years since adoption of C++ 11, everything should work and the skids should be smooth and well greased.
Of course not!
Windows seems to like UTF-16. That's fine. Other operating systems seem to like UTF-8 or UTF-16.
Out of the box on Mac and Ubuntu I can do something like the following:
std::cout << u8"Α-Ωα-ω\n";
Of course on Windows not only is the u8" syntax not yet supported, but even if you get a std::string with a proper UTF-8 encoding that won't work either.
Turns out that I can set the cmd.exe console encoding to use UTF-8, and it works great if I use printf for my string, but std::cout doesn't work. To top things off Microsoft decided to explicitly disallow UTF-8 in their std::locale implementation. So I can't tell std::cout to send things to the console as UTF-8. Instead it appears that I will need to use printf or find another obscure way of outputting Unicode in my unit test console based application.
I'm not sure what this means, but it does give hope to those worried that the machines will take over. It will likely take them several decades to figure out the mess we have made with software.
g++ is dead
Lately I've been working on a project that involves using C++ to develop libraries for Windows, Mac, Ubuntu, iOS and Android, with an eye toward quality and portability.
Recently I decided that I needed to build a Unicode class to support a forward thinking basis for things. Of course C++ is generally not Unicode friendly, but it isn't all that unfriendly either. Especially with C++ 11.
So I dig in and start learning and coding and came up with a first pass of my class on Windows using Visual Studio 2013. So far so good.
Now let's go over to Ubuntu where where we have compilers like g++ 4.8 and Clang 3.3 that are reportedly more compliant than Visual Studio.
The first thing I notice is a missing include. The next thing I notice is that the API for std::basic_string that I was trying to emulate is well, just plain wrong! Not even close!
I start digging and find that the GNU standard C++ library is about 6 sigmas off of supporting C++ 11. How can you claim your compiler is C++ 11 compliant but the library is not?
Fortunately, I eventually got to the point of using the new libc++ library which Apple switched to awhile ago and things work, in clang. But when I try to build for Android I can't get the experimental clang support to work. Probably because of 2 or 3 really important steps that I missed, but nonetheless I gave up.
Working through the problem I manged to hack around the deficiencies in the GNU standard library with regard to std::basic_string, but got nowhere with trying to get the codecvt stuff working. Turns out that the other option of using iconv isn't built into Android. I would have needed to jump through several major hoops to compile it and hook it in.
I eventually wrote my own conversion routines between UTF-8, UTF-16 and UTF-32.
So to the point of my post.
It is 2014. C++ 11 was ratified, in March of 2011 and formally adopted in August of 2011. So why, 3 years later, is the GNU library languishing? I would not have expected full support on day 1 or even 1 year after adoption, but 3 years? And the thing that is particularly annoying is that some of the stuff that isn't supported is trivial to correct.
My sense is that the ideas behind GNU and the "Free" software thing are fallacious, which I speculate is a major reason the Apple decided to ditch GNU and that we now see that a more open "open source" implementation in Clang and libc++ is now displacing GNU.
While I will likely continue to keep an eye on the GNU C++ compiler for awhile, it is now a distant #3 in my compiler recommendation list after Visual Studio and Clang. This leads me to the prediction that G++ will be a footnote in the history of C++ compilers. Although they could turn this around, but I'm not going to hold my breath.
Recently I decided that I needed to build a Unicode class to support a forward thinking basis for things. Of course C++ is generally not Unicode friendly, but it isn't all that unfriendly either. Especially with C++ 11.
So I dig in and start learning and coding and came up with a first pass of my class on Windows using Visual Studio 2013. So far so good.
Now let's go over to Ubuntu where where we have compilers like g++ 4.8 and Clang 3.3 that are reportedly more compliant than Visual Studio.
The first thing I notice is a missing include
I start digging and find that the GNU standard C++ library is about 6 sigmas off of supporting C++ 11. How can you claim your compiler is C++ 11 compliant but the library is not?
Fortunately, I eventually got to the point of using the new libc++ library which Apple switched to awhile ago and things work, in clang. But when I try to build for Android I can't get the experimental clang support to work. Probably because of 2 or 3 really important steps that I missed, but nonetheless I gave up.
Working through the problem I manged to hack around the deficiencies in the GNU standard library with regard to std::basic_string, but got nowhere with trying to get the codecvt stuff working. Turns out that the other option of using iconv isn't built into Android. I would have needed to jump through several major hoops to compile it and hook it in.
I eventually wrote my own conversion routines between UTF-8, UTF-16 and UTF-32.
So to the point of my post.
It is 2014. C++ 11 was ratified, in March of 2011 and formally adopted in August of 2011. So why, 3 years later, is the GNU library languishing? I would not have expected full support on day 1 or even 1 year after adoption, but 3 years? And the thing that is particularly annoying is that some of the stuff that isn't supported is trivial to correct.
My sense is that the ideas behind GNU and the "Free" software thing are fallacious, which I speculate is a major reason the Apple decided to ditch GNU and that we now see that a more open "open source" implementation in Clang and libc++ is now displacing GNU.
While I will likely continue to keep an eye on the GNU C++ compiler for awhile, it is now a distant #3 in my compiler recommendation list after Visual Studio and Clang. This leads me to the prediction that G++ will be a footnote in the history of C++ compilers. Although they could turn this around, but I'm not going to hold my breath.
Thursday, January 30, 2014
Software Quality Ceiling
Over the past 3 years I have worked on a project porting a large legacy C++ code base so that it will run on Mac, Windows and in the future other platforms. In doing so we used a number of third party toolkits including Qt.
Portable libraries are nice to use when they work right. The problem is that sometimes you run up against quality issues with the libraries. Things that work right on one platform and not on another or things that simply don't work right on any platform.
Qt is a nice toolkit, but it has many issues. We worked closely with the support team at Digia to correct many issues, but with every software update a new set of problems are added. This means a long and laborious process of first creating a reproducible bit of code, reporting it, sometimes needing to convince the support people that it really is a bug, getting a patch, trying it out and often iterating several times until the patch is correct. Then we hope that the problem is corrected in the next update, which it often is, but about as often has new issues.
Of course Qt is not the only problem. Yesterday I was investigating the new threading library for C++ 11 in Visual Studio 2013. Everything was going along just fine as I was working on a simple ThreadPool class. It worked great except that it would hang when shutting down. I finally did a search and ran across the following http://connect.microsoft.com/VisualStudio/feedback/details/747145. A bug report against Visual Studio 2012. Marked Closed Deferred.
I'm able to workaround the problem by ensuring that my thread pool shuts down before returning from main, but now I'm faced with how to write a class that logically would be global that will correctly shutdown in Visual Studio. I have yet to test it on Mac or Ubuntu, but I'm fairly confident that I won't have the same problem there.
Over the years I have encountered numerous code generation problems with compilers. The good news is that I haven't run across anything recently. But in the back of my mind I continue to worry that one may show up. This is really bad because unless you actually run the code affected in the manner that would exhibit the bug you may never know you have this problem.
Then there are things like the openGL drivers for video cards. In some cases something works fine on one card but not on another. At that point you are faced with answering the question, did I do something wrong? Sometimes I did. In some cases the fact that it worked on one card was wrong. But in other cases I used the API correctly and there is a bug in the driver.
Then there is the unspecified behavior clause in languages. In C++ there are many places in the standard where the behavior is listed as unspecified or compiler dependent. In these cases I would prefer that it didn't compile or at the very least it should crash when I try to use it. In my recent porting project I ran across several cases where the compiler on the Mac crashed on badly written code, but on Windows it didn't. I was happy to find and fix those issues. But they should have not worked in the first place.
All of these issues result in a quality ceiling. My goal is to write code that is 100% bug free. Occasionally, I achieve that for simple cases. More often I have a few bugs that testing should eventually uncover. But if I'm working with tools, compilers, libraries, operating systems, etc. that are buggy I will always be faces with the fact that my software will be of a lower quality than I'm theoretically capable of writing.
As an industry we need to figure out how to break through the quality ceiling.
With that I close, because I have no answers.
Wednesday, November 30, 2011
Optimizing C++ Build Times
I've been fighting improving build times under C++ for years and over that time I've managed to put together a few good practices to help.
1) Use good header inclusion isolation.
Use forward declarations as much as possible.
Use the PIMPL idiom to provide better isolation.
Avoid include only libraries. Some are marginally OK, such as most of the STL. Others, especially certain boost libraries, can create enormous burdens. Wrap these using PIMPL or "has-a" instead of inheritance.
Use good encapsulation by splitting your classes into reasonably small DLLs. This reduces link times and helps enforce other good design practices.
In theory this should be all you need.
But there are some other things that work well.
2) Use IncrediBuild if you are building under Windows.
There are other distributed build systems as well, but at the moment I've not had any luck improving over local builds on today's multi-core hardware.
3) Use multi-core machines.
The more cores the better.
4) Use SSD drives to eliminate disk bottlenecks.
Depending on the amount of code you are compiling you may not need an SSD drive. However, in my current project I've seen a 30% improvement in build times.
I also have found that when using IncrediBuild in combination with an SSD can really improve the throughput allowing you to use many more machines. I've recently done tests where I ran up to 100 agents seeing significant improvements between 50 and 100.
5) Figure out what the hardware bottlenecks really are.
Memory speed. This is still a potential bottleneck in today's hardware.
Memory amount. This is easy, if you are using more than about 3/4 of your memory typically while building, get more.
Disk speed. The disk cache can get into a swamped state where it takes as much as several minutes to finish writing all the cached data. This can lead to randomly long link or compilation times. Use SSD drives to take care of this.
Network speed. If you are using distributed builds then go to at least a gigabit network. Beyond that I'm unsure that there would be any gain. Below that it is clearly a bottleneck.
CPU speed. Faster is better unless the bottleneck is memory speed, disk speed, or network speed.
Number of cores. Since compilation can be highly parallel more is better, unless memory speed or disk speed is an issue. Usually more cores means faster memory so it is normally just disk speed that can slow you down.
6) Turn off active virus scanning.
This should be obvious, but every .obj, .dll, or .exe that you create would get scanned thus slowing your compilations considerably.
I find it amazing how often this is overlooked.
Does this work? Yes. I've taken a build that was pushing 15 minutes and reduced it to about 8 using the methods in 1). I've take a build that was pushing 10 minutes and dropped it to about 7 using and SSD drive. I further cut the same build down to about 4 minutes by taking the number of IncrediBuild clients from 20 to 100, 5 minutes for 50 clients.
Slow builds are productivity suckers. The slowest build I ever saw was 30 hours. I only did that once. At one point I worked on a project that went from 5 minutes to 1 1/2 hours in year.
Somewhere not far past 5 minutes will put you into a state where flow gets interrupted. If you find your flow getting interrupted often, you will be far less productive. Don't let that happen.
Ideally you should strive to keep your builds under 1 minute. I believe this is possible for typical builds with the right encapsulation of data regardless of the total size of your project. While an entire system built from scratch could still take days, your normal builds should not need to take long. If they do, you are wasting resources.
Subscribe to:
Posts (Atom)
