Doug's Software Development Notes

Optimizing C++ Build Times

2011-11-30T22:17:00.001-08:00

I've been fighting improving build times under C++ for years and over that time I've managed to put together a few good practices to help.

1) Use good header inclusion isolation.

Use forward declarations as much as possible.

Use the PIMPL idiom to provide better isolation.

Avoid include only libraries. Some are marginally OK, such as most of the STL. Others, especially certain boost libraries, can create enormous burdens. Wrap these using PIMPL or "has-a" instead of inheritance.

Use good encapsulation by splitting your classes into reasonably small DLLs. This reduces link times and helps enforce other good design practices.

In theory this should be all you need.

But there are some other things that work well.

2) Use IncrediBuild if you are building under Windows.

There are other distributed build systems as well, but at the moment I've not had any luck improving over local builds on today's multi-core hardware.

3) Use multi-core machines.

The more cores the better.

4) Use SSD drives to eliminate disk bottlenecks.

Depending on the amount of code you are compiling you may not need an SSD drive. However, in my current project I've seen a 30% improvement in build times.

I also have found that when using IncrediBuild in combination with an SSD can really improve the throughput allowing you to use many more machines. I've recently done tests where I ran up to 100 agents seeing significant improvements between 50 and 100.

5) Figure out what the hardware bottlenecks really are.

Memory speed. This is still a potential bottleneck in today's hardware.

Memory amount. This is easy, if you are using more than about 3/4 of your memory typically while building, get more.

Disk speed. The disk cache can get into a swamped state where it takes as much as several minutes to finish writing all the cached data. This can lead to randomly long link or compilation times. Use SSD drives to take care of this.

Network speed. If you are using distributed builds then go to at least a gigabit network. Beyond that I'm unsure that there would be any gain. Below that it is clearly a bottleneck.

CPU speed. Faster is better unless the bottleneck is memory speed, disk speed, or network speed.

Number of cores. Since compilation can be highly parallel more is better, unless memory speed or disk speed is an issue. Usually more cores means faster memory so it is normally just disk speed that can slow you down.

6) Turn off active virus scanning.

This should be obvious, but every .obj, .dll, or .exe that you create would get scanned thus slowing your compilations considerably.

I find it amazing how often this is overlooked.

Does this work? Yes. I've taken a build that was pushing 15 minutes and reduced it to about 8 using the methods in 1). I've take a build that was pushing 10 minutes and dropped it to about 7 using and SSD drive. I further cut the same build down to about 4 minutes by taking the number of IncrediBuild clients from 20 to 100, 5 minutes for 50 clients.

Slow builds are productivity suckers. The slowest build I ever saw was 30 hours. I only did that once. At one point I worked on a project that went from 5 minutes to 1 1/2 hours in year.

Somewhere not far past 5 minutes will put you into a state where flow gets interrupted. If you find your flow getting interrupted often, you will be far less productive. Don't let that happen.

Ideally you should strive to keep your builds under 1 minute. I believe this is possible for typical builds with the right encapsulation of data regardless of the total size of your project. While an entire system built from scratch could still take days, your normal builds should not need to take long. If they do, you are wasting resources.

What is the Best Software Process?

2011-11-27T10:16:00.001-08:00

In order to write good software you need some sort of process. As it turns out it doesn't really matter much which process you choose but there are a few key elements to a good process.

1) It must focus on your customer.

The first thing to get right is to solve the problem that is most needed by your customer. This can be a bit hard to determine because in most cases the customer will tell you what they think you can solve rather than what you can solve.

This means that sometimes they want you to do the impossible and you need to scale back their expectations or that they have already scaled back their expectations and what they really need to be solved is easy to do.

There is a real art in figuring this out but it is mostly about spending enough time understanding what it is that your customer does. This often means you need to learn something new outside the domain of computer science.

To me the chance to learn something new is the best part.

2) The process must be iterative.

No matter how you end up doing things there needs to be some iteration involved. This has led to many different process methodologies. All the best ones use some sort of iteration.

Basically this recognizes the fact that you can't create a great design without writing some code. Eventually you need to rewrite the code to deal with changes. Sometimes you get it right to start with, but usually that is for a trivial problem that you can already write the code for in your head.

3) The process should be collaborative.

Writing code by yourself is OK, but two heads are better than one. Code that is written by and maintained by multiple people will get better or worse over time depending on the team.

If it is getting worse then you have a problem with your team. Either you or your teammates need to learn to work better together. Almost always it is you that needs to improve. How can that be you ask?

Well what it boils down to is that most software engineers think their code is better than their colleagues. While that may be true in some cases it is almost always true that you can learn something from your teammates about writing better code.

Looking for that and adopting the best of others will make you a better engineer. When you get to this point then it is probably true that your teammates need to improve. They will be a lot more likely to adopt your good habits if you have first adopted theirs.

4) The process must include testing.

This is obvious, but there are some good and bad ways of testing. While you should test your own code it should always be tested by someone else. Testing should be independent and done by a team that doesn't also do development.

The testing team should develop somewhat of an adversarial role with respect to the code. However, it should be more of a friendly rivalry. The goal of the developer is zero bugs. The goal of the tester is to find many bugs.

Failure to achieve either of these should be a sign that the respective team has room to improve.

5) The process should include documentation.

The first documentation should be written by the software engineer. It should be written, no videos, and it should clearly state what the feature should do (requirements) and how it works (specification).

Each of these can be used to produce other documents, a test plan, user documentation, etc.

6) There needs to be accountability.

This can come in many forms. For example source control maintains a record of who did what and when, a bug tracking database records defect and is used to assign them to the appropriate party, a schedule gives management, marketing, and other stakeholders a prediction as to when they can expect delivery so that they can plan accordingly.

7) Return on Investment Must be Estimated.

Technically this goes under accountability, but it is so often ignored that I call it out specifically.

Software that costs more to develop than it saves should never be started. For example if it takes $100,000 to automate a shipping process where the savings is only $20,000 over the next 5 years then there is probably better things to do.

Not only is this a losing proposition, it is likely that the project will get canceled making it an even bigger waste of money and time. It will also piss off the engineers that work on it.

If you are an engineer starting work on a project like this then make sure management knows it is a waste of resources. If they don't listen, move on. You don't need the aggravation, and there are plenty of software development shops around that do work on profitable projects.

On the other hand if adding a feature for $100,000 will add $1,000,000 a year in revenue then it is most likely a good thing to work on.

Spend your development resources where they will make you the most money. It's good business and it's good for the team.

8) Focus on the End Game.

Often from the time of feature completeness to release can take several weeks. Often one thing, such as finishing documentation, web site, marketing materials, etc. can end up taking a surprisingly long time. These needs can often push back the date of feature completeness by a month or two for a given release date.

Gantt charts are very helpful in planning your end game to ensure that everything that has dependencies on the software being complete are done.

9) Use Technology Only When Appropriate.

There are some technologies that are pretty much a given. For example you need a compiler.

Other things should be a given such as using source control and a bug tracking database.

Then there are technologies that may or may not be helpful. Most things can be done manually. Such as managing requirements, specifications, etc. Tools exist to do these things, but are not necessary to write software. Use them only if they really help you.

If they are black holes where information is poured but never used then get rid of them as they are just taking up time.

10) The Process Must be Simple.

If you can't explain it in 10 minutes to your mother and have her understand it, then it's too complicated.

A complicated process will fail. It won't get buy in from your team. It will be interpreted differently by different team members. It can take a long time to be fully adopted.

11) Use Iteration to Improve the Process

With each release you can sit down and examine your process. Figure out what worked well and what didn't work so well. Focus on the areas that didn't work well and figure out how to make them work better.

One thing that happens frequently over time is that as things improve you hit a release where a the quality slips. This tends to come from complacency. You have to keep working to improve.

One way to deal with this is to keep raising the bar. Recognize that you can always improve some part of the process as there is not a perfect process.

12) Adapt Process to Your Needs.

Don't just adopt a process. No process is perfect and every team has slightly different needs.

For example, a young inexperienced team can greatly benefit from pairs programming while a highly seasoned team would only be slowed by pairs programming. Some would argue that point with me, but 30 years of development experience tells me otherwise.

Adapt process ideas to your team needs. Don't just blindly adopt a process as it won't work if you and your team don't understand why you are doing things in a particular way.

Conclusion

With a little effort you can write good software. With a lot more effort you may be able to write slightly better software. But if it doesn't make your company money, it's a bad process.

The traveling salesman problem is a great example of a problem where achieving the perfect solution can take so much compute time that the salesman could have gone with a quick less than optimal solution and be back from his trip before the computation of the perfect solution is finished.

Don't let your process become such a large cost that you can't get the job done.

Boost Threads and the Data Hiding Zoo

2011-01-08T10:14:00.000-08:00

In C++ the concept of data hiding is implemented via private data. This is a fairly nice concept but it is really more like putting the data behind bars and telling the viewer not to touch it.

Very much like visiting animals at a zoo you can look at all the data and so long as you don't touch it everyone is happy.

However, this also means that we have to see the data even when we don't want to. This has a lot of detrimental effects, not the least of which is that it adds to compile times because of C++'s lame include model.

C++ and for that matter pretty much all languages give us the ability to not only hide the data but to make it invisible. The simplest and most effective method is to put the data that you don't need to know about into a DLL and then provide a simple function call to perform the action that you want to make public.

A good example of this is a dialog. Let's assume that you have a class that has an associated dialog that is designed to edit the class. All you need is a simple function call that takes a reference to the data that you want modified. Then the only public interface to the dialog is a plain old function call.

In some recent work I've been doing involving reducing the time it takes to compile our program I have run across cases where the simple function call prototype and the class for the dialog are in the same header file. This may seem logical since they are related, but by the time you count up all the lines that the dialog class needs to include from all the 100's of header files that it ultimately includes you can easily have 300,000 lines of included files to get at the 1 line prototype that you need.

The problem get's compounded in the file that is used to dispatch dialogs which can end up including dozens of similar headers. Certainly there is overlap so in the end the file may only bloat to about half a million lines of code when they are all included. But why pay the cost of half a million lines of code when a 100 lines will do?

I also spent part of last week encapsulating the boost thread locks and mutexes which on Windows end up pulling in 400,000 lines of code. I ended up with a nearly identical templated set of classes where the headers run less than 100 lines of code total including everything that they include. This was done by using the pimpl (Pointer to Implementation) idiom.

In talking to one of my colleagues I commented that it seemed wholly surprising to me that people as smart as the guys that put together the boost library would impose such a horrendous overhead when it was nearly trivial to avoid it. He commented that this is what you get when you have a header only library. To which I replied, but boost threads requires that you link against a DLL.

The absurdity of a 4 orders of magnitude difference in number of lines of code to implement a simple concept makes me wonder what the implementors were thinking. I can understand an inexperienced or even a somewhat experienced person making this sort of rookie blunder, but these are supposed to be experts.

I am becoming more and more conscious of the fact that many so called experts in the C++ community are really not. They are really smart, but that doesn't make them experts. That just makes them dangerous.

While there are some really good top notch open source software projects out there the vast majority, including boost, are putting out average or below average software.

The same can be said about the standard template library. While many implementations of the STL are good, I have yet to find one that includes less than 20,000 lines of code for something as simple as a string. Ask yourself why std::string has to be template. The correct answer is that there is no good reason.

Template library creators have lost sight or never grasped the basic programming concepts of data hiding, data invisibility, or the proper design of a library.

Rather than hide the implementation in a .cpp file they expose everything to your compilation unit through header includes and force the compiler to work excessively hard to compile the same thing over and over again.

Rather than give us the option of using the explicit instantiation model and do forward declarations of template classes they force upon us the one size fits all mentality of exposing our compilation unit to tens or hundreds of thousands of lines of code when a couple of dozen is all that is needed.

I don't get it. Why isn't the C++ community up in arms about this irresponsible imposition of insanely heavy compilation costs?

I suppose part of the problem is that adding 1 more straw to the pile isn't noticeable. Eventually you break the back of the proverbial camel or boil the proverbial frog alive.

The problem is that the library creators aren't trying to use their libraries in real world applications. Their trivial test applications compile in seconds and so they don't notice and don't have to pay the cost of the fatal flaw in their designs.

Demand better. If the "experts" aren't going to make the data invisible then we are forced to do it. That adds a little work for all of us that could be done once.

Code Invariance and C++ Templates

2010-12-12T11:45:00.000-08:00

I have been working to improve compilation times recently using all the usual things, interfaces, pointer to implementation (PIMPL) wrappers, forward declarations, etc.

In doing this my biggest target is the boost libraries. These are orders of magnitude larger than you might think. In analyzing the amount of headers that a typical boost header includes I've found them to be 2-10 times the size of a typical STL header.

The STL headers are already bloated. For example <string> runs to about 30,000 lines typically.

First of all std::string is a template to produce 2 common variants std::string and std::wstring. While it could produce more I can't think of any that I would typically need or want.

Design Guideline:

Don't use templates when a non-template implementation is just as good or better.

If you look at the std::string API you discover that you should be able to prototype it in less than 200 lines of code. Now there are arguments for inlining for performance reasons that make inlining it a nice option.

However, most of the time I don't need the performance benefits. It is also very difficult to forward declare a string and the STL implementors don't provide us with one that we can use. They provide but no other forward declarations for any of the container classes.

The result is that any file, even a 10 line file, that I want to use a string in becomes effectively a 30,000 line file.

This is a huge price to pay when the code it generates is invariant. Now I know that compilers are smart and can cache this, and some do a pretty good job. And computers are a lot faster today. But when I end up spending several hours a day waiting for compiles, productivity suffers.

The point here is that templates are effectively programs that generate code. That code only needs to change when the template changes and the generated code needs only to be done once for each combination of types that you are using.

With the design of C++ this is done once for each compilation unit. It is redone anytime a compilation unit is invalid. This leads to an N^2 algorithm on top of needing to parse thousands of lines of code.

There are other invariants in compilation as well. In fact any code that hasn't changed and won't generate different machine code is invariant. Compilers and programmers move invariant code out of repetitive tasks.

We need a compiler design that does the same thing.

Today we see typical compilation times of minutes and hours when in reality they could often be milliseconds. Current if I change 1 character in a header that is included in all of my code then all of it needs to be recompiled. If that character results in a 1 byte difference in the generated machine code, it should be instantaneous.

Effectively this is the same problem as we used to have with typewriters. If I add 1 character I may need to retype the whole page. If I add a paragraph I may need to retype all the following pages. This is why we used to have many secretaries in businesses. Now we have word processors and we are far more efficient.

Language designers need to apply the lessons of complexity to the design of the language. Library writers need to apply the lessons of complexity to the design of libraries.

A small amount of work on the part of the STL library designers would allow us to have massive savings in the time it takes to compile.

The C++ language already gives us options for making the use of templates less invariant.

Let's take std::string as an example. While I don't think std::string as a template is even marginally a good idea, let's assume for a minute that it is. To give C++ users the most flexibility we need the following:

<string_fwd> - Provides a forward declaration of std::string in as few lines as possible so it can be used to reduce the header load in header files that only need a forward declaration.

<string_proto> - Provides the prototype for std::string without implementing any methods using as few lines of code as possible. This allows me to use std::string in compilation units without having to pay the price at compile time of compiling all the methods in string and then optimizing their inline use and then emitting the template code.

<string> - Provides the full std::string implementation as it stands now so that I can use it in cases where performance is critical and can also explicitly instantiate it where needed.

This pattern applied to all of the STL would be easy to implement and give us more ability to control how long compiles take.

While I suppose I could create a wrapper class for std::string, std::vector and the other containers are much more compelling at templates. I have been playing with doing a forward declaration and that seems to be feasible, but suffers from some extra maintenance.

With any luck someone will start listening and fix some of these issues.

Boost Filesystem Version 3

2010-12-12T10:28:00.000-08:00

I've been looking at the boost filesystem as a possible replacement for file manipulation and directory parsing.

In doing so I discovered that the comparison operator for boost::filesystem3::path is horribly slow. I'm not entirely sure why, it may be the iterator.

In doing this I implemented a simplistic wrapper to avoid the massive include load. The filesystem 3 path.hpp include looks like it includes well over 100,000 lines of headers. My wrapper runs only about 30,000 lines almost all of which is because I'm including <string>

In my operator< comparison function I replaced the compare with a .native() string comparison and found what appears to be about a 10X improvement in performance on the Mac.

The more I dig into boost the more I'm convinced that it must be used with extreme caution.

The Include Model: Yet Another Bad Idea

2010-12-04T11:48:00.000-08:00

It's probably been at least 20 years now since I first recognized that the include model for programming languages was a bad idea. Others have been saying this for much longer.

First let me explain the model. The idea is that you have a set of items that you want to be the same in many contexts. For example in C++ you have a class prototype, or defines, etc. In HTML you may have style sheets and other things.

So what happens is you code up these items in a file that gets included at the point they are needed. You write it once and then use it many times. So far so good.

Now the not so obvious problem is that each include may include several others. And those in turn may include more. The result is an explosion of code. In C++ we put include guards in to prevent multiple inclusion and this helps a lot, but even so a single include can lead to thousands of lines of code.

In the C++ standard template library the inclusion of something like <vector> is about 20,000 lines of code.

So why is it that we keep making the same language design mistake?

Probably because the model is so simple.

The result is that we can end up with compilers taking several minutes or even hours to compile code.

Why should it take more than a few milliseconds to update my executable when all I did was change one line of code?

My challenge to the C++ and other standardization committees is to once and for all expunge the idea of an inclusion model for the language from all standards.

In reality all that is needed is to move to a database model of compilation. Conversion from the current text based model would be required, but in reality this is already done by compilers now.

I'm sure there are challenges, like what to do with preprocessing, but those can be overcome.

While I'm not an expert on compilers, I do know something having written a couple of them in my career. I've also talked to people who write compilers that agree that this is feasible and in fact behind the scenes a lot of stuff is done that is very close to what we need.

All that we are missing is for the standardization committees to move forward.

Let's not waste time on more language niceties until we fix the fundamental flaws.

The include model is a fatal fundamental flaw of many languages that collectively costs software companies billions of dollars in lost productivity.

As I write this I also am reminded that char * is another bad idea that came from the same guys that brought us #include. char * as a C and C++ language construct is also responsible for billions of dollars of lost productivity over the years. Think about how much time used to be wasted looking for that off by 1 error or security holes that were caused by buffer overruns in char * arrays...

It is unfortunate that some of these ideas don't get fixed sooner. I sometimes wonder if the language creators are writing code. If they were it would seem like they would be a lot more motivated to fix these issues.

I would love to hear some counter point arguments as to why the include model is still with us.

Unexpected Technology

2010-12-04T11:06:00.000-08:00

It is the technology that is unanticipated that excites us the most. For example we have recently seen the ability to scan credit cards on a smartphone. This is probably something that many have thought of over the years, but it is something that I personally didn’t think of before it happened.

When people see technology that they didn’t anticipate it’s like a magic trick where the outcome is a total surprise to the audience.

The real trick is coming up with a bit of technological magic that is unanticipated.

Blindly Following the Advice of Respected Writers

2010-12-04T10:40:00.000-08:00

Over the last couple of weeks I have had two occasions where I found fatally flawed arguments in the writings of very respected writers in the C++ community.

In both cases my gut told me when I first read these things that they were wrong. In one case I and others took the advice and ran with it for several years only to be burned later on by what should have been obvious.

Case 1: In Scott Meyers book on Effective STL he gives the advice to use std::for_each for three reasons. He contends that it is faster, less prone to error and more readable. However, after using this advice and seeing how these things were implemented, especially in conjunction with the use of boost::bind or tr1::bind it is clear that all three arguments are simply wrong.

First it is not faster. If you analyze how an stl iterator translates into C++ code you find that iterators are in fact almost identical to the fastest possible hand coded C++ loop that you can write. In addition he contends that the implementors may not write the fastest possible iterators and then in almost the same breath he says that the implementors know more about their implementation than we do so they can write a faster for_each.

To which I say if they can’t write a fast iterator what makes you think they will write a faster for_each.

In analyzing the for_each implementations they are a thin wrapper over a for loop. But with a side effect. The side effect is that end() is called only once before the loop. Most of the time this isn’t an issue, but sometimes it is.

The second contention is that it is less prone to error. Unfortunately, this is simply not true. The syntax is harder to decipher and because of that it is very easy to create a functor that does extra copies and so in the end it is slower.

Finally he contends that it is more readable. Which in my experience is clearly false. Stuffing too much information on one line is bad.

Consider

for(int i = 0; i < n; i++) { temp[i] = b[i]; b[i] = a[i]; a[i] = temp[i]; }

vs.

for(int i = 0; i < n; i++) {

temp[i] = b[i];

b[i] = a[i];

a[i] = temp[i];

}

Which is easier to read. To me the second is the clear choice and most people I know would agree.

Stuffing this into a std::for_each would mean writing a functor, using something like lambda operators etc. However, the more obvious solution to making this more readable is to write a function.

So in the end in the light of day the analysis, my gut, and actual experience says that it is bad advice.

Case 2: In the book C++ Coding Standards the authors make the statement that you shouldn’t have a standard for the placement of things that don’t affect the readability of the code such as curly braces. The interesting thing is if you carefully read what they say they state that they have chosen a formatting for the book to enhance readability. Which on the surface should be enough to call into question their logic.

They also suggest that you be consistent and that you use the standard in the file you are in.

The problem is that if you work with a team of people in C++ you are often adding virtual functions to a class hierarchy, which may be a symptom of bad design, but is more often than not just part of extending the capability of the software. In doing this you will often be working across a class hierarchy that is maintained by more than one person.

This leads to a clear problem. Let’s say you are adding a virtual function. Do I write it like this:

bool T::supportsMyNewFeature() const {

return true;

} 

bool T::supportsMyNewFeature() const

{

return true;

}

If one file uses one style of curly brace formatting and another the other then I’m constantly having to switch between styles depending on the file.

If I impose my style instead of using the team standard then files quickly become inconsistent and there is no file specific style so the only reasonable option is to default to the team standard.

If I use the team standard then files that don’t use the standard quickly become inconsistent and so the only reasonable option is to default back to the team standard.

If I use the team standard then there is no problem.

There are cases where imposing a team style may not be needed. Such as cases where one person is responsible for an entire module. While this works, generally over time that module will eventually be taken over by someone else or by several people who may not share you view on formatting. They either have to use your formatting or reformat to theirs or to the team standard.

Team standards make these decisions easier.

So back to the main point of this post. Don’t believe everything you read. The writers of these books are very smart people, but they do make mistakes. Some of them are obvious when you examine them. Others may not be so obvious and lead you down a path that leads to problems. When you realize you have been mislead it is time to let everyone know and change how you do things.

BOOST_FOREACH: A Bad Implementation of a Good Idea

2010-12-04T09:58:00.001-08:00

I am more and more often running across constructs, especially from boost, that I find to be intuitively poor.

BOOST_FOREACH is the latest one that I've looked at.

First let me state that I like the concept of for each as is implemented in many languages. C++ doesn't happen to implement it and some really smart people decided that it would be cool to create something that has the expressive niceties of for each.

One result is BOOST_FOREACH. However, the include file runs to over 67,000 lines of code and in my opinion the readability is worse.

Why?

Consider the following snippet.

BOOST_FOREACH(T & a, myClass.getAVector()) {

foo(a);

}

Here is another way to write it:

T &aVec = myClass.getAVector();

for(std::vector::iterator it = aVec.begin(); it != aVec.end(); ++it) {

foo(a);

}

And another which is more like std::for_each:

T &aVec = myClass.getAVector();

std::vector::iterator begin = aVec.begin();

std::vector::iterator end = aVec.end();

for(std::vector::iterator it = begin; it != end; ++it) {

foo(a);

}

Which is easier to understand?

Is the invariant getAVector() removed from the loop?

Is end called for each loop or once at the beginning like std::for_each?

Did you know that std::for_each is more like my second example than the first?

If there is a side-effect of calling getAVector() what is the behavior?

What happens if the size of the container changes during the loop?

Is the expanded macro efficient?

Can everyone on your team answer these questions?

Can new developers that may come into your team answer these questions?

Should we spend 67,000 lines of extra code for every compilation unit that uses BOOST_FOREACH to save a few dozen characters?

My conclusion is that BOOST_FOREACH is not more readable and the cost is not worth the header load even if it were.

I will not be using it.

Boost: Not Ready for Professional Projects?

2010-12-04T09:53:00.000-08:00

Over the last several weeks I’ve been doing some general code maintenance that is a necessary part of software development. One of the problems with using C++, or any include based language for that matter, is the issue of the geometric growth in number of lines of code that can potentially get included.

If you have an include file that includes 2 other files and those in turn include 2 others you can see that very quickly you can get to a point where the total number of lines of code included runs into the hundreds of thousands of lines.

I got curious about just how big of a hit this was and decided to write a simple include file analyzer to see how potentially big this was. My analyzer is pretty brain dead in that all it looks for is #include statements. It doesn’t respect the preprocessor so the numbers it spits back are pretty much worst case. But it does give some very useful results.

For example many of the stl containers like map, vector etc. tend to run about 20,000 lines and a couple of hundred includes. This was surprisingly large to me. The good news is that there is a lot of overlap, so if I include both map and vector it’s only incrementally larger.

In our work we use many of the boost constructs, like boost shared_ptr. These are some very nice and useful constructs that are an important conceptual addition to the language.

When I analyzed the boost headers I was very unpleasantly surprised at how big some things were. For example the boost shared_ptr was running about 200,000 lines. Of course some of this is platform specific code, but the point is that it is a massive hit.

As a sanity check one of my colleagues ran all of our code through the preprocessors and dumped the output into a single file. It was about 25 gigabytes of data!

While it is obvious that we should be aware of these issues, in general it is really easy to miss the cost of including header files.

Boost is providing a great service to the C++ community, but they too have fallen victim to the geometric growth of include based libraries.

Since shared_ptr is now part of tr1 we have moved over to using it. In analyzing the tr1 memory file inclusion it is an order of magnitude smaller than the boost implementation.

I write this not to say that boost is bad, but to say that you need to be aware of the cost of using it. To that end I suggest the following:

Don’t include any boost header in any commonly used header file. Use the Pointer to Implementation (PIMPL) idiom, forward declarations or simply don’t use boost.
Prefer stl solutions, although you still have a massively large header file load. But 20,000 lines is nothing compared to 200,000 lines.
Always check the cost of our includes. It is often a bit like boiling a frog in water in that a single change is hardly noticeable. But then one day you look up and your compile time is 30 minutes.

In any case it is my hope that the writers of boost take a look at putting it on a diet. It has become a massively bloated beast.

Problem Solving Skills Revisted

2010-10-16T14:02:00.000-07:00

Over the past several years I have been contemplating the idea that one needs to have good problem solving skills to be a good software engineer. I happen to agree with this assessment and work pretty hard to find engineers that are good at solving problems.

One way to do this is to ask them to solve certain programming problems, such as reversing a string, or writing a recursive or iterative factorial function.

We also ask less well defined problem solving questions, such as how many gas stations are there in Seattle. All of these questions are designed to figure out how a candidate approaches problem solving.

This works really well, but in the end while solving problems is a great skill to have most software suffers from a set of problems that are conceptually trivial to solve.

These problems mostly fall into the category of poor organization. It is often the case that highly creative people are often horrible about organization. While they know intellectually how to organize, they don't.

In some cases this leads to code that is highly disorganized which becomes a big problem to maintain.

It is my opinion that a software team needs a variety of skill sets. One of which is the organizer. These are people who love organizing code to make it more readable.

They understand well what a good and poor class hierarchy is and like to make the code better by massaging code through refactoring to be better. Much of the time these types of task require 1% inspiration and 99% perspiration.

The skill set of this type of engineer is one who has the ability to stick with a repetitive task for a long period of time, one who enjoys making the code better, has reasonable problem solving skills, and the ability to leverage the rest of the team to figure out the best way to solve problems.

In addition to this there is also the need for a manager of the team to be a good politician/salesman that gets the rest of the team on-board with all the tasks that the organizer does. One thing that I find frustrating is how often a small percentage of engineers have a very strong affinity to a code formatting style. Selling these engineers on accepting the team formatting style is very important.

The organizer must also support and understand well the team formatting style as they will be imposing this on the code as they go through it. If they disagree with the team standard and don't follow it they become useless as an organizer.

While it is nice to get both a highly skilled problem solver who is also an organizer, often these two skills don't appear in the same individual and there is no need to have exceptional problem solving skills to organize code.

When building a team you should try to get a good mix of skill sets. I'm not entirely sure how many organizers you need on your team, but I'm inclined to think that you need at least as many organizers as problem solvers.

What do Computer Scientists Do?

2010-10-16T12:00:00.000-07:00

Over the past couple of weeks I have had the opportunity to talk to some computer science students as well as a number of professors and others involved in a Computer Science (CS) education. This topic came up several times.

In one context it was an issue that some students get into a CS program based on some vision of what it is that a computer scientist would do, only to find out that it is quite different. The result is that these students drop out of computer science.

While learning that CS is not for you early is great, when we look at this problem from the flip side this may mean that many who would like to study CS think it is something else and thus don't even consider CS as an educational choice.

In recruiting interns I ask what they expect to get out of an internship. Often the answer is that they want to find out what it is like to work in industry.

The answer to this question is that it depends. There are many routes that one can take with a CS degree.

One can variously be a software engineer, do research, teach, be a software test engineer, be a user interface designer, etc.

Since my expertise is primarily in the area of software engineering I will limit this discussion to that area.

In a nutshell a software engineer is responsible for creating a software solution to solve a customer problem.

An example of this is the blog you are reading now. The problem being solved is to provide a simple way of posting a blog. A collection of tools that allow editing of the blog, the style, etc. have been created that make this easier than writing straight html.

In software engineering one is expected to become an expert in at least two knowledge domains. The primary domain is that of designing, writing and testing software. The secondary domain is in the problem that you are trying to solve.

For example at one point in my career I worked on printing for a CAD program. To do printing correctly I had to learn about many things including paper sizes. I ended up learning more about paper sizes for CAD than pretty much any of our customers that use the program.

Often software engineers don't spend enough time in the secondary domain learning what their customer needs. This can lead to poorly designed solutions to the customer problem.

Let's take an aside here and talk about customer need vs. want. It is a classic mistake to give a customer what they ask for. The problem is this, the customer is not an expert in the software domain. Thus when they ask for something they are asking from the standpoint of their limited knowledge of software.

By taking the time to really learn what the problem is that they are trying to solve you as a software engineer become an expert in that problem and can apply your expertise of software to come up with a solution that the customer may not have thought possible that is far better than anything they could have suggested.

On the flip side one shouldn't discount what the customer is asking for as it may be that their idea is better than what you would come up with.

Software is tightly coupled to technology. Often it is the technology that attracts students to CS. As a software engineer it is your responsibility to match the right technology to solving the customer problem.

There can be many competing technologies. For example there are many programming languages such as Java, C, C++, C#, Perl, Python, Ruby, Basic, Fortran, Cobol, Lisp, and dozens of others.

Choosing the right technology can be tough. That is where continuous education in what is available is important. It is also true that in some cases there is no best choice, or that there are many good choices. In some cases the right choice may be to use the technology you are most familiar with as that will allow you to finish the task soonest.

In other cases it may mean you need to learn something new. For example in the last year I've spent time learning more about XML, .NET, Qt, Ruby, CMAKE, and several other technologies.

Over time it becomes more and more difficult to be an expert in various technologies so often software engineers specialize in certain areas. For example I've invested a great deal of time in learning how to program in C++. While I can program in many other languages I spend most of my time using C++. As such it becomes the language I gravitate to when there is no clearly better choice.

So once you have made your choice of technologies to use to solve the problem we get into the real grunt work, writing the software. This is typically where most engineers spend the majority of their time. The decisions that you make in choosing the technology can be crucial to how hard or easy this is, but ultimately it gets down to making it all work.

Being able to write bug free, maintainable software is a skill that requires a lot of effort. Some that may be able to make good decisions about hardware or solving problems don't have the attention to detail that is required to be good at this.

Some people view software as a means to an end. They think that it doesn't matter what it looks like so long as it gets the job done. These are the guys that would be happy with a sports car that has a big engine, great suspension, transmission but the body has dents, bondo and rust.

The code you write should be a work of art. Most people who drive a car don't really care what the engine looks like so long as it is reliable and does what they expect. To a car enthusiast what the engine looks like is as important as the paint job, interior, etc.

One way to put this is that the ugly of software will eventually become visible to the customer. Usually this is in the form of bugs. Pride of workmanship, attention to detail, continuous improvement to the code are hallmarks of what a software engineer does.

This can be as trivial as formatting the code in a consistent way or abiding by a consistent naming convention or as complex as sweating the details in a very complicated algorithm.

Another aspect of software engineering is one of economics. As a software engineer you are expected to keep the bottom line in mind. Unless you are independently wealthy and can do whatever you like eventually you need to make some money. Whether you are working for yourself or a company the bottom line is important. The work you do needs to pay for the time you invest in it.

This is where you need to become good at time management. People who are poor at time management make poor software engineers or they require a lot more handholding from their managers. Managers don't like doing this in general so you need to be good at time management.

You need to know how long it takes to do stuff. I have found that most people tend to be horrible at this. I have lost count of the number of times that I have underestimated the time it will take to do a task. What this means is that you need to account for this tendency when you estimate how long it will take to do things. Here are some of my thoughts on time estimation.

There is obviously a lot more to software engineering that I can cover in this post, but the basic idea is that software engineering is about communicating between people. Sometimes people have a picture of a software engineer as someone who spends all night programming and is very non-social.

While there is a place for people that fall into this category in software engineering they need a lot of extra direction and management to keep them on track. Often these highly creative people come up with brilliant solutions to problems that no one cares about.

The really good engineers spend as much time working with people as they do with their nose in code. As a software engineer it is your job to bridge the gap between real life and technology.

Evolution and Software Engineering, "Survival of the Fittest Code"

2010-03-16T21:43:00.000-07:00

The theory of evolution, as postulated by Charles Darwin, and further extended by Herbert Spencer, who coined the term "Survival of the Fittest" has an interesting parallel in software development.

Anyone who has worked on any code base for an extended period of time knows that some code is in a constant state of flux. Often that same code is where most of the defects occur as well. At the same time there is code that almost never changes.

In software development methods a lot of emphasis is place on things such as code reviews, design patterns, requirements, testing and many other concepts that are supposed to create good maintainable code that is relatively defect free. In practice these methods generally don't produce exceptionally better code.

I have seen a lot of well written, well designed code that is easy to read and maintain that is also in a state of flux. Usually this is only for a relatively short time, but sometimes things take a really long time jell no matter how much good design work was done up front.

I have also seen a lot of poorly written, badly designed code that is hard to read and difficult to maintain that hardly ever changes. And in many cases this code is core to the survival of the software program.

Often very good programers who take a look at the code label it as "bad code" because it is really hard to understand.

And to a large extent they are correct.

The problem is, it works, it does the job well, it's bullet proof, not by design, but because over time it evolved through many fixes to the point that it works really well.

This code survives. Often much longer than really well written code.

In the end who is to say that ugly code is bad code. Sure it may be hard to maintain, but if it does the job it was designed to do and doesn't require much or any maintenance then maybe it really is good code.

Now I'm not advocating that we write ugly code. However, this points out that the ultimate measure of good software is that it does what it was designed to do.

When we write code we should strive to make it maintainable and readable by others, but in the end if it doesn't do what it is supposed to do it is garbage that will get thrown away or refactored to do the job.

Another concept that "Survival of the Fittest Code" encompasses is that code evolves over time. In other words creating good software can be achieved by constant fixing and massaging of the code until the customer is happy with it. The fewer iterations to achieve this the more successful we will be.

Traditionally we call this the "Test and Fix" paradigm of software development.

When we look at DNA we find that a random or planned breeding produces results that are possibly good and possibly bad. The customer is the environment that the offspring has to live in. If the environment is really harsh to the offspring it may die before it reproduces. If on the other hand the environment is really easy for the offspring to survive in then it will thrive.

I have observed cases where an unintended "bug" became a useful "feature". In some cases this feature was unknown to the engineer and later on when they fixed the bug the customer gets really upset that the feature they were using is now gone.

This sort of interaction will more than likely result in the reinstatement of the feature. Since the engineer now knows about the use case of the customer they have the opportunity to evolve the software in another direction that the customer likes.

"Survival of the Fittest Code" can be an effective software development strategy.

In fact I believe it may be the single most effective software development strategy.

Just think how much more effective it could be if we combine it with just a little bit of planning and good coding habits.

Future of Operating Systems

2009-10-12T10:34:00.000-07:00

I've been watching with interest how the operating system wars are playing out. The traditional war has been dominated by Microsoft with players like Mac and linux picking up a significant but minor role.

All of that is changing now. Today smart phones are more capable as computing devices than the hardware that these other operating systems were developed on. Which begs the question why don't they use one of these operating systems?

In fact they do. Google, Nokia and probably others are basing their operating systems on linux and of course the iPhone from Apple is based on the Mac OS which is based on unix.

They try to hide this as much as possible, but it gives us a peek at the future. The biggest reason that linux or unix work so well is that these operating systems were well designed from the start and run on very light weight hardware.

Of course Microsoft has their CE operating system and will continue to push that as well.

One thing that has puzzled me for awhile is why each vendor seems to want to create a closed system. Software developers who develop for one smart phone will no doubt want to develop for all smart phones. The typical way this is done is by supporting applications that run within the web browser. This is a great strategy, but I think it is a bit short sighted.

In talking to people and using online applications, such as the editor for this blog, I have found that while browser based applications seem to work, they have issues. The most notable is frequent updates that break things. This is fine for social networking or casual blogging, but it isn't fine for a mission critical application that you use to make money.

For example, if you are a reporter that has a daily blog, you want the software that you use for blogging to work the same today as it did yesterday. If one day you start using it and it's broken because a bug was introduced, you want the option to fall back to the version you used yesterday.

Some sites do a good job of this, but many do not.

Also, while many applications do require internet access, some do not. If an application doesn't require internet access or it only requires occasional access then it seems logical that it could be independent of the browser.

Most applications written up until a few years ago didn't have any internet connectivity.

What we have is a very large set of applications that could be potentially ported to a smart phone and possibly be very useful there. These applications are often only Windows based or linux, or Mac. Some have been ported to all the major platforms.

One of the portable application frameworks is QT. QT was recently acquired by Nokia and now Nokia is creating a smart phone operating system that supports QT. I believe this is a very astute move by Nokia. It is the kind of move that I would hope other vendors recognize and step up to supporting QT on their smart phones as well.

While every vendor would love to have their smart phone be the phone that everyone uses, the fact remains that the more applications that you can offer for your phone the better it will sell.

We saw Apple initially close the door on apps saying that they should be developed to run in Safari. They quickly changed that and we saw a huge number of apps written in a short period of time.

Now we are seeing all the smart phone vendors putting together their answer to the iPhone. If they want to beat Apple then they need to adopt a standard API so that software developers can write applications that can run on any phone. Open systems will ultimately win.

We have seen linux start to take over the OS market and ultimately I believe all operating systems will be based on linux. There is really no reason that even Microsoft can't move to linux and in fact there have been rumors that they are ultimately going that way.

Creating standard API's that are supported on all devices is reasonable and I believe necessary in order for hardware vendors to stay in business. Think of the operating system as if it were the road and the application as if it is the car. Every car can drive on every reasonably smooth highway. Certainly there are special cases, but in general if you were to build a car that could only be driven on half the highways you it wouldn't sell.

I will watch with interest the unfolding of the smart phone operating system wars. It is my belief that the winners will be those that embrace open standards, like linux and QT. The losers will be those that try to create a closed standard.

While Apple hit a home run with the iPhone if they want to remain a viable competitor they need to embrace an open standard so that developers can develop applications for all smart phones. While they clearly have an advantage today, if the combined market of other smart phone makers were to adopt many open standards they could be a minor player a few years from now.

Ooh! Shiny Thing

2009-09-30T07:10:00.000-07:00

Over the years I have seen a lot of new technology come and go. Some of it sticks, some of it was just a bad idea. The problem is that often we as software engineers tend to get excited about some new bit of technology or a design pattern etc. This is the "Ooh! Shiny Thing" syndrome that many of us share.

While I'm a fan of new technology, it is a double edged sword. What tends to happen is that some new technology is used in one place in the code but other places that were using and older technology don't get updated. Now there are two ways of doing the same thing. Then someone else comes along and uses a third technology for the same task.

Maintaining a code base having several ways of doing the exact same thing means that you need to learn each method. While not a bad thing to learn, it takes time. Time devoted to maintenance can be costly. Given that pretty much any technology will have bugs, you are also taking on that problem as well.

Over the years I've found that the shininess tends to wear off of new technology pretty fast.

Use new technology, but be careful when doing so. In an ideal world the new technology should replace the existing technology that does the same thing. But only after it has been proven to be reliable and viable for the future.

One of your best gauges is the old-guy. Sure he may be old and set in his ways, but there are good reasons for this. It isn't that he can't learn new things. Certainly as we age it is more difficult to learn new stuff. Of course there are some that are unwilling to learn anything new. You should ignore them.

The ones you want to use as a gauge are those that are continually learning new stuff. Look at what they are learning. Let them know about what you are interested in. They may have some insight that say's it's good or bad. Take that input and use it to make a more informed decision.

Recently I've invested quite a bit of time learning stuff out of the boost library. Some of it is very good. For example shared_ptr is so essential that it has been pulled into the next C++ standard.

Other things are less compelling. Some of them while nice in theory will put a world of hurt on your compiler. For example boost::format while a really nice way of doing type safe formatting in a printf sort of way will produce a very large amount of template code. In many cases the much more lightweight stringstream is just as readable.

This isn't to say that you shouldn't use it, but what appears nice on the surface can have some very serious side-effects that are not at all apparent.

Another easy to abuse boost function is bind. In some cases it can make C++ into a write only language. Once you get a bit beyond the simple syntax to bind a function into a std::for_each the code can become almost unreadable and also is also susceptible to errors such as passing by value instead of by reference.

I'm sure that you can look at your own code base and find similar cases where someone was overly clever.

Just because it can be done doesn't mean it should.

Be careful with your new technology. Before you use new technology make sure the rest of your team is OK with it. Getting their buy-in is important and the act of doing so may change your mind about the viability of the technology.

Do What You Enjoy

2009-09-19T09:32:00.000-07:00

While at some point in time we all need to get to the bottom line of making enough money to live and perhaps a bit more to do some of the things we enjoy, doing a job just because it makes you money is ultimately unsatisfying, unless making money is what excites you.

I happen to enjoy writing software. I also enjoy other things that are creative, such as the occasional remodeling job, blogging, cycling etc. The software thing is what currently pays the bills and as it turns out is enjoyable to me most of the time as well.

Every once in awhile you will get excited about something. Perhaps it's a new technology. When that excitement peaks you should follow it. Over the years I've seen many great ideas thrown around that eventually turned out to really big. Very often those ideas were implemented by others who also had that idea, but instead of pushing it into the background pursued it.

The reasons for this are varied. Often the idea seems impossible to implement. I'm reminded of an idea that some of us tossed around when I was in college studying mechanical engineering. We thought it would be cool to have a pen that would be able to write by itself. It would have tiny gyroscopes in it to control it's movement. We also thought that we could have the pen record what you were writing using accelerometers.

Interestingly the second part of that idea is now a reality in the "Pulse Smartpen". This pen uses a camera and tiny dots on the paper to record what you write. In fact it goes even further in that it knows where on the paper you are and so can play back stuff.

Obviously back in 1980 when we had this idea there were massive impediments to doing what we talked about and it was really only an idle conversation that none of us were even slightly motivated to try.

The thing is that there were many more practical ideas that could have been implemented given the technology of the day that I and others were sometimes excited about. But we often let the practicalities of life get in the way of pursuing them.

Or worse we knew they couldn't be done.

Many of the greatest inventions were done by those that didn't know it couldn't be done. Don't let someone else talk you out of doing something you are excited about. Push forward, even if it takes years of working nights and weekends to get the job done. While you are doing that you will learn something. And maybe that will lead to a breakthrough. When it's no longer fun put it away and do something else.

In any case always have something that you are pursuing that excites you.

America's Got Talent?

2009-09-19T08:51:00.000-07:00

I recently watched the "America's Got Talent" show. Interestingly the winner was not the most talented person. Two singers got the top two spots, one country and western the other opera. The opera singer was in my opinion and others that know music a significantly better singer.

This got me to thinking about what makes software popular. In my experience the best written software from user interface design and from the point of view of software quality is not necessarily the most popular. The program that I work on now, "Chief Architect", is much more popular than the program I worked on before "HiQ".

The quality of the code I'm working on now started out significantly worse than what I had thought was average quality code in the industry today. Even now I don't believe the code quality is significantly better than average even with all the work our team has done to improve it. It has improved dramatically over the years as has been attested to by our customers and our software metrics.

The interesting thing is this poor quality code produced a consumer product that has several times been the #1 product in it's category. So clearly code quality isn't what makes a program popular.

The user interface design in this example is also interesting in that it had and still has numerous cases where the design is inconsistent, cumbersome and visually unappealing.

What is it that makes one product more appealing than the other? In the case of "America's Got Talent" there are a lot more people who like country and western music than those that like opera. The same idea applies to "Chief Architect" and "HiQ". "Chief Architect" appeals to anyone who wants to design a house while "HiQ" appeals mainly to engineers who are trying to solve some hairy computational problem.

This ties in with my previous blog post "Know Your Market". If your goal is to make the most profit then the idea of working on things that are really useful should be extended to include the idea of working on things that appeal to a wide audience.

Know Your Market

2009-08-31T08:01:00.000-07:00

A recent article by Tom Demarco "Software Engineering: An Idea Whose Time Has Come and Gone?" is a must read. It makes the "obvious" point that we should be working on projects that are really useful and not working on projects that are relatively useless.

Knowing the bottom line is implicit in the article, but often ignored by some really smart people. Most of the projects I've worked on over the years, even very successful ones, have rarely done the leg work to get estimates of how much money a project could potentially make and even more rare have looked at what the project could realistically make.

Instead the project is often done because the person funding it thinks it's a good idea or the person selling the idea to them is really good at selling the idea.

I like giving engineers the freedom to create and work on things that they are passionate about. This is generally a good idea because they work on things they like and generally if one person likes it many will. However, sometimes this isn't true.

So before you start your next project spend some time figuring out the bottom line. If it's marginal go on to another idea.

Converting a Monolithic Application to DLLs

2009-08-29T14:59:00.000-07:00

Over the last several months I have embarked on a rather difficult task of converting a monolithic application into several DLLs. This application has been in need of this for about 10 years now, but we kept putting it off because it was hard and adding new features seemed to be the thing to do.

Plan from the beginning to use DLLs. It is easy to do, leads to organizational improvements, and will pay off many times over in time savings.

The hard part in all of this is to determine what to move first. This is where being familiar with your code base can pay off. The first thing we did was to start moving some of the core items. As it turned out IO code and error handling came first for us. Then we were able to hit some of the core classes.

When working on this project I kept thinking that I would hit a point where things would be easier to move. However, it seems that there is a never ending supply of couplings caused by poor choices about where code was placed.

There are several design choices that make moving code to a DLL hard. One is when base classes rely on derived classes. Intuitively this seems like it is generally a bad idea. It is amazing how easy it is for this occur though. Often the reasons for doing this don't seem horrible at the time and there often aren't any short term repercussions that make doing this obvious.

About all you have to prevent this sort of thing is peer code reviews and developing better coding habits.

Another problem is collections of classes that are all stored in a single file. Adding a new file when you create a new class isn't difficult, but it does take a couple of minutes to do and it is awfully tempting to save those couple of minutes for that little class you are creating.

Yet another problem are methods that don't really belong in a class. It is a bit of a trap to think that all methods should belong in a class. However, there are cases where you have what I would call a bridging routine that deals with two or more classes but doesn't clearly belong in either. Often what I see is the method put into one of the classes taking as an argument the other class or classes.

This creates a coupling between the classes. It is very easy to end up with couplings like this that end up linking loosely related classes. In a social network it doesn't take very many hops until you find a group of people that you don't know. It is the same with classes. These loose couplings create a network of interdependency. Social networks also often lead back to you. The same can happen with classes.

When trying to move a set of classes that are coupled like this it is very easy for a few loose couplings to spread out and pull in very large chunks of your application. A few cuts here and there is often all that is required break a single class out. The more couplings the harder it is to break out.

Obviously there need to be couplings between classes otherwise you can't have an application. In observing these couplings it occurred to me that just as in other networks there are nodes that naturally form in classes. Some classes have very few links and can be thought of a little bit as leaves. They are referred to by other classes but don't refer to any other classes.

I've observed two categories of nodes. Node classes which have methods that dispatch to many classes and base classes which have many other classes derived from them.

When moving to a DLL the base class nodes need to move early. The dispatch nodes end up needing to move late.

I've been thinking about this observation of node classes and have come to the conclusion that we should design node classes to do little more than be dispatchers to other classes. They shouldn't do any work beyond what is required to do the dispatching.

These node classes will form naturally as your classes form and start interacting. If you don't recognize the formation of one of these classes you can easily find that a node class has turned into a monolithic class that does too much. Since all roads tend to lead to these classes it is a natural reaction to put more and more stuff into them.

In object oriented design there are guidelines, like a base class should be a pure virtual. This idea is supported by the observation of node classes. Make your base classes as simple as possible. Possibly without data or any methods. When you add methods they should be only to support the required communication between classes. If you need more function in a derived class then create a new class that provides that function.

The second form of node class is the dispatcher. It is easily identified because it's implementation will include many header files. These tend to be more problematic for moving into a DLL as they have many couplings. They are also problematic for other reasons.

The problem that I've observed is that these classes are also often classes that were designed to perform a specific task and they grew into dispatchers that pulled in a lot of other classes because of poor design choices.

One trick to dealing with these classes when you port is to create a pure virtual mix-in class that defines pure virtual declarations of the methods you need to access in the DLL. In the end the set of these methods that get created can indicate a set of possible functions that should be pulled into a mix-in class that defines the communication interface.

After working on this project for awhile I'm wholly convinced that all applications should be built as libraries from the very start with a very minimal main executable to kick things off. Besides enforcing better modularity in design this also helps to cut down build and link times which can easily get out of hand.

We use a build tool called IncrediBuild to speed up the compile process. Without it we would be waiting hours to build the entire application. With it the build can complete in minutes. With good modularization you should be able to treat your DLLs like third party products that only get updated infrequently. The more of these you have the less you have to recompile when making changes.

Hopefully you find this information useful. Even with my many years of experience I was unprepared for the cost of moving things to a DLL. Hopefully you can prevent your projects from getting into this state or justify starting to work on this sooner. It will cost more and more the longer you put this off. So getting started on it early.

Bug Debt

2009-08-27T06:54:00.000-07:00

I work on a product that has a backlog of logged bugs that is larger than the team can fix within a single release cycle. It got this way because of a number of factors. Regardless of the reasons we have what I call a bug debt.

Just like real debt bug debt has an interest penalty. There is the cost of entering the bug, the cost of prioritizing the bug, etc. Often there is also an associated support cost with customers who run across the bug. This can also result in lost income due to returns or failure to buy. In addition there can be the penalty of working around the bug in new code.

The point of this is bug debt is real. Given enough time you can create enough bug debt that you don't have time to work on new features. The interest penalty can take enough time out of your day that eventually you could find yourself working 100% of your time but still have a growing number of bugs.

To calculate your bug debt look at the number of bugs you have assigned to you. Then look at the number of bugs you fix on average for a typical period of time when you are writing new code. Use those numbers to calculate how many weeks it will take you to fix those bugs assuming no new bugs are reported. That is your debt.

I like to keep my debt under 1 month. It is generally impossible to keep it at zero as bugs tend to come in at a fairly constant rate.

Once you have your debt calculated you can also look at the average number of bugs reported and figure out how long it will take to work off your debt.

In the long run keeping your bug debt low will allow you to be more productive and spend more time working on new projects.

Of course if you don't create the bug in the first place that is even better. But that is a subject for another time.

Bad code is an infection that spreads

2009-08-27T06:23:00.000-07:00

I've worked on my current project for many years now. When I first started I was appalled at the quality of the code. But I jumped in thinking that I could clean things up and make it better.

While I have made it better over the years, I am disappointed that the code is still no where near the quality level that I would like.

My question is why after nearly 10 years of working on this code base can't I make more progress in cleaning it up. I know my team is capable of writing excellent code and when I look at the new code they write it's a pleasure to work with.

However, when I look at code they and I write that interacts with bad code I find that the bad code doesn't get better. It seems to grow and infect the new code around it. It's a lot like a fungus, like athletes foot. If you don't entirely eliminate the bad code it will come back.

We are often faced with the tradeoff of delivering a new product vs. cleaning up a bad design. If necessary make the tradeoff of buttoning things up and delivering the product. But, when you have to go back into the bad code to fix bugs or to extend functionality, clean it up first. Then fix you bug.

My nephew taught me that when building a house spend a lot of time getting the foundation spot on. If the foundation is off level by a half inch then you have to adapt the framing to correct for that. This domino effect runs all the way up the house.

So clean up that bad code before you extend the functionality or it will infect your new code.

Vista Experience--Moving on to Windows 7

2009-06-14T09:37:00.000-07:00

Since my last post about Vista I have come to realize that it actually is a pretty decent operating system. While I do have to say that MS screwed up big time in several key areas, for the most part they are moving in a pretty good direction.

What MS screwed up

Mainly the biggest problem is that MS insisted on keeping old APIs around rather than deprecating them and moving them out after a bit. In doing this they have allowed applications to continue using outdated modes of operation that went away several OS versions ago. While in the short term this would lead to more complaints, in the long run it allows the OS to lose complexity. We all know that complexity is the biggest issue in software development.

The second thing that MS screwed up was to not recognize that their longstanding effort to keep old APIs around had led to a lot of software that wasn't ready for Vista. Indeed most of it wasn't ready for NT. In particular much software didn't support limited access accounts. This caused one of Vista's biggest features, limited access, to be turned off on a regular basis. I would wager than probably at least 25% of all Vista boxes out there have this turned off so that one or another software can run.

OpenGL support was another area that they messed up. But it isn't just Vista that is a problem. We have found that transitioning from XP to Vista is a trade off in bugs. XP had certain bugs that Vista fixed and Vista brought new ones to the table. Supporting both systems has led to certain compromises.

Finally, they screwed up because they didn't deal with the press that popped up with faulty claims. For example, I've heard that Vista is a resource hog and is slow. The problem is that I've found it to be significantly faster in many cases. Mostly due to the fact that with desktop composition turned on it doesn't need to redraw repeatedly.

There are cases where it does seem to bog down from time to time, but XP did the same thing.

Vista is Good, Windows 7 is Better

I've more recently switched to using the Windows 7 beta and subsequently the RC. It is stable and has some nice additions to the interface that are pretty nice. It is in fact becoming more Mac Like.

However, if you are waiting for Windows 7 to cure all the Vista problem, think again. While MS has addressed some of the compatibility issues with old software there are still problems with UAC. I'm still having to run DevStudio with UAC turned off in order to use IncrediBuild. And there are certain debugging tasks that can't be done without turning it off system wide.

If you can't make your development tools play well with your OS then what kind of a message are you sending developers?

Window 7 looks like it will be well received and should help MS turn the tide on losing ground to the Mac. Although I hope not. I think that we need more parity in OS market share percentages in order to improve the overall competition.

Heres hoping that Apple releases a nice response to Windows 7 that pushes things to the next level.

Vista Woes

2007-12-09T11:28:00.001-08:00

Yeah, I know, everyone has been whining about Vista. After more than a year of sitting on the side running XP 64 I finally decided it was time to start fixing some bugs that only occur under Vista. So I had my IT guy install Vista 32 and 64 bit along with the XP 64 so I could boot to any of the systems.

After 4 days I finally had to insist on getting at least my old XP 64 back up and running so that I could finish off some work that I had started there. At the moment I'm told that there are two versions of Vista on the machine that I can't boot to.

I'm not sure why things took so long but it seems that a large percentage of the time was spent trying to get the installation CD Keys accepted by Vista.

I don't know about anyone else but 4 days to add a hard disk and install the two systems seems a tad bit excessive.

I say this while happily blogging from my Mac which I upgraded to Leopard a week after it was released without any trouble at all.

Recruiting -- Did I Miss the Dress Code Memo?

2007-11-09T19:15:00.000-08:00

Last week I was a the University of Washington recruiting computer scientists. When we go recruiting we tend to wear what most would consider business casual. No ties, suits, etc. but definitely not jeans and t-shirts. When I showed up I felt like I had walked in on a competition for who could find the rattiest jeans contest. The thing is that some of the clothing I saw was in worse shape than what I wear working in my yard on the weekend.

While I understand that software engineers don't want to work in an environment where they need to dress in anything more than jeans and t-shirts, it is common courtesy to dress with some respect for the notion that you are representing your company and not a bunch of yahoos that just walked in from a weekend of yard work.

The thing is that this attitude of disrespectful dress basically tells me that these people aren't professionals and don't expect professional work from the students they are recruiting. While I wouldn't make a decision about hiring someone based on how they dress, I have noticed that those that dress well fall into two camps. Those that know they need an edge and those that are truly professionals. The smart ones that don't dress well tend not to be well rounded. These are people you don't want interfacing with customers.

Since a good portion of software engineering is actually getting the right requirements from customers it makes sense that you need at least some engineers that can do this. While we would probably hire someone who didn't have these skills if they were very smart in other areas and worked well with the team, we would rather have more well rounded people as they will tend to solve the right problems.

In any case it was astounding to me that so many big well respected companies seem to feel that software engineers will be attracted to a company that sends under dressed recruiters.

The "What's Important Today" Problem

2007-09-29T09:13:00.000-07:00

In a typical day I have at least one person walk in my office and say something along the lines of "This feature is critical to our customers and we need to do something about it now." The problem is the same person will stop by the next day or next week with another really important problem. I'm almost always working on something or another and so can't just drop everything and start a new project. If I did nothing would get done.

So what to do about this? The problem is that we all have a tendency to make the issue that we ran into today a top priority. But then tomorrow, something else is a top priority. So it kind of becomes a time averaging problem. If you create a list priorities then re-prioritize, add to and remove items on it on a daily basis, then over time the highest priority items will float to the surface.

This works, as long as you aren't putting today's problem at the top of the list. So it makes sense to put today's problem in the middle of the list and then let it float. If it really is a high priority item then it will float to the top quickly.

Time Estimation Classic Mistakes

2007-08-11T16:56:00.000-07:00

When I first started working as a mechanical engineer my boss would routinely double any estimate I gave him. On average my recollection is that he was pretty much always right. I usually had time to do a little more than what needed to be done so I had time to pursue some diversions.

Most of the work I did during those years was writing software to support the team. It was with some surprise that years later when I was working in software development groups time estimates were always taken at face value.

To me this is intuitively wrong, but for some reason most people don't seem to see this. There seems to be an assumption that an engineer will pad his estimates. The problem is most people don't account for the obvious. Things like vacations, and sick time are so obvious that you would think they wouldn't be missed. Given industry averages for sick time, vacations and holidays subtract 1 month per year.

Other things like meeting overhead go equally unnoticed. Typically you are going to spend, if you are lucky, 4 hours a week in meetings. That's about 25 man days a year. Subtract 1 more month.

Then there is the unknown nature of certain tasks. Let's you estimate that a task will take 4 days. Let's assume for a second that you missed something and it takes 2 days longer. Let's also say that you overlooked a quicker way of doing it and it takes only 2 days. So on average the estimate of 4 days is good. Wrong!

Let's say you were way off and it takes 16 days. There is no way that it could take -12 days. The problem is that you need to view time passage as doubling or halving of the estimation. This means that you need to do estimation on more of logarithmic scale.

So assuming an engineer estimated correctly on average to plus or minus 2 times his estimate you need to calculate the actual average of (x/2 + 2x)/2. Or x * 1.25. If they are off typically by 4x then it becomes (x/4 + 4x)/2 or x * 2.125. Or if they are off by only 10% typically (x/1.1 + 1.1x)/2 or 1.0045. Even someone that is very accurate will tend to be on average under. I've never met anyone that was nearly that good. Someone who is accurate by 2x is rare.

The thing is that the longer the time estimate, the more likely that it will be off. I have read in some places that in the early part of a project the time estimates may be off by as much as 4X. Given that the longer a project takes the more likely there will more things missed in the estimation this makes sense. Using the 2 times rule of thumb for estimating should give you good results most of the time assuming you have also accurately accounted for the obvious overhead of vacations and such. The overhead we estimated above amounts to about 17%.

Revisiting overhead a little more we realize that there is more. How much time in a day is lost to idle conversation. How much time is lost due to other factors such as computer failures, upgrades, recruiting, answering questions for support, etc. All told this adds up to a fairly significant percentage. Since most engineers don't estimate the time required for maintenance, bug fixing, documentations etc. this can easily extend the 17% to as much as 50%.

There are many other factors as well. For example the motivation factor. While it is true that people work harder when they have a deadline that is looming than when they don't most managers will overestimate the effect of this motivation. My observation is that it is never more than a small benefit and often runs the wrong way. When an engineer is faced with a deadline that they know they cannot meet, they don't work harder. They give up.

Since most of the time estimates are on the wrong side when the deadline gets to where it is looming it is almost always a foregone conclusion that the engineer will know it cannot be met. And even if the estimate is horribly short but the time frame is more than a month or so, there is no motivation from this at all. Bottom line, forget about it. It is a transitory effect that those who do good estimation enjoy toward the end of a cycle that you pay back at the beginning of the next cycle. If you don't do good estimation you never enjoy it and you have a net negative impact.

On the other hand excessivly padding a schedule doesn't work either as it you will suffer from the problem that someone else will beat you to market.

The best choice is to get the estimate right in the first place and to correct for mistakes as you go. It a terrible disservice to employees to have a schedule slip as they may schedule a vacation based on original estimates so they aren't gone during a critical time only to find out that because of slippage the vacation they scheduled falls at a very inopportune time. This makes them look bad and feel bad if they take the time, but they should take the time because it was planned for.

Other things I see happen all the time is the unplanned release. Someone in upper management sees a business opportunity and takes it. Unfortunately, the development team is already behind schedule on their current tasks. This can terribly demoralize a team. Even if the new opportunity pans out, which often as not it doesn't, one or the other or both suffer. If on the other hand you plan for this, yes I said plan for the unexpected opportunity, you will be ready to leap and can have success on all fronts.

If the unplanned never materializes, what happens? You ship early! This is a morale booster in everyones book. A team that consistently hits it's targets will start taking pride in that and will become even more productive. A team that consistently fails to meet it's targets will remain unproductive and those that are on the team that take pride in their work will start looking for a successful team.

One problem that I have never seen on any team that I have worked on, although I'm sure it exists somewhere, is an engineer that pads estimates so much that they can spend most of their time playing games. If you hire good people you don't need to worry about this. They are competitive and if one engineer thinks they can get the job done faster than another they will challenge an unrealistic estimate.

Another factor is the generally optimistic nature of people. Of course this is what you want. I was just telling a junior engineer the other day to ignore the advice of a more senior engineer that said "you don't want to touch that code". While that senior engineer may be right, many really great things have been invented by those that didn't know better. If you have a good group of engineers they are optimistic and that optimism will be reflected in their time estimates as well.

When it gets down to it estimation is an as much art as it is science. The problem I see is that problems occur most often when the science is ignored. Here I have clearly shown that logic dictate multiplying all estimates by some factor. 2 Seems as good as any and as we have seen is probably still optimistic. However, it is much closer to reality than 1.

No One Knows How to Build Software Right

2007-08-11T09:34:00.000-07:00

I was reading an interview with Joel Spolsky who writes Joel on Software this AM in the ACM Queue. Much of the time I agree with what Joel has to say, but something he said in the article really rubbed me the wrong way.

He was going on and on about how there isn't any real science in all the software engineering books and articles. Things like programmers having offices being based off of one weekend study done years ago and simply parroted by others over time. While he is right to say that the software engineering texts aren't giving us the best advice, he is very, very wrong to say that we don't know anything about this.

Every time I watch a large construction project or watch a show on TV about how some huge project was done and that it is either ahead of or behind schedule, I know that those projects face exactly the same kinds of issues as software development.

Someone changed the plans again, or the welds on the beams from a supplier are failing. We just released a product to manufacturing yesterday. It was quite a successful project so far, hopefully it sells well when it hits the stores. It was a project that had a very tight and short schedule due to a hard deadline.

While we pulled it off there were several things that occurred along the way that were classic problems for any business. At one point someone suddenly realized that one of our key people was going to be on vacation for a week and all of a sudden the whole project slipped.

This actually resulted in us devoting more time to development which made for a better product. However, it would have been an even better product if we hadn't switched gears in the middle. The new features that were added had a domino effect of requiring a user interface redesign. If we had known we had more time to start with we could have planned the UI better and most likely would have had time for another feature or two.

We also had last minute problems due to a misunderstanding about how we needed to branch our software to avoid problems. This resulted in a lost day due to a failed automatic build that occurred because of the side-effect of fixing an entirely unrelated bug in another product that is only loosely related to the failed product. Lesson learned.

The point of all this is that every complex project out there works under the same fundamental principals. We are dealing with people of various skill sets and they make mistakes. The projects that are completed on time are the ones with the best managers that understand principals of time management and anticipate the unexpected by putting time into the schedule for these items.

The software industry could learn a lot from the construction industry in the area of project management. Certainly, there are differences in specifics. It's a heck of a lot easier to change a broken foundational class than it is to fix a badly poured concrete foundation in a building. The ramifications of the change in the software and the resulting issues in the building can be far reaching. It's way better in both cases to get it right at the start.

Because it seems so easy to change the design of the software at any point along the way, we can get complacent about building things well. The problem is later on you have all of these things that adapt to the bad design and in a sense come to depend on it. When you go back months or years later and need to fix the bad design you are also faced with needing to identify and fix all the places that depend on it. Faced with an impending release date you may end up opting not to fix it and add yet another band aid fix that makes fixing it next time around even harder.

In any case, the next time someone says we need to study the problem of software engineering more, or that there is no science that helps us do this, slap them soundly and point them at the nearest large construction project.

Flexible vs Inflexible User Interface

2007-08-10T19:22:00.000-07:00

Today while reviewing a project to add customizable hot keys to our product that a couple of our summer interns were working on a suggestion was made that got me to thinking about flexible vs. inflexible user interfaces and what is better.

The suggestion was that we not allow the arrow keys to be reassigned to other commands. On the surface this seems like a bad idea. Why would we prevent the customer from making the commands that are currently associated with the arrow keys be reassigned to something else?

To answer that question is to understand a bit more about what makes a good user interface.

A flexible user interface is nice because it allows you to do pretty much anything you want. On the other hand it may make doing very common simple tasks harder to do. An example of this is what makes computers work. Machine language is a very flexible user interface. You can use it to create any software that you want. On the other hand it can be nearly impossible to deal with for a reasonably complicated system.

A higher level language is less flexible but you can be much more productive. In C++, for example, a single line can generate more instructions than a room full of monkeys.

This gets to main point, simple interface may be the best interface. A great example of this is the iPod. It does only a very few things, but it does them very well and simply.

It is tempting to take the easy way out and create a general interface for a particular task. In this case we are actually doing just that. In fact, it is easier to generate the interface to allow someone to customize hot keys than it is to come up with a set of hot keys that is really good. That's why we are doing this in the first place as it allows multiple different viewpoints of what make a good set of hot keys to exist within the same application.

However, in the case of the arrow keys we have a modal interface that allows different behavior depending on the mode based on an arrow key being depressed. These are all commands that affect navigation in a way that is intuitively associated with arrow keys.

Of course now we get into the realm of trying to predict what the customer wants. This is like trying to predict whether it will rain on Sunday or not. Pretty much whatever you decide is the desirable outcome will not be good.

The case of the arrow keys isn't nearly as compelling as many other cases. For example the tools for the blog I'm using now are quite limiting from a general word processing point of view. However, because they are so simple, even my wife can figure them out. Actually, she knows them far better than I as she has her own blog. In any case the point is that an interface is should be as simple as possible but no simpler.

Egoless Programming

2007-07-31T20:43:00.000-07:00

The concept of egoless programming has been around for years. The thing that really surprises me is how much ego seems to creep into the development process. Unfortunately, it seems that some of the smartest software engineers out there tend to have very large egos associated with the code they write. Of course this is a generalization as I have encountered engineers that weren't very good that had large egos and those that were really smart that don't have much of an ego.

The real question is how important is the concept of egoless programming? In reality the question should be how important is teamwork. Egoless programming is all about team work. That reminds me that there is no "I" in team work...

Analysis Paralysis

2007-07-21T19:22:00.000-07:00

Sometimes we as engineers fall into the trap of searching for the one best solution to a problem. Algorithmically speaking I suspect this is probably NP-complete. This places it in the same category as the traveling salesman and knapsack problems in terms of complexity to solve.

For anyone who is familiar with these types of problems you should realize the futility of trying to find the best solution. Instead what we should be doing is attempting to come up with a measure that says that a solution is good enough.

Almost every problem has multiple qualitative variables such as ease of implementation, speed of execution, simplicity of maintenance, etc. And many problems have qualitative solutions, such as deciding whether one user interface design is better than another.

When you are working on a problem a good guideline to follow is to ask yourself whether the solution at the moment is good enough to satisfy minimum requirements in all areas. Then ask yourself if you can see an obvious modification that significantly improves one or more of these areas that doesn't significantly erode any other areas. If you are only seeing marginal gains in any area or the work to make it better is difficult, then you are probably done.

Knowing Your Customer

2007-07-11T21:26:00.000-07:00

One of the hardest things to do in software is to really understand what your customer wants. Most software engineers make the mistake of assuming that what they like is what the customer will like. While this is often the case it is often not the case as well.

You can probably make the assumption that if you don't like something that some customers won't like it either. You can also assume that if you like something that some customers will also like it. However, whether this is a large or small percentage is hard to tell without spending some time researching the problem.

Another thing to keep in mind that many very creative people are terribly disorganized. Engineers like to have nicely orthogonal solutions. Everything lines up nicely and is consistent. The problem is the world isn't this way. It is often haphazard, disorganized, inconsistent etc. Creative people can often be difficult to deal with and make conceptual leaps that don't make any logical sense.

A foolish consistency is the hobgoblin of little minds. -- Ralph Waldo Emerson

Don't get me wrong. Consistency is often a very good thing as it tends to simplify the learning process. But on the other hand it may get in the way of efficiently doing your job.

My brother runs a support group and one of the things he focuses on is an accurate statement of the problem. An accurate problem statement is key. It's as obvious as the nose on your face that if you solve the wrong problem the customer won't be happy. So you must first understand the problem.

Here we get into requirements gathering. If you state a problem incorrectly you will not be able to produce an accurate set of requirements. If you think about it a little you will realize that requirements are simply another way of expressing a problem. Rather than stating what isn't happening, the problem, you are stating what should happen, the requirements.

A single problem may result in multiple requirements. And a single requirement may partially or totally solve many problems.

Once you know what your customer wants, you aren't really done yet. Let's take a simple example. When working with CAD there are quite a few large format paper sizes. For example Arch A, B, C, D and E. D is very commonly used. So if you talked to several customers you might come to the conclusion that you only need to support D. Or if you are having a particularly lucid moment, you might say that we should support all these sizes.

You would be wrong. The fact is that there is also an ANSI set of paper sizes. An ISO set of sizes and a variety of others. You will not find out about these by talking to your customers. You must become an expert in paper sizes in order to design a system that will work well for pretty much any customer.

The point is that when you start working on software you need to know more about the topic you are working on than most of your customers. When you do this you will be able to write software that not only does what you customer asked for, but also does what your customer doesn't know they need yet.

Now you are probably saying that you don't have time to do all this research. The problem is that if you don't do the research, you will be revisiting your design later. The less you know about a problem the more likely it is that your design will require significant rework. For anyone that has maintained legacy code, you will realize that not only will you need to make things work write, you will also need to make sure that the way it did work is still supported or at least provide a migration path.

The redesign may take as much time as the original and the migration path will probably be more work on top of that. If it took you a week to do the original project and a week or so to do the rework plus half a week doing research to figure out what you should have done then spending half a week doing the research to begin with and not having to do the rework will save you at least a week later on.

If on the other hand you fail to do the research the second time you are likely to repeat the process several times until you get it right. Each time incurring additional overhead to support all of those legacy migration paths. Eventually, you will have spent at least a week in the school of hard knocks learning about why your original design was wrong.

Not only that, you will suffer from less commercial success because the customers aren't happy with what you provided them.

So bottom line talk to your customers. Really understand what they want and how they want to do it. Read about what they do from whatever sources you can obtain or get training from others. Perhaps even a course at your local college would be in order. In any case, when you get done with all of this you should be in a position to not only know what and how your customers work, but also be able to teach them a thing or two as well.

Nothing makes a customer happier than having a solution that anticipates their needs. Nothing frustrates a customer more than having to jump through hoops to do very common tasks.

So, know your customer better than they know themselves and you will never fail to produce top quality solutions.

Subversive Programming

2007-07-09T22:17:00.001-07:00

I program mainly in C++. As such I know the language fairly well. If you are beyond the beginner stage of learning C++ you know that const correctness is important. It is a contract between you and the reader of the code that if they call this method they will not modify the data of the object in an observable way using the public API of the class.

Observable is important here because in some cases it is important to be able to have data that is modifiable in a class. The case that I find most compelling is when you need to cache a calculation for performance reasons.

Now there are ways around this using perfectly legal C++ syntax. I once saw some code where this was done and even proudly commented as "fooling the compiler". The problem is that the compiler isn't the only one being fooled. When this sort of thing is done you are basically lying to the poor maintainer of the code. If this happens frequently then code maintenance is more about navigating a field of land mines.

Most of the time engineers aren't intentionally subversive. In the example I cited this resulted from not knowing that this was subversive. A great engineer will actively learn what is subversive and endeavor to avoid it.

Be a better engineer by learning about subversive coding techniques and expunging them from your vocabulary. The following link is a good start http://mindprod.com/jgloss/unmain.html.

Endurance

2007-07-08T09:53:00.000-07:00

Often software development is more about endurance than being smart. While it really helps to be smart, there are many things that don't come easily. I often see young software engineers wanting to give up on a project because they aren't making the kind of progress they would like to see. They feel like they are wasting time and especially for someone who is used to being successful, it makes them feel like they are failing.

I like to ride my bicycle, often for several hours at a time. In a lot of cases I end up overextending myself and the last few miles on the road back can be very difficult. The same can be said of software. Especially, if you are working on a project and you get down toward the end and you find that there is a very difficult problem to solve that you didn't anticipate.

It is during these times that often the successful engineer isn't the smartest one, but the one with the most mental endurance.

When I ride my bike I like to contrive to make the ride back the easy half. So I'll ride uphill to start or into the wind. Sometimes this doesn't work out, but if you can tackle the hard stuff first, you will be in a better position to finish the project. And, if it turns out that the hard stuff can't be solved, you call it quits and you have lost less time.

Endurance isn't everything. Sometimes you need to know when to take a break. I will very often figure out a nasty problem on the drive home from work. The simple act of taking my mind off the problem or being distracted by other things seems to help my mind figure out problems. If you find yourself struggling with a problem well after everyone else has gone home, go home. You are not doing anyone any favors by continuing to beat your head against a problem when you are tired.

Just as 30 minute break to eat and drink and generally relax in the middle of a bike ride can make the rest of the ride go a lot easier, the same is true for your brain. It works better when it is rested.

Another way to make things easier to deal with is to break things into chunks that you can deal with. When I'm riding, I will set my sights on the next easily achievable goal. It might be the top of the next hill, the next mile, the halfway mark or so on. When you are working on a project you should keep the larger picture of where you are at in mind, but focus on the next thing. It may seem that months or years of time devoted to a project is insurmountable, but if you take it one easy step at a time you will soon start to see the results.

Sometimes I see engineers that are more like sprinters. These are the guys that are working really long hours in the last week or so of a project to get it done. My question is where were they in the first few weeks? You obviously don't want to get burned out early so you have to pace yourself, but there is no reason to work excessivly hard at the end of a project, unless you didn't plan well. I never liked cramming for tests either. I figured if I didn't know it by the time I took the test then I didn't know it and what's the point of taking the class in the first place.

So the next time you feel like the project you are working on is never going to get done. Take a step back examine the long term goals, look at where you have been and then refocus on the next step.

When to Worry About Performance

2007-07-05T19:15:00.000-07:00

I spent the day wrestling with slow code. Most of it was very easy to spot where the slowness was. I like using the poor man's profiler to spot these. What's the poor man's profiler you ask? Basically it boils down to using the debugger to break already executing code. The theory is that in all probability it will break in or near the code that is taking a lot of time. It's quick and should work with nearly any debugger. The assumption, of course, is that you can make something slow enough to start it and then cause it to break before it gets past the slow code.

In any case that isn't the point of this post. The real question is when should you worry about performance. One developer I worked with many years ago said this. "Make work, make it work right, make it work fast." I like this quote but I add to it, when you are working on an algorithm take 5 minutes and pick a fast one.

Most of the time I encounter slowness when there is an O(n^2), O(n log(n)) or O(n) algorithm in use when an O(n) or O(log(n)) algorithm is easy to choose at development time. The problem is that often that slower algorithm is excessively difficult to replace. Often, the slow algorithm isn't discovered until the customer pushes the limits of the program.

I lost count many, many years ago of the number of times where the engineer came to the conclusion that no one would ever need more than n of these things. Then the customer comes up with a case that the engineer didn't think about where 10 * n is a small number.

Rule of thumb. Think of the largest number of items that you can imagine that a customer would ever need then multiply it by 100 or 1000. Then do your design based on that assumption.

Another idea that makes my blood boil is the whole Moore's Law thing. We have effectively seen the end of this about 4 years ago. That was when memory speed stopped keeping up with CPU speed. The words I hear are just buy a faster computer. Or in a year or two it won't matter.

The problem with this is that if computers double in speed every 18 months or so, you are talking about an algorithm that is O(n log(n)). If you are throwing O(n^2) algorithms at this they will swamp any CPU speed gains you get. Couple this with Moore's Law not keeping up with expectations and ever more efficient ways of increasing n and performance suddenly becomes a huge issue.

The current trend in multi-core processors is also interesting. However, if an algorithm is O(n) or worse on memory access with only 1 path to memory, no amount of additional cores will help.

So the bottom line is that if you write any code that is essentially unbounded on n, spend a few minutes coming up with an algorithm that is optimal for large n.

I leave you with one caution. If n is only very rarely large and is very frequently on the order of magnitude of 10, then that better algorithm may in fact be slow if you are working with a large number of items.

Accountability in Software

2007-07-03T22:05:00.000-07:00

I must confess that I am a Mechanical Engineer by training. Everything I learned about software engineering has been on the job and a lot reading and a fair amount of trial and error. When I first started my career as a Mechanical Engineer the group I was working with had written some software as part of a NASA contract. Since I had an interest, and as it turns out, an aptitude in dealing with programming, I naturally migrated into supporting this software. This software was written in Fortran (ugh)... At the time I thought it was a great programming language. In comparison to using Basic on a TRS80 (a whole 4K of memory) this was pretty much a step up.

As a Mechanical Engineer I had to take the Fundamentals of Engineering exam when I graduated. Depending on the field that I went into, I would have eventually needed to take another exam to become a licensed professional engineer. As it turns out, I took a different route and never crossed that bridge.

Should software engineers be licensed? Theoretically, no. Practically, maybe. The issue is really accountability. Most good software gets that way because of several factors. Mainly by the simple fact that bad software doesn't usually make money. Or if it does, eventually someone else will come along and do a better job.

The other way that software gets good, is because of the person who wrote it made it good.

Engineers are licensed because of loss of life due to things like bridge failures. Even so, there are still failures that occur even though an engineer signed off on the design. The real bottom line is whether you can make money or not. However, licensing does leverage into the idea that an individual will not want to sign off on something if their neck or license is on the line. Given the large number of people out there that picked up a "Dummies" programming book and are now "experts", I wonder if maybe this would help to separate the true engineers from the code jockeys.

Don't get me wrong, just because I have an engineering degree doesn't qualify me as a software engineer, and just as before professional mechanical engineering degree programs existed there were many who were highly qualified engineers. I feel that we are in a similar time frame where good software engineering training is more the exception than the rule.

I do a fair amount of college recruiting. However, our local university doesn't have a software engineering degree. They offer computer engineering and computer science. The good news is that these programs are part of the college of engineering rather than the college of science. They have a really nice senior design program that emphasizes software engineering.

There are still gaps though. It never ceases to amaze me how little a CS student knows about practical software engineering when they graduate. Their grasp of the languages that they used is usually very weak. The ones that actually do know something about it spend a lot of time learning on their own. This is a double edged sword though.

If you learn on your own you can often build very bad habits that are hard to correct later on. In addition, most of the professors teaching CS haven't spent that much time learning languages. This leads to huge gaps of knowledge or misplaced emphasis on what is important in a language.

I happen to have spent many years writing in C++. I consider myself to be advanced but by no means an expert in C++. I am continually learning something new about the language. In any case I digress from the point of this blog.

The point is that even if we did decide that licensing is a good thing, which it probably is, I wouldn't trust the knowledge base that might be required for licensing. Perhaps the best solution in the short term is to require software teaching credentials to include 10+ years of industry experience. If we get the professors on the right page in terms of what is important knowledge in the industry, then the rest will follow.

This will naturally occur over time, but will probably take several generations for the profession of software engineering education to catch up with industry.

On the other hand, it will take some pretty hard knocks for some of the industry to catch up with what is important. When you have pressures to push a product out the door to capture market share, or just to keep a company from going under and no personal accountability, we get varying results.

This too will naturally get corrected over time as these products and companies fail.

The bottom line though, is that you as a software engineer need to take personal responsibility for what you write. Become a craftsman. Make your code and the end product a thing of beauty. Write code that is defect free. Make sure you know your customer. Take responsibility for your code.

I doubt that this will become the norm, so when licensing becomes necessary, and you can't pass the test, do us all a favor and go work at a fast food place where your sloppiness and lack of accountability will be expected and rewarded with the salary that you deserve.

On the other hand, if you really are a great software engineer, no worries...

Rule of Hand

2007-07-02T21:25:00.000-07:00

I have a guideline that I call the "Rule of Hand". If a function takes up more lines than you cover with your hand it's probably too big.

I'm reminded of a post that I saw just the other day that was a very long paragraph. Probably about 4 times longer than this post. It was very difficult to read simply because it put too much information into a single spot.

The idea here is that small bits of information are much easier to read and understand.

One final comment is that this is a guideline and as a guideline there are always exceptions.

Code Comments

2007-07-02T20:56:00.000-07:00

When I first started writing software the common mantra was to put in lots of comments. At times I did this and other times I didn't. During most of my career I have worked on code bases that someone else started. In that time I have rarely seen code comments that really helped. Often I see comments that basically restate in English what the code is already saying. Even worse the comments tell me that the code does something different than what it actually does.

In more recent years I have found myself reading comments as a last resort. Usually this is after I have determined that the code isn't working the way it seems like it should. At this point I'm looking for a clue as to what the original author, sometimes myself, was thinking when they wrote the code.

So, long story short, if you feel like you need to write a comment to clarify some code, you probably need to rewrite the code so that it is more readable. Every once in awhile I get someone who is asking me to put in comments to clarify what is going on. As it turns out, usually the problem is that what I thought was obvious when I wrote it wasn't. This makes it very difficult to comment code effectively.

Which gets to the point of my post. Learn to read code. No matter what language you are using, you need to be fluent in both writing and reading the language. To be really fluent you need to read code written by a large number of writers. What you will find is that there are certain dialects that creep into language. The more code you read the better you will be at figuring out what is wrong with a program.

Ultimately, the speed at which you can correct problems in your and others code will translate into how valuable you are as an engineer.

The other side of this is to write self documenting code. This usually involves reasonably descriptive variable names. Beware of too long names as this can also detract from readability.

Package up logical things into functions or methods that do only one thing. That way it is really clear what is happening.

Mostly just use common sense and keep things simple.

Software Engineering Analogies

2007-07-01T21:26:00.000-07:00

I have seen many comparisons to software engineering that like to make software development fit into the engineering metaphor. Sometimes it goes like this, a software engineer designs the software and then passes it on to a programmer to implement. The analogy is that software is like constructing a building or a bridge or something like that. The architect or engineer doesn't do the construction of the building therefore the software architect doesn't do so either.

It sounds good and it actually does work. However, almost all software isn't developed using this "sound" method. Why?

Because it doesn't work that way. A whole industry of software engineering methodology is based off of the assumption that the analogy is correct. Which makes a lot of software engineers and non engineers who read this stuff think that the way we develop software is wrong. This doesn't do anybody any good. Often you get people outside of the development team telling you that your methods are wrong. When or if you try to apply these methods they tend to cost too much and don't get significantly better results. In some cases the results are worse.

I submit there is a better analogy. In order to understand the analogy, we need to first understand what the end product of a software engineer is.

Most people will state the obvious, the end product is the executable. It might be an embedded system that runs a cell phone, or a sales contact management database or a software program like the web browser you are using.

While this is the deliverable of the company producing the software, it isn't the deliverable of the software engineer. The software engineer delivers code that can be compiled into the final product.

So what does an architect deliver that is designing a building? Not the building. The deliverable is the working drawings. They stick with the process through construction and continue to fix the drawings as problems during construction are identified.

If you haven't guessed already analogy I like is that the software engineer is like an architect. Just as an architect interfaces with customers to define the requirements for a building, the software engineer also interfaces with customers to define the requirements for the software. Just as an architect will iterate through several designs working with pen and paper or a CAD system a software engineer does the same. The working drawings can be reproduced at will, just like software.

This analogy makes a lot more sense when you realize that most software is implemented by the designer of the software. This works really well when there is just one person and when you are working in a small group of engineers. It also scales pretty well to larger groups, but does rely on more documentation to keep everyone on the same page.

In any case, I think if you start your analysis of how to develop software with a faulty analogy, you get faulty results.

Assuming you accept my analogy that a software engineer is like an architect, an examination of how architects work would be in order. In fact when you look at this model you find that most of the stuff that an architect does a software engineer also does. You interface with your customer, you show them preliminary designs, you take feedback and iterate on them.

It is just as easy to change a CAD drawing as it is to change software so iterating on the software is something that can be done. Just as architects leverage building blocks like CAD details that they have placed into a library a software engineer leverages code reuse.

Another aspect of this is that an architect needs to make all the working drawings in a set look consistent even if several people are working on them. Have you ever had trouble with a team member not sticking to coding standards because they liked their style instead of the team standard?

It starts getting you to thinking, if the end product is the code, then there should be some pride of workmanship in the code. A mechanical engineer designing a high end car will spend a lot of time making the engine compartment easy to work on and to some extent it will be a work of art. If you ever get a chance it is interesting to take a look at the large steam engines that Henry Ford used in his factories. These aren't ugly, they are works of art.

If you look at code and it looks like it went through a blender, you can tell the person who wrote it was less concerned with the implementation than simply getting the job done. Back to a set of architectural drawings. A sketch on a napkin may suffice for a very simple project, like building a deck, as long as you are the one doing the construction. But the more complicated the design and the more people involved, the more accurate and clear the working drawings have to be.

Another consideration is the idea of writing in pseudo-code. This is useful if you are going to be writing in assembly, but these days our languages are more expressive, readable and precise than pseudo-code. I liken this to writing the outline of an English novel in French. The same idea applies to other tools like UML etc. While I do believe UML has value when communicating with someone who doesn't understand the language you are writing in, it really doesn't help a lot when communicating to a expert in the language.

Don't look to hard at this analogy or you will start to find cracks in it. As with all good analogies, they break down after a certain point.

So the next time you think about software engineering, think about the architect.