Tuesday, July 31, 2007

Egoless Programming

The concept of egoless programming has been around for years. The thing that really surprises me is how much ego seems to creep into the development process. Unfortunately, it seems that some of the smartest software engineers out there tend to have very large egos associated with the code they write. Of course this is a generalization as I have encountered engineers that weren't very good that had large egos and those that were really smart that don't have much of an ego.

The real question is how important is the concept of egoless programming? In reality the question should be how important is teamwork. Egoless programming is all about team work. That reminds me that there is no "I" in team work...

Saturday, July 21, 2007

Analysis Paralysis

Sometimes we as engineers fall into the trap of searching for the one best solution to a problem. Algorithmically speaking I suspect this is probably NP-complete. This places it in the same category as the traveling salesman and knapsack problems in terms of complexity to solve.

For anyone who is familiar with these types of problems you should realize the futility of trying to find the best solution. Instead what we should be doing is attempting to come up with a measure that says that a solution is good enough.

Almost every problem has multiple qualitative variables such as ease of implementation, speed of execution, simplicity of maintenance, etc. And many problems have qualitative solutions, such as deciding whether one user interface design is better than another.

When you are working on a problem a good guideline to follow is to ask yourself whether the solution at the moment is good enough to satisfy minimum requirements in all areas. Then ask yourself if you can see an obvious modification that significantly improves one or more of these areas that doesn't significantly erode any other areas. If you are only seeing marginal gains in any area or the work to make it better is difficult, then you are probably done.

Wednesday, July 11, 2007

Knowing Your Customer

One of the hardest things to do in software is to really understand what your customer wants. Most software engineers make the mistake of assuming that what they like is what the customer will like. While this is often the case it is often not the case as well.

You can probably make the assumption that if you don't like something that some customers won't like it either. You can also assume that if you like something that some customers will also like it. However, whether this is a large or small percentage is hard to tell without spending some time researching the problem.

Another thing to keep in mind that many very creative people are terribly disorganized. Engineers like to have nicely orthogonal solutions. Everything lines up nicely and is consistent. The problem is the world isn't this way. It is often haphazard, disorganized, inconsistent etc. Creative people can often be difficult to deal with and make conceptual leaps that don't make any logical sense.

A foolish consistency is the hobgoblin of little minds. -- Ralph Waldo Emerson

Don't get me wrong. Consistency is often a very good thing as it tends to simplify the learning process. But on the other hand it may get in the way of efficiently doing your job.

My brother runs a support group and one of the things he focuses on is an accurate statement of the problem. An accurate problem statement is key. It's as obvious as the nose on your face that if you solve the wrong problem the customer won't be happy. So you must first understand the problem.

Here we get into requirements gathering. If you state a problem incorrectly you will not be able to produce an accurate set of requirements. If you think about it a little you will realize that requirements are simply another way of expressing a problem. Rather than stating what isn't happening, the problem, you are stating what should happen, the requirements.

A single problem may result in multiple requirements. And a single requirement may partially or totally solve many problems.

Once you know what your customer wants, you aren't really done yet. Let's take a simple example. When working with CAD there are quite a few large format paper sizes. For example Arch A, B, C, D and E. D is very commonly used. So if you talked to several customers you might come to the conclusion that you only need to support D. Or if you are having a particularly lucid moment, you might say that we should support all these sizes.

You would be wrong. The fact is that there is also an ANSI set of paper sizes. An ISO set of sizes and a variety of others. You will not find out about these by talking to your customers. You must become an expert in paper sizes in order to design a system that will work well for pretty much any customer.

The point is that when you start working on software you need to know more about the topic you are working on than most of your customers. When you do this you will be able to write software that not only does what you customer asked for, but also does what your customer doesn't know they need yet.

Now you are probably saying that you don't have time to do all this research. The problem is that if you don't do the research, you will be revisiting your design later. The less you know about a problem the more likely it is that your design will require significant rework. For anyone that has maintained legacy code, you will realize that not only will you need to make things work write, you will also need to make sure that the way it did work is still supported or at least provide a migration path.

The redesign may take as much time as the original and the migration path will probably be more work on top of that. If it took you a week to do the original project and a week or so to do the rework plus half a week doing research to figure out what you should have done then spending half a week doing the research to begin with and not having to do the rework will save you at least a week later on.

If on the other hand you fail to do the research the second time you are likely to repeat the process several times until you get it right. Each time incurring additional overhead to support all of those legacy migration paths. Eventually, you will have spent at least a week in the school of hard knocks learning about why your original design was wrong.

Not only that, you will suffer from less commercial success because the customers aren't happy with what you provided them.

So bottom line talk to your customers. Really understand what they want and how they want to do it. Read about what they do from whatever sources you can obtain or get training from others. Perhaps even a course at your local college would be in order. In any case, when you get done with all of this you should be in a position to not only know what and how your customers work, but also be able to teach them a thing or two as well.

Nothing makes a customer happier than having a solution that anticipates their needs. Nothing frustrates a customer more than having to jump through hoops to do very common tasks.

So, know your customer better than they know themselves and you will never fail to produce top quality solutions.

Monday, July 9, 2007

Subversive Programming

I program mainly in C++. As such I know the language fairly well. If you are beyond the beginner stage of learning C++ you know that const correctness is important. It is a contract between you and the reader of the code that if they call this method they will not modify the data of the object in an observable way using the public API of the class.

Observable is important here because in some cases it is important to be able to have data that is modifiable in a class. The case that I find most compelling is when you need to cache a calculation for performance reasons.

Now there are ways around this using perfectly legal C++ syntax. I once saw some code where this was done and even proudly commented as "fooling the compiler". The problem is that the compiler isn't the only one being fooled. When this sort of thing is done you are basically lying to the poor maintainer of the code. If this happens frequently then code maintenance is more about navigating a field of land mines.

Most of the time engineers aren't intentionally subversive. In the example I cited this resulted from not knowing that this was subversive. A great engineer will actively learn what is subversive and endeavor to avoid it.

Be a better engineer by learning about subversive coding techniques and expunging them from your vocabulary. The following link is a good start http://mindprod.com/jgloss/unmain.html.

Sunday, July 8, 2007

Endurance

Often software development is more about endurance than being smart. While it really helps to be smart, there are many things that don't come easily. I often see young software engineers wanting to give up on a project because they aren't making the kind of progress they would like to see. They feel like they are wasting time and especially for someone who is used to being successful, it makes them feel like they are failing.

I like to ride my bicycle, often for several hours at a time. In a lot of cases I end up overextending myself and the last few miles on the road back can be very difficult. The same can be said of software. Especially, if you are working on a project and you get down toward the end and you find that there is a very difficult problem to solve that you didn't anticipate.

It is during these times that often the successful engineer isn't the smartest one, but the one with the most mental endurance.

When I ride my bike I like to contrive to make the ride back the easy half. So I'll ride uphill to start or into the wind. Sometimes this doesn't work out, but if you can tackle the hard stuff first, you will be in a better position to finish the project. And, if it turns out that the hard stuff can't be solved, you call it quits and you have lost less time.

Endurance isn't everything. Sometimes you need to know when to take a break. I will very often figure out a nasty problem on the drive home from work. The simple act of taking my mind off the problem or being distracted by other things seems to help my mind figure out problems. If you find yourself struggling with a problem well after everyone else has gone home, go home. You are not doing anyone any favors by continuing to beat your head against a problem when you are tired.

Just as 30 minute break to eat and drink and generally relax in the middle of a bike ride can make the rest of the ride go a lot easier, the same is true for your brain. It works better when it is rested.

Another way to make things easier to deal with is to break things into chunks that you can deal with. When I'm riding, I will set my sights on the next easily achievable goal. It might be the top of the next hill, the next mile, the halfway mark or so on. When you are working on a project you should keep the larger picture of where you are at in mind, but focus on the next thing. It may seem that months or years of time devoted to a project is insurmountable, but if you take it one easy step at a time you will soon start to see the results.

Sometimes I see engineers that are more like sprinters. These are the guys that are working really long hours in the last week or so of a project to get it done. My question is where were they in the first few weeks? You obviously don't want to get burned out early so you have to pace yourself, but there is no reason to work excessivly hard at the end of a project, unless you didn't plan well. I never liked cramming for tests either. I figured if I didn't know it by the time I took the test then I didn't know it and what's the point of taking the class in the first place.

So the next time you feel like the project you are working on is never going to get done. Take a step back examine the long term goals, look at where you have been and then refocus on the next step.

Thursday, July 5, 2007

When to Worry About Performance

I spent the day wrestling with slow code. Most of it was very easy to spot where the slowness was. I like using the poor man's profiler to spot these. What's the poor man's profiler you ask? Basically it boils down to using the debugger to break already executing code. The theory is that in all probability it will break in or near the code that is taking a lot of time. It's quick and should work with nearly any debugger. The assumption, of course, is that you can make something slow enough to start it and then cause it to break before it gets past the slow code.

In any case that isn't the point of this post. The real question is when should you worry about performance. One developer I worked with many years ago said this. "Make work, make it work right, make it work fast." I like this quote but I add to it, when you are working on an algorithm take 5 minutes and pick a fast one.

Most of the time I encounter slowness when there is an O(n^2), O(n log(n)) or O(n) algorithm in use when an O(n) or O(log(n)) algorithm is easy to choose at development time. The problem is that often that slower algorithm is excessively difficult to replace. Often, the slow algorithm isn't discovered until the customer pushes the limits of the program.

I lost count many, many years ago of the number of times where the engineer came to the conclusion that no one would ever need more than n of these things. Then the customer comes up with a case that the engineer didn't think about where 10 * n is a small number.

Rule of thumb. Think of the largest number of items that you can imagine that a customer would ever need then multiply it by 100 or 1000. Then do your design based on that assumption.

Another idea that makes my blood boil is the whole Moore's Law thing. We have effectively seen the end of this about 4 years ago. That was when memory speed stopped keeping up with CPU speed. The words I hear are just buy a faster computer. Or in a year or two it won't matter.

The problem with this is that if computers double in speed every 18 months or so, you are talking about an algorithm that is O(n log(n)). If you are throwing O(n^2) algorithms at this they will swamp any CPU speed gains you get. Couple this with Moore's Law not keeping up with expectations and ever more efficient ways of increasing n and performance suddenly becomes a huge issue.

The current trend in multi-core processors is also interesting. However, if an algorithm is O(n) or worse on memory access with only 1 path to memory, no amount of additional cores will help.

So the bottom line is that if you write any code that is essentially unbounded on n, spend a few minutes coming up with an algorithm that is optimal for large n.

I leave you with one caution. If n is only very rarely large and is very frequently on the order of magnitude of 10, then that better algorithm may in fact be slow if you are working with a large number of items.

Tuesday, July 3, 2007

Accountability in Software

I must confess that I am a Mechanical Engineer by training. Everything I learned about software engineering has been on the job and a lot reading and a fair amount of trial and error. When I first started my career as a Mechanical Engineer the group I was working with had written some software as part of a NASA contract. Since I had an interest, and as it turns out, an aptitude in dealing with programming, I naturally migrated into supporting this software. This software was written in Fortran (ugh)... At the time I thought it was a great programming language. In comparison to using Basic on a TRS80 (a whole 4K of memory) this was pretty much a step up.

As a Mechanical Engineer I had to take the Fundamentals of Engineering exam when I graduated. Depending on the field that I went into, I would have eventually needed to take another exam to become a licensed professional engineer. As it turns out, I took a different route and never crossed that bridge.

Should software engineers be licensed? Theoretically, no. Practically, maybe. The issue is really accountability. Most good software gets that way because of several factors. Mainly by the simple fact that bad software doesn't usually make money. Or if it does, eventually someone else will come along and do a better job.

The other way that software gets good, is because of the person who wrote it made it good.

Engineers are licensed because of loss of life due to things like bridge failures. Even so, there are still failures that occur even though an engineer signed off on the design. The real bottom line is whether you can make money or not. However, licensing does leverage into the idea that an individual will not want to sign off on something if their neck or license is on the line. Given the large number of people out there that picked up a "Dummies" programming book and are now "experts", I wonder if maybe this would help to separate the true engineers from the code jockeys.

Don't get me wrong, just because I have an engineering degree doesn't qualify me as a software engineer, and just as before professional mechanical engineering degree programs existed there were many who were highly qualified engineers. I feel that we are in a similar time frame where good software engineering training is more the exception than the rule.

I do a fair amount of college recruiting. However, our local university doesn't have a software engineering degree. They offer computer engineering and computer science. The good news is that these programs are part of the college of engineering rather than the college of science. They have a really nice senior design program that emphasizes software engineering.

There are still gaps though. It never ceases to amaze me how little a CS student knows about practical software engineering when they graduate. Their grasp of the languages that they used is usually very weak. The ones that actually do know something about it spend a lot of time learning on their own. This is a double edged sword though.

If you learn on your own you can often build very bad habits that are hard to correct later on. In addition, most of the professors teaching CS haven't spent that much time learning languages. This leads to huge gaps of knowledge or misplaced emphasis on what is important in a language.

I happen to have spent many years writing in C++. I consider myself to be advanced but by no means an expert in C++. I am continually learning something new about the language. In any case I digress from the point of this blog.

The point is that even if we did decide that licensing is a good thing, which it probably is, I wouldn't trust the knowledge base that might be required for licensing. Perhaps the best solution in the short term is to require software teaching credentials to include 10+ years of industry experience. If we get the professors on the right page in terms of what is important knowledge in the industry, then the rest will follow.

This will naturally occur over time, but will probably take several generations for the profession of software engineering education to catch up with industry.

On the other hand, it will take some pretty hard knocks for some of the industry to catch up with what is important. When you have pressures to push a product out the door to capture market share, or just to keep a company from going under and no personal accountability, we get varying results.

This too will naturally get corrected over time as these products and companies fail.

The bottom line though, is that you as a software engineer need to take personal responsibility for what you write. Become a craftsman. Make your code and the end product a thing of beauty. Write code that is defect free. Make sure you know your customer. Take responsibility for your code.

I doubt that this will become the norm, so when licensing becomes necessary, and you can't pass the test, do us all a favor and go work at a fast food place where your sloppiness and lack of accountability will be expected and rewarded with the salary that you deserve.

On the other hand, if you really are a great software engineer, no worries...