Interviewing developers for legacy application codebase

Many of us are working on legacy codebase and cursing the previous developers for what they did? Especially on code quality. It may be very difficult to understand others code, if there are no comments or documents on why they did something in a particular way. Then we have to assume what was going on their head when they were writing that piece of code. Based on that we have to take new decisions. If our assumptions are wrong, our decision will also get wrong and it produce more damage to the system.

So to work on legacy code, developers needs special skills to read previous developers mind by looking at code. Its something like investigating a criminal case or researching on fossils to find what and why it happened in the past.

What is legacy codebase?

My simplest definition to the legacy code is as follows.

Any code which contains technical debt is legacy code.

Technical debt makes code resist to changes. Now we have to understand what is technical debt in software engineering. Its nothing but the damage caused to software because of bad decisions. One will not take bad decisions intentionally. They might be forced to take bad decisions because of many reasons. When we say taking bad decisions, it may be taken by higher managers as well as developers. Don't think that developers are not involved in producing technical debt. Below are some reasons for tech debt.

• Deadlines because of budget, competition & time
• Lack of analysis, planning & design.
• Lack of awareness.

In one way tech debt is good and that is what gives job to most of us :-)

Coming back to legacy systems. If we google, we get different views on legacy code. Below are some of the factors people are saying what makes a codebase legacy

1. The system or the target platform for code is out of common use or runs in compatibility mode
2. The codebase contains code for handling old and outdated things such as reading floppy disk or serial port etc...
3. The code which works in production but difficult to maintain because
1. The original developers are not around
2. No comments or docs.
3. No code quality such as no inheritance & overriding, no patterns, not following SOLID etc...
4. So many short cuts.
5. There are many dependent systems which restrict changes
6. The third party library versions are no longer supported by vendors
7. No unit tests to validate new changes.
4. Any production code is legacy

If we look closely, all these reasons points to technical debt. Ideally we should pay off tech debt at some point. But nobody does because of budget and release timeline constraints hence the code becomes legacy.

But it didn't mean we can just keep the legacy code as is and go forward. There should be somebody to maintain the legacy systems as there were good investments went to build such systems.

Also we cannot guarantee that the new systems we build will never became legacy.  

Tips to interview for legacy codebase

As we seen in the beginning engineers who are going to maintain the legacy codebase needs special skills. Below are some techniques to be tried to assess them during the interview.

Inform them

The first and important thing is to inform them that they are going to work on a legacy codebase. That gives them a chance to rethink and reject the offer. There are not many developers with the skill to work on legacy codebase. As mentioned above it requires special skill sets than just writing code. If their body language changes on hearing the word legacy systems, we need to be cautious. There are still programmers who don't know what is legacy code and of course they cannot take any action based on this information.

I was part of interviews to hire for legacy codebase and I asked about ADO.Net knowing that the person is not at all going to write any ADO.Net code.  Most of the legacy systems will be having a working data layer and developers will be getting required object(s) when they make a call to DL.Also asked about new language features knowing that its not going to be upgraded in recent years.

Puzzle to find out the logic

If the candidate is willing to work on legacy code, the next step is to evaluate them whether they are fit for the job. One of the important skill required for legacy code maintainers is the ability to understand the intention of previous coder by looking at / debugging the code. There is no guarantee that there will be enough comments, docs or best practices. So the best way to evaluate them is to show a piece of poorly written legacy code and ask to find out what is going on.

If we have legacy code, we don't have to think about what to ask. We can get it from our codebase itself. one sample is below.

int a=0;
private void Timer_Tick(object sender, EventArgs e)
{
    if ((a++ & 0x10) != 0)
    {
        Foo();
        a = 1;
    }
    DoWork();
}
 
private void Foo()
{
    Console.WriteLine("Foo");
}
 
private void DoWork()
{
    Console.WriteLine("Do Work");
}

What is happening in this timer tick event handler? Most of the graduates who just came out of college can say the output of this code. But they may not understand the thought process of the developer who wrote this code. The history goes as follows.

Developer X wrote the original code. At that time there was only DoWork(). For every timer hit, say 1 second the function was getting executed. Later developer Y came and he had to do something every 16 seconds. By that time, this code became legacy. X is not there in company or too busy to help Y. So instead of introducing new timer and all, Y simply added a counting mechanism using variable a so that only in every 16th second the Foo() gets executed. Y might have added simple comparison of variable a to 16 and reset the same.

Later Z came to fix performance with  hardcore bits and bytes computer science background and he changed the simple comparison to bitwise operation.

If somebody thinks like this, he is perfect fit for legacy systems.

Puzzle to fix something with minimal changes

Another important ability required for legacy developers is to do things with minimal changes. Better they should fix the defects by removing code than adding more. We can easily find out these type of scenarios in the legacy systems. If the candidate is able to fix by removing code, hire him immediately.

Refactoring with tools

Refactoring is important when working with legacy systems. It doesn't mean one person should sit for months and refactor entire code base. Instead whenever we do a change do a small refactor. Instead of doing it by hand, it should be done with tools such as refactor menu in Visual Studio. Testing a candidate for refactoring skill requires machine test where we can observe what is his/her approach towards refactoring. They should be refactoring only after making sure its not invoked from outside via reflection. Also look whether the names they are giving are meaningful or not. If someone don't know how to name things and refactoring, its same as creating legacy code in the first place.    

Code quality in legacy projects - Different approach

Dealing with legacy project / code base is a tricky thing. This issue is there in other engineering disciplines such as civil too. Think about a civil engineer, who want to do something to an old house. It may be taking an extension or building upstairs. Should he spent time on fixing the issues in the existing house or concentrate on the new construction? If he spend more time on fixing existing house, such as strengthening, that work may not be visible to the stakeholder ie the house owner. But that is required to carry out the extension/new tasks. Convincing house owner on how much to spend and where to spend is a difficult job for the engineer.

Lets come to the software engineering. Engineer is asked to work on a project to add features which has legacy code. After the first sight of code, we could see its not following any basic principles of coding. Classes with 1000s of lines and all methods written as public, everything coupled each other etc...How do we convince the stakeholders, it require refactoring before we start the new features?

Sometime in new projects itself, we feel that developer are spending more time to fix issues and all due to bad code and it requires refactoring. But how can we identify what areas require refactoring. The ideal case is to refactor the entire application, but most of the time, if the code base is large, its not practical and cost effective.

Lets go back to civil engineering. If civil engineer has some proofs such as cracks on walls, leaks or thin walls he can present the same to the owner and get the approval for strengthening. But how can a software engineer find out such proof to do refactoring?

The answer is he need to find out the hot-spots where the code is hot.

Cold code

Lets understand what is cold code. This is the code which is dead/unused/less changed or just works! We can see in any legacy systems more than 50-60% code is not changing though those are written badly. Sometimes that code is not getting called or even its getting called, it does it duty. Bad code doesn't mean that its not doing it duty. Bad code here refers to wrongly structured code which is difficult to understand and maintain. Maintain means adding features and fixing defects. If the code is working as per the expectation, should we change it just to keep up with the coding standards and to make the code beautiful?

Can we write perfect code

Programming is more to art than engineering. In art there is no perfection. Our today's perfection may not be tomorrow's. It also changes from person to person. So whatever we do to our code to beautify today, after some time we feel that its legacy and needs refactoring. One another reason for that is, in programming there are many ways to achieve same thing. One person think one way is great, the next person thinks great way is something different.

Unless we have unlimited funding, its not practical to refactor a large code base entirely. So we need to find out the hot spots which require refactoring to add value.

Hot code

What is hot code? We can say that its the code which needs to be refactored first. Below are some examples. We require the help from source control systems to find hot spots where hot code is located.

Code that continuously changing

If a portion of code getting continuously changing, we can say its the hot code. Remember the open closed principle. This continuously modified code is not following that principle and refactoring that area will help us in long term.

Code that changed my different teams

If a portion of code is getting changed by multiple functional teams, we can smell one more area of hot code. Ideally, if that portion is following Single Responsibility Principle, only one functional team will be touching that code. Another problem, if multiple teams are touching same code is lack of responsibility. If something breaks in that area, nobody will take responsibility as many people are working on it. Refactoring that portion will help to avoid many things.

The best example will be a serializer / other generic utilities getting changed by multiple teams. If a team doing 'User Registration' feature is changing the serializer, we can easily identify that serializer is not generic and needs refactoring.

As we have seen, these method work by analyzing the source control systems.

Bug prediction

Even if we present the above proofs which demands refactoring, stakeholders may not agree for refactoring. So we need to explain other advantages of finding hot-spots. One is bug prediction. 

The concept is simple. If some code is changing and the reason for change is defect fix, we can predict that new changes in the area will introduce more defects unless we refactor. If we refactor that area, we can reduce the cost for defect fixing. Stakeholders always looks at money and quality to the end user. If the defects are getting reduced which improve the quality to end user, they will agree to refactor.

Below are some links related to how google used this source control based technique.

http://google-engtools.blogspot.com/2011/12/bug-prediction-at-google.html
http://www.google.com/patents/US20140033174

After implementing, there were studies on whether this really helps humans and could see that its not able to tell developer, what is the bug exactly. It just tells there are chances for bug in the file as its changed frequently.

Below are some implementations of the google bug prediction algorithm.
https://github.com/igrigorik/bugspots
https://github.com/niedbalski/python-bugspots

Tools

There are other tools as well for analyzing source control systems and extract information.

http://people.engr.ncsu.edu/ermurph3/papers/icsm11.pdf
Code maat

Moral of the story

Next time do not just say, we need to refactor the entire code base and stay without any answer when stakeholders ask why we need to refactor and what is the estimate . 

Instead present proof that these are the hot-spots we need to refactor and we need these much time.