Software Development Principles

Hi all
I have often thought about what the proper software methodology should be for the various robots that I build. My thoughts have evolved over time as I have seen these tool work. I do not have any formal software engineering training, however these are things that I have seen, heard, read, etc.. that I believe in (at 2am while I write this).

Disclaimer: Most of the stuff here I have taken from other sources. Please check out the links attached for more information.

I am starting with how to write code since that is the basis of everything else. After that we will move into the software engineering infrastructure to make the software work properly.

General Programming Rules

Without having basic rules that you follow when coding, your code is doomed to fail and be unreliable. Everything else is built upon this. Many people start programming and want to use crazy template magic with everything inheriting something else, however you must STOP and write simple code that is easy to understand and test.

• Keep control structures simple – Limit your branching, don’t have long nested if, case, and loop statements, no recursive magic.
• Endless loops – Do not use boundless loops unless you really want to be stuck there forever (such as the main loop). Everything else must have an upper/lower bound for exiting the loop. Think more about for loops than while loops.
• No dynamic memory – Forget that malloc exists. It can cause programs to act differently at each run time, also can add memory leaks which can be hard to detect.
• asserts – asserts are good since you can enforce that the proper thing is helping in each function. I generally do not like the default assert functions that just cause an exit; sometimes that is appropriate but in many cases you should try recovering from the error instead of just exiting. You need to evaluate which is the safer approach in your application.
• Keep scope – Try to assign and initialize variables, structures, etc.. in as small of a scope as possible. Included in this is limiting your usage of global variables.
• Pointers – Limit your use of pointers when easily possible. You should really avoid multiple levels of pointers. They can be very confusing to follow and lead to all sorts of problems if you reference memory addresses that you should not be touching.
• Check return value from all (non void) functions – If a function can (and should be able to) return an error value make sure to catch it and act on it. It can also be good to check the bounds of a value returned instead of blindly trusting it.
• Turn on all compiler warnings – So you catch things early. This can especially be true when formatting text, working with memory, and when bit banging.
• Use parentheses to control order of operations – I have seen many people assume or forget about the proper order of operations. Using parentheses can clarify and enforce the desired result.
• Hard coded number – Please use variables for any value that might need to be changed by a programmer. If you have an even slightly non obvious number, give a comment about the source of that number and how to derive it.
• Preprocesor – Limit the preprocessor to “simple” things, such as #define and #if. Using complex macros can lead to hard to understand code, poor portability, and a increased probability of bugs.
• Standards – Keep your code consistent. This includes naming, white space, and where your curly braces go. This makes it easier to follow the code. If you are editing existing code use the format of the existing code.
• Sleep with noop’s or empty for loops – These should be avoided since they will run differently on different systems. This is primarily for embedded systems.
• Document your code – Need I say more…

Links:
Rules for defensive C programming
The Power of Ten – Rules for Developing Safety Critical Code

Software Repositories

Having a code repo should be a given for every programmer and programming team. There are many free tools available so you have no excuse for not using them.

Having a software repository lets you save your code at various points so that you can roll back to prior working versions. Part of the repository tools is having a log of what was changed to each file and by whom. These tools are also great ways to share code amongst a team so that you are not emailing code back and forth among people. You can also put documentation in the repo so that is is always available.

There are many options to choose from cvs, svn, mercurial, git, bitkeeper, etc.. Just pick one!

Lately I have been using a hosted repo service called github.com since I like all of the tools that are packaged with git on the website. bitbucket.org is another similar option.

I generally like creating repositories with the following directories in the root directory:

• bin – a place for all of the binaries after being compiled. By default this directory should have nothing checked into the repository.
• config – If I have lots of config files I will put them all in this directory.
• doc – documentation for the system (datasheets, diagrams, notes, etc..)
• externals – source code and libraries from outside parties (ie. you did not write them)(
• includes – header files that other multiple programs want access to at compile time
• libs – a place for all of the libraries after being compiled. By default this directory should have nothing checked into the repository.
• src – this is where all of the coded we write should live.
• tools – this is for random “tools” that do not fit anywhere else.

In addition to those directories above. I will usually put the main make file for the project and a README in the root directory.

If using ROS the directory structure will be different than the one above to conform with ROS’s ideas on packages and using catkin as a build tool.

Links:
Software repositories or just plain repo

Bug/Issue tracking

Having a central database to track bugs is very important for developing great code that runs bug free. This lets anyone identify a bug and then report it in a way that it does not get lost. Just telling the developer or emailing them is not sufficient. People forget things easily. Also you do not want the developer to close a bug; you want somebody to report the bug, the developer fixes the bug, and then the original person who identified the bug should verify that it was properly fixed.

When you file a bug report it needs 3 things:

1. What you expected to happen
2. What did happen
3. Steps to reproduce observed behavior (ie bug). Try to minimize the number of steps needed to reproduce error. Include versions of code that you are using.

Links:
Painless Bug Tracking

Automated Testing

Each of your processes/functions should have a set of tests (often called “unit tests”) performed on them to verify functionality. These test are good to make sure that functions work with various input extremes and combinations. Ideally you should have a test for each possible control path within a function.

You can easily write these test for example if you are testing a “sum(number1, number2)” function you can have the following test file:

int main () {
    assert( 2 = sum(1,1));
    assert( 0 = sum(-1,1));
    assert( NULL = sum(1,NULL)); // or whatever you want it to be doing…
    assert( 0 = sum(0,0));
    assert( 1999998 = sum(999999,999999));
    assert( 5 = sum(4.2,1)); // assuming integers and that floats are rounded.

    RETURN (TRUE);
}

In this example assert will print an error message about where it failed and exit.
(This is just my quick example. It can definitely be expanded and improved. It can also be changed to fail at compile time so that you do not need to “run” it.)

Automatic (Daily) build system

Having a build server that builds every time somebody commits code or at minimum every night, is valuable for identifying inadvertent bugs introduced into your system and automatically running your automated tests.

Often you will make a change to one module which will cause another module to fail. But as developers we get focused on our modules and tend to not try recompiling all of the code to make sure that we did not break anything. It is always better to find out you broke something earlier than later.

Certain things tend to cause other modules to break more than others. For example if you have a widely used header file or message definition file, making a change to those can widely affect other items.

This also makes programmers only commit code that compiles to the repo and not broken code. If they commit broken code the build tools can/should sent out emails to notify people that the build failed (and whose fault it was).

You can either just make a quick script to check out the repo, build it, parse the output and email the results; or you can get a proper build system such as Jenkins. Generally using somebody else’s product will be more reliable and a better use of your time.

Links:
Daily Builds Are Your Friend

Code Reviews

Code reviews are tough, but they are also one of the best ways to reduce the number of bugs in a piece of software. The traditional code review where a person gets up and talks others in the room through the code (that the people have never seen before); DOES NOT WORK! This might be good for getting people familiar with the programs operation, but it is not the best way to do a code review and find problems.

To do a code review you should have the programmer send a review request to 1-3 other software engineers. The review request should have an overview of the code, a flowchart, and directions to access the code. The number of engineers and the experience level of those engineers that need to review the code should be based on how critical the code it. After sending the code each reviewer should:

1. Review the code (obviously) – Don’t dwell on style, you have better things to do with your time.
2. Develop a list of questions about the code
3. Write some tests and make sure they pass (these can be committed to always run in the nightly build)

When reviewing the code before the large review some of the things the reviewers should watch out for are:

• Does the code do what it is supposed to do (and not do what it is not supposed to do)
• Should the code/function be written in a more efficient manner
• Is the control flow simple/clean/good (if, else, return, etc..)
• Do switch statements have a default catch all
• Is the code commented adequately
• Are variables declared at the proper scope (minimizing globals)
• Memory leaks
• What happens if things return NULL?
• Error/Exception handling
• Buffer overflows (doing raw things with char’s, int’s, etc.. memory locations)
• Pointer and pointer based math
• Array indexing
• Closing all handles (files, ports, etc..)
• Macro usage
• 64bit vs 32 bit modifications
• Multithreading (is it needed?, number of threads, race points, deadlocks, data passing, priority inversion, etc..)
• Operators (precedence and proper usage [ & vs &&, = vs == ])
• If input items from users and functions is checked before being used

(Did you notice how this list is similar to the good programming practices above)

After doing the above steps it is now OK to hold the big meeting where people sit down and walk through and review the code; although this is not strictly necessary.

Reviewing code can be hard. Getting your programmers used to reading code and reviewing it will help strengthen their own coding. It can also be useful to have dedicated QA people who review code.

Links:
Embedded System Code Review Checklist
Microsoft: Best Practices: Code Reviews

Build System

You should be using a build tool so that you are not manually compiling each file. Probably the most common one is make.

Each make file should have at minimum the following directives:

• all – Build all of the production code
• clean – Deletes the binary and object files. This lets the next build be completely clean.
• install – Installs the binaries and libraries into the proper location. This can be a process such as:
  1. Copy old binary to <binaryName>_<archivedDateTime>
  2. Copy new binary to location with active processes
• test – Builds a set of unit/system tests that verify operation and make sure bugs were not introduced.

I think you should be able to build, install, test, or clean your entire system with one command. As such I will normally create a make file for each processes, and then create one master make file that calls all of the individual make files. Having a single make file that can build everything is good for installing a new system and makes setting up the nightly build easier.

Summary

• Write good code.
• Use a repository service (assuming it is not super sensitive data) such as github or bitbucket. That gives you the repo, the issue tracking, and the ability to easily do code reviews.
• Setup a server that builds your repository and runs your tests, at least once a day

Do you have other favorite tools to make your software more reliable? Please leave comments below.

Check this article out in Chinese: http://www.labazhou.net/2015/04/software-engineering-principles/

Main image based on sample code from wikipedia and hammer from pixabay.com/.

Liked it? Take a second to support David Kohanbash on Patreon!