TDD and design

Earlier I wrote about the long distance I went with mac driver design (link). The current design is sketched below with examples of functions, and responsibilities (green). The clouds are C files. The design has 95% unit test line coverage. Tests have proven their power. I refactored the code using a local repository while traveling on vacation on remote island (during off-days from diving) and obviously no access to real target for testing. I made 53 commits. My commit frequency is very high, so many of the refactorings were just renaming and extracting helpers, but there were also more fundamental design changes. When I finaly, and sadly, made it back to the lab, I was kind of afraid that the code won't run and the fastest thing to do is to throw away all the refactorings. The next fastest thing would be to repeat them one by one in real repository. I gotta say I was surprised when the code worked right out the cross-compiler and all I needed to do was one massive merge from local to real repository. This is very rewarding. During the refactoring there was a handfull of incidents when tests caught a stupid mistake made by me. This is worthy even if you had the access to real target. The nice thing is that unit tests on dev environment tell it right away.

There is no need to make tradeoffs and large/long changes without feedback because of lengthy burning times, or lengthty stepping path to debug newly written code.

But it wasn't the biggest learning. The biggest learning was that the design resulted quite the different from what one would expect. I base this claim to investigation of several example MAC driver source codes available in the internet. The design has proven to be good, in terms of testability (that was the driver) and adaptability (this was the proof). More about adaptability later below. The thing that differentiates this style of design is the emphasis it puts to testing. Design is good if it is easy to test. If the tests are complicated to understand or difficult to write all together, then there is a good chance that design has flauses.

I think what I experienced is well explained by Michael Feathers in his talk "The Deep Synergy Between Testability and Good Design" (video). Take a look, and don't think this applies only to OO languages. You'd be wrong. The driver we are talking about here is written in C.

Current design was tested when the hw team decided to have a second option for MAC driver. They wanted the final pcb so they could proceed with emission tests and we together did not have enough information to do the decision either way. The candidate for production pcb has routing for both options. One implementation of set based design. But back to the sw side of it...

The concepts in driver design kept most of the files completely untouched. SPI and DMA drivers were independent compile units and they needed no touching. This also means that no code was duplicated in the production code mass.

The original design was done with just testability in mind. At that time there was no knowledge about the extra hardware the design needs to comply with.

In my opinion the code became adaptable and reusable by designing it for testability.

I don't think writing code this way, by seprating concerns, focusing on single responsibility, and not mixing abstraction levels, is slower to write. Is it different? Oh yeah. You have to really develop a new sense for good coding. As a remark, which I did earlier, I wouldn't refactor the code after fiddling around, but write a decent design based on learning from exploring, aka spike.

In the current MAC driver code one detailed design decision may make you raise your eye browse; Single function ClockByte() is in separate file. This is because I wanted to assert through just the bytes been send, not through processor register dummies. Other option would have been to inject a function pointer for this. I chose to use link time seam.

On the other hand just few simple tests for basic correctness of ClockByte() function are enough. This can also been seen as principle of separation of concerns and keeping the files at the same level of abstraction.

Current design is far away from being perfect. It is not what I think should be achieved. It continues to offer me opportunities for deeper understanding of tdd, and synergy between design and tests, more deeply. The next lesson will be available when it gets factored to enable irq based interfacing between uC and mac driver. So far it has been just message polling.

Those Stupid Moments - Round and Round Again with Unit Test Tools

It is time to feel stupid again. We have been experimenting TDD of firmware as you know if you have been reading this blog. We started with Embedded Unit framework which I was familiar with. We wanted to make it simpler to be able to run unit tests on target with very low resource budget. Then we realized that Unity framework is exactly that. Atomic Object has documented their thorough test-driven approach and open sourced many of their made-in-process Ruby tools. There's a nice article on the approach in Methods&Tools.

At first we did not get to use the other Ruby tools from Atomic Object apart Unity unit test framework, but implemented similar, but simpler, in Python. That was pretty much because we did not "get" the tools, being electrical engineers and all.


Then we realized that we do not have the time and energy, nor the skills, to develop these tools seriously, but should use existing tools out of the box as much as possible. We started using Atomic Object's Ruby tools. So now the same project is using its third tool set for unit testing.

Rake for building the tests

I can't do magic with Rake - yet. I get the build script to work, but it is structured pretty much like scripting languages I'm used to, and most of the things are more complex than they should. I however have gotten a sight of its power and it is definatelly my choise of build scripting at the moment. The online manual has simple examples of using Rake to build C projects. It makes a good starting point. Advanced examples can be found in Atomic Object's tool distribution. You can check some more in-depth analysis on Rake by Martin Fowler here.

Unity unit test framework

Just above in complexity of pure macro implementation, or standard assert's, is Unity. It's written in C, with few helper macros. It's intelligence is in it's simplicity. There is no support for suites or anything like that. What to build and run can be grouped in build script.

Here's what you need for a unit test in Unity:

static void testAfterAdding4NodesTheSizeOfListShouldBe4(void)
{
  int a;
  struct list list;
 
  list_Init (&list);
  list_AddHead (&list, &a);
  list_AddHead (&list, &a);
  list_AddHead (&list, &a);
  list_AddHead (&list, &a);

  TEST_ASSERT_EQUAL(4, list_GetSize (&list));
}

int main(void)
{
  Unity.TestFile = __FILE__;

  UnityBegin();
  // RUN_TEST calls runTest
  RUN_TEST(testAfterAdding4NodesTheSizeOfListShouldBe4);
  UnityEnd();
  return (int) 0;
}

If you don't use mocking techniques you can just define RUN_TEST macro to call your setUp, test, and tearDown functions. That's it. In full approach it will call your mock module setup and verify functions also.

Cmock - automatic mock file generator

The other day a colleague interested in TDD'ing C said to me; "the biggest problem to me is that in C we allways need some real friends for a module under test. We do not have real mechanism to support interfaces." Well yes, and no. We concider interface to be the .h file, or in more advanced cases a struct of function pointers. After that we get to automating stuff. Cmock is a tool that generates mock functionality based on .h file. It goes through all .h files in your source directory and generates a corresponding mock.c files. These mocks can be linked together with the module under test and can be teached to expect calls and to return values. Here's an simple example of a function definition and resulting teaching function:

uint8_t Protection_filter_getState( uint8_t me_index);

void Protection_filter_getState_ExpectAndReturn( uint8_t me_index, uint8_t toReturn);

Argent automatic test code generator

It's a pain to add mock init and verify functions, as it is to add a call to test function to main function, or just to change a name of test in two different places. That's the pain - Argent is the relief. It uses a simple markup scheme, and interprets your test code in order to automatically insert the calls to appropriate test functions, and mock module initialization and verifications.

//[[$argent require 'cmock_injector.rb'; inject_mocks("lists");$]]

// Calls to mock objects will be inserted here accoring the included mock headers.

//[[$end$]]

//[[$argent require 'generate_unity.rb'; generate_unity();$]]

// Calls to test functions (identified by test prefix) will be inserted here.

//[[$end$]]

Mock dependency check

It's also a pain to add mock dependencies to linker command. Using the same interpreter as argent the mock dependencies can be checked at build time automatically. So all that is needed to take a mock object into build is to have it's header included in the test file.

So that's how far we are now. Now we need some experience to get our routines right.

I quess this is a good example of having courage to improve when we see an opportunity, and strenght to admit being dead wrong - over and over again. This is truly a learning experience.

TDD Tools

Using TDD for firmware in C can be quite laborious. Creating a new module requires at least steps like:

1. Create a test .c file
2. Add created test to test build
3. Create implementation .c file
4. Create implementation .h file
5. Add implementation to production build
6. Add implementation to previously created test build
7. Create fake object(s) to make the test build to pass
8. Add fake object files to test build

When you adapt the habit of testing everything you quickly get bored repeating these tasks. You end up looking for tools to automate this repeating work. At least this is what you should end up doing, and not giving up.

Atomic Object has created tools in Ruby to do the trick and some. At ESC2007 they demonstrated their tools and already gave the tools on CD, but there still was some minor problems getting the tools to build. Now they have released their tools as open source and fixed the small problems in the earlier distribution. In Summer Issue of Methods & Tools is an article elaborating Atomic Objects approach to agile firmware development infrastructure. Highly recommended starting point, and offers also a path for advanced testing.

Just tried out the package that is now available from their site - Worked like a charm!

Tools do really not need to be complex to make a difference, propably quite the opposite holds. We made file templates and bunch of scripts running sed to replace key words to automate C unit testing steps. We learned this from James Grenning's CppTestTools. From Atomic Object's package we adapted their simple unit test framework Unity. We stripped it down even further, and also mixed it up with the idea of different outputters libraries, like in Embedded Unit. We also adapted their idea of automated mock object creation, but ended up writing similar tool in Python (basically because we wanted to fully understand what we are doing). All our tools are tuned for our use and constantly evolving, but I think towards similar solution as already in use by Atomic Object, but in step-wise manner. We believe that you need to understand the reason, give it the context, for any new skill or practice in order to make it truly work.

Wrap It Thinly

The major concern about TDD in firmware development I hear is the hardware relation.

-"We don't have hardware to run the code in."
-"Every part of the code is hardware dependant."

I noticed I wrote about these approximately a year ago for the first time (I, II, and III). That was more on the theoretical side. This time I have a bit more experience on the subject. We have found Alistair Cocburn's article on hexagonal architecture valuable in philosophizing this issue. Many firmware developers are not familiar with design patterns, but a good starting point is above mentioned article. Writing adapters for your hardware, grouping them to ports, keeping the number of them low, and only allowing them to be responsible for the lowest-level of hardware interfacing, will help you isolate hardware dependant parts. Here is a picture of hexagonal architecture for the case project we are working on.




Then there is a philosophical issue of testing the adapters, or wrappers. In many cases binary input/output handlings are just like accessors, getters and setters. In TDD literature it is generally agreed that these do not need unit tests. When writing hardware wrappers we need to think a bit harder. We could understand the unit tests for hardware wrappers as our firmware/hardware interface specification. Then for sure it makes sense to write these tests. My thinking goes along with "to be able to judge what to unit test and what not to unit test, you have to be able to unit test everything".

Test the Goal of the Code

It is time to get technical again. We have been writing unit tests for some time. We have tried test-driven development. However we have never been disciplined in this. Sometimes we don't write tests at all. Sometimes we don't fix the test suite when code is needed to change. Sometimes we just decide that well, this project or that part of the project does not need tests (typically when it seems to get complex or difficult to test!). We have come to a conclusion that with this approach we will never totally master the skill, and not get the benefit it is supposed to bring.

We decided to try another approach. We are writing a very real hard-time constrained, very hardware related code with 100% test-driven development (TDD) in C. A code exists to do the job, but we are gonna completely re-write it. It is originally ported to C from assembly, and this is done maybe four years ago. Since then it has been improved, modified, and adapted to new situations. What has happened is the code has rotten, it is a jungle of conditional compilation, and so forth. So what is better place to do a tiny bit of always refreshing skunk work and completely re-write the whole piece of art?

Right after we made the decision that we are gonna write EVERYTHING with test-driven development we noticed a change in thinking. We constantly think for the simplest possible thing to do, constantly re-evaluate the module (C file) interfaces and so on. If you are gonna do everything in this manner, there just is no backdoor for you.

Another big change was (well, we still need constantly remind ourselves about this) the shift in thinking towards writing the tests as specification. We often ended up thinking about the solution and then writing tests to check the solution. This is maybe caused by the fact that one shot on the code exists and is thoroughly known. However, you will create novel, simple solutions for your tests only if you think about what you want your next small code-to-be do, not how it will do it. Keith Ray has listed questions to help you in this.

Sean Shubin has done a nice work and listed first guidelines to TDD. Again:

Test the goal of the code, not the implementation

Story: Spike Up Your Coctail

I posted earlier about short, 6 week, experiment where we had weekly deliveries of firmware and true parallel co-design of plastics, electronics, and software/firmware. I mentioned I would try to write down experience report. Well, it became more like story, and you can find the whole of it here (pdf). (LINK CORRECTED 27/6/2007)


Abstract. Agile development is a term used for wide variety of lightweight software development methods following shared values1. Many of these methods and practices however can be applied to more general new product development. This paper describes a six week project using some of the practices from agile development. These practices included self organizing team, collective ownership, continuous integration, iterative planning, iteration demos, team retrospective meetings, wall work queue, information radiator etc. The project crystallized a vague idea and a draft of electronics schematic into two fully functioning prototypes. It is shown that meaningful functionality can be developed in just six weeks. This is remarkable when reflected against the effort needed in so called traditional process models with formal analysis/design and theoretical proof of concept with heavy review processes. This is possible because of today’s advanced development tools and prototyping technology. Stefan Thomke (2001) calls this an era of enlightened experimentation. Building the working prototype is more cost effective way of reliable proof of concept. Using this approach already in the fuzzy front-end phase of the project would result in huge savings in overall project schedule and budget. It was also noticed that all stakeholders of this project appreciated the approach and considered it as “common sense”. The positive side effect was the team building effect this period had.

What does it mean to be agile?

OK, the ESC2007 Conference is over. Many of the classes and especially Peer Roundtable discussions at the conference, Beer Roundtable discussions at the bars after conference hours, and long flight hours with a colleague, woke several questions, among them:

-What does it mean to be agile?
-When is an organization/team considered agile?

In his review (pdf) Dr. Pekka Abrahamsson showed the software life-cycle support of different agile methods. We know that Scrum does not have any engineering practices described in detail, but focuses on project management. That's why it has been adapted also outside software development. It is simple. This is why majority of the so called agile adaptations at least say they follow Scrum. As have we; we have used it with cross-disciplined embedded system development teams to develop a complete embedded system as fast paced iterative development. I have also written in my paper how I saw reflective iterative development as a tool itself for figuring out what practices you need. This said I share Simon Baker's post

Are you missing the point?

As long as you share the underlying values of agile development, and keep the obvious practices like short iteration, demonstrable progress and retrospectives, you are going to figure out the rest of it - or die trying or at least you figure out the need for something and can start looking. If your survival anxiety in your current environment is high, but you have juniority prevailing in your team, you can still gain a lot from practicing "just" Scrum. Scrum puts the basic structure in place with its simple rules and in my experience this is an immediate relief to chaos. This does not mean that it is easy. It will make everything visible; good and bad and needs a lot of discipline and nurturing.

With this approach, you being now out of the chaos and anarchy, it is time to bring in the engineering practices. Our case has shown that they are needed. We have experienced with eXtreme Programming (XP) practices; collocated team, pair programming (development), test driven development, coding standard, metaphor, and so on. We have seen the value in them, but not been able to actually master them in firmware or embedded system development in order to put the discipline into using them. Still we feel that we gain from practicing Scrum, and agile development as learning-driven thinking. This is the very reason why we still keep learning more skills and practices.

Are we agile? I do not know.

If your environment is functioning and open minded, and you have master mind developers in your team, why not go full blown XP right away? We have manifested that those practices are of high value, but they are hard for beginners. We now have also hard evidence to support the effectiveness of these methos (see for example the AGILE-ITEA project, run by the same Dr. Pekka Abrahamsson btw). It may be that with this approach you find out that you need some project management practices from Scrum. Who knows.

I wrote in my original paper about agile firmware development that lot of the agile programming practices for firmware development need innovative thinking. I think last week at the ESC2007 I met some of the people capable doing this. In the front line James Grenning from Object Mentor and Mike Karlesky and Greg Williams from Atomic Object. They have put these engineering practices into full use (also) in low end firmware projects. They are also willing to share their knowledge by publishing their tools as open source, and having discussions about the methods. Maybe we even become clients of these guys some day... Furthermore during the classes and discussions I got the feeling that a fair amount of people are actually practicing.

We have just scratched the surface, but after last week I believe that it is time to put some serious effort into agile programming practices, and actual firmware development. After all that is where we started this journey. It just kind a slipped into system development.

Are we agile after adapting these practices? I do not know.

TDD'ing State Machines

I'm an electrical engineer. Everything in programming seems to be very very hard for me. Last week I spent 40 hours in aeroplanes, so I had some quality reading time. I did read Agile Estimating and Planning and Test-Driven Development from cover-to-cover. I had quickly gone through them earlier, but this was enlightening experience.

First of all, I have told that we have been trying to figure out effective TDD for our FW development. To be honest, we've struggled. We wrote tests for a state machine implementation. We expected the tests to be run in certain sequence, and that the state was always left where it should be after the test. We of course ended up in lot of rework on test code when our state machine design emerged. We thought that we need to design the state machine up-front in order to be able effectively write tests for it. This did not sound like test driven development at all. This required some more thorough thinking.

So back to my flight. Kent Beck writes that coupling is bad also between tests. A test should leave the system as it was. So writing unit tests in the same sequence as you expect your state machine to work does not seem like a right thing to do. You should be able to execute individual tests and to change the order of individual tests. So instead we are starting to recognize a single state as something to test.

Ph.D. Miro Samek writes about state machines being a killer app for function pointers. He further defines a design pattern for state chart implementation. Our state chart interface has become to be something like:

construct()
dispatch(event)

construct is self explaining and dispatch dispatches new event. We have specific event tick that gives resource time to state machine every 10ms (based on our 50Hz mains). We could also implement a timeout as OS service, and not to have local counters for timed events. Anyway, if we implement each state as a function as Mr. Samek proposes, and we hold our state as a function pointer we can write

dispatch( me, event )
{
me.myState(event);
}

We get rid of cryptic switch -structures. This solution helps us to unit test each state as a function with natural feel.

I'm going to write some benchmarking code to see the actual overhead, but at the moment I like this idea.

Driving Embedded Firmware C Project on CruiseControl

I finally made it. A dedicated page for configuring CruiseControl for embedded firmware C project. It combines some of my earlier posts, but also tries to add detail. Take a look here, and tell me what is missing.

Weekly Delivery of Firmware

Yesterday we finished a six week project developing technology platform for User Interface innovation using some practices familiar from agile methods. The project delivered two different configurations of home automation user interface device; one for wall mounting and one for remote control. Some facts:

• Vague or missing requirements
• New 8-bit microcontroller
• Lots of new sensor technology
• New embedded firmware team of 3+1, including one consultant (in house)
• Some drivers developed by outsourcing
• Final project consisted of 20KLOC of firmware C source (bad meter, but...)

Some findings after initial reflection:

• Weekly delivery (not release) of firmware is possible, at least in some cases.
• Short term goals and daily Scrum meeting effectively work as motivators, and enforce full learning and getting things done done.
• There is not small enough project to not use version control and continuous integration (we had no way to have common version control with outsourced team: large org, IT, you know the drill...)
• HW evolved from microcontroller starter kit to final electronics via several bread board mocks and one halfway PCB prototype. This causes extra assembly work when looked at the surface, but this overhead is by far overcome by enabling the concurrent engineering instead of waiting for final version.

The last point holds only if all the disciplines work as One Team.

We did not have cross-functional team as we did in earlier larger experiment. Schematics, PCB layout and mechanics were developed concurrent, but they were not part of the team - nor did they follow any of so called agile practices. Firmware planning was tried to be synchronized with deliveries from other disciplines. In several occasions these deliveries were delayed, and a common goal needed for a team to actually be a team was somewhat missing. This is where I would like to do things differently if I was to do it all over again.

Occasionally we heard sentences like "If everything goes smoothly we will be ready on Tuesday, but we will be ready at least on Thursday". At this year's Agile Business Conference in London David Taylor (The Naked Leader) said "Don't have a plan B because you will always achieve it." Just guess if the delivery was on Tuesday or Thursday - or Friday?

The whole project was started by extremely vague idea. In few hours we worked out initial backlog and a delivery (release) plan based on simple themes. From that point on plan, design and requirements emerged during the 6 weeks nicely synchronized with weekly demonstration of prototype and new planning session. This experiment supports the belief that good people can work this way, and that we truly live the era of enlightened experimentation. We believe that we could not have performed any better with thorough analysis, design, and specification phases - at least not to cover the cost of them. During the development there were several points where it was not possible to proceed as we had thought. A comprehensive plan based on these initial assumptions would have failed miserably, and the time creating them would have been wasted. In this project the whole team worked out an alternative solution, adjusted the next week's plan, and the project was fine...

I will write an experience report and hopefully make it available here.

The Lenghty Debugging Phase

To give embedded firmware TDD a kick start we have had two sessions with a colleague. During these sessions we have created scripts to automate the unit test structure and build process and discussed how the make as thin wrappers for HW as possible. However we learned also other valuable lessions during these sessions. Lesson number one is the "debugging phase" being so strongly in our domain culture. We often hear a firmware developer saying, "well, it takes couple of days to implement it, but you can never tell how long the debugging is going to take". Of course debugging phase is where all the firmware heros truly shine. That's another unfortunate culture issue in firmware development.

In our first session we ran into problem of not getting one test running. Immediately we started debugging. Printf's fly in here and there (as an advanced host run I/O twist). The change to this mode was so rapid, that neither of us realized what is going on until later. After some 30 minutes(!) we recognized that we are not moving, but just desperarately experimenting with printf's, and it hit us; we have not changed our behavior as we were supposed to. We should have picked a more simple thing to implement. When we realized we can not chew what we had chosen, we should have started all over again with simpler thing.

It is our belief that by doing TDD for firmware we should get rid of, or at least dramatically reduce, the debugging phase and thus create an atmosphere of success, and improve the reliability of estimates.

This type of sessions are great tools for learning. I strongly share Michael Harmer's opinion on The Training Course Scam.

Ken Schwaber on Quality of Code

Ken's presentation "A Canary in a Coal Mine" at Agile2006 conference is available via InfoQ. Ken talks about quality and the danger of Scrum teams droping quality issues to get higher velocity and to be sure to hit the date. He calls this a re-definition of "done", or "somewhat done". We have manifested an increasing technical debt in our iterative work. So in the last project we were putting more focus to acceptance testing inside each sprint and introduced something we called "cross-testing". This means that other pair needed to test the feature implemented by other pair in order to claim feature "done". Embedded firmware systems need a lot of manual acceptance testing (at least our current practice) and it is easy to get slobby testing your own work over and over again.

Was the technical debt there because of Scrum? Well, no - it was made visible by Scrum and the team was able to decide to do something about it early enough.

----

Ken's another speech about Scrum is available at Google Video. (You will find out that Scrum works with idiots as well)

I posted about Jeff Sutherland's Scrum speech earlier. It is also available via InfoQ.