A New Home

A fiend of mine did fantastic job on my new site. My fuzzy ideas and his zero-whining attitude made a good pair. It was truly an agile project. The site was up and running from day 1, and it emerged during number of customer-developer pairing sessions. Final outlook is a result of countless "you know, why don't we just try it"s.

Embedded Agile, ESC2010, Boston

Here are the slides and technical paper from my talk on Embedded Agile at Embedded Systems Conference 2010, Boston.

I had a good time at the conference. Hope you can find the material useful!

Abstract. New product development (NPD) is getting more and more challenging. Change happens all the time in all dimensions, including own organization, technology, competition, and marketplace. Agile development is targeted at working in a turbulent environment driven by continuous learning. Originated from software industry, its applicability to embedded system development has been analyzed over the years. In this paper, I present some observations on implications of embedded system development to agile development. I introduce findings on frequent releasing, automated testing, co-design including non-SW development and quality systems like ISO9001.

See you in Grenoble

I'll be in Grenoble, France November 22.-24.11.2010 to attend Agile Grenoble.

If you happen to be around, get in touch.

See you in Boston

I'll be attending ESC 2010 in Boston in a couple of weeks. You can catch me speaking about embedded agile on Tuesday 21st. You can find me somewhere in the conference throughout the week. Let me know if you're around.

Fiddling around before TDD

The first of the Uncle Bob's Three Rules of tdd states:

1. You are not allowed to write any production code unless it is to make a failing unit test pass.

A few recent discussions among embedded developers revealed that this rule has caused some confusion among fellow beginners of embedded tdd. So, let me first point you to another tip from the same source(Uncle Bob advises to do fiddling around on things that you are not sure of how they work). I will share a short story:

I've been writing a proof-of-concept driver for serial to Ethernet controller. It was not sure if the serial port on uC could be configured to work with the MAC controller, or if it would be possible to use the DMA controller to manage longer transfers. I needed the proof-of-concept for the hardware team fast. I learned how to use the controller by trial and error, glueing together several bits from application notes and examples and running them in combination of two evaluation kits. Needless to say it was a mess. It even turned out I couldn't get the job done without a few circuits from my hardware pals. It would have been really awkward and laborous to have tests written during this fast paced back and forth experimenting based on a sample code which of course didn't come with tests. All this was done in C, with the tools for C.

After I knew which bits worked and which didn't, I wanted to illustrate this learning in tests. I harnessed the quickly hammered code with tests and then massaged the tests and code hand in hand into better shape. In retrospective, I should have treated the original code as a throw away prototype (aka code from Spike). I thought I would be faster by continuing to work with the code I had. Sad, but lacking the discipline made my overall cycle time propably massively longer. I believe that this is more of a rule.

It might be from the first XP book, I'm not sure, but when I was first introduced to agile methods people always listed the last rule; rules are just rules. Based on this experience I do believe it is pragmatic to fiddle things around without tests. But when you start sculpting the solution towards production code, you should take a fresh start and drive test-first based on your newly acquired knowledge. I know I will next time.

Test Driven Development for Embedded C in beta

James Grenning's book Test Driven Development for Embedded C is now available in beta from The Pragmatic Bookshelf.

I have taken a peek and checking it out is strongly recommended.

Bowling Game Kata in C

Olve Maudal has shared his presentation of Bowling Game Kata using C. You can find the slides directly here (pdf).

I'll add a link to my TDD in C delicious.

Coverage with lcov, and so what?

A while back we ran an experimental line coverage analysis on our acceptance test suite. The result was 68% on the code for the main control board. I got the result from nightly build and mentioned it in Daily Scrum, and prompted "so what do we think about it, should we track it"? Everyone on the team had a blank stare and then finally a team member came forward "yeah, that's a good question. So, what?"

Coverage is information. It is just that, an additional piece of information, not by any means the final truth. I don't remember who teached me this, but;

"If you take all your assert's away you still have the same coverage. You just ain't testing anything at all."

This has been explained here and of course in Brian Marick's classic How to Misuse Code Coverage (pdf).

Well, maybe the good coverage can not say anything about the quality of your tests, but poor coverage can certainly say thing or two of opposite nature. If your coverage is 20% we can say quite confidently that you ain't there yet.

I started with acceptance test line coverage, but the rest is about unit test line coverage. Some embedded teams use gcov and I have heard people fiddling the data to generate fancier reports. Being lazy as I am I didn't do it myself. I did what I'm good at and searched for what others have already done. I found lgov, which is a tool in Perl to format gcov data.

We run lcov under Cygwin. You can get lcov for example from here, extract, and execute "make install". Next compile and link unit tests with gcc using flags "-fprofile-arcs" and "-ftest-coverage". We have a special build target for intrumenting the unit test executables with debug and coverage information so that we don't unnecessary slow down the bulk of builds. Next execute your full suite just like you normally would.

In our case all .obj files from test build are in ./bin directory, and that's where all the coverage data files go to. Our unit test script moves them to ./bin/code_coverage directory away from .obj files, and we want the final html report to be in ./build/test/code_coverage. Now we have the information necessary to create a shell script to do the actual analysis and reporting of coverage data:

path=`dirname $0`

lcov --directory $path/bin/code_coverage -b $path --capture --output-file $path/ic.info

genhtml -o $path/build/test/code_coverage $path/ic.info

Vola', your disappointing(?) results are ready to be browsed, like so:

What the heck, it's all green, while you only have tests for simple utils files? In this approach there is a limitation - you only get coverage information for the files that are involved with your test suite. With huge legacy code, this would yield too promising picture early on. Again you need to think for yourself.

Experiment with coverage in your team, I think it's worth every penny but even when you start closing 100%, remember to keep analyzing, "so what?"

Learning to cope with legacy C

New responsibilities during the past year have been a great learning experience. The key learning is that now I really know how incompetent I am. I can’t wait to move again and learn how many more things I do really badly, or what would be even better, can't do at all. This is a brief story of one such finding during this joyrney.

For the past year we have focused on ATDD with our own framework written in Python. We have 200+ automated acceptance tests for the system. With unit tests we however have struggled. While we have over 100 (well, it’s a start) of them, without the exception of the latest ones they are not really meaningful.

What's different with the latest tests then? They focus on higher level. I’m not sure what these tests as programmer tests should be called, but a programmer test will do for now. I do believe unit tests should be focused when doing TDD, but, wait, wait, I have an excuse… The code is old. It has its dependencies, and while maybe not the worst case in the world, it is a pain to get something compiled in isolation. The code has responsibility based structure (or should have had), and this structure is expressed in the source code folder structure. Each of the responsible "modules", or folders, typically contain own task. A typical task looks something like this:

task_specific_inits();

for(;;) {
s = OS_wait_for_something();
switch(s) {
case 1:
do_something1(s);
break;
}
}

Sometimes do_something1(s) is inlined and you may get a bitter sweet taste of those infamous 1000+ line functions. Other times you are lucky and the whole high level event parsing is already done in own function, along with lines do_something_with_the_event_from_X(s). This function continues the handling with loooong switch case, hopefully just calling further functions.

So, when we decide to test something inside a selected "module", or a folder in our case, we compile and link single test file, all the production code from a single responsible module/folder, production code for everything considered utils, like linked lists etc., and fake everything else. For faking we use Atomic Object's Cmock and manually written stuff when appropriate. We choose the task handling for injecting the test actions.

We arrange the test execution environment as we wish by initializing all the parties to expected state and teaching the mocked neighbours accordingly. We inject a single event, or short sequence of events, into task's handling routine and we try to find ways to assert if everything went as we wished for. Sometimes we can use this to learn what really happens when you give such and such event. After all the default assumption is that the code works, as it has been in production for years. We want to make sure it stays that way, when we change it. We have several options for observing the behavior:

1. Automatically generated mocks will tell us if the interaction was as expected
2. We can use getters of utilities, like linked lists
3. We can sense the internal status of any of the production code files with few nasty little tricks like #define STATIC

When the first test, and maybe her friend, is running it is time to start refactoring your code. Refactoring your test code, that is. If you take a closer look on what you have done, you most likely see 1-2 300 lines long test cases, which look pretty much the same. Now it is a good time to start extracting helpers. When creating an event sequence to be run you probably generate similar data structures. These can be extracted into functions. You probably do a bunch of similar assertions on many of your test. These can be extracted to helper functions. And so on, and so on. Each refactoring is likely to reveal more opportunities for cleaning the code. This can't be emphasized more. It is important to keep the code clean from the beginning. Otherwise you will have a 10KLOC test file on your hands, and it is much more work to start cleaning it only at that point.

This is very far from TFD (test first design). It is a battle to get some tests going to be in better place to continue improving and changing the code. The code is not going to disappear anywhere soon, so there will be lots of changes.

Why it took us a year to get to this point? Blame is on me. I got bitten by the test bug while writing a really hard real-time firmware app with a former colleague bunch of years back, and we learned that small exact tests leading into small steps of coding lead into zero debugging time. This was type of SW where we earlier had spent majority of our time debugging the code with oscilloscope and manually monitoring led blinks with throw away debugging code. During that experiment I saw the light (as saw my colleague), and thought that this is how also firmware should be written. Write each line of code to make a small test pass. However it is fairly rare in embedded domain to get your hands on a green project. This may not be a characteristic of just embedded sw, but sw in general today. We mostly write enhancements to existing products. Existing products in 2009 are not typically delivered with automated tests, and even less so developed this in mind. There is going to be plenty of opportunities for battles like this. Letting go on the ideal very low level unit testing took a year for me. It is still my ideal way of coding, but we can not get there overnight with legacy code.

If getting first tests in place sounds easy(?), calm down. It is only a starting place. You will notice how hard it is to test your code for example because of scattered initialization routines or that there is no structure in the first place. You should concider all these problems as good things. They are indicators for places of improvement. Those problems are in the code, building tests only make it more visible. If you work on those problems, you should be able to see more dependency breaking opportunities and eventually get to more focused tests. That’s the plan at the moment.

Michael Feathers uses term pinch point in his book about working with legacy code. Pinch point is a function or small collection of functions that you can write tests against and cover changes in many more functions. I guess event handlers for tasks are our first natural pinch points. This at least is the current step on the learning ladder for me. Hope the ladders won’t fall.

James Grenning also made a nice job articulating the whole legacy code testing process in C language (link).

Atomic Object also presented the importance of refactoring the test code from the beginning (link).

My Delicious Embedded TDD Links

After a hint from a friend I recalled that I actually did start to collect embedded TDD links on delicious.com a while ago.

So not having anything more interesting to do, I updated the collection a bit and placed a link on the sidebar.

My Delicious Links on Embedded TDD

The Complex Velocity of True Teams

Using velocity for long term planning of a project is a wonderful idea. With minimal effort we get fairly good estimates. At least this would be the case in ideal situations, meaning new team without maintenance or obsolete component replacement obligations etc. If you are a believer in true teams as I am, you do realize that all the urgent work and even absence due personal issues affect team dynamics. Just calculating velocity in terms of team member availability isn't realistic with true teams. The velocity, or team achievement, is not linear function of sum of available team member hours. If one person is for example pulled away from team activities for taking care of urgent manufacturing issue, this will affect the team dynamics. This has deeper impact than just missing those hours. To be honest, in some teams the impact can be positive as well. In that case I suppose they were non functioning work group, which now due the changed dynamics has a chance to become a team.

This said, no, we have not solved the issue of maintenance and other urgent work disrupting normal work. We are struggling with bouncing velocity. This is for several reasons, but it also woke me to think about true teams again. Anyway, I'd like to pay more attention to this and give true teams a chance to grow.

Even More Guns for Embedded Development

A while back Benoit saw the light of using modern languages (Ruby in that case) to help in developing embedded software, which still in most cases is developed in C for very well known reasons. I though I will throw additional ingradient to this soup. We are also using Python for our test-driven development, for acceptance test-driven development to be precise. There are few concepts that has lead us to do this.


1. You remember wrapping thinly with hexagonal architecture, right?

2. Company that I work for has a long history of using embedded PC simulation system called E-SIM from Mentor Graphics. This product has evolved, but we are using the old version.

3. E-SIM simulation tool has a Python API wrapper

A continuum is to develop own embedded acceptance test framework using Python, right? Well, we did it anyway. It is fairly simple, really. E-SIM communicates with the application interfaces using the mechanism called channels. There is a Windows service running called Communication Manager. And this fellow is responsible for gathering and sharing values in these channels to various C applications and our test driver written in Python. Indeed, we can have several applications running on one PC and simulate multi-device/processor/board systems as we wish. A channel can be a package in communication, a binary input as push button, binary output as LED, and so on.

Now, wanting to be able to write acceptance tests in a domain language that could be more readily understood by the business, we created a script parser using Python's reflection. This test driver parses the test script file and launches all the application .exe's as needed and then stimulates and senses the system using of E-SIM channel accessors. Sounds simple? The nice part is that it really is simple.

These tests run automatically on our continuous integration server, but we also have a GUI which simulates UI of a real system. Now we can watch the UI in action during the test sequence scripts.

Here is a snippet of .test acceptance test script written in (almost) domain language

sendFire 0,1,2,3,Delay Fire
wait 2
sendDelayed 0,1,100
wait 2
checkRow 1,FIRE in zone 00021/1

Here is the reflection part of the parser in Python. I have no idea how good of a Python code this is being an average embedded C programmer, but it looks simple to me.

for line in tp.getTestLines():
  self.currentScriptLine = line
  self.currentScriptLineNumber += 1
   
  if len(line) > 1:
    elements = line.split(' ',1)

    try:
      if len(elements) > 1:
        getattr(self, elements[0])(elements[1].lstrip(' '))
      else:
        getattr(self, elements[0])()
    except:
      self.myReporter.addWarning()
      self.warning("Could not execute line: %s" % line)

And finally here is a simple method in the TestDriver class. You can see that the method's name matches the action word in the test script snippet above. The rest of the test script line is passed as an argument to the method.

  def checkRow(self, parameters):
    elements = parameters.split(',')
    row = int(elements[0])
  
    expected = string.replace(elements[1], "\n", "")
    actual = self.currentLCD_Rows(row)
 
    self.checkEqual( expected, actual )

Similar approaches are Atomic Object's Systir, and Exoftware's Exactor. These can be labeled as script frameworks in contrast to table frameworks like FIT/Fitnesse.

Impossible, I Have Never Done That

When adopting agile planning techniques to non-software development activities an often heard argument against estimating the complexity of, say, schematic is:

"it is impossible to say, I have never done that before"

Well, firstly, in embedded software that's the situation pretty much allways, yet agile planning has proven to be efficient enough over and over again.

Secondly, we are living the era of enlightened experimentation. Prototyping hardware is drop dead cheap and fast nowadays. Buying yourself a status of "I have done that once" instead of "haven't done that before" is as easy as deciding to do so.

Thirdly, schematics can be estimated with relative complexity.

Fourthly, Parkinson's law works everywhere where humans are involved. Timeboxing early prototypes will help making the progress reliable and visible. This is however understandably very counter intuitive to the "get it right the first time" camp.

This is what is called up-front prototyping. In contrast to traditional hardware prototyping trying to validate something at the end, up-front prototyping focuses on learning. We may even know that the prototype will not work on most parts, but we just want a reliable measure of how far are we. It is even advisable to design a prototype to prove just one thing. More than one uncertanty will make the work unnecessarily complex.

This distinction between the goals of traditional validating and new era learning prototyping has gotten me concidering a new word to replace "prototype". So far the best candidate is 'product'. Using a word product throughout the lifecycle we would realise that this is aiming to production quality, but the maturity of the desing is evolving.

Of course I'm talking about prototypes that can be assembled within reasonable cycle cost. The bar however is getting lower and lower every day. Interesting exercise is to take your latest embedded development project, draw a bar of development salary cost and a second bar illustrating your prototype cycle cost. You may be surprised how cheap "expensive unnecessary prototyping" is in the big picture.

At least have a taste

We teach our children to at least have a taste before they judge anything as "yak".

We also teach engineers to try out practices before they judge anything as "yak".

"Just try it, if it does not work - ditch it".

TDD (test-driven development) is probably the most difficult technical skill associated with so called agile development especially in constrained embedded environment. It is also probably the most rewarding when in use.

The problem with TDD is that it has a learning curve. To make things worse the earlier debug-later development model has an unlearning curve. Most of the true benefits can concretely be seen after a fairly long period of time. It is not enough just to have a taste before judging, you actually need a prolonged tryout period. You most likely need some external help in the beginning. Your short term productivity suffers because of learning new skills and solving new kinds of problems.

You need someone in customer's role who is willing to talk about long term productivity as well.

Never the less I recommend to just have a taste.

Just remembered when I was just nodding on embedded TDD

Recently I have been talking a lot about embedded TDD to different, mostly experienced, embedded programmers. I thought I would jump right into a bit more advanced issues, to start a discussion, and maybe walk out with few new things myself. After all they were excellent programmers. I didn't want to waste the opportunity to learn.


During these sessions I have seen a lot of nodding people. I have though that "wow, I'm pretty good at explaining this". I have left with a smile on my face after a huge sense of agreement. I know better now. It's a much longer journey than one day seminar.

The challenge, or one of them, as I realize now, is that if you give a basic introduction with mickey mouse stuff examples to experienced embedded developers you will be immediatelly shot down with "that won't work in our environment, because [choose your favorite item identified on Embedded Agile Yahoo group]".


On the other hand if you rush straight into embedded TDD specific stuff, people are going to just nod.

And this is the other end of the same challenge. They are nodding so that you would go away faster. Not understanding fully is scary for a seasoned engineer who has not opened a book in the past 15 years or so. This nodding experience alone is not going to make anyone start practicing TDD. The best is to give a hands-on experience on TDD itself, show the benefit, and not on mickey mouse stuff but on real deal.


Advice that I'm going to follow is to try to remember when I was also just nodding and how many years, books, articles, conferences, talks, discussions, trials, failures, small successes, and so on, it took for me to get it. And I'm still only beginning to understand TDD.