Those Stupid Moments - Round and Round Again with Unit Test Tools

It is time to feel stupid again. We have been experimenting TDD of firmware as you know if you have been reading this blog. We started with Embedded Unit framework which I was familiar with. We wanted to make it simpler to be able to run unit tests on target with very low resource budget. Then we realized that Unity framework is exactly that. Atomic Object has documented their thorough test-driven approach and open sourced many of their made-in-process Ruby tools. There's a nice article on the approach in Methods&Tools.

At first we did not get to use the other Ruby tools from Atomic Object apart Unity unit test framework, but implemented similar, but simpler, in Python. That was pretty much because we did not "get" the tools, being electrical engineers and all.

Then we realized that we do not have the time and energy, nor the skills, to develop these tools seriously, but should use existing tools out of the box as much as possible. We started using Atomic Object's Ruby tools. So now the same project is using its third tool set for unit testing.

Rake for building the tests

I can't do magic with Rake - yet. I get the build script to work, but it is structured pretty much like scripting languages I'm used to, and most of the things are more complex than they should. I however have gotten a sight of its power and it is definatelly my choise of build scripting at the moment. The online manual has simple examples of using Rake to build C projects. It makes a good starting point. Advanced examples can be found in Atomic Object's tool distribution. You can check some more in-depth analysis on Rake by Martin Fowler here.

Unity unit test framework

Just above in complexity of pure macro implementation, or standard assert's, is Unity. It's written in C, with few helper macros. It's intelligence is in it's simplicity. There is no support for suites or anything like that. What to build and run can be grouped in build script.

Here's what you need for a unit test in Unity:

static void testAfterAdding4NodesTheSizeOfListShouldBe4(void)

list list;

list_Init (&list);
list_AddHead (&list, &a);
list_AddHead (&list, &a);
list_AddHead (&list, &a);
list_AddHead (&list, &a);

TEST_ASSERT_EQUAL(4, list_GetSize (&list));

int main(void)

Unity.TestFile = __FILE__;

// RUN_TEST calls runTest
(int) 0;

If you don't use mocking techniques you can just define RUN_TEST macro to call your setUp, test, and tearDown functions. That's it. In full approach it will call your mock module setup and verify functions also.

Cmock - automatic mock file generator

The other day a colleague interested in TDD'ing C said to me; "the biggest problem to me is that in C we allways need some real friends for a module under test. We do not have real mechanism to support interfaces." Well yes, and no. We concider interface to be the .h file, or in more advanced cases a struct of function pointers. After that we get to automating stuff. Cmock is a tool that generates mock functionality based on .h file. It goes through all .h files in your source directory and generates a corresponding mock.c files. These mocks can be linked together with the module under test and can be teached to expect calls and to return values. Here's an simple example of a function definition and resulting teaching function:

uint8_t Protection_filter_getState( uint8_t me_index);

Protection_filter_getState_ExpectAndReturn( uint8_t me_index, uint8_t toReturn);

Argent automatic test code generator

It's a pain to add mock init and verify functions, as it is to add a call to test function to main function, or just to change a name of test in two different places. That's the pain - Argent is the relief. It uses a simple markup scheme, and interprets your test code in order to automatically insert the calls to appropriate test functions, and mock module initialization and verifications.

//[[$argent require 'cmock_injector.rb'; inject_mocks("lists");$]]

// Calls to mock objects will be inserted here accoring the included mock headers.


//[[$argent require 'generate_unity.rb'; generate_unity();$]]

// Calls to test functions (identified by test prefix) will be inserted here.


Mock dependency check

It's also a pain to add mock dependencies to linker command. Using the same interpreter as argent the mock dependencies can be checked at build time automatically. So all that is needed to take a mock object into build is to have it's header included in the test file.

So that's how far we are now. Now we need some experience to get our routines right.

I quess this is a good example of having courage to improve when we see an opportunity, and strenght to admit being dead wrong - over and over again. This is truly a learning experience.

CSM Course At Our Premises

Jeff Sutherland and Jens Oestergaard are going to give a Certified Scrum Master (CSM) course at our premises on 3.-4. Dec. 2007 in Espoo, Finland. It's an open class, so if you are interested check out if there's still seats available.


Link: Waterfall Manifesto

If you are not completely satisfied with agile manifesto - There is an alternative.

Check out: Waterfall Manifesto

It follows on the lines of popular Waterfall 2006 Conference.


Is There Any Value in Values?

Recently I was blessed with an opportunity to facilitate a project retrospective for globally distributed project at the end of their 9th Sprint. Project has development in Europe, China, and Mexico. It is an embedded system development project involving electronics, plastics, mechanics, and firmware development. For all the participants this is the first contact with Agile project management. As a starting exercise we had each participant try to describe the project so far with a single word. The words we got:

  • "Chido" (This is something like "cool")

  • "Good"

  • "Good"

  • "Skilled"

  • "Challenging"

  • "Spectacular"

  • "Fun"

  • "Speed"

  • "Improvement"

  • "Interesting"

  • "To Be Finished..."

This was interesting in itself.

An article "Manage Your Embedded Projects" in Embedded Systems Design described developer motivators:

  • "Achievement"

  • "Growth"

  • "Work itself"

  • "Personal Life"

  • "Technical supervision"

  • "Advancement"

  • "Relationship with peers"

  • "Recognition"

  • "Salary"

  • "Responsibility"

This data is adapted from Boehm, Software Engineering Economics (1981) and Herzberg, "One more time: how do you motivate employees?" Harvard Business Review (1987). So it is a bit outdated. The data in the article is used to demonstrate the difference in motivators between developers and managers. Similar findings are presented in Motivators of Software Process Improvement: an analysis of practitioners' views (2002) and De-motivators for software process improvement: an analysis of practitioners' views (2003), both by Nathan Baddoo and Tracy Hall.

We can go on to generalize this further, and think about differences of people and organizations. Value system of people has been recorded and studied for a long time. Von Rosenstiel and Koch (1) reported that the values have changed towards respecting the uniqueness of people, individualism, enjoyment and discipline,self-development and also material values. That is not enjoyment, it is enjoyment AND discipline among other things. To me the "just words" exercise showed that agile work actually builds on these values; the project was challenging and needed lot of skill, but that just made it interesting and fun - chido. In many cases this however is not following the values of the company, which often are still drawn from old school thinking, like punctuality, hard work, modesty and other values related to duty and acceptance. This discrepancy between values of employee and employer affects the commitment level. Lack of commitment, in turn, reduces employee motivation and effort.

Of course in large organizations values are mostly just lip service, but nevertheless above should lead organizations to rethink the value in their values.

(1) von Rosenstiel, Lutz and Koch, Stefan, Change in Socioeconomic Values as a Trigger of organizational Learning, Published in (Dierkes, Meinolf, Berthoin Antal, Ariane, Child, John, and Nonaka, Ikujiro, Handbook of Organizational Learning & Knowledge, Oxford University Press Inc., New York, 2001).


Yet Another Agile Seminar in Finland

It seems to be the time for another Agile Finland Seminar. It is organized by Agile Alliance, Reaktor Innovations, and F-Secure. Guys have set out to break the old record for short notice:

On October 14th, 2006, I sent an email to the AgileFinland Yahoo! group about the next Agile Seminar taking place only 9 days later. Guess what? Everything went well even with such a short notice so this time we decided to improve and give you an even shorter notice.

At the time of writing this there are 84 seats available. Go get yours...


Agile Embedded Seminar at ELKOM2007

On Wednesday Tieturi organized a mini seminar on Agile and embedded in Helsinki, Finland. The seminar had 3 talks. First Pentti Virtanen gave a general background introduction to agile development. This was pretty much your average pure vanilla intro to agile -type presentation. The second part by Tomi Juhola was titled agile embedded in practice. I had high hopes for this as I though I would actually get to hear some new concrete stories from other Finnish companies practicing agile development. Well, it turned out that the cases presented were first the nice work of Nancy Van Schooenderwoert, and the second Jeff Sutherlands PatientKeeper. No new evidence there. The last talk by Teppo Heikurinen was about testing in agile development. The main topic seemed to be TDD. I have realized that TDD is not so much about testing, but more about design. There was some critique from audience that TDD is just like testing your own work and it has known problems. Yes in that sense it does, but TDD is a design practice which just happens to have a positive side effect of providing the safety net in a form of automated regression tests. At least this is my current understanding. This point was totally left out. Automated acceptance testing was mentioned, but that's about it. Other thing was that the value of TDD was verified with case studies from mainstream programming. I think the biggest issue is to convince the embedded crowd with studies from their own field, since basically people believe that agile has value but are not sure if it can be applied to their domain. Teppo mentioned that there are no studies from embedded and TDD, but Nancy's work is just that! I guess he forgot his colleagues presentation.

The room was full with 60+ listeners, so there is interest towards agile and embedded. I think a lot of people present were not practicing but more curious about what is all that buzz around agile and whether these methods are applicable to embedded (firmware) software development. I hope the event increased the interest even further and in the near future we would get more stories from Finnish companies practicing agile and embedded. It's nice to notice that Tieturi is taking a strong role in promoting these methods for embedded as well.


There's More Under Agile Umbrella Than XP and Scrum

Just a reminder. While Scrum and eXtreme Programming are without a doubt most widely adapted agile methods there are more. I studied many of them while reviewing their applicability to firmware development. Rod Coffin and Derek Lane have also done a nice job writing a two part article summarizing seven different methods that fall under the agile umbrella.


Agile2007: Embedded Agile Discovery Session

On Wednesday at Agile2007 conference James Grenning and Dan Pierce pulled together a discovery session for Agile and Embedded. The room was packed with people with experience on applying agile methods to embedded software development. Well, packed in this context means just over a dozen in a conference with 1150 attendees. Anyway. You really should sign up to Yahoo Agile Embedded group to follow the work of others. There are some remarkable people in this small community.

The discussion had two main topics. First, why embedded agile is getting foothold in the industry as slowly as it is. Several items were identified and the discussion is continued in Yahoo group. Second, that's correct, what else, but agile testing. People had different levels of practical experience, but the main theme was that it is important and it is definatelly possible. On the topic of safety critical embedded software and agile methods, the verdict was that this is not a problem. Safety critical embedded software actually SHOULD be developed using agile methods.

90 minutes was way too short time for anything concrete, but at least the event proved that agile embedded community exists. Thanks to James and Dan.

Jack Ganssle was also present and he posted about the conference in his Break Points column at embedded.com.

The Agile 2007 conference catered mostly to PC types, but some embedded heads showed up.

That post also made it to the latest Carnival of Agilists. This edition of Carnival of Agilists also mentions Atomic Object's approach to mocking the embedded world.

Agile embedded gets some visibility.


Seminar on Agile and Embedded in Helsinki

At ELKOM07 fair in Helsinki, Finland, Tieturi organizes a free 4h mini seminar on Agile and embedded system development on 5.9.2007. While it's free you still need to register.


Does eXtreme Development Exist?

I spent last week in Washington D.C. participating Agile2007 conference. The conference program was overwhelming with over 300 events. However while we have many implications to agile embedded systems development the agile embedded community is still small. So next time you see an agile conference ad, sign up!

Jim Highsmith is a person that has brought together agile community and general product development. I have followed his work and writings closely. He has brought work of people like Stefan Thomke, Preston Smith, and Donald Rainertsen (Reinertsen & Associates) to my knowledge. The latter two co-wrote the book Developing Products in Half the Time. In the panel discussion about agile manifesto there was a Q if the panelists believed that values of manifesto are applicable to development outside SW. (This is what tried also!). Highsmith mentioned a pure HW team that they successfully coached based on eXtreme Programming techniques. In that particular project there was no SW at all!

While browsing for the links, I found out that there is a new book out from Preston Smith:

Flexible Product Development: Building Agility for Changing Markets. It's on top of my wish list.


The Pain of Choosing the Refactoring Tool

Eclipse SDK version 4.0 was released earlier this summer. My hopes were high especially after listening Doug Schaefer, the Eclipse SDK project lead, at ESC2007. I just quickly took a look at the long waited refactoring tools.

Well, it's not as impressive as could, but it's a start. We know that writing refactoring tools for C is difficult, as with the preprocessing a C compiler can be made to understand virtually anything. Doug mentioned their struggle when he commented earlier on this blog. Good news is that it's now easier to start using Eclipse for C projects, as it is possible to get Eclipse and SDK in same package.

We have been using SlickEdit for a while. Slick has a bit more refactorings for C - actually all three of them.

Refactoring is an important ingredient in test-driven development and while these tools are not much, they still take the bar for quick refactoring lower.

And that is good news. If you are not able to decide from this myriad of choises, you can choose the middle way - get the SlickEdit plugin for Eclipse.


TDD Tools

Using TDD for firmware in C can be quite laborious. Creating a new module requires at least steps like:

  1. Create a test .c file
  2. Add created test to test build
  3. Create implementation .c file
  4. Create implementation .h file
  5. Add implementation to production build
  6. Add implementation to previously created test build
  7. Create fake object(s) to make the test build to pass
  8. Add fake object files to test build

When you adapt the habit of testing everything you quickly get bored repeating these tasks. You end up looking for tools to automate this repeating work. At least this is what you should end up doing, and not giving up.

Atomic Object has created tools in Ruby to do the trick and some. At ESC2007 they demonstrated their tools and already gave the tools on CD, but there still was some minor problems getting the tools to build. Now they have released their tools as open source and fixed the small problems in the earlier distribution. In Summer Issue of Methods & Tools is an article elaborating Atomic Objects approach to agile firmware development infrastructure. Highly recommended starting point, and offers also a path for advanced testing.

Just tried out the package that is now available from their site - Worked like a charm!

Tools do really not need to be complex to make a difference, propably quite the opposite holds. We made file templates and bunch of scripts running sed to replace key words to automate C unit testing steps. We learned this from James Grenning's CppTestTools. From Atomic Object's package we adapted their simple unit test framework Unity. We stripped it down even further, and also mixed it up with the idea of different outputters libraries, like in Embedded Unit. We also adapted their idea of automated mock object creation, but ended up writing similar tool in Python (basically because we wanted to fully understand what we are doing). All our tools are tuned for our use and constantly evolving, but I think towards similar solution as already in use by Atomic Object, but in step-wise manner. We believe that you need to understand the reason, give it the context, for any new skill or practice in order to make it truly work.


Wrap It Thinly

The major concern about TDD in firmware development I hear is the hardware relation.

-"We don't have hardware to run the code in."
-"Every part of the code is hardware dependant."

I noticed I wrote about these approximately a year ago for the first time (I, II, and III). That was more on the theoretical side. This time I have a bit more experience on the subject. We have found Alistair Cocburn's article on hexagonal architecture valuable in philosophizing this issue. Many firmware developers are not familiar with design patterns, but a good starting point is above mentioned article. Writing adapters for your hardware, grouping them to ports, keeping the number of them low, and only allowing them to be responsible for the lowest-level of hardware interfacing, will help you isolate hardware dependant parts. Here is a picture of hexagonal architecture for the case project we are working on.

Then there is a philosophical issue of testing the adapters, or wrappers. In many cases binary input/output handlings are just like accessors, getters and setters. In TDD literature it is generally agreed that these do not need unit tests. When writing hardware wrappers we need to think a bit harder. We could understand the unit tests for hardware wrappers as our firmware/hardware interface specification. Then for sure it makes sense to write these tests. My thinking goes along with "to be able to judge what to unit test and what not to unit test, you have to be able to unit test everything".


Test the Goal of the Code

It is time to get technical again. We have been writing unit tests for some time. We have tried test-driven development. However we have never been disciplined in this. Sometimes we don't write tests at all. Sometimes we don't fix the test suite when code is needed to change. Sometimes we just decide that well, this project or that part of the project does not need tests (typically when it seems to get complex or difficult to test!). We have come to a conclusion that with this approach we will never totally master the skill, and not get the benefit it is supposed to bring.

We decided to try another approach. We are writing a very real hard-time constrained, very hardware related code with 100% test-driven development (TDD) in C. A code exists to do the job, but we are gonna completely re-write it. It is originally ported to C from assembly, and this is done maybe four years ago. Since then it has been improved, modified, and adapted to new situations. What has happened is the code has rotten, it is a jungle of conditional compilation, and so forth. So what is better place to do a tiny bit of always refreshing skunk work and completely re-write the whole piece of art?

Right after we made the decision that we are gonna write EVERYTHING with test-driven development we noticed a change in thinking. We constantly think for the simplest possible thing to do, constantly re-evaluate the module (C file) interfaces and so on. If you are gonna do everything in this manner, there just is no backdoor for you.

Another big change was (well, we still need constantly remind ourselves about this) the shift in thinking towards writing the tests as specification. We often ended up thinking about the solution and then writing tests to check the solution. This is maybe caused by the fact that one shot on the code exists and is thoroughly known. However, you will create novel, simple solutions for your tests only if you think about what you want your next small code-to-be do, not how it will do it. Keith Ray has listed questions to help you in this.

Sean Shubin has done a nice work and listed first guidelines to TDD. Again:
Test the goal of the code, not the implementation


Embedded TDD Link List

I created a del.icio.us bookmarks for web resources I have found useful for understanding TDD for embedded software.

my del.icio.us bookmarks for embedded TDD


Embedded Blogosphere

Yesterday was yet another record breaking hot day in Helsinki, and what did I do? That's right, searched the net for new embedded slash agile blogs. I periodically do this, but normally I come empty handed. This time I was happily surprised to find Low Country Software Ramblings by Benoit Lavigne. Benoit was also wondering "Am I alone?", and later found out that "I am not alone after all!"

Via Benoit's blog I discovered Embedded Cork by Ralph Depping and Embedded Gurus. Embedded gurus seem to be a bit quiet, but there is some recent stuff as well. Names behind Embedded Gurus group are impressive; Nigel Jones, Michael Barr, Larry Mittag, Mike Anderson, and Miro Samek.

Keep in mind the Atomic Object Spin Embedded Corner, and But Uncle Bob Archive (James Grenning).

I linked them in sidebar - give them a look.


Story: Spike Up Your Coctail

I posted earlier about short, 6 week, experiment where we had weekly deliveries of firmware and true parallel co-design of plastics, electronics, and software/firmware. I mentioned I would try to write down experience report. Well, it became more like story, and you can find the whole of it here (pdf). (LINK CORRECTED 27/6/2007)

Abstract. Agile development is a term used for wide variety of lightweight software development methods following shared values1. Many of these methods and practices however can be applied to more general new product development. This paper describes a six week project using some of the practices from agile development. These practices included self organizing team, collective ownership, continuous integration, iterative planning, iteration demos, team retrospective meetings, wall work queue, information radiator etc. The project crystallized a vague idea and a draft of electronics schematic into two fully functioning prototypes. It is shown that meaningful functionality can be developed in just six weeks. This is remarkable when reflected against the effort needed in so called traditional process models with formal analysis/design and theoretical proof of concept with heavy review processes. This is possible because of today’s advanced development tools and prototyping technology. Stefan Thomke (2001) calls this an era of enlightened experimentation. Building the working prototype is more cost effective way of reliable proof of concept. Using this approach already in the fuzzy front-end phase of the project would result in huge savings in overall project schedule and budget. It was also noticed that all stakeholders of this project appreciated the approach and considered it as “common sense”. The positive side effect was the team building effect this period had.


Camp Fire

A week ago on Tuesday we organized a camp fire event at our premises. Camp fire is an event about sharing agile experiences, and this was second of such events. Roughly twenty people gathered to an auditorium after the office hours. The audience consisted of people from few different organizations inside the company; developers, managers and marketers. We had two talks; Vasco Duarte from F-Secure was invited as quest storyteller and Mikko Kaijärvi from Schneider Electric volunteered to share his recent experience.

Vasco's presentation was strong, as usual. He talked about top-down agile adaptation at F-Secure in a very realistic way. He did everything but painted a pretty picture about the journey. This is according to his experience, and literature as well, expected. When everything is laid down on the table, it tends to get ugly from time to time. It was interesting to see how within top-down approach they were able to put the whole organization into "agile rhythm". Vasco mentioned that lack of outside help, or not getting outside support enough, was the main thing he would do differently if need ever arises. Vasco presented also some cold quantitative data from theirs and others agile adaptation to support the reasoning for change. It just happens to look like we have more data supporting agile than sequential models ever had. The presentation and especially the Q&A and discussion parts showed Vasco's deep knowledge and understanding of modern project work.

Mikko presented another interesting story in a lively style. His project has applied agile project management and Scrum practices in globally distributed embedded system development project. He estimated that only 5% of development effort is software and firmware, and in addition this is coming from non-agile team. Agile practices were seen as the only option for this project to be successful. The project was started with very vague requirements, barely a vision. Teams were pretty much given. One team in Shanghai, China and another in Monterrey, Mexico, and technology and software development in Europe. The offshore teams had minimal domain knowledge and a huge amount of knowledge needed to be transferred fast. Continuous integration of mechanical and electronics prototypes was a key catalyst for communication. Mikko further identified a clear transition in team behavior from hierarchical - through a single point communication - to self managing emergent behavior actively seeking for best possible solutions in this challenging environment. The project has only been active for five months, but they expect to deliver it in September as originally scheduled. Compared to average project that would be fast according Mikko. Furthermore active early participation of marketing is expected to result in better product as well. This has potential of making yet another example of agile project management practices being succesfull also outside pure software projects.

As conclusion both talks were excellent, delivered by guys that are experienced in both, the project work and giving presentations. Hopefully people were able to walk out with something new and to be able to connect some more dots in their thinking.

After the official part the discussion continued over snack and refreshments. It was agreed that lot of the agile adaptations outside so called agile homeground is about attitude. It is about the shift in thinking towards the art of possible. Just changing your de facto answer from starting with "Yes, but..." to "Yes, and..." offers a huge potential. Vasco's message was "if you really want it, you can do it." The reward demonstrably is there.


Enough of processes, let's do labels

The title of course follows the Dr.Dobbs article by Ivar Jacobsen, Pan-Wei Ng, and Ian Spence "Enough of Processes, let's do practices Part I".

Venkatesh Krishnamurthy has an article in March 2007 Agile Journal about top-down commitment in agile adaptation. He defines three layers of stakeholders in organization; Sponsors, Mid Level Management, and Development Team. As a conclusion he presents an argument that highest possibility for successful agile adaptation is a right mixture of bottom-up and top-down strategies. Dave Nicolette reviews the article in his post and shares his experience:

The agile group had started life as a skunkworks operation. When the group was brought back into the IT department formally, IT management dismantled the group and morphed the project process, management tools, and development methods back into conventional ones, although with "agile" labels.

There are propably a lot of stories like that. Based on the initial moderate success in agile adaptation on the technical level, agile gets to be the new silver bullet and buzz word. A new group of people are expected to automatically form a team, jell, and "just do agile". "Just do agile" means that traditional PM starts using agile jargon without bothering to look what these terms mean, not to mention that she would try to master the agile values, principles and practices, adjust them, and further develop them. "Just do agile" seems like a little bit too thin statement regarding that agile adaptation is described to be a paradigm shift, a completely new philosophy requiring a new mental package. "Just do agile" could be called doing practices, but that would be being nice. Instead it is only doing labels as Dave called the phenomenon. Few years back I was teached to avoid new labels, like agile ones, when introducing change. Now I have learned that some people only want new labels. That's probably why the silver bullet business is doing so well. Calling everything agile, Scrum, and/or Sprint is not much of a change, and will not automatically make everything visible, consistent, or reliable. Somehow the point is completely lost. Of course people who expected agile to be a silver bullet or magic wand are going to be disappointed and call agile just another fad. Some people who actual get agile, and understand that it is a bit more than "doing some stuff in 30 or so days and using words like agile, Scrum, and Sprint a lot", are also quite frustrated. Dave's post sets out the more than likely dark symptom of this:

As I write this, just over a year has elapsed since the process of dismantling agile began. Of the 60+ people who were part of the agile group at its peak, only 4 individuals who were in that group are still employed by the company.

It is equally true according to my experience that commitment from all three categories mentioned by mr. Krishnamurthy is needed to continue agile adaptation for any prolonged periods . Both orders in which this adaptation happens have been reported as successful, and yes indeed, also as failures. Mixture of both strategies sounds appealing.

Mike Vizdos writes about similar experiences in his RIP Scrum -post and cartoon. He calls this a "Death by a thousand copies".


Conference Programs Announced

Several agile conferences have their programs out.

Oresund Agile 2007
Copenhagen, 11-14 June

Como, 18-22 June

Washington D.C., 13-17 August

At Agile2007 Dan Pierce and James Grenning are giving a talk "Embedded Agile - Issues and Practices". Should be valuable, as these two gentlemen have been talking about and practicing agile development in embedded domain for a long time. Of course lots of other interesting stuff as well, 5 days with 8 or more tracks...


Edward Bear Does Not Have Time to Improve.

In many companies the software assets and practices are left to rot. This indifference to fundamental quality issues is justified with phrases like "we are in a hurry", "we need to create new stuff", and other nonsense. Software is going to require maintenance and further development for its whole life. If you neglect to keep it fit it is going to haunt you back for sure. And it gets worse. If you leave your code base and practices to rot, you will eventually paralyse your whole productiveness. See Ken Schwaber explain it in this InfoQ video.

I have used the story about timber jack being so late at his logging site, that he does not have time to sharpen his saw. Some time ago I bumped into another quote that makes the same point.

“HERE is Edward Bear, coming downstairs now, bump, bump, bump, on the back of his head, behind Christopher Robin. It is, as far as he knows, the only way of coming downstairs, but sometimes he feels that there really is another way, if only he could stop bumping for a moment and think of it.”

It really gets painful sometimes.


It's less chaotic today than it was yesterday

SD Times reviews the new, 2006 Chaos Report from the Standish Group. According the article:

35 percent of software projects started in 2006 can be categorized as
successful, meaning they were completed on time, on budget and met user

I'm not sure if 35 percent is a good number, but it surely beats 16.2 percent in the 1994 report.

It would be nice to know how the size and complexity of average software project has developed during those 12 years. Maybe that is in the report, we just have to wait a bit more (and get 500$ for the report from somewhere).

The report lists three reasons for the improvement:
  • better project management,
  • iterative development, and
  • the emerging Web infrastructure.

I don't know if better project management in this context means agile project management. A quote “Managers have a better understanding of the dynamics of a project.” hints to that direction. Iterative development at least is something that we believe is essential for success, and it is nice to get more and more data to show that. Web infrastructure plays quite insignificant role in firmware development.

If you happen to know recent data about new product development success rate in general, or maybe even embedded software numbers, I would be most interested seeing them.


What does it mean to be agile?

OK, the ESC2007 Conference is over. Many of the classes and especially Peer Roundtable discussions at the conference, Beer Roundtable discussions at the bars after conference hours, and long flight hours with a colleague, woke several questions, among them:

-What does it mean to be agile?
-When is an organization/team considered agile?

In his review (pdf) Dr. Pekka Abrahamsson showed the software life-cycle support of different agile methods. We know that Scrum does not have any engineering practices described in detail, but focuses on project management. That's why it has been adapted also outside software development. It is simple. This is why majority of the so called agile adaptations at least say they follow Scrum. As have we; we have used it with cross-disciplined embedded system development teams to develop a complete embedded system as fast paced iterative development. I have also written in my paper how I saw reflective iterative development as a tool itself for figuring out what practices you need. This said I share Simon Baker's post

Are you missing the point?

As long as you share the underlying values of agile development, and keep the obvious practices like short iteration, demonstrable progress and retrospectives, you are going to figure out the rest of it - or die trying or at least you figure out the need for something and can start looking. If your survival anxiety in your current environment is high, but you have juniority prevailing in your team, you can still gain a lot from practicing "just" Scrum. Scrum puts the basic structure in place with its simple rules and in my experience this is an immediate relief to chaos. This does not mean that it is easy. It will make everything visible; good and bad and needs a lot of discipline and nurturing.

With this approach, you being now out of the chaos and anarchy, it is time to bring in the engineering practices. Our case has shown that they are needed. We have experienced with eXtreme Programming (XP) practices; collocated team, pair programming (development), test driven development, coding standard, metaphor, and so on. We have seen the value in them, but not been able to actually master them in firmware or embedded system development in order to put the discipline into using them. Still we feel that we gain from practicing Scrum, and agile development as learning-driven thinking. This is the very reason why we still keep learning more skills and practices.

Are we agile? I do not know.

If your environment is functioning and open minded, and you have master mind developers in your team, why not go full blown XP right away? We have manifested that those practices are of high value, but they are hard for beginners. We now have also hard evidence to support the effectiveness of these methos (see for example the AGILE-ITEA project, run by the same Dr. Pekka Abrahamsson btw). It may be that with this approach you find out that you need some project management practices from Scrum. Who knows.

I wrote in my original paper about agile firmware development that lot of the agile programming practices for firmware development need innovative thinking. I think last week at the ESC2007 I met some of the people capable doing this. In the front line James Grenning from Object Mentor and Mike Karlesky and Greg Williams from Atomic Object. They have put these engineering practices into full use (also) in low end firmware projects. They are also willing to share their knowledge by publishing their tools as open source, and having discussions about the methods. Maybe we even become clients of these guys some day... Furthermore during the classes and discussions I got the feeling that a fair amount of people are actually practicing.

We have just scratched the surface, but after last week I believe that it is time to put some serious effort into agile programming practices, and actual firmware development. After all that is where we started this journey. It just kind a slipped into system development.

Are we agile after adapting these practices? I do not know.


The Vacation is Over - Officially

It's Monday. Back at work after seven weeks of vacation. It actually feels almost as bad as it sounds. I did spend over six weeks in Thailand. Lots of sun, beer, good food, nice people, and of course technical diving. Technical diving is a nicely balancing hobby for an agile proponent, as it relies so much in up-front plan:

Plan the dive, dive the plan.

There are very few situations where this rule may be broken. It is the opposite to my beliefs in new product development. I'm a strong believer in:

Plans are useless, planning is everything.

This Friday I'm off to Embedded Systems Conference 2007. We're planning to get together with people from AgileEmbedded Yahoo group during the conference. If you you are planning to be in San Jose keep an eye for that. If you are not planning to be there, it's better get started.



Embedded Systems Conference 2007 in San Jose, California, is having some interesting events. James Grenning is giving several sessions and people from Atomic Object are coming to tell about their low level system agile testing methods.

The best news is that right after my vacation (still have some three to four weeks to burn) I'm going to attend the conference with a colleague. Talking about nice return to work! I'll try to have energy to write something about the conf'.


Scrum Simulation Rocks

Last week I gave an introduction to agile development and Scrum for a global project organization in Shanghai, China. After the lunch on first day it was time for 59min Scrum simulation. I have written about bad experiences when trying to achieve too much, so it was back to the basics.

There were participants from Finland, China, and Mexico. We used the product backlog for Family Cookbook. Only addition to original Scrum simulation was the use of relative complexity estimations on product backlog items. This time it worked. Estimating the tasks in numbers gave birth to discussion on whether something is simple, or not, not just dividing an item into tasks. Some items were also soon identified to be too large to fit in one 14 minutes Sprint. Team was immediately forced to seek for collaboration with Product Owner.

On retrospective the main thing experienced was the time pressure. I do not see this as a bad thing, it is just the beginning of understanding the discipline needed for time-boxed development. Time pressure will decrease when the team builds trust and improves estimating.

Everyone valued the exercise and though that it was a good lesson. I might consider adding a separate exercise for planning and estimating before the final Scrum Simulation. With this early experience I do not think that I will ever try meaningful work inside the simulation again. The difference was so clear.


TDD'ing State Machines

I'm an electrical engineer. Everything in programming seems to be very very hard for me. Last week I spent 40 hours in aeroplanes, so I had some quality reading time. I did read Agile Estimating and Planning and Test-Driven Development from cover-to-cover. I had quickly gone through them earlier, but this was enlightening experience.

First of all, I have told that we have been trying to figure out effective TDD for our FW development. To be honest, we've struggled. We wrote tests for a state machine implementation. We expected the tests to be run in certain sequence, and that the state was always left where it should be after the test. We of course ended up in lot of rework on test code when our state machine design emerged. We thought that we need to design the state machine up-front in order to be able effectively write tests for it. This did not sound like test driven development at all. This required some more thorough thinking.

So back to my flight. Kent Beck writes that coupling is bad also between tests. A test should leave the system as it was. So writing unit tests in the same sequence as you expect your state machine to work does not seem like a right thing to do. You should be able to execute individual tests and to change the order of individual tests. So instead we are starting to recognize a single state as something to test.

Ph.D. Miro Samek writes about state machines being a killer app for function pointers. He further defines a design pattern for state chart implementation. Our state chart interface has become to be something like:


construct is self explaining and dispatch dispatches new event. We have specific event tick that gives resource time to state machine every 10ms (based on our 50Hz mains). We could also implement a timeout as OS service, and not to have local counters for timed events. Anyway, if we implement each state as a function as Mr. Samek proposes, and we hold our state as a function pointer we can write

dispatch( me, event )

We get rid of cryptic switch -structures. This solution helps us to unit test each state as a function with natural feel.

I'm going to write some benchmarking code to see the actual overhead, but at the moment I like this idea.


Driving Embedded Firmware C Project on CruiseControl

I finally made it. A dedicated page for configuring CruiseControl for embedded firmware C project. It combines some of my earlier posts, but also tries to add detail. Take a look here, and tell me what is missing.


Book: The Tipping Point

I finally read The Tipping Point, by Malcolm Gladwell. The Tipping Point tells us how little things can make a huge difference whether an idea will success or not. Will it tip or not. Examples and cases presented by Mr. Gladwell range from young men's high suicide rate in Micronesia, via fashion trends, cigarettes, children's tv shows, all the way to warning mechanism in surroundings of Boston about the upcoming attack of Brittish in 1775.

Malcolm explains, in a very understandable way, the law of the few; Mavens, Connectors and Salesmen. These are the three different personalities needed for a new idea to become epidemical. He talks about social epidemics. He then further explains the concept of stickiness factor, and the power of context.

What has this to do with things we are interested in? Graig Larman calls introduction of agile development methods a paradigm shift. When you introduce an idea of this magnitude, what you are doing, is asking people to unlearn much of their current knowledge. We of course know that the old knowledge is by no means useless, we just need to update it with some new knowledge. Never the less, introduction of agile methods and lean thinking is a novel idea in most organizations. The theories and concepts presented in The Tipping Point help us to understand the process of making this change and getting it to last. I have learned a long time ago, that it is not enough that you understand, you need to get others to get it as well.

Let's see what these theories in my opinion tell about agile intervention. In many cases, especially early in the decade, agile methods were introduced to organizations by younger champion developer, usually from position of no power. They just happened to be interested in learning and continuous improvement. They read a lot, they go to conferences, and they talk to fellow craftsmen. Furthermore they like to analyse the new things, and then tell these ideas to others. They are the Mavens. If things go nicely they will meet a Connector at some point. Connector knows everyone inside the organization, and knows who to talk to in order to get the change started. After the initial start, to make the idea of agile development really tip, we need extra ordinary people called Salesmen. They are the ones that can translate the ideas and experience reports so that we can communicate with lots of different people in order to get real structural changes to happen in an organization. Agile coaches and consultants are a good bet at this if you can not find someone in your organization fast enough. On top of that agile is a nice sounding word, and it can help to increase the Stickiness Factor. The first agile pilot project should have importance and wide enough commitment, creating the right Context for the intervention to tip.

Other literature (for example Pinchot, Intrapreneuring: Why You Don't Have to Leave Corporation to Become an Entrapreneuer, 1985) documents the same kind of process. Terms used elsewhere include 'Brain workers' who have the intellectual ability to think in novel ways and are able to argue their decisions. 'Intrapreneuers' are daring in using their vision of innovation regardless of for example political resistance. 'Gate keepers' have strong contacts to the organization's environment and have communicational competencies for making initiatives 'digestible' for the organization. Further a top management mentors, godfathers, or corporate angles are terms used for expressing the need for management commitment in such a venture.
Look at the world around you. It may seem like an immovable, implacable place. It is not. With the slightest push - in just the right place - it can be tipped.


Blogger is out of beta

Blogger is out of beta and I thought I finally upgrade my template. I'm glad I did, managing the appearance of my blog is now a breeze.

I did some rework on links as well. If you are interested in agile development methods for embedded software and firmware you should join the discussion at Agile Embedded Yahoo Group. There are some brilliant people in the group, but the group could use some additional energy.

I seem to have few regular readers, so if this change messed up your reader - my sincere apologies. This was just something that needed to be done at some point.