Embedded Blogosphere Gets Stronger

A while ago we took a look at embedded blogospehere. That sphere is now even stronger - James Grenning has a new blog. He is "blogging about Agile Development, especially embedded". That should be interesting enough. There's already two posts about techniques that can be used to test-drive hw driver development (1, 2).

Only few seats left in Agile Finland Seminar

Agile Finland is organizing yet another seminar. It takes place March 4th in Helsinki. This morning there was only some twenty seats available.

Islands of Expertise Can Get You Lost

You may be familiar with the following scenario. The embedded software development department has finally launched a large new system. The system obviously immediately gets new feature requests and other modification needs. During the initial development each developer has found a pet part in the overall system. When the improvements/modifications start it is intuitive to assign these smaller projects to a person whose area the change mostly concerns. In this approach we launch several, as many as there are developers, projects to run in parallel. This seems like a natural way to go.

While tempting at the first look, there are several problems in the above scenario. First is the customer’s role. It is difficult enough to get a functioning product owner for one project. Running multiple projects at the same time is ridiculously challenging. If we get one person to act as a customer proxy for all these small projects, this person soon becomes a bottleneck for all these projects. A project needs collaboration with the customer representative, and the sooner we get the feedback the better. The customer needs to be available at all times, and this is easier to achieve for one project at the time.

Second problem follows from the fact that it is rare that all these projects are completely independent. While we think that no one works on more than one project at the time, we actually have created a highly complex hidden multi-project environment. Typically one, or few, developers know something about everything. This means that people that are supposed to be working for their own project need to consult other projects continuously. These people become the bottlenecks for other projects, and their own project is impossible to estimate. As a net result we push all our projects to be more, and more, late without any control to what is going on.

Thirdly, assigning a developer to her previous pet part of the system further develops the specialization in a narrow part of the overall system, creating small islands of expertice. More and more closed silos of expertice emerge in the department. This is a high risk path as in an unfortunate case of loosing that person for one reason, or another we halt the development of that part for unpredictable period of time. Yes, I have heard that documentation should be there for allowing anyone to work on new part. Well, having been in the development game for nearly 15 years I kind of have lost faith to that approach. Maybe if the previous developers have focused their energy mainly on enabling future work –type of documentation. Never the less, safer path is to several developers to know the work.

The fourth problem is the lack of possibility to gain from the superior performance of true teams. Assigning each person to a single small project will at its best only accomplish the sum of each developer in development department. If we are successful in getting a true team to jell we can accomplish a lot higher performance that the sum of team members. This is important especially when the work we are engaged with gets more complex, and needs higher problem solving capability.

Changing to work in teams and in the same part of the system may feel difficult and weird at the beginning. Progress may also appear slow as most of the team members are unfamiliar with the parts of the system they need to work with. It may also take old school engineers by the surprise that they also need the softer skills, interaction and communication. It is crucial that collective ownership is still enforced. Teams have a remarkable capability in finding the ways to work. The practices supporting the work of team will merge rapidly. You don’t have to know everything to start working this way. Just follow the basics;

1. do fast paced iterative development,
2. have reflective retrospectives, and
3. create open, learning-driven culture.
   

Post leak

A friend told me that one of my drafts leaked into feed. It was probably an user error by yours truly. I apologize for your incovenience.

What comes to that draft, I hope to find time to finish writing about Preston Smith's latest book "Flexible Product Development" and agile in non-software development soon.

Is It Death of Innovation - or Birth of Innovative Modifications

In "Your NPD portfolio may be harmful to your business health" (Cooper, 2005) presents data showing that true new product development has decreased by 43.7% between years 1990-2004. According the same article improvements and modifications to existing products has increased by 80.1% during the same time period.

September 2007 issue of Embedded System Design Journal revealed the results of reader's vote. Parts of the results add recent data to above observation. According to reader's vote the fraction of "new to the world; a new project from the scratch" compared to "an upgrade or improvement to an earlier or existing product" products has decreased from 48% to 39% in the past three years. 79% of those upgrades or improvements included "New or different software features".

Do not feel sad if it's been a while since you started to work on a brand new product. You are not alone. Working with new product seems to be getting more and more rare situation. If you get to do this, enjoy while you can. The data shows that there is a fair chance that you will be working with that product for a long time.

From the agile development point this data shows that it is ever more important to invest in automated regression test suite and refactored, clean, code. It is a safe bet that the software will be under heavy modification.

Very few of us have a chance to work in true high-tech fields which are generally understood to require innovation. Products developed are increasing in size and complexity. Building something from scratch based on a novel idea is most of the times in mega-project category. Mega-projects will take a long time, and thus include a mega-risk. This means that they are not so attractive to management. They do not get the funding so easily. However to me innovation means combining some, typically existing, technologies and ideas in a novel way to create some value to some one. By this definition adding ethernet connection to traditional product, and figuring out a valuable meaning for it, is innovation. We also need innovation in designing and improving the products so that they more easily adapt even more change. The need for innovation has not disappeared, it is just the nature of it that is shifting.

New Year, New Tricks

There has been a quiet period again in writing. There's a reason, it's the same as always - I've been busy. Now it is new year, and we kind of think we can have a fresh start. We all know that this is not exactly going to work like that, but this time the new year happens to actually be interesting for me; I have a new employer. I still remained within the same organization, but I'm now working at completely different business unit. I'm still going to work with agile methods, probably more so than ever. Of course it is still embedded stuff, no question, but there are major changes:

I'm now engaged with much larger systems.
Application changed from lighting control to fire alarm system.
Resource budget has also increased quite a bit. Basic setup now contains:

Renesas H8
2M program flash
2M RAM
Commercial RTOS

As a result I'm learning a bunch of new stuff every day. I have a good feeling about this year.

Those Stupid Moments - Round and Round Again with Unit Test Tools

It is time to feel stupid again. We have been experimenting TDD of firmware as you know if you have been reading this blog. We started with Embedded Unit framework which I was familiar with. We wanted to make it simpler to be able to run unit tests on target with very low resource budget. Then we realized that Unity framework is exactly that. Atomic Object has documented their thorough test-driven approach and open sourced many of their made-in-process Ruby tools. There's a nice article on the approach in Methods&Tools.

At first we did not get to use the other Ruby tools from Atomic Object apart Unity unit test framework, but implemented similar, but simpler, in Python. That was pretty much because we did not "get" the tools, being electrical engineers and all.


Then we realized that we do not have the time and energy, nor the skills, to develop these tools seriously, but should use existing tools out of the box as much as possible. We started using Atomic Object's Ruby tools. So now the same project is using its third tool set for unit testing.

Rake for building the tests

I can't do magic with Rake - yet. I get the build script to work, but it is structured pretty much like scripting languages I'm used to, and most of the things are more complex than they should. I however have gotten a sight of its power and it is definatelly my choise of build scripting at the moment. The online manual has simple examples of using Rake to build C projects. It makes a good starting point. Advanced examples can be found in Atomic Object's tool distribution. You can check some more in-depth analysis on Rake by Martin Fowler here.

Unity unit test framework

Just above in complexity of pure macro implementation, or standard assert's, is Unity. It's written in C, with few helper macros. It's intelligence is in it's simplicity. There is no support for suites or anything like that. What to build and run can be grouped in build script.

Here's what you need for a unit test in Unity:

static void testAfterAdding4NodesTheSizeOfListShouldBe4(void)
{
  int a;
  struct list list;
 
  list_Init (&list);
  list_AddHead (&list, &a);
  list_AddHead (&list, &a);
  list_AddHead (&list, &a);
  list_AddHead (&list, &a);

  TEST_ASSERT_EQUAL(4, list_GetSize (&list));
}

int main(void)
{
  Unity.TestFile = __FILE__;

  UnityBegin();
  // RUN_TEST calls runTest
  RUN_TEST(testAfterAdding4NodesTheSizeOfListShouldBe4);
  UnityEnd();
  return (int) 0;
}

If you don't use mocking techniques you can just define RUN_TEST macro to call your setUp, test, and tearDown functions. That's it. In full approach it will call your mock module setup and verify functions also.

Cmock - automatic mock file generator

The other day a colleague interested in TDD'ing C said to me; "the biggest problem to me is that in C we allways need some real friends for a module under test. We do not have real mechanism to support interfaces." Well yes, and no. We concider interface to be the .h file, or in more advanced cases a struct of function pointers. After that we get to automating stuff. Cmock is a tool that generates mock functionality based on .h file. It goes through all .h files in your source directory and generates a corresponding mock.c files. These mocks can be linked together with the module under test and can be teached to expect calls and to return values. Here's an simple example of a function definition and resulting teaching function:

uint8_t Protection_filter_getState( uint8_t me_index);

void Protection_filter_getState_ExpectAndReturn( uint8_t me_index, uint8_t toReturn);

Argent automatic test code generator

It's a pain to add mock init and verify functions, as it is to add a call to test function to main function, or just to change a name of test in two different places. That's the pain - Argent is the relief. It uses a simple markup scheme, and interprets your test code in order to automatically insert the calls to appropriate test functions, and mock module initialization and verifications.

//[[$argent require 'cmock_injector.rb'; inject_mocks("lists");$]]

// Calls to mock objects will be inserted here accoring the included mock headers.

//[[$end$]]

//[[$argent require 'generate_unity.rb'; generate_unity();$]]

// Calls to test functions (identified by test prefix) will be inserted here.

//[[$end$]]

Mock dependency check

It's also a pain to add mock dependencies to linker command. Using the same interpreter as argent the mock dependencies can be checked at build time automatically. So all that is needed to take a mock object into build is to have it's header included in the test file.

So that's how far we are now. Now we need some experience to get our routines right.

I quess this is a good example of having courage to improve when we see an opportunity, and strenght to admit being dead wrong - over and over again. This is truly a learning experience.

CSM Course At Our Premises

Jeff Sutherland and Jens Oestergaard are going to give a Certified Scrum Master (CSM) course at our premises on 3.-4. Dec. 2007 in Espoo, Finland. It's an open class, so if you are interested check out if there's still seats available.

Link: Waterfall Manifesto

If you are not completely satisfied with agile manifesto - There is an alternative.

Check out: Waterfall Manifesto

It follows on the lines of popular Waterfall 2006 Conference.

Is There Any Value in Values?

Recently I was blessed with an opportunity to facilitate a project retrospective for globally distributed project at the end of their 9th Sprint. Project has development in Europe, China, and Mexico. It is an embedded system development project involving electronics, plastics, mechanics, and firmware development. For all the participants this is the first contact with Agile project management. As a starting exercise we had each participant try to describe the project so far with a single word. The words we got:

• "Chido" (This is something like "cool")


• "Good"


• "Good"


• "Skilled"


• "Challenging"


• "Spectacular"


• "Fun"


• "Speed"


• "Improvement"


• "Interesting"


• "To Be Finished..."

This was interesting in itself.

An article "Manage Your Embedded Projects" in Embedded Systems Design described developer motivators:

• "Achievement"


• "Growth"


• "Work itself"


• "Personal Life"


• "Technical supervision"


• "Advancement"


• "Relationship with peers"


• "Recognition"


• "Salary"


• "Responsibility"

This data is adapted from Boehm, Software Engineering Economics (1981) and Herzberg, "One more time: how do you motivate employees?" Harvard Business Review (1987). So it is a bit outdated. The data in the article is used to demonstrate the difference in motivators between developers and managers. Similar findings are presented in Motivators of Software Process Improvement: an analysis of practitioners' views (2002) and De-motivators for software process improvement: an analysis of practitioners' views (2003), both by Nathan Baddoo and Tracy Hall.

We can go on to generalize this further, and think about differences of people and organizations. Value system of people has been recorded and studied for a long time. Von Rosenstiel and Koch (1) reported that the values have changed towards respecting the uniqueness of people, individualism, enjoyment and discipline,self-development and also material values. That is not enjoyment, it is enjoyment AND discipline among other things. To me the "just words" exercise showed that agile work actually builds on these values; the project was challenging and needed lot of skill, but that just made it interesting and fun - chido. In many cases this however is not following the values of the company, which often are still drawn from old school thinking, like punctuality, hard work, modesty and other values related to duty and acceptance. This discrepancy between values of employee and employer affects the commitment level. Lack of commitment, in turn, reduces employee motivation and effort.

Of course in large organizations values are mostly just lip service, but nevertheless above should lead organizations to rethink the value in their values.

(1) von Rosenstiel, Lutz and Koch, Stefan, Change in Socioeconomic Values as a Trigger of organizational Learning, Published in (Dierkes, Meinolf, Berthoin Antal, Ariane, Child, John, and Nonaka, Ikujiro, Handbook of Organizational Learning & Knowledge, Oxford University Press Inc., New York, 2001).

Yet Another Agile Seminar in Finland

It seems to be the time for another Agile Finland Seminar. It is organized by Agile Alliance, Reaktor Innovations, and F-Secure. Guys have set out to break the old record for short notice:

On October 14th, 2006, I sent an email to the AgileFinland Yahoo! group about the next Agile Seminar taking place only 9 days later. Guess what? Everything went well even with such a short notice so this time we decided to improve and give you an even shorter notice.

At the time of writing this there are 84 seats available. Go get yours...

Agile Embedded Seminar at ELKOM2007

On Wednesday Tieturi organized a mini seminar on Agile and embedded in Helsinki, Finland. The seminar had 3 talks. First Pentti Virtanen gave a general background introduction to agile development. This was pretty much your average pure vanilla intro to agile -type presentation. The second part by Tomi Juhola was titled agile embedded in practice. I had high hopes for this as I though I would actually get to hear some new concrete stories from other Finnish companies practicing agile development. Well, it turned out that the cases presented were first the nice work of Nancy Van Schooenderwoert, and the second Jeff Sutherlands PatientKeeper. No new evidence there. The last talk by Teppo Heikurinen was about testing in agile development. The main topic seemed to be TDD. I have realized that TDD is not so much about testing, but more about design. There was some critique from audience that TDD is just like testing your own work and it has known problems. Yes in that sense it does, but TDD is a design practice which just happens to have a positive side effect of providing the safety net in a form of automated regression tests. At least this is my current understanding. This point was totally left out. Automated acceptance testing was mentioned, but that's about it. Other thing was that the value of TDD was verified with case studies from mainstream programming. I think the biggest issue is to convince the embedded crowd with studies from their own field, since basically people believe that agile has value but are not sure if it can be applied to their domain. Teppo mentioned that there are no studies from embedded and TDD, but Nancy's work is just that! I guess he forgot his colleagues presentation.

The room was full with 60+ listeners, so there is interest towards agile and embedded. I think a lot of people present were not practicing but more curious about what is all that buzz around agile and whether these methods are applicable to embedded (firmware) software development. I hope the event increased the interest even further and in the near future we would get more stories from Finnish companies practicing agile and embedded. It's nice to notice that Tieturi is taking a strong role in promoting these methods for embedded as well.

There's More Under Agile Umbrella Than XP and Scrum

Just a reminder. While Scrum and eXtreme Programming are without a doubt most widely adapted agile methods there are more. I studied many of them while reviewing their applicability to firmware development. Rod Coffin and Derek Lane have also done a nice job writing a two part article summarizing seven different methods that fall under the agile umbrella.

Agile2007: Embedded Agile Discovery Session

On Wednesday at Agile2007 conference James Grenning and Dan Pierce pulled together a discovery session for Agile and Embedded. The room was packed with people with experience on applying agile methods to embedded software development. Well, packed in this context means just over a dozen in a conference with 1150 attendees. Anyway. You really should sign up to Yahoo Agile Embedded group to follow the work of others. There are some remarkable people in this small community.

The discussion had two main topics. First, why embedded agile is getting foothold in the industry as slowly as it is. Several items were identified and the discussion is continued in Yahoo group. Second, that's correct, what else, but agile testing. People had different levels of practical experience, but the main theme was that it is important and it is definatelly possible. On the topic of safety critical embedded software and agile methods, the verdict was that this is not a problem. Safety critical embedded software actually SHOULD be developed using agile methods.

90 minutes was way too short time for anything concrete, but at least the event proved that agile embedded community exists. Thanks to James and Dan.

Jack Ganssle was also present and he posted about the conference in his Break Points column at embedded.com.

The Agile 2007 conference catered mostly to PC types, but some embedded heads showed up.

That post also made it to the latest Carnival of Agilists. This edition of Carnival of Agilists also mentions Atomic Object's approach to mocking the embedded world.

Agile embedded gets some visibility.

Seminar on Agile and Embedded in Helsinki

At ELKOM07 fair in Helsinki, Finland, Tieturi organizes a free 4h mini seminar on Agile and embedded system development on 5.9.2007. While it's free you still need to register.