The Complex Velocity of True Teams

Using velocity for long term planning of a project is a wonderful idea. With minimal effort we get fairly good estimates. At least this would be the case in ideal situations, meaning new team without maintenance or obsolete component replacement obligations etc. If you are a believer in true teams as I am, you do realize that all the urgent work and even absence due personal issues affect team dynamics. Just calculating velocity in terms of team member availability isn't realistic with true teams. The velocity, or team achievement, is not linear function of sum of available team member hours. If one person is for example pulled away from team activities for taking care of urgent manufacturing issue, this will affect the team dynamics. This has deeper impact than just missing those hours. To be honest, in some teams the impact can be positive as well. In that case I suppose they were non functioning work group, which now due the changed dynamics has a chance to become a team.

This said, no, we have not solved the issue of maintenance and other urgent work disrupting normal work. We are struggling with bouncing velocity. This is for several reasons, but it also woke me to think about true teams again. Anyway, I'd like to pay more attention to this and give true teams a chance to grow.

Even More Guns for Embedded Development

A while back Benoit saw the light of using modern languages (Ruby in that case) to help in developing embedded software, which still in most cases is developed in C for very well known reasons. I though I will throw additional ingradient to this soup. We are also using Python for our test-driven development, for acceptance test-driven development to be precise. There are few concepts that has lead us to do this.


1. You remember wrapping thinly with hexagonal architecture, right?

2. Company that I work for has a long history of using embedded PC simulation system called E-SIM from Mentor Graphics. This product has evolved, but we are using the old version.

3. E-SIM simulation tool has a Python API wrapper

A continuum is to develop own embedded acceptance test framework using Python, right? Well, we did it anyway. It is fairly simple, really. E-SIM communicates with the application interfaces using the mechanism called channels. There is a Windows service running called Communication Manager. And this fellow is responsible for gathering and sharing values in these channels to various C applications and our test driver written in Python. Indeed, we can have several applications running on one PC and simulate multi-device/processor/board systems as we wish. A channel can be a package in communication, a binary input as push button, binary output as LED, and so on.

Now, wanting to be able to write acceptance tests in a domain language that could be more readily understood by the business, we created a script parser using Python's reflection. This test driver parses the test script file and launches all the application .exe's as needed and then stimulates and senses the system using of E-SIM channel accessors. Sounds simple? The nice part is that it really is simple.

These tests run automatically on our continuous integration server, but we also have a GUI which simulates UI of a real system. Now we can watch the UI in action during the test sequence scripts.

Here is a snippet of .test acceptance test script written in (almost) domain language

sendFire 0,1,2,3,Delay Fire
wait 2
sendDelayed 0,1,100
wait 2
checkRow 1,FIRE in zone 00021/1

Here is the reflection part of the parser in Python. I have no idea how good of a Python code this is being an average embedded C programmer, but it looks simple to me.

for line in tp.getTestLines():
  self.currentScriptLine = line
  self.currentScriptLineNumber += 1
   
  if len(line) > 1:
    elements = line.split(' ',1)

    try:
      if len(elements) > 1:
        getattr(self, elements[0])(elements[1].lstrip(' '))
      else:
        getattr(self, elements[0])()
    except:
      self.myReporter.addWarning()
      self.warning("Could not execute line: %s" % line)

And finally here is a simple method in the TestDriver class. You can see that the method's name matches the action word in the test script snippet above. The rest of the test script line is passed as an argument to the method.

  def checkRow(self, parameters):
    elements = parameters.split(',')
    row = int(elements[0])
  
    expected = string.replace(elements[1], "\n", "")
    actual = self.currentLCD_Rows(row)
 
    self.checkEqual( expected, actual )

Similar approaches are Atomic Object's Systir, and Exoftware's Exactor. These can be labeled as script frameworks in contrast to table frameworks like FIT/Fitnesse.

Impossible, I Have Never Done That

When adopting agile planning techniques to non-software development activities an often heard argument against estimating the complexity of, say, schematic is:

"it is impossible to say, I have never done that before"

Well, firstly, in embedded software that's the situation pretty much allways, yet agile planning has proven to be efficient enough over and over again.

Secondly, we are living the era of enlightened experimentation. Prototyping hardware is drop dead cheap and fast nowadays. Buying yourself a status of "I have done that once" instead of "haven't done that before" is as easy as deciding to do so.

Thirdly, schematics can be estimated with relative complexity.

Fourthly, Parkinson's law works everywhere where humans are involved. Timeboxing early prototypes will help making the progress reliable and visible. This is however understandably very counter intuitive to the "get it right the first time" camp.

This is what is called up-front prototyping. In contrast to traditional hardware prototyping trying to validate something at the end, up-front prototyping focuses on learning. We may even know that the prototype will not work on most parts, but we just want a reliable measure of how far are we. It is even advisable to design a prototype to prove just one thing. More than one uncertanty will make the work unnecessarily complex.

This distinction between the goals of traditional validating and new era learning prototyping has gotten me concidering a new word to replace "prototype". So far the best candidate is 'product'. Using a word product throughout the lifecycle we would realise that this is aiming to production quality, but the maturity of the desing is evolving.

Of course I'm talking about prototypes that can be assembled within reasonable cycle cost. The bar however is getting lower and lower every day. Interesting exercise is to take your latest embedded development project, draw a bar of development salary cost and a second bar illustrating your prototype cycle cost. You may be surprised how cheap "expensive unnecessary prototyping" is in the big picture.

At least have a taste

We teach our children to at least have a taste before they judge anything as "yak".

We also teach engineers to try out practices before they judge anything as "yak".

"Just try it, if it does not work - ditch it".

TDD (test-driven development) is probably the most difficult technical skill associated with so called agile development especially in constrained embedded environment. It is also probably the most rewarding when in use.

The problem with TDD is that it has a learning curve. To make things worse the earlier debug-later development model has an unlearning curve. Most of the true benefits can concretely be seen after a fairly long period of time. It is not enough just to have a taste before judging, you actually need a prolonged tryout period. You most likely need some external help in the beginning. Your short term productivity suffers because of learning new skills and solving new kinds of problems.

You need someone in customer's role who is willing to talk about long term productivity as well.

Never the less I recommend to just have a taste.

Just remembered when I was just nodding on embedded TDD

Recently I have been talking a lot about embedded TDD to different, mostly experienced, embedded programmers. I thought I would jump right into a bit more advanced issues, to start a discussion, and maybe walk out with few new things myself. After all they were excellent programmers. I didn't want to waste the opportunity to learn.


During these sessions I have seen a lot of nodding people. I have though that "wow, I'm pretty good at explaining this". I have left with a smile on my face after a huge sense of agreement. I know better now. It's a much longer journey than one day seminar.

The challenge, or one of them, as I realize now, is that if you give a basic introduction with mickey mouse stuff examples to experienced embedded developers you will be immediatelly shot down with "that won't work in our environment, because [choose your favorite item identified on Embedded Agile Yahoo group]".


On the other hand if you rush straight into embedded TDD specific stuff, people are going to just nod.

And this is the other end of the same challenge. They are nodding so that you would go away faster. Not understanding fully is scary for a seasoned engineer who has not opened a book in the past 15 years or so. This nodding experience alone is not going to make anyone start practicing TDD. The best is to give a hands-on experience on TDD itself, show the benefit, and not on mickey mouse stuff but on real deal.


Advice that I'm going to follow is to try to remember when I was also just nodding and how many years, books, articles, conferences, talks, discussions, trials, failures, small successes, and so on, it took for me to get it. And I'm still only beginning to understand TDD.

Embedded Blogosphere Gets Stronger

A while ago we took a look at embedded blogospehere. That sphere is now even stronger - James Grenning has a new blog. He is "blogging about Agile Development, especially embedded". That should be interesting enough. There's already two posts about techniques that can be used to test-drive hw driver development (1, 2).

Only few seats left in Agile Finland Seminar

Agile Finland is organizing yet another seminar. It takes place March 4th in Helsinki. This morning there was only some twenty seats available.

Islands of Expertise Can Get You Lost

You may be familiar with the following scenario. The embedded software development department has finally launched a large new system. The system obviously immediately gets new feature requests and other modification needs. During the initial development each developer has found a pet part in the overall system. When the improvements/modifications start it is intuitive to assign these smaller projects to a person whose area the change mostly concerns. In this approach we launch several, as many as there are developers, projects to run in parallel. This seems like a natural way to go.

While tempting at the first look, there are several problems in the above scenario. First is the customer’s role. It is difficult enough to get a functioning product owner for one project. Running multiple projects at the same time is ridiculously challenging. If we get one person to act as a customer proxy for all these small projects, this person soon becomes a bottleneck for all these projects. A project needs collaboration with the customer representative, and the sooner we get the feedback the better. The customer needs to be available at all times, and this is easier to achieve for one project at the time.

Second problem follows from the fact that it is rare that all these projects are completely independent. While we think that no one works on more than one project at the time, we actually have created a highly complex hidden multi-project environment. Typically one, or few, developers know something about everything. This means that people that are supposed to be working for their own project need to consult other projects continuously. These people become the bottlenecks for other projects, and their own project is impossible to estimate. As a net result we push all our projects to be more, and more, late without any control to what is going on.

Thirdly, assigning a developer to her previous pet part of the system further develops the specialization in a narrow part of the overall system, creating small islands of expertice. More and more closed silos of expertice emerge in the department. This is a high risk path as in an unfortunate case of loosing that person for one reason, or another we halt the development of that part for unpredictable period of time. Yes, I have heard that documentation should be there for allowing anyone to work on new part. Well, having been in the development game for nearly 15 years I kind of have lost faith to that approach. Maybe if the previous developers have focused their energy mainly on enabling future work –type of documentation. Never the less, safer path is to several developers to know the work.

The fourth problem is the lack of possibility to gain from the superior performance of true teams. Assigning each person to a single small project will at its best only accomplish the sum of each developer in development department. If we are successful in getting a true team to jell we can accomplish a lot higher performance that the sum of team members. This is important especially when the work we are engaged with gets more complex, and needs higher problem solving capability.

Changing to work in teams and in the same part of the system may feel difficult and weird at the beginning. Progress may also appear slow as most of the team members are unfamiliar with the parts of the system they need to work with. It may also take old school engineers by the surprise that they also need the softer skills, interaction and communication. It is crucial that collective ownership is still enforced. Teams have a remarkable capability in finding the ways to work. The practices supporting the work of team will merge rapidly. You don’t have to know everything to start working this way. Just follow the basics;

1. do fast paced iterative development,
2. have reflective retrospectives, and
3. create open, learning-driven culture.
   

Post leak

A friend told me that one of my drafts leaked into feed. It was probably an user error by yours truly. I apologize for your incovenience.

What comes to that draft, I hope to find time to finish writing about Preston Smith's latest book "Flexible Product Development" and agile in non-software development soon.

Is It Death of Innovation - or Birth of Innovative Modifications

In "Your NPD portfolio may be harmful to your business health" (Cooper, 2005) presents data showing that true new product development has decreased by 43.7% between years 1990-2004. According the same article improvements and modifications to existing products has increased by 80.1% during the same time period.

September 2007 issue of Embedded System Design Journal revealed the results of reader's vote. Parts of the results add recent data to above observation. According to reader's vote the fraction of "new to the world; a new project from the scratch" compared to "an upgrade or improvement to an earlier or existing product" products has decreased from 48% to 39% in the past three years. 79% of those upgrades or improvements included "New or different software features".

Do not feel sad if it's been a while since you started to work on a brand new product. You are not alone. Working with new product seems to be getting more and more rare situation. If you get to do this, enjoy while you can. The data shows that there is a fair chance that you will be working with that product for a long time.

From the agile development point this data shows that it is ever more important to invest in automated regression test suite and refactored, clean, code. It is a safe bet that the software will be under heavy modification.

Very few of us have a chance to work in true high-tech fields which are generally understood to require innovation. Products developed are increasing in size and complexity. Building something from scratch based on a novel idea is most of the times in mega-project category. Mega-projects will take a long time, and thus include a mega-risk. This means that they are not so attractive to management. They do not get the funding so easily. However to me innovation means combining some, typically existing, technologies and ideas in a novel way to create some value to some one. By this definition adding ethernet connection to traditional product, and figuring out a valuable meaning for it, is innovation. We also need innovation in designing and improving the products so that they more easily adapt even more change. The need for innovation has not disappeared, it is just the nature of it that is shifting.

New Year, New Tricks

There has been a quiet period again in writing. There's a reason, it's the same as always - I've been busy. Now it is new year, and we kind of think we can have a fresh start. We all know that this is not exactly going to work like that, but this time the new year happens to actually be interesting for me; I have a new employer. I still remained within the same organization, but I'm now working at completely different business unit. I'm still going to work with agile methods, probably more so than ever. Of course it is still embedded stuff, no question, but there are major changes:

I'm now engaged with much larger systems.
Application changed from lighting control to fire alarm system.
Resource budget has also increased quite a bit. Basic setup now contains:

Renesas H8
2M program flash
2M RAM
Commercial RTOS

As a result I'm learning a bunch of new stuff every day. I have a good feeling about this year.

Those Stupid Moments - Round and Round Again with Unit Test Tools

It is time to feel stupid again. We have been experimenting TDD of firmware as you know if you have been reading this blog. We started with Embedded Unit framework which I was familiar with. We wanted to make it simpler to be able to run unit tests on target with very low resource budget. Then we realized that Unity framework is exactly that. Atomic Object has documented their thorough test-driven approach and open sourced many of their made-in-process Ruby tools. There's a nice article on the approach in Methods&Tools.

At first we did not get to use the other Ruby tools from Atomic Object apart Unity unit test framework, but implemented similar, but simpler, in Python. That was pretty much because we did not "get" the tools, being electrical engineers and all.


Then we realized that we do not have the time and energy, nor the skills, to develop these tools seriously, but should use existing tools out of the box as much as possible. We started using Atomic Object's Ruby tools. So now the same project is using its third tool set for unit testing.

Rake for building the tests

I can't do magic with Rake - yet. I get the build script to work, but it is structured pretty much like scripting languages I'm used to, and most of the things are more complex than they should. I however have gotten a sight of its power and it is definatelly my choise of build scripting at the moment. The online manual has simple examples of using Rake to build C projects. It makes a good starting point. Advanced examples can be found in Atomic Object's tool distribution. You can check some more in-depth analysis on Rake by Martin Fowler here.

Unity unit test framework

Just above in complexity of pure macro implementation, or standard assert's, is Unity. It's written in C, with few helper macros. It's intelligence is in it's simplicity. There is no support for suites or anything like that. What to build and run can be grouped in build script.

Here's what you need for a unit test in Unity:

static void testAfterAdding4NodesTheSizeOfListShouldBe4(void)
{
  int a;
  struct list list;
 
  list_Init (&list);
  list_AddHead (&list, &a);
  list_AddHead (&list, &a);
  list_AddHead (&list, &a);
  list_AddHead (&list, &a);

  TEST_ASSERT_EQUAL(4, list_GetSize (&list));
}

int main(void)
{
  Unity.TestFile = __FILE__;

  UnityBegin();
  // RUN_TEST calls runTest
  RUN_TEST(testAfterAdding4NodesTheSizeOfListShouldBe4);
  UnityEnd();
  return (int) 0;
}

If you don't use mocking techniques you can just define RUN_TEST macro to call your setUp, test, and tearDown functions. That's it. In full approach it will call your mock module setup and verify functions also.

Cmock - automatic mock file generator

The other day a colleague interested in TDD'ing C said to me; "the biggest problem to me is that in C we allways need some real friends for a module under test. We do not have real mechanism to support interfaces." Well yes, and no. We concider interface to be the .h file, or in more advanced cases a struct of function pointers. After that we get to automating stuff. Cmock is a tool that generates mock functionality based on .h file. It goes through all .h files in your source directory and generates a corresponding mock.c files. These mocks can be linked together with the module under test and can be teached to expect calls and to return values. Here's an simple example of a function definition and resulting teaching function:

uint8_t Protection_filter_getState( uint8_t me_index);

void Protection_filter_getState_ExpectAndReturn( uint8_t me_index, uint8_t toReturn);

Argent automatic test code generator

It's a pain to add mock init and verify functions, as it is to add a call to test function to main function, or just to change a name of test in two different places. That's the pain - Argent is the relief. It uses a simple markup scheme, and interprets your test code in order to automatically insert the calls to appropriate test functions, and mock module initialization and verifications.

//[[$argent require 'cmock_injector.rb'; inject_mocks("lists");$]]

// Calls to mock objects will be inserted here accoring the included mock headers.

//[[$end$]]

//[[$argent require 'generate_unity.rb'; generate_unity();$]]

// Calls to test functions (identified by test prefix) will be inserted here.

//[[$end$]]

Mock dependency check

It's also a pain to add mock dependencies to linker command. Using the same interpreter as argent the mock dependencies can be checked at build time automatically. So all that is needed to take a mock object into build is to have it's header included in the test file.

So that's how far we are now. Now we need some experience to get our routines right.

I quess this is a good example of having courage to improve when we see an opportunity, and strenght to admit being dead wrong - over and over again. This is truly a learning experience.

CSM Course At Our Premises

Jeff Sutherland and Jens Oestergaard are going to give a Certified Scrum Master (CSM) course at our premises on 3.-4. Dec. 2007 in Espoo, Finland. It's an open class, so if you are interested check out if there's still seats available.

Link: Waterfall Manifesto

If you are not completely satisfied with agile manifesto - There is an alternative.

Check out: Waterfall Manifesto

It follows on the lines of popular Waterfall 2006 Conference.

Is There Any Value in Values?

Recently I was blessed with an opportunity to facilitate a project retrospective for globally distributed project at the end of their 9th Sprint. Project has development in Europe, China, and Mexico. It is an embedded system development project involving electronics, plastics, mechanics, and firmware development. For all the participants this is the first contact with Agile project management. As a starting exercise we had each participant try to describe the project so far with a single word. The words we got:

• "Chido" (This is something like "cool")


• "Good"


• "Good"


• "Skilled"


• "Challenging"


• "Spectacular"


• "Fun"


• "Speed"


• "Improvement"


• "Interesting"


• "To Be Finished..."

This was interesting in itself.

An article "Manage Your Embedded Projects" in Embedded Systems Design described developer motivators:

• "Achievement"


• "Growth"


• "Work itself"


• "Personal Life"


• "Technical supervision"


• "Advancement"


• "Relationship with peers"


• "Recognition"


• "Salary"


• "Responsibility"

This data is adapted from Boehm, Software Engineering Economics (1981) and Herzberg, "One more time: how do you motivate employees?" Harvard Business Review (1987). So it is a bit outdated. The data in the article is used to demonstrate the difference in motivators between developers and managers. Similar findings are presented in Motivators of Software Process Improvement: an analysis of practitioners' views (2002) and De-motivators for software process improvement: an analysis of practitioners' views (2003), both by Nathan Baddoo and Tracy Hall.

We can go on to generalize this further, and think about differences of people and organizations. Value system of people has been recorded and studied for a long time. Von Rosenstiel and Koch (1) reported that the values have changed towards respecting the uniqueness of people, individualism, enjoyment and discipline,self-development and also material values. That is not enjoyment, it is enjoyment AND discipline among other things. To me the "just words" exercise showed that agile work actually builds on these values; the project was challenging and needed lot of skill, but that just made it interesting and fun - chido. In many cases this however is not following the values of the company, which often are still drawn from old school thinking, like punctuality, hard work, modesty and other values related to duty and acceptance. This discrepancy between values of employee and employer affects the commitment level. Lack of commitment, in turn, reduces employee motivation and effort.

Of course in large organizations values are mostly just lip service, but nevertheless above should lead organizations to rethink the value in their values.

(1) von Rosenstiel, Lutz and Koch, Stefan, Change in Socioeconomic Values as a Trigger of organizational Learning, Published in (Dierkes, Meinolf, Berthoin Antal, Ariane, Child, John, and Nonaka, Ikujiro, Handbook of Organizational Learning & Knowledge, Oxford University Press Inc., New York, 2001).