The Complex Velocity of True Teams

Using velocity for long term planning of a project is a wonderful idea. With minimal effort we get fairly good estimates. At least this would be the case in ideal situations, meaning new team without maintenance or obsolete component replacement obligations etc. If you are a believer in true teams as I am, you do realize that all the urgent work and even absence due personal issues affect team dynamics. Just calculating velocity in terms of team member availability isn't realistic with true teams. The velocity, or team achievement, is not linear function of sum of available team member hours. If one person is for example pulled away from team activities for taking care of urgent manufacturing issue, this will affect the team dynamics. This has deeper impact than just missing those hours. To be honest, in some teams the impact can be positive as well. In that case I suppose they were non functioning work group, which now due the changed dynamics has a chance to become a team.

This said, no, we have not solved the issue of maintenance and other urgent work disrupting normal work. We are struggling with bouncing velocity. This is for several reasons, but it also woke me to think about true teams again. Anyway, I'd like to pay more attention to this and give true teams a chance to grow.

Even More Guns for Embedded Development

A while back Benoit saw the light of using modern languages (Ruby in that case) to help in developing embedded software, which still in most cases is developed in C for very well known reasons. I though I will throw additional ingradient to this soup. We are also using Python for our test-driven development, for acceptance test-driven development to be precise. There are few concepts that has lead us to do this.


1. You remember wrapping thinly with hexagonal architecture, right?

2. Company that I work for has a long history of using embedded PC simulation system called E-SIM from Mentor Graphics. This product has evolved, but we are using the old version.

3. E-SIM simulation tool has a Python API wrapper

A continuum is to develop own embedded acceptance test framework using Python, right? Well, we did it anyway. It is fairly simple, really. E-SIM communicates with the application interfaces using the mechanism called channels. There is a Windows service running called Communication Manager. And this fellow is responsible for gathering and sharing values in these channels to various C applications and our test driver written in Python. Indeed, we can have several applications running on one PC and simulate multi-device/processor/board systems as we wish. A channel can be a package in communication, a binary input as push button, binary output as LED, and so on.

Now, wanting to be able to write acceptance tests in a domain language that could be more readily understood by the business, we created a script parser using Python's reflection. This test driver parses the test script file and launches all the application .exe's as needed and then stimulates and senses the system using of E-SIM channel accessors. Sounds simple? The nice part is that it really is simple.

These tests run automatically on our continuous integration server, but we also have a GUI which simulates UI of a real system. Now we can watch the UI in action during the test sequence scripts.

Here is a snippet of .test acceptance test script written in (almost) domain language

sendFire 0,1,2,3,Delay Fire
wait 2
sendDelayed 0,1,100
wait 2
checkRow 1,FIRE in zone 00021/1

Here is the reflection part of the parser in Python. I have no idea how good of a Python code this is being an average embedded C programmer, but it looks simple to me.

for line in tp.getTestLines():
  self.currentScriptLine = line
  self.currentScriptLineNumber += 1
   
  if len(line) > 1:
    elements = line.split(' ',1)

    try:
      if len(elements) > 1:
        getattr(self, elements[0])(elements[1].lstrip(' '))
      else:
        getattr(self, elements[0])()
    except:
      self.myReporter.addWarning()
      self.warning("Could not execute line: %s" % line)

And finally here is a simple method in the TestDriver class. You can see that the method's name matches the action word in the test script snippet above. The rest of the test script line is passed as an argument to the method.

  def checkRow(self, parameters):
    elements = parameters.split(',')
    row = int(elements[0])
  
    expected = string.replace(elements[1], "\n", "")
    actual = self.currentLCD_Rows(row)
 
    self.checkEqual( expected, actual )

Similar approaches are Atomic Object's Systir, and Exoftware's Exactor. These can be labeled as script frameworks in contrast to table frameworks like FIT/Fitnesse.

Impossible, I Have Never Done That

When adopting agile planning techniques to non-software development activities an often heard argument against estimating the complexity of, say, schematic is:

"it is impossible to say, I have never done that before"

Well, firstly, in embedded software that's the situation pretty much allways, yet agile planning has proven to be efficient enough over and over again.

Secondly, we are living the era of enlightened experimentation. Prototyping hardware is drop dead cheap and fast nowadays. Buying yourself a status of "I have done that once" instead of "haven't done that before" is as easy as deciding to do so.

Thirdly, schematics can be estimated with relative complexity.

Fourthly, Parkinson's law works everywhere where humans are involved. Timeboxing early prototypes will help making the progress reliable and visible. This is however understandably very counter intuitive to the "get it right the first time" camp.

This is what is called up-front prototyping. In contrast to traditional hardware prototyping trying to validate something at the end, up-front prototyping focuses on learning. We may even know that the prototype will not work on most parts, but we just want a reliable measure of how far are we. It is even advisable to design a prototype to prove just one thing. More than one uncertanty will make the work unnecessarily complex.

This distinction between the goals of traditional validating and new era learning prototyping has gotten me concidering a new word to replace "prototype". So far the best candidate is 'product'. Using a word product throughout the lifecycle we would realise that this is aiming to production quality, but the maturity of the desing is evolving.

Of course I'm talking about prototypes that can be assembled within reasonable cycle cost. The bar however is getting lower and lower every day. Interesting exercise is to take your latest embedded development project, draw a bar of development salary cost and a second bar illustrating your prototype cycle cost. You may be surprised how cheap "expensive unnecessary prototyping" is in the big picture.

At least have a taste

We teach our children to at least have a taste before they judge anything as "yak".

We also teach engineers to try out practices before they judge anything as "yak".

"Just try it, if it does not work - ditch it".

TDD (test-driven development) is probably the most difficult technical skill associated with so called agile development especially in constrained embedded environment. It is also probably the most rewarding when in use.

The problem with TDD is that it has a learning curve. To make things worse the earlier debug-later development model has an unlearning curve. Most of the true benefits can concretely be seen after a fairly long period of time. It is not enough just to have a taste before judging, you actually need a prolonged tryout period. You most likely need some external help in the beginning. Your short term productivity suffers because of learning new skills and solving new kinds of problems.

You need someone in customer's role who is willing to talk about long term productivity as well.

Never the less I recommend to just have a taste.

Just remembered when I was just nodding on embedded TDD

Recently I have been talking a lot about embedded TDD to different, mostly experienced, embedded programmers. I thought I would jump right into a bit more advanced issues, to start a discussion, and maybe walk out with few new things myself. After all they were excellent programmers. I didn't want to waste the opportunity to learn.


During these sessions I have seen a lot of nodding people. I have though that "wow, I'm pretty good at explaining this". I have left with a smile on my face after a huge sense of agreement. I know better now. It's a much longer journey than one day seminar.

The challenge, or one of them, as I realize now, is that if you give a basic introduction with mickey mouse stuff examples to experienced embedded developers you will be immediatelly shot down with "that won't work in our environment, because [choose your favorite item identified on Embedded Agile Yahoo group]".


On the other hand if you rush straight into embedded TDD specific stuff, people are going to just nod.

And this is the other end of the same challenge. They are nodding so that you would go away faster. Not understanding fully is scary for a seasoned engineer who has not opened a book in the past 15 years or so. This nodding experience alone is not going to make anyone start practicing TDD. The best is to give a hands-on experience on TDD itself, show the benefit, and not on mickey mouse stuff but on real deal.


Advice that I'm going to follow is to try to remember when I was also just nodding and how many years, books, articles, conferences, talks, discussions, trials, failures, small successes, and so on, it took for me to get it. And I'm still only beginning to understand TDD.

Embedded Blogosphere Gets Stronger

A while ago we took a look at embedded blogospehere. That sphere is now even stronger - James Grenning has a new blog. He is "blogging about Agile Development, especially embedded". That should be interesting enough. There's already two posts about techniques that can be used to test-drive hw driver development (1, 2).

Only few seats left in Agile Finland Seminar

Agile Finland is organizing yet another seminar. It takes place March 4th in Helsinki. This morning there was only some twenty seats available.

Islands of Expertise Can Get You Lost

You may be familiar with the following scenario. The embedded software development department has finally launched a large new system. The system obviously immediately gets new feature requests and other modification needs. During the initial development each developer has found a pet part in the overall system. When the improvements/modifications start it is intuitive to assign these smaller projects to a person whose area the change mostly concerns. In this approach we launch several, as many as there are developers, projects to run in parallel. This seems like a natural way to go.

While tempting at the first look, there are several problems in the above scenario. First is the customer’s role. It is difficult enough to get a functioning product owner for one project. Running multiple projects at the same time is ridiculously challenging. If we get one person to act as a customer proxy for all these small projects, this person soon becomes a bottleneck for all these projects. A project needs collaboration with the customer representative, and the sooner we get the feedback the better. The customer needs to be available at all times, and this is easier to achieve for one project at the time.

Second problem follows from the fact that it is rare that all these projects are completely independent. While we think that no one works on more than one project at the time, we actually have created a highly complex hidden multi-project environment. Typically one, or few, developers know something about everything. This means that people that are supposed to be working for their own project need to consult other projects continuously. These people become the bottlenecks for other projects, and their own project is impossible to estimate. As a net result we push all our projects to be more, and more, late without any control to what is going on.

Thirdly, assigning a developer to her previous pet part of the system further develops the specialization in a narrow part of the overall system, creating small islands of expertice. More and more closed silos of expertice emerge in the department. This is a high risk path as in an unfortunate case of loosing that person for one reason, or another we halt the development of that part for unpredictable period of time. Yes, I have heard that documentation should be there for allowing anyone to work on new part. Well, having been in the development game for nearly 15 years I kind of have lost faith to that approach. Maybe if the previous developers have focused their energy mainly on enabling future work –type of documentation. Never the less, safer path is to several developers to know the work.

The fourth problem is the lack of possibility to gain from the superior performance of true teams. Assigning each person to a single small project will at its best only accomplish the sum of each developer in development department. If we are successful in getting a true team to jell we can accomplish a lot higher performance that the sum of team members. This is important especially when the work we are engaged with gets more complex, and needs higher problem solving capability.

Changing to work in teams and in the same part of the system may feel difficult and weird at the beginning. Progress may also appear slow as most of the team members are unfamiliar with the parts of the system they need to work with. It may also take old school engineers by the surprise that they also need the softer skills, interaction and communication. It is crucial that collective ownership is still enforced. Teams have a remarkable capability in finding the ways to work. The practices supporting the work of team will merge rapidly. You don’t have to know everything to start working this way. Just follow the basics;

1. do fast paced iterative development,
2. have reflective retrospectives, and
3. create open, learning-driven culture.
   

Post leak

A friend told me that one of my drafts leaked into feed. It was probably an user error by yours truly. I apologize for your incovenience.

What comes to that draft, I hope to find time to finish writing about Preston Smith's latest book "Flexible Product Development" and agile in non-software development soon.

Is It Death of Innovation - or Birth of Innovative Modifications

In "Your NPD portfolio may be harmful to your business health" (Cooper, 2005) presents data showing that true new product development has decreased by 43.7% between years 1990-2004. According the same article improvements and modifications to existing products has increased by 80.1% during the same time period.

September 2007 issue of Embedded System Design Journal revealed the results of reader's vote. Parts of the results add recent data to above observation. According to reader's vote the fraction of "new to the world; a new project from the scratch" compared to "an upgrade or improvement to an earlier or existing product" products has decreased from 48% to 39% in the past three years. 79% of those upgrades or improvements included "New or different software features".

Do not feel sad if it's been a while since you started to work on a brand new product. You are not alone. Working with new product seems to be getting more and more rare situation. If you get to do this, enjoy while you can. The data shows that there is a fair chance that you will be working with that product for a long time.

From the agile development point this data shows that it is ever more important to invest in automated regression test suite and refactored, clean, code. It is a safe bet that the software will be under heavy modification.

Very few of us have a chance to work in true high-tech fields which are generally understood to require innovation. Products developed are increasing in size and complexity. Building something from scratch based on a novel idea is most of the times in mega-project category. Mega-projects will take a long time, and thus include a mega-risk. This means that they are not so attractive to management. They do not get the funding so easily. However to me innovation means combining some, typically existing, technologies and ideas in a novel way to create some value to some one. By this definition adding ethernet connection to traditional product, and figuring out a valuable meaning for it, is innovation. We also need innovation in designing and improving the products so that they more easily adapt even more change. The need for innovation has not disappeared, it is just the nature of it that is shifting.

New Year, New Tricks

There has been a quiet period again in writing. There's a reason, it's the same as always - I've been busy. Now it is new year, and we kind of think we can have a fresh start. We all know that this is not exactly going to work like that, but this time the new year happens to actually be interesting for me; I have a new employer. I still remained within the same organization, but I'm now working at completely different business unit. I'm still going to work with agile methods, probably more so than ever. Of course it is still embedded stuff, no question, but there are major changes:

I'm now engaged with much larger systems.
Application changed from lighting control to fire alarm system.
Resource budget has also increased quite a bit. Basic setup now contains:

Renesas H8
2M program flash
2M RAM
Commercial RTOS

As a result I'm learning a bunch of new stuff every day. I have a good feeling about this year.