Timo's weblog

TDD and design

2011-12-28T18:14:00.000+01:00

Earlier I wrote about the long distance I went with mac driver design (link). The current design is sketched below with examples of functions, and responsibilities (green). The clouds are C files. The design has 95% unit test line coverage. Tests have proven their power. I refactored the code using a local repository while traveling on vacation on remote island (during off-days from diving) and obviously no access to real target for testing. I made 53 commits. My commit frequency is very high, so many of the refactorings were just renaming and extracting helpers, but there were also more fundamental design changes. When I finaly, and sadly, made it back to the lab, I was kind of afraid that the code won't run and the fastest thing to do is to throw away all the refactorings. The next fastest thing would be to repeat them one by one in real repository. I gotta say I was surprised when the code worked right out the cross-compiler and all I needed to do was one massive merge from local to real repository. This is very rewarding. During the refactoring there was a handfull of incidents when tests caught a stupid mistake made by me. This is worthy even if you had the access to real target. The nice thing is that unit tests on dev environment tell it right away.

There is no need to make tradeoffs and large/long changes without feedback because of lengthy burning times, or lengthty stepping path to debug newly written code.

But it wasn't the biggest learning. The biggest learning was that the design resulted quite the different from what one would expect. I base this claim to investigation of several example MAC driver source codes available in the internet. The design has proven to be good, in terms of testability (that was the driver) and adaptability (this was the proof). More about adaptability later below. The thing that differentiates this style of design is the emphasis it puts to testing. Design is good if it is easy to test. If the tests are complicated to understand or difficult to write all together, then there is a good chance that design has flauses.

I think what I experienced is well explained by Michael Feathers in his talk "The Deep Synergy Between Testability and Good Design" (video). Take a look, and don't think this applies only to OO languages. You'd be wrong. The driver we are talking about here is written in C.

Current design was tested when the hw team decided to have a second option for MAC driver. They wanted the final pcb so they could proceed with emission tests and we together did not have enough information to do the decision either way. The candidate for production pcb has routing for both options. One implementation of set based design. But back to the sw side of it...

The concepts in driver design kept most of the files completely untouched. SPI and DMA drivers were independent compile units and they needed no touching. This also means that no code was duplicated in the production code mass.

The original design was done with just testability in mind. At that time there was no knowledge about the extra hardware the design needs to comply with.

In my opinion the code became adaptable and reusable by designing it for testability.

I don't think writing code this way, by seprating concerns, focusing on single responsibility, and not mixing abstraction levels, is slower to write. Is it different? Oh yeah. You have to really develop a new sense for good coding. As a remark, which I did earlier, I wouldn't refactor the code after fiddling around, but write a decent design based on learning from exploring, aka spike.

In the current MAC driver code one detailed design decision may make you raise your eye browse; Single function ClockByte() is in separate file. This is because I wanted to assert through just the bytes been send, not through processor register dummies. Other option would have been to inject a function pointer for this. I chose to use link time seam.

On the other hand just few simple tests for basic correctness of ClockByte() function are enough. This can also been seen as principle of separation of concerns and keeping the files at the same level of abstraction.

Current design is far away from being perfect. It is not what I think should be achieved. It continues to offer me opportunities for deeper understanding of tdd, and synergy between design and tests, more deeply. The next lesson will be available when it gets factored to enable irq based interfacing between uC and mac driver. So far it has been just message polling.

Slides from Agile 2011

2011-09-10T07:05:00.008+02:00

Agile 2011 ended in Salt Lake City, Utah a few weeks ago. It was again great to meet the growing circle of friends in agile community. This year my own conference was a bit different as I submitted two talks and participated in the review process for Agile for Embedded Systems Development -track.

Indeed, we had dedicated track for embedded stuff. It was the 10 year anniversary for agile manifesto and this was the first time the embedded got this much attention. The track had quality sessions and averaged around 20 attendees for each session.

You can find my slide decks via the links below:

Embedded Testing Cycle - the First 3 Years, Markku Åhman and Timo Punkka

James Grenning presented the embedded test-driven development (TDD) cycle already in 2004. Whispers on the hallways of conference hotels tell that somebody is actually implementing this idea. However, there are only few documented implementation details available. Schneider Electric’s fire security team has been implementing TDD cycle as an integral part of the development process for 3 years. Come to learn from their real-life experience and mistakes in automated testing at different levels: unit testing, acceptance testing using simulation, and in real target hardware.

Agile Hardware and Co-Design, Timo Punkka

Agile software development is getting attention also in embedded software development. Embedded system development on the other hand requires different engineering disciplines working together. When embedded software team starts using agile methods, it affects also other disciplines. Agile development emphasizes continuous learning through experimenting and collaboration instead of following a detailed up-front plan. Agile embedded software team expects different behavior in system co-design. This talk discusses reasons and ways to adapt agile development to co-design of system development.

Lots of other presentations, including all other embedded track presentations, are available via the conference program site.

A New Home

2011-08-18T15:06:00.006+02:00

A fiend of mine did fantastic job on my new site. My fuzzy ideas and his zero-whining attitude made a good pair. It was truly an agile project. The site was up and running from day 1, and it emerged during number of customer-developer pairing sessions. Final outlook is a result of countless "you know, why don't we just try it"s.

Embedded Agile, ESC2010, Boston

2010-12-17T20:20:00.003+01:00

Here are the slides and technical paper from my talk on Embedded Agile at Embedded Systems Conference 2010, Boston.

I had a good time at the conference. Hope you can find the material useful!

Abstract. New product development (NPD) is getting more and more challenging. Change happens all the time in all dimensions, including own organization, technology, competition, and marketplace. Agile development is targeted at working in a turbulent environment driven by continuous learning. Originated from software industry, its applicability to embedded system development has been analyzed over the years. In this paper, I present some observations on implications of embedded system development to agile development. I introduce findings on frequent releasing, automated testing, co-design including non-SW development and quality systems like ISO9001.

See you in Grenoble

2010-11-08T17:10:00.002+01:00

I'll be in Grenoble, France November 22.-24.11.2010 to attend Agile Grenoble.

If you happen to be around, get in touch.

See you in Boston

2010-09-11T19:21:00.004+02:00

I'll be attending ESC 2010 in Boston in a couple of weeks. You can catch me speaking about embedded agile on Tuesday 21st. You can find me somewhere in the conference throughout the week. Let me know if you're around.

Fiddling around before TDD

2010-08-27T18:51:00.001+02:00

The first of the Uncle Bob's Three Rules of tdd states:

1. You are not allowed to write any production code unless it is to make a failing unit test pass.

A few recent discussions among embedded developers revealed that this rule has caused some confusion among fellow beginners of embedded tdd. So, let me first point you to another tip from the same source(Uncle Bob advises to do fiddling around on things that you are not sure of how they work). I will share a short story:

I've been writing a proof-of-concept driver for serial to Ethernet controller. It was not sure if the serial port on uC could be configured to work with the MAC controller, or if it would be possible to use the DMA controller to manage longer transfers. I needed the proof-of-concept for the hardware team fast. I learned how to use the controller by trial and error, glueing together several bits from application notes and examples and running them in combination of two evaluation kits. Needless to say it was a mess. It even turned out I couldn't get the job done without a few circuits from my hardware pals. It would have been really awkward and laborous to have tests written during this fast paced back and forth experimenting based on a sample code which of course didn't come with tests. All this was done in C, with the tools for C.

After I knew which bits worked and which didn't, I wanted to illustrate this learning in tests. I harnessed the quickly hammered code with tests and then massaged the tests and code hand in hand into better shape. In retrospective, I should have treated the original code as a throw away prototype (aka code from Spike). I thought I would be faster by continuing to work with the code I had. Sad, but lacking the discipline made my overall cycle time propably massively longer. I believe that this is more of a rule.

It might be from the first XP book, I'm not sure, but when I was first introduced to agile methods people always listed the last rule; rules are just rules. Based on this experience I do believe it is pragmatic to fiddle things around without tests. But when you start sculpting the solution towards production code, you should take a fresh start and drive test-first based on your newly acquired knowledge. I know I will next time.

Test Driven Development for Embedded C in beta

2010-04-03T08:13:00.004+02:00

James Grenning's book Test Driven Development for Embedded C is now available in beta from The Pragmatic Bookshelf.

I have taken a peek and checking it out is strongly recommended.

Bowling Game Kata in C

2009-12-19T07:35:00.003+01:00

Olve Maudal has shared his presentation of Bowling Game Kata using C. You can find the slides directly here (pdf).

I'll add a link to my TDD in C delicious.

Coverage with lcov, and so what?

2009-06-30T07:22:00.000+02:00

A while back we ran an experimental line coverage analysis on our acceptance test suite. The result was 68% on the code for the main control board. I got the result from nightly build and mentioned it in Daily Scrum, and prompted "so what do we think about it, should we track it"? Everyone on the team had a blank stare and then finally a team member came forward "yeah, that's a good question. So, what?"

Coverage is information. It is just that, an additional piece of information, not by any means the final truth. I don't remember who teached me this, but;

"If you take all your assert's away you still have the same coverage. You just ain't testing anything at all."

This has been explained here and of course in Brian Marick's classic How to Misuse Code Coverage (pdf).

Well, maybe the good coverage can not say anything about the quality of your tests, but poor coverage can certainly say thing or two of opposite nature. If your coverage is 20% we can say quite confidently that you ain't there yet.

I started with acceptance test line coverage, but the rest is about unit test line coverage. Some embedded teams use gcov and I have heard people fiddling the data to generate fancier reports. Being lazy as I am I didn't do it myself. I did what I'm good at and searched for what others have already done. I found lgov, which is a tool in Perl to format gcov data.

We run lcov under Cygwin. You can get lcov for example from here, extract, and execute "make install". Next compile and link unit tests with gcc using flags "-fprofile-arcs" and "-ftest-coverage". We have a special build target for intrumenting the unit test executables with debug and coverage information so that we don't unnecessary slow down the bulk of builds. Next execute your full suite just like you normally would.

In our case all .obj files from test build are in ./bin directory, and that's where all the coverage data files go to. Our unit test script moves them to ./bin/code_coverage directory away from .obj files, and we want the final html report to be in ./build/test/code_coverage. Now we have the information necessary to create a shell script to do the actual analysis and reporting of coverage data:


path=`dirname $0` 
lcov --directory $path/bin/code_coverage -b $path --capture --output-file $path/ic.info
genhtml -o $path/build/test/code_coverage $path/ic.info

Vola', your disappointing(?) results are ready to be browsed, like so:

What the heck, it's all green, while you only have tests for simple utils files? In this approach there is a limitation - you only get coverage information for the files that are involved with your test suite. With huge legacy code, this would yield too promising picture early on. Again you need to think for yourself.

Experiment with coverage in your team, I think it's worth every penny but even when you start closing 100%, remember to keep analyzing, "so what?"

Learning to cope with legacy C

2009-02-06T11:32:00.002+01:00

New responsibilities during the past year have been a great learning experience. The key learning is that now I really know how incompetent I am. I can’t wait to move again and learn how many more things I do really badly, or what would be even better, can't do at all. This is a brief story of one such finding during this joyrney.

For the past year we have focused on ATDD with our own framework written in Python. We have 200+ automated acceptance tests for the system. With unit tests we however have struggled. While we have over 100 (well, it’s a start) of them, without the exception of the latest ones they are not really meaningful.

What's different with the latest tests then? They focus on higher level. I’m not sure what these tests as programmer tests should be called, but a programmer test will do for now. I do believe unit tests should be focused when doing TDD, but, wait, wait, I have an excuse… The code is old. It has its dependencies, and while maybe not the worst case in the world, it is a pain to get something compiled in isolation. The code has responsibility based structure (or should have had), and this structure is expressed in the source code folder structure. Each of the responsible "modules", or folders, typically contain own task. A typical task looks something like this:

task_specific_inits();

for(;;) {
s = OS_wait_for_something();
switch(s) {
case 1:
do_something1(s);
break;
}
}

Sometimes do_something1(s) is inlined and you may get a bitter sweet taste of those infamous 1000+ line functions. Other times you are lucky and the whole high level event parsing is already done in own function, along with lines do_something_with_the_event_from_X(s). This function continues the handling with loooong switch case, hopefully just calling further functions.

So, when we decide to test something inside a selected "module", or a folder in our case, we compile and link single test file, all the production code from a single responsible module/folder, production code for everything considered utils, like linked lists etc., and fake everything else. For faking we use Atomic Object's Cmock and manually written stuff when appropriate. We choose the task handling for injecting the test actions.

We arrange the test execution environment as we wish by initializing all the parties to expected state and teaching the mocked neighbours accordingly. We inject a single event, or short sequence of events, into task's handling routine and we try to find ways to assert if everything went as we wished for. Sometimes we can use this to learn what really happens when you give such and such event. After all the default assumption is that the code works, as it has been in production for years. We want to make sure it stays that way, when we change it. We have several options for observing the behavior:

1. Automatically generated mocks will tell us if the interaction was as expected
2. We can use getters of utilities, like linked lists
3. We can sense the internal status of any of the production code files with few nasty little tricks like #define STATIC

When the first test, and maybe her friend, is running it is time to start refactoring your code. Refactoring your test code, that is. If you take a closer look on what you have done, you most likely see 1-2 300 lines long test cases, which look pretty much the same. Now it is a good time to start extracting helpers. When creating an event sequence to be run you probably generate similar data structures. These can be extracted into functions. You probably do a bunch of similar assertions on many of your test. These can be extracted to helper functions. And so on, and so on. Each refactoring is likely to reveal more opportunities for cleaning the code. This can't be emphasized more. It is important to keep the code clean from the beginning. Otherwise you will have a 10KLOC test file on your hands, and it is much more work to start cleaning it only at that point.

This is very far from TFD (test first design). It is a battle to get some tests going to be in better place to continue improving and changing the code. The code is not going to disappear anywhere soon, so there will be lots of changes.

Why it took us a year to get to this point? Blame is on me. I got bitten by the test bug while writing a really hard real-time firmware app with a former colleague bunch of years back, and we learned that small exact tests leading into small steps of coding lead into zero debugging time. This was type of SW where we earlier had spent majority of our time debugging the code with oscilloscope and manually monitoring led blinks with throw away debugging code. During that experiment I saw the light (as saw my colleague), and thought that this is how also firmware should be written. Write each line of code to make a small test pass. However it is fairly rare in embedded domain to get your hands on a green project. This may not be a characteristic of just embedded sw, but sw in general today. We mostly write enhancements to existing products. Existing products in 2009 are not typically delivered with automated tests, and even less so developed this in mind. There is going to be plenty of opportunities for battles like this. Letting go on the ideal very low level unit testing took a year for me. It is still my ideal way of coding, but we can not get there overnight with legacy code.

If getting first tests in place sounds easy(?), calm down. It is only a starting place. You will notice how hard it is to test your code for example because of scattered initialization routines or that there is no structure in the first place. You should concider all these problems as good things. They are indicators for places of improvement. Those problems are in the code, building tests only make it more visible. If you work on those problems, you should be able to see more dependency breaking opportunities and eventually get to more focused tests. That’s the plan at the moment.

Michael Feathers uses term pinch point in his book about working with legacy code. Pinch point is a function or small collection of functions that you can write tests against and cover changes in many more functions. I guess event handlers for tasks are our first natural pinch points. This at least is the current step on the learning ladder for me. Hope the ladders won’t fall.

James Grenning also made a nice job articulating the whole legacy code testing process in C language (link).

Atomic Object also presented the importance of refactoring the test code from the beginning (link).

My Delicious Embedded TDD Links

2009-01-27T11:51:00.004+01:00

After a hint from a friend I recalled that I actually did start to collect embedded TDD links on delicious.com a while ago.

So not having anything more interesting to do, I updated the collection a bit and placed a link on the sidebar.

My Delicious Links on Embedded TDD

The Complex Velocity of True Teams

2008-11-10T08:27:00.001+01:00

Using velocity for long term planning of a project is a wonderful idea. With minimal effort we get fairly good estimates. At least this would be the case in ideal situations, meaning new team without maintenance or obsolete component replacement obligations etc. If you are a believer in true teams as I am, you do realize that all the urgent work and even absence due personal issues affect team dynamics. Just calculating velocity in terms of team member availability isn't realistic with true teams. The velocity, or team achievement, is not linear function of sum of available team member hours. If one person is for example pulled away from team activities for taking care of urgent manufacturing issue, this will affect the team dynamics. This has deeper impact than just missing those hours. To be honest, in some teams the impact can be positive as well. In that case I suppose they were non functioning work group, which now due the changed dynamics has a chance to become a team.

This said, no, we have not solved the issue of maintenance and other urgent work disrupting normal work. We are struggling with bouncing velocity. This is for several reasons, but it also woke me to think about true teams again. Anyway, I'd like to pay more attention to this and give true teams a chance to grow.

Even More Guns for Embedded Development

2008-07-12T21:45:00.001+02:00

A while back Benoit saw the light of using modern languages (Ruby in that case) to help in developing embedded software, which still in most cases is developed in C for very well known reasons. I though I will throw additional ingradient to this soup. We are also using Python for our test-driven development, for acceptance test-driven development to be precise. There are few concepts that has lead us to do this.

1. You remember wrapping thinly with hexagonal architecture, right?

2. Company that I work for has a long history of using embedded PC simulation system called E-SIM from Mentor Graphics. This product has evolved, but we are using the old version.

3. E-SIM simulation tool has a Python API wrapper

A continuum is to develop own embedded acceptance test framework using Python, right? Well, we did it anyway. It is fairly simple, really. E-SIM communicates with the application interfaces using the mechanism called channels. There is a Windows service running called Communication Manager. And this fellow is responsible for gathering and sharing values in these channels to various C applications and our test driver written in Python. Indeed, we can have several applications running on one PC and simulate multi-device/processor/board systems as we wish. A channel can be a package in communication, a binary input as push button, binary output as LED, and so on.

Now, wanting to be able to write acceptance tests in a domain language that could be more readily understood by the business, we created a script parser using Python's reflection. This test driver parses the test script file and launches all the application .exe's as needed and then stimulates and senses the system using of E-SIM channel accessors. Sounds simple? The nice part is that it really is simple.

These tests run automatically on our continuous integration server, but we also have a GUI which simulates UI of a real system. Now we can watch the UI in action during the test sequence scripts.

Here is a snippet of .test acceptance test script written in (almost) domain language



sendFire 0,1,2,3,Delay Fire
wait 2
sendDelayed 0,1,100
wait 2
checkRow 1,FIRE in zone 00021/1

Here is the reflection part of the parser in Python. I have no idea how good of a Python code this is being an average embedded C programmer, but it looks simple to me.




for line in tp.getTestLines():
  self.currentScriptLine = line
  self.currentScriptLineNumber += 1
   
  if len(line) > 1:
    elements = line.split(' ',1)

    try:
      if len(elements) > 1:
        getattr(self, elements[0])(elements[1].lstrip(' '))
      else:
        getattr(self, elements[0])()
    except:
      self.myReporter.addWarning()
      self.warning("Could not execute line: %s" % line)

And finally here is a simple method in the TestDriver class. You can see that the method's name matches the action word in the test script snippet above. The rest of the test script line is passed as an argument to the method.


  def checkRow(self, parameters):
    elements = parameters.split(',')
    row = int(elements[0])
  
    expected = string.replace(elements[1], "\n", "")
    actual = self.currentLCD_Rows(row)
 
    self.checkEqual( expected, actual )

Similar approaches are Atomic Object's Systir, and Exoftware's Exactor. These can be labeled as script frameworks in contrast to table frameworks like FIT/Fitnesse.

Impossible, I Have Never Done That

2008-06-18T09:00:00.000+02:00

When adopting agile planning techniques to non-software development activities an often heard argument against estimating the complexity of, say, schematic is:

"it is impossible to say, I have never done that before"

Well, firstly, in embedded software that's the situation pretty much allways, yet agile planning has proven to be efficient enough over and over again.

Secondly, we are living the era of enlightened experimentation. Prototyping hardware is drop dead cheap and fast nowadays. Buying yourself a status of "I have done that once" instead of "haven't done that before" is as easy as deciding to do so.

Thirdly, schematics can be estimated with relative complexity.

Fourthly, Parkinson's law works everywhere where humans are involved. Timeboxing early prototypes will help making the progress reliable and visible. This is however understandably very counter intuitive to the "get it right the first time" camp.

This is what is called up-front prototyping. In contrast to traditional hardware prototyping trying to validate something at the end, up-front prototyping focuses on learning. We may even know that the prototype will not work on most parts, but we just want a reliable measure of how far are we. It is even advisable to design a prototype to prove just one thing. More than one uncertanty will make the work unnecessarily complex.

This distinction between the goals of traditional validating and new era learning prototyping has gotten me concidering a new word to replace "prototype". So far the best candidate is 'product'. Using a word product throughout the lifecycle we would realise that this is aiming to production quality, but the maturity of the desing is evolving.

Of course I'm talking about prototypes that can be assembled within reasonable cycle cost. The bar however is getting lower and lower every day. Interesting exercise is to take your latest embedded development project, draw a bar of development salary cost and a second bar illustrating your prototype cycle cost. You may be surprised how cheap "expensive unnecessary prototyping" is in the big picture.

At least have a taste

2008-05-06T20:05:00.002+02:00

We teach our children to at least have a taste before they judge anything as "yak".

We also teach engineers to try out practices before they judge anything as "yak".

"Just try it, if it does not work - ditch it".

TDD (test-driven development) is probably the most difficult technical skill associated with so called agile development especially in constrained embedded environment. It is also probably the most rewarding when in use.

The problem with TDD is that it has a learning curve. To make things worse the earlier debug-later development model has an unlearning curve. Most of the true benefits can concretely be seen after a fairly long period of time. It is not enough just to have a taste before judging, you actually need a prolonged tryout period. You most likely need some external help in the beginning. Your short term productivity suffers because of learning new skills and solving new kinds of problems.

You need someone in customer's role who is willing to talk about long term productivity as well.

Never the less I recommend to just have a taste.

Just remembered when I was just nodding on embedded TDD

2008-04-01T22:40:00.000+02:00

Recently I have been talking a lot about embedded TDD to different, mostly experienced, embedded programmers. I thought I would jump right into a bit more advanced issues, to start a discussion, and maybe walk out with few new things myself. After all they were excellent programmers. I didn't want to waste the opportunity to learn.

During these sessions I have seen a lot of nodding people. I have though that "wow, I'm pretty good at explaining this". I have left with a smile on my face after a huge sense of agreement. I know better now. It's a much longer journey than one day seminar.

The challenge, or one of them, as I realize now, is that if you give a basic introduction with mickey mouse stuff examples to experienced embedded developers you will be immediatelly shot down with "that won't work in our environment, because [choose your favorite item identified on Embedded Agile Yahoo group]".

On the other hand if you rush straight into embedded TDD specific stuff, people are going to just nod.

And this is the other end of the same challenge. They are nodding so that you would go away faster. Not understanding fully is scary for a seasoned engineer who has not opened a book in the past 15 years or so. This nodding experience alone is not going to make anyone start practicing TDD. The best is to give a hands-on experience on TDD itself, show the benefit, and not on mickey mouse stuff but on real deal.

Advice that I'm going to follow is to try to remember when I was also just nodding and how many years, books, articles, conferences, talks, discussions, trials, failures, small successes, and so on, it took for me to get it. And I'm still only beginning to understand TDD.

Embedded Blogosphere Gets Stronger

2008-03-10T05:12:00.001+01:00

A while ago we took a look at embedded blogospehere. That sphere is now even stronger - James Grenning has a new blog. He is "blogging about Agile Development, especially embedded". That should be interesting enough. There's already two posts about techniques that can be used to test-drive hw driver development (1, 2).

Only few seats left in Agile Finland Seminar

2008-02-29T04:53:00.003+01:00

Agile Finland is organizing yet another seminar. It takes place March 4th in Helsinki. This morning there was only some twenty seats available.

Islands of Expertise Can Get You Lost

2008-01-30T09:56:00.000+01:00

You may be familiar with the following scenario. The embedded software development department has finally launched a large new system. The system obviously immediately gets new feature requests and other modification needs. During the initial development each developer has found a pet part in the overall system. When the improvements/modifications start it is intuitive to assign these smaller projects to a person whose area the change mostly concerns. In this approach we launch several, as many as there are developers, projects to run in parallel. This seems like a natural way to go.

While tempting at the first look, there are several problems in the above scenario. First is the customer’s role. It is difficult enough to get a functioning product owner for one project. Running multiple projects at the same time is ridiculously challenging. If we get one person to act as a customer proxy for all these small projects, this person soon becomes a bottleneck for all these projects. A project needs collaboration with the customer representative, and the sooner we get the feedback the better. The customer needs to be available at all times, and this is easier to achieve for one project at the time.

Second problem follows from the fact that it is rare that all these projects are completely independent. While we think that no one works on more than one project at the time, we actually have created a highly complex hidden multi-project environment. Typically one, or few, developers know something about everything. This means that people that are supposed to be working for their own project need to consult other projects continuously. These people become the bottlenecks for other projects, and their own project is impossible to estimate. As a net result we push all our projects to be more, and more, late without any control to what is going on.

Thirdly, assigning a developer to her previous pet part of the system further develops the specialization in a narrow part of the overall system, creating small islands of expertice. More and more closed silos of expertice emerge in the department. This is a high risk path as in an unfortunate case of loosing that person for one reason, or another we halt the development of that part for unpredictable period of time. Yes, I have heard that documentation should be there for allowing anyone to work on new part. Well, having been in the development game for nearly 15 years I kind of have lost faith to that approach. Maybe if the previous developers have focused their energy mainly on enabling future work –type of documentation. Never the less, safer path is to several developers to know the work.

The fourth problem is the lack of possibility to gain from the superior performance of true teams. Assigning each person to a single small project will at its best only accomplish the sum of each developer in development department. If we are successful in getting a true team to jell we can accomplish a lot higher performance that the sum of team members. This is important especially when the work we are engaged with gets more complex, and needs higher problem solving capability.

Changing to work in teams and in the same part of the system may feel difficult and weird at the beginning. Progress may also appear slow as most of the team members are unfamiliar with the parts of the system they need to work with. It may also take old school engineers by the surprise that they also need the softer skills, interaction and communication. It is crucial that collective ownership is still enforced. Teams have a remarkable capability in finding the ways to work. The practices supporting the work of team will merge rapidly. You don’t have to know everything to start working this way. Just follow the basics;

do fast paced iterative development,
have reflective retrospectives, and
create open, learning-driven culture.

Post leak

2008-01-22T05:39:00.000+01:00

A friend told me that one of my drafts leaked into feed. It was probably an user error by yours truly. I apologize for your incovenience.

What comes to that draft, I hope to find time to finish writing about Preston Smith's latest book "Flexible Product Development" and agile in non-software development soon.

Is It Death of Innovation - or Birth of Innovative Modifications

2008-01-18T09:06:00.000+01:00

In "Your NPD portfolio may be harmful to your business health" (Cooper, 2005) presents data showing that true new product development has decreased by 43.7% between years 1990-2004. According the same article improvements and modifications to existing products has increased by 80.1% during the same time period.

September 2007 issue of Embedded System Design Journal revealed the results of reader's vote. Parts of the results add recent data to above observation. According to reader's vote the fraction of "new to the world; a new project from the scratch" compared to "an upgrade or improvement to an earlier or existing product" products has decreased from 48% to 39% in the past three years. 79% of those upgrades or improvements included "New or different software features".

Do not feel sad if it's been a while since you started to work on a brand new product. You are not alone. Working with new product seems to be getting more and more rare situation. If you get to do this, enjoy while you can. The data shows that there is a fair chance that you will be working with that product for a long time.

From the agile development point this data shows that it is ever more important to invest in automated regression test suite and refactored, clean, code. It is a safe bet that the software will be under heavy modification.

Very few of us have a chance to work in true high-tech fields which are generally understood to require innovation. Products developed are increasing in size and complexity. Building something from scratch based on a novel idea is most of the times in mega-project category. Mega-projects will take a long time, and thus include a mega-risk. This means that they are not so attractive to management. They do not get the funding so easily. However to me innovation means combining some, typically existing, technologies and ideas in a novel way to create some value to some one. By this definition adding ethernet connection to traditional product, and figuring out a valuable meaning for it, is innovation. We also need innovation in designing and improving the products so that they more easily adapt even more change. The need for innovation has not disappeared, it is just the nature of it that is shifting.

New Year, New Tricks

2008-01-02T21:04:00.000+01:00

There has been a quiet period again in writing. There's a reason, it's the same as always - I've been busy. Now it is new year, and we kind of think we can have a fresh start. We all know that this is not exactly going to work like that, but this time the new year happens to actually be interesting for me; I have a new employer. I still remained within the same organization, but I'm now working at completely different business unit. I'm still going to work with agile methods, probably more so than ever. Of course it is still embedded stuff, no question, but there are major changes:

I'm now engaged with much larger systems.
Application changed from lighting control to fire alarm system.
Resource budget has also increased quite a bit. Basic setup now contains:

Renesas H8
2M program flash
2M RAM
Commercial RTOS

As a result I'm learning a bunch of new stuff every day. I have a good feeling about this year.

Those Stupid Moments - Round and Round Again with Unit Test Tools

2007-11-28T21:32:00.000+01:00

It is time to feel stupid again. We have been experimenting TDD of firmware as you know if you have been reading this blog. We started with Embedded Unit framework which I was familiar with. We wanted to make it simpler to be able to run unit tests on target with very low resource budget. Then we realized that Unity framework is exactly that. Atomic Object has documented their thorough test-driven approach and open sourced many of their made-in-process Ruby tools. There's a nice article on the approach in Methods&Tools.

At first we did not get to use the other Ruby tools from Atomic Object apart Unity unit test framework, but implemented similar, but simpler, in Python. That was pretty much because we did not "get" the tools, being electrical engineers and all.

Then we realized that we do not have the time and energy, nor the skills, to develop these tools seriously, but should use existing tools out of the box as much as possible. We started using Atomic Object's Ruby tools. So now the same project is using its third tool set for unit testing.

Rake for building the tests

I can't do magic with Rake - yet. I get the build script to work, but it is structured pretty much like scripting languages I'm used to, and most of the things are more complex than they should. I however have gotten a sight of its power and it is definatelly my choise of build scripting at the moment. The online manual has simple examples of using Rake to build C projects. It makes a good starting point. Advanced examples can be found in Atomic Object's tool distribution. You can check some more in-depth analysis on Rake by Martin Fowler here.

Unity unit test framework

Just above in complexity of pure macro implementation, or standard assert's, is Unity. It's written in C, with few helper macros. It's intelligence is in it's simplicity. There is no support for suites or anything like that. What to build and run can be grouped in build script.

Here's what you need for a unit test in Unity:

static void testAfterAdding4NodesTheSizeOfListShouldBe4(void)
{
  int a;
  struct list list;
 
  list_Init (&list);
  list_AddHead (&list, &a);
  list_AddHead (&list, &a);
  list_AddHead (&list, &a);
  list_AddHead (&list, &a);

  TEST_ASSERT_EQUAL(4, list_GetSize (&list));
}

int main(void)
{
  Unity.TestFile = __FILE__;

  UnityBegin();
  // RUN_TEST calls runTest
  RUN_TEST(testAfterAdding4NodesTheSizeOfListShouldBe4);
  UnityEnd();
  return (int) 0;
}

If you don't use mocking techniques you can just define RUN_TEST macro to call your setUp, test, and tearDown functions. That's it. In full approach it will call your mock module setup and verify functions also.

Cmock - automatic mock file generator

The other day a colleague interested in TDD'ing C said to me; "the biggest problem to me is that in C we allways need some real friends for a module under test. We do not have real mechanism to support interfaces." Well yes, and no. We concider interface to be the .h file, or in more advanced cases a struct of function pointers. After that we get to automating stuff. Cmock is a tool that generates mock functionality based on .h file. It goes through all .h files in your source directory and generates a corresponding mock.c files. These mocks can be linked together with the module under test and can be teached to expect calls and to return values. Here's an simple example of a function definition and resulting teaching function:

uint8_t Protection_filter_getState( uint8_t me_index);

void Protection_filter_getState_ExpectAndReturn( uint8_t me_index, uint8_t toReturn);

Argent automatic test code generator

It's a pain to add mock init and verify functions, as it is to add a call to test function to main function, or just to change a name of test in two different places. That's the pain - Argent is the relief. It uses a simple markup scheme, and interprets your test code in order to automatically insert the calls to appropriate test functions, and mock module initialization and verifications.

//[[$argent require 'cmock_injector.rb'; inject_mocks("lists");$]]

// Calls to mock objects will be inserted here accoring the included mock headers.

//[[$end$]]

//[[$argent require 'generate_unity.rb'; generate_unity();$]]

// Calls to test functions (identified by test prefix) will be inserted here.

//[[$end$]]

Mock dependency check

It's also a pain to add mock dependencies to linker command. Using the same interpreter as argent the mock dependencies can be checked at build time automatically. So all that is needed to take a mock object into build is to have it's header included in the test file.

So that's how far we are now. Now we need some experience to get our routines right.

I quess this is a good example of having courage to improve when we see an opportunity, and strenght to admit being dead wrong - over and over again. This is truly a learning experience.

CSM Course At Our Premises

2007-11-28T07:22:00.000+01:00

Jeff Sutherland and Jens Oestergaard are going to give a Certified Scrum Master (CSM) course at our premises on 3.-4. Dec. 2007 in Espoo, Finland. It's an open class, so if you are interested check out if there's still seats available.

Link: Waterfall Manifesto

2007-11-07T06:48:00.000+01:00

If you are not completely satisfied with agile manifesto - There is an alternative.

Check out: Waterfall Manifesto

It follows on the lines of popular Waterfall 2006 Conference.

Is There Any Value in Values?

2007-10-26T11:34:00.000+02:00

Recently I was blessed with an opportunity to facilitate a project retrospective for globally distributed project at the end of their 9th Sprint. Project has development in Europe, China, and Mexico. It is an embedded system development project involving electronics, plastics, mechanics, and firmware development. For all the participants this is the first contact with Agile project management. As a starting exercise we had each participant try to describe the project so far with a single word. The words we got:

"Chido" (This is something like "cool")

"Good"

"Good"

"Skilled"

"Challenging"

"Spectacular"

"Fun"

"Speed"

"Improvement"

"Interesting"

"To Be Finished..."

This was interesting in itself.

An article "Manage Your Embedded Projects" in Embedded Systems Design described developer motivators:

"Achievement"

"Growth"

"Work itself"

"Personal Life"

"Technical supervision"

"Advancement"

"Relationship with peers"

"Recognition"

"Salary"

"Responsibility"

This data is adapted from Boehm, Software Engineering Economics (1981) and Herzberg, "One more time: how do you motivate employees?" Harvard Business Review (1987). So it is a bit outdated. The data in the article is used to demonstrate the difference in motivators between developers and managers. Similar findings are presented in Motivators of Software Process Improvement: an analysis of practitioners' views (2002) and De-motivators for software process improvement: an analysis of practitioners' views (2003), both by Nathan Baddoo and Tracy Hall.

We can go on to generalize this further, and think about differences of people and organizations. Value system of people has been recorded and studied for a long time. Von Rosenstiel and Koch (1) reported that the values have changed towards respecting the uniqueness of people, individualism, enjoyment and discipline,self-development and also material values. That is not enjoyment, it is enjoyment AND discipline among other things. To me the "just words" exercise showed that agile work actually builds on these values; the project was challenging and needed lot of skill, but that just made it interesting and fun - chido. In many cases this however is not following the values of the company, which often are still drawn from old school thinking, like punctuality, hard work, modesty and other values related to duty and acceptance. This discrepancy between values of employee and employer affects the commitment level. Lack of commitment, in turn, reduces employee motivation and effort.

Of course in large organizations values are mostly just lip service, but nevertheless above should lead organizations to rethink the value in their values.

(1) von Rosenstiel, Lutz and Koch, Stefan, Change in Socioeconomic Values as a Trigger of organizational Learning, Published in (Dierkes, Meinolf, Berthoin Antal, Ariane, Child, John, and Nonaka, Ikujiro, Handbook of Organizational Learning & Knowledge, Oxford University Press Inc., New York, 2001).

Yet Another Agile Seminar in Finland

2007-10-02T07:51:00.000+02:00

It seems to be the time for another Agile Finland Seminar. It is organized by Agile Alliance, Reaktor Innovations, and F-Secure. Guys have set out to break the old record for short notice:

On October 14th, 2006, I sent an email to the AgileFinland Yahoo! group about the next Agile Seminar taking place only 9 days later. Guess what? Everything went well even with such a short notice so this time we decided to improve and give you an even shorter notice.

At the time of writing this there are 84 seats available. Go get yours...

Agile Embedded Seminar at ELKOM2007

2007-09-10T12:55:00.000+02:00

On Wednesday Tieturi organized a mini seminar on Agile and embedded in Helsinki, Finland. The seminar had 3 talks. First Pentti Virtanen gave a general background introduction to agile development. This was pretty much your average pure vanilla intro to agile -type presentation. The second part by Tomi Juhola was titled agile embedded in practice. I had high hopes for this as I though I would actually get to hear some new concrete stories from other Finnish companies practicing agile development. Well, it turned out that the cases presented were first the nice work of Nancy Van Schooenderwoert, and the second Jeff Sutherlands PatientKeeper. No new evidence there. The last talk by Teppo Heikurinen was about testing in agile development. The main topic seemed to be TDD. I have realized that TDD is not so much about testing, but more about design. There was some critique from audience that TDD is just like testing your own work and it has known problems. Yes in that sense it does, but TDD is a design practice which just happens to have a positive side effect of providing the safety net in a form of automated regression tests. At least this is my current understanding. This point was totally left out. Automated acceptance testing was mentioned, but that's about it. Other thing was that the value of TDD was verified with case studies from mainstream programming. I think the biggest issue is to convince the embedded crowd with studies from their own field, since basically people believe that agile has value but are not sure if it can be applied to their domain. Teppo mentioned that there are no studies from embedded and TDD, but Nancy's work is just that! I guess he forgot his colleagues presentation.

The room was full with 60+ listeners, so there is interest towards agile and embedded. I think a lot of people present were not practicing but more curious about what is all that buzz around agile and whether these methods are applicable to embedded (firmware) software development. I hope the event increased the interest even further and in the near future we would get more stories from Finnish companies practicing agile and embedded. It's nice to notice that Tieturi is taking a strong role in promoting these methods for embedded as well.

There's More Under Agile Umbrella Than XP and Scrum

2007-09-04T19:11:00.000+02:00

Just a reminder. While Scrum and eXtreme Programming are without a doubt most widely adapted agile methods there are more. I studied many of them while reviewing their applicability to firmware development. Rod Coffin and Derek Lane have also done a nice job writing a two part article summarizing seven different methods that fall under the agile umbrella.

Agile2007: Embedded Agile Discovery Session

2007-08-30T18:46:00.000+02:00

On Wednesday at Agile2007 conference James Grenning and Dan Pierce pulled together a discovery session for Agile and Embedded. The room was packed with people with experience on applying agile methods to embedded software development. Well, packed in this context means just over a dozen in a conference with 1150 attendees. Anyway. You really should sign up to Yahoo Agile Embedded group to follow the work of others. There are some remarkable people in this small community.

The discussion had two main topics. First, why embedded agile is getting foothold in the industry as slowly as it is. Several items were identified and the discussion is continued in Yahoo group. Second, that's correct, what else, but agile testing. People had different levels of practical experience, but the main theme was that it is important and it is definatelly possible. On the topic of safety critical embedded software and agile methods, the verdict was that this is not a problem. Safety critical embedded software actually SHOULD be developed using agile methods.

90 minutes was way too short time for anything concrete, but at least the event proved that agile embedded community exists. Thanks to James and Dan.

Jack Ganssle was also present and he posted about the conference in his Break Points column at embedded.com.

The Agile 2007 conference catered mostly to PC types, but some embedded heads showed up.

That post also made it to the latest Carnival of Agilists. This edition of Carnival of Agilists also mentions Atomic Object's approach to mocking the embedded world.

Agile embedded gets some visibility.

Seminar on Agile and Embedded in Helsinki

2007-08-23T12:36:00.000+02:00

At ELKOM07 fair in Helsinki, Finland, Tieturi organizes a free 4h mini seminar on Agile and embedded system development on 5.9.2007. While it's free you still need to register.

Does eXtreme Development Exist?

2007-08-21T12:10:00.000+02:00

I spent last week in Washington D.C. participating Agile2007 conference. The conference program was overwhelming with over 300 events. However while we have many implications to agile embedded systems development the agile embedded community is still small. So next time you see an agile conference ad, sign up!

Jim Highsmith is a person that has brought together agile community and general product development. I have followed his work and writings closely. He has brought work of people like Stefan Thomke, Preston Smith, and Donald Rainertsen (Reinertsen & Associates) to my knowledge. The latter two co-wrote the book Developing Products in Half the Time. In the panel discussion about agile manifesto there was a Q if the panelists believed that values of manifesto are applicable to development outside SW. (This is what tried also!). Highsmith mentioned a pure HW team that they successfully coached based on eXtreme Programming techniques. In that particular project there was no SW at all!

While browsing for the links, I found out that there is a new book out from Preston Smith:

Flexible Product Development: Building Agility for Changing Markets. It's on top of my wish list.

The Pain of Choosing the Refactoring Tool

2007-08-02T07:50:00.000+02:00

Eclipse SDK version 4.0 was released earlier this summer. My hopes were high especially after listening Doug Schaefer, the Eclipse SDK project lead, at ESC2007. I just quickly took a look at the long waited refactoring tools.

Well, it's not as impressive as could, but it's a start. We know that writing refactoring tools for C is difficult, as with the preprocessing a C compiler can be made to understand virtually anything. Doug mentioned their struggle when he commented earlier on this blog. Good news is that it's now easier to start using Eclipse for C projects, as it is possible to get Eclipse and SDK in same package.

We have been using SlickEdit for a while. Slick has a bit more refactorings for C - actually all three of them.

Refactoring is an important ingredient in test-driven development and while these tools are not much, they still take the bar for quick refactoring lower.

And that is good news. If you are not able to decide from this myriad of choises, you can choose the middle way - get the SlickEdit plugin for Eclipse.

TDD Tools

2007-07-30T09:32:00.000+02:00

Using TDD for firmware in C can be quite laborious. Creating a new module requires at least steps like:

Create a test .c file
Add created test to test build
Create implementation .c file
Create implementation .h file
Add implementation to production build
Add implementation to previously created test build
Create fake object(s) to make the test build to pass
Add fake object files to test build

When you adapt the habit of testing everything you quickly get bored repeating these tasks. You end up looking for tools to automate this repeating work. At least this is what you should end up doing, and not giving up.

Atomic Object has created tools in Ruby to do the trick and some. At ESC2007 they demonstrated their tools and already gave the tools on CD, but there still was some minor problems getting the tools to build. Now they have released their tools as open source and fixed the small problems in the earlier distribution. In Summer Issue of Methods & Tools is an article elaborating Atomic Objects approach to agile firmware development infrastructure. Highly recommended starting point, and offers also a path for advanced testing.

Just tried out the package that is now available from their site - Worked like a charm!

Tools do really not need to be complex to make a difference, propably quite the opposite holds. We made file templates and bunch of scripts running sed to replace key words to automate C unit testing steps. We learned this from James Grenning's CppTestTools. From Atomic Object's package we adapted their simple unit test framework Unity. We stripped it down even further, and also mixed it up with the idea of different outputters libraries, like in Embedded Unit. We also adapted their idea of automated mock object creation, but ended up writing similar tool in Python (basically because we wanted to fully understand what we are doing). All our tools are tuned for our use and constantly evolving, but I think towards similar solution as already in use by Atomic Object, but in step-wise manner. We believe that you need to understand the reason, give it the context, for any new skill or practice in order to make it truly work.

Wrap It Thinly

2007-07-11T08:48:00.000+02:00

The major concern about TDD in firmware development I hear is the hardware relation.

-"We don't have hardware to run the code in."
-"Every part of the code is hardware dependant."

I noticed I wrote about these approximately a year ago for the first time (I, II, and III). That was more on the theoretical side. This time I have a bit more experience on the subject. We have found Alistair Cocburn's article on hexagonal architecture valuable in philosophizing this issue. Many firmware developers are not familiar with design patterns, but a good starting point is above mentioned article. Writing adapters for your hardware, grouping them to ports, keeping the number of them low, and only allowing them to be responsible for the lowest-level of hardware interfacing, will help you isolate hardware dependant parts. Here is a picture of hexagonal architecture for the case project we are working on.

Then there is a philosophical issue of testing the adapters, or wrappers. In many cases binary input/output handlings are just like accessors, getters and setters. In TDD literature it is generally agreed that these do not need unit tests. When writing hardware wrappers we need to think a bit harder. We could understand the unit tests for hardware wrappers as our firmware/hardware interface specification. Then for sure it makes sense to write these tests. My thinking goes along with "to be able to judge what to unit test and what not to unit test, you have to be able to unit test everything".

Test the Goal of the Code

2007-06-18T17:51:00.000+02:00

It is time to get technical again. We have been writing unit tests for some time. We have tried test-driven development. However we have never been disciplined in this. Sometimes we don't write tests at all. Sometimes we don't fix the test suite when code is needed to change. Sometimes we just decide that well, this project or that part of the project does not need tests (typically when it seems to get complex or difficult to test!). We have come to a conclusion that with this approach we will never totally master the skill, and not get the benefit it is supposed to bring.

We decided to try another approach. We are writing a very real hard-time constrained, very hardware related code with 100% test-driven development (TDD) in C. A code exists to do the job, but we are gonna completely re-write it. It is originally ported to C from assembly, and this is done maybe four years ago. Since then it has been improved, modified, and adapted to new situations. What has happened is the code has rotten, it is a jungle of conditional compilation, and so forth. So what is better place to do a tiny bit of always refreshing skunk work and completely re-write the whole piece of art?

Right after we made the decision that we are gonna write EVERYTHING with test-driven development we noticed a change in thinking. We constantly think for the simplest possible thing to do, constantly re-evaluate the module (C file) interfaces and so on. If you are gonna do everything in this manner, there just is no backdoor for you.

Another big change was (well, we still need constantly remind ourselves about this) the shift in thinking towards writing the tests as specification. We often ended up thinking about the solution and then writing tests to check the solution. This is maybe caused by the fact that one shot on the code exists and is thoroughly known. However, you will create novel, simple solutions for your tests only if you think about what you want your next small code-to-be do, not how it will do it. Keith Ray has listed questions to help you in this.

Sean Shubin has done a nice work and listed first guidelines to TDD. Again:

Test the goal of the code, not the implementation

Embedded TDD Link List

2007-06-09T08:11:00.000+02:00

I created a del.icio.us bookmarks for web resources I have found useful for understanding TDD for embedded software.

my del.icio.us bookmarks for embedded TDD

Embedded Blogosphere

2007-06-07T13:08:00.000+02:00

Yesterday was yet another record breaking hot day in Helsinki, and what did I do? That's right, searched the net for new embedded slash agile blogs. I periodically do this, but normally I come empty handed. This time I was happily surprised to find Low Country Software Ramblings by Benoit Lavigne. Benoit was also wondering "Am I alone?", and later found out that "I am not alone after all!"

Via Benoit's blog I discovered Embedded Cork by Ralph Depping and Embedded Gurus. Embedded gurus seem to be a bit quiet, but there is some recent stuff as well. Names behind Embedded Gurus group are impressive; Nigel Jones, Michael Barr, Larry Mittag, Mike Anderson, and Miro Samek.

Keep in mind the Atomic Object Spin Embedded Corner, and But Uncle Bob Archive (James Grenning).

I linked them in sidebar - give them a look.

Story: Spike Up Your Coctail

2007-06-01T06:29:00.000+02:00

I posted earlier about short, 6 week, experiment where we had weekly deliveries of firmware and true parallel co-design of plastics, electronics, and software/firmware. I mentioned I would try to write down experience report. Well, it became more like story, and you can find the whole of it here (pdf). (LINK CORRECTED 27/6/2007)

Abstract. Agile development is a term used for wide variety of lightweight software development methods following shared values1. Many of these methods and practices however can be applied to more general new product development. This paper describes a six week project using some of the practices from agile development. These practices included self organizing team, collective ownership, continuous integration, iterative planning, iteration demos, team retrospective meetings, wall work queue, information radiator etc. The project crystallized a vague idea and a draft of electronics schematic into two fully functioning prototypes. It is shown that meaningful functionality can be developed in just six weeks. This is remarkable when reflected against the effort needed in so called traditional process models with formal analysis/design and theoretical proof of concept with heavy review processes. This is possible because of today’s advanced development tools and prototyping technology. Stefan Thomke (2001) calls this an era of enlightened experimentation. Building the working prototype is more cost effective way of reliable proof of concept. Using this approach already in the fuzzy front-end phase of the project would result in huge savings in overall project schedule and budget. It was also noticed that all stakeholders of this project appreciated the approach and considered it as “common sense”. The positive side effect was the team building effect this period had.

Camp Fire

2007-05-29T20:24:00.000+02:00

A week ago on Tuesday we organized a camp fire event at our premises. Camp fire is an event about sharing agile experiences, and this was second of such events. Roughly twenty people gathered to an auditorium after the office hours. The audience consisted of people from few different organizations inside the company; developers, managers and marketers. We had two talks; Vasco Duarte from F-Secure was invited as quest storyteller and Mikko Kaijärvi from Schneider Electric volunteered to share his recent experience.

Vasco's presentation was strong, as usual. He talked about top-down agile adaptation at F-Secure in a very realistic way. He did everything but painted a pretty picture about the journey. This is according to his experience, and literature as well, expected. When everything is laid down on the table, it tends to get ugly from time to time. It was interesting to see how within top-down approach they were able to put the whole organization into "agile rhythm". Vasco mentioned that lack of outside help, or not getting outside support enough, was the main thing he would do differently if need ever arises. Vasco presented also some cold quantitative data from theirs and others agile adaptation to support the reasoning for change. It just happens to look like we have more data supporting agile than sequential models ever had. The presentation and especially the Q&A and discussion parts showed Vasco's deep knowledge and understanding of modern project work.

Mikko presented another interesting story in a lively style. His project has applied agile project management and Scrum practices in globally distributed embedded system development project. He estimated that only 5% of development effort is software and firmware, and in addition this is coming from non-agile team. Agile practices were seen as the only option for this project to be successful. The project was started with very vague requirements, barely a vision. Teams were pretty much given. One team in Shanghai, China and another in Monterrey, Mexico, and technology and software development in Europe. The offshore teams had minimal domain knowledge and a huge amount of knowledge needed to be transferred fast. Continuous integration of mechanical and electronics prototypes was a key catalyst for communication. Mikko further identified a clear transition in team behavior from hierarchical - through a single point communication - to self managing emergent behavior actively seeking for best possible solutions in this challenging environment. The project has only been active for five months, but they expect to deliver it in September as originally scheduled. Compared to average project that would be fast according Mikko. Furthermore active early participation of marketing is expected to result in better product as well. This has potential of making yet another example of agile project management practices being succesfull also outside pure software projects.

As conclusion both talks were excellent, delivered by guys that are experienced in both, the project work and giving presentations. Hopefully people were able to walk out with something new and to be able to connect some more dots in their thinking.

After the official part the discussion continued over snack and refreshments. It was agreed that lot of the agile adaptations outside so called agile homeground is about attitude. It is about the shift in thinking towards the art of possible. Just changing your de facto answer from starting with "Yes, but..." to "Yes, and..." offers a huge potential. Vasco's message was "if you really want it, you can do it." The reward demonstrably is there.

Enough of processes, let's do labels

2007-05-21T11:56:00.000+02:00

The title of course follows the Dr.Dobbs article by Ivar Jacobsen, Pan-Wei Ng, and Ian Spence "Enough of Processes, let's do practices Part I".

Venkatesh Krishnamurthy has an article in March 2007 Agile Journal about top-down commitment in agile adaptation. He defines three layers of stakeholders in organization; Sponsors, Mid Level Management, and Development Team. As a conclusion he presents an argument that highest possibility for successful agile adaptation is a right mixture of bottom-up and top-down strategies. Dave Nicolette reviews the article in his post and shares his experience:

The agile group had started life as a skunkworks operation. When the group was brought back into the IT department formally, IT management dismantled the group and morphed the project process, management tools, and development methods back into conventional ones, although with "agile" labels.

There are propably a lot of stories like that. Based on the initial moderate success in agile adaptation on the technical level, agile gets to be the new silver bullet and buzz word. A new group of people are expected to automatically form a team, jell, and "just do agile". "Just do agile" means that traditional PM starts using agile jargon without bothering to look what these terms mean, not to mention that she would try to master the agile values, principles and practices, adjust them, and further develop them. "Just do agile" seems like a little bit too thin statement regarding that agile adaptation is described to be a paradigm shift, a completely new philosophy requiring a new mental package. "Just do agile" could be called doing practices, but that would be being nice. Instead it is only doing labels as Dave called the phenomenon. Few years back I was teached to avoid new labels, like agile ones, when introducing change. Now I have learned that some people only want new labels. That's probably why the silver bullet business is doing so well. Calling everything agile, Scrum, and/or Sprint is not much of a change, and will not automatically make everything visible, consistent, or reliable. Somehow the point is completely lost. Of course people who expected agile to be a silver bullet or magic wand are going to be disappointed and call agile just another fad. Some people who actual get agile, and understand that it is a bit more than "doing some stuff in 30 or so days and using words like agile, Scrum, and Sprint a lot", are also quite frustrated. Dave's post sets out the more than likely dark symptom of this:

As I write this, just over a year has elapsed since the process of dismantling agile began. Of the 60+ people who were part of the agile group at its peak, only 4 individuals who were in that group are still employed by the company.

It is equally true according to my experience that commitment from all three categories mentioned by mr. Krishnamurthy is needed to continue agile adaptation for any prolonged periods . Both orders in which this adaptation happens have been reported as successful, and yes indeed, also as failures. Mixture of both strategies sounds appealing.

Mike Vizdos writes about similar experiences in his RIP Scrum -post and cartoon. He calls this a "Death by a thousand copies".

Conference Programs Announced

2007-05-09T07:22:00.000+02:00

Several agile conferences have their programs out.

Oresund Agile 2007
Copenhagen, 11-14 June

XP2007
Como, 18-22 June

Agile2007
Washington D.C., 13-17 August

At Agile2007 Dan Pierce and James Grenning are giving a talk "Embedded Agile - Issues and Practices". Should be valuable, as these two gentlemen have been talking about and practicing agile development in embedded domain for a long time. Of course lots of other interesting stuff as well, 5 days with 8 or more tracks...

Edward Bear Does Not Have Time to Improve.

2007-05-02T08:49:00.000+02:00

In many companies the software assets and practices are left to rot. This indifference to fundamental quality issues is justified with phrases like "we are in a hurry", "we need to create new stuff", and other nonsense. Software is going to require maintenance and further development for its whole life. If you neglect to keep it fit it is going to haunt you back for sure. And it gets worse. If you leave your code base and practices to rot, you will eventually paralyse your whole productiveness. See Ken Schwaber explain it in this InfoQ video.

I have used the story about timber jack being so late at his logging site, that he does not have time to sharpen his saw. Some time ago I bumped into another quote that makes the same point.

“HERE is Edward Bear, coming downstairs now, bump, bump, bump, on the back of his head, behind Christopher Robin. It is, as far as he knows, the only way of coming downstairs, but sometimes he feels that there really is another way, if only he could stop bumping for a moment and think of it.”

It really gets painful sometimes.

It's less chaotic today than it was yesterday

2007-04-19T07:12:00.000+02:00

SD Times reviews the new, 2006 Chaos Report from the Standish Group. According the article:

35 percent of software projects started in 2006 can be categorized as
successful, meaning they were completed on time, on budget and met user
requirements.

I'm not sure if 35 percent is a good number, but it surely beats 16.2 percent in the 1994 report.

It would be nice to know how the size and complexity of average software project has developed during those 12 years. Maybe that is in the report, we just have to wait a bit more (and get 500$ for the report from somewhere).

The report lists three reasons for the improvement:

better project management,
iterative development, and
the emerging Web infrastructure.

I don't know if better project management in this context means agile project management. A quote “Managers have a better understanding of the dynamics of a project.” hints to that direction. Iterative development at least is something that we believe is essential for success, and it is nice to get more and more data to show that. Web infrastructure plays quite insignificant role in firmware development.

If you happen to know recent data about new product development success rate in general, or maybe even embedded software numbers, I would be most interested seeing them.

What does it mean to be agile?

2007-04-13T07:55:00.000+02:00

OK, the ESC2007 Conference is over. Many of the classes and especially Peer Roundtable discussions at the conference, Beer Roundtable discussions at the bars after conference hours, and long flight hours with a colleague, woke several questions, among them:

-What does it mean to be agile?
-When is an organization/team considered agile?

In his review (pdf) Dr. Pekka Abrahamsson showed the software life-cycle support of different agile methods. We know that Scrum does not have any engineering practices described in detail, but focuses on project management. That's why it has been adapted also outside software development. It is simple. This is why majority of the so called agile adaptations at least say they follow Scrum. As have we; we have used it with cross-disciplined embedded system development teams to develop a complete embedded system as fast paced iterative development. I have also written in my paper how I saw reflective iterative development as a tool itself for figuring out what practices you need. This said I share Simon Baker's post

Are you missing the point?

As long as you share the underlying values of agile development, and keep the obvious practices like short iteration, demonstrable progress and retrospectives, you are going to figure out the rest of it - ~~or die trying~~ or at least you figure out the need for something and can start looking. If your survival anxiety in your current environment is high, but you have juniority prevailing in your team, you can still gain a lot from practicing "just" Scrum. Scrum puts the basic structure in place with its simple rules and in my experience this is an immediate relief to chaos. This does not mean that it is easy. It will make everything visible; good and bad and needs a lot of discipline and nurturing.

With this approach, you being now out of the chaos and anarchy, it is time to bring in the engineering practices. Our case has shown that they are needed. We have experienced with eXtreme Programming (XP) practices; collocated team, pair programming (development), test driven development, coding standard, metaphor, and so on. We have seen the value in them, but not been able to actually master them in firmware or embedded system development in order to put the discipline into using them. Still we feel that we gain from practicing Scrum, and agile development as learning-driven thinking. This is the very reason why we still keep learning more skills and practices.

Are we agile? I do not know.

If your environment is functioning and open minded, and you have master mind developers in your team, why not go full blown XP right away? We have manifested that those practices are of high value, but they are hard for beginners. We now have also hard evidence to support the effectiveness of these methos (see for example the AGILE-ITEA project, run by the same Dr. Pekka Abrahamsson btw). It may be that with this approach you find out that you need some project management practices from Scrum. Who knows.

I wrote in my original paper about agile firmware development that lot of the agile programming practices for firmware development need innovative thinking. I think last week at the ESC2007 I met some of the people capable doing this. In the front line James Grenning from Object Mentor and Mike Karlesky and Greg Williams from Atomic Object. They have put these engineering practices into full use (also) in low end firmware projects. They are also willing to share their knowledge by publishing their tools as open source, and having discussions about the methods. Maybe we even become clients of these guys some day... Furthermore during the classes and discussions I got the feeling that a fair amount of people are actually practicing.

We have just scratched the surface, but after last week I believe that it is time to put some serious effort into agile programming practices, and actual firmware development. After all that is where we started this journey. It just kind a slipped into system development.

Are we agile after adapting these practices? I do not know.

The Vacation is Over - Officially

2007-03-26T07:24:00.000+02:00

It's Monday. Back at work after seven weeks of vacation. It actually feels almost as bad as it sounds. I did spend over six weeks in Thailand. Lots of sun, beer, good food, nice people, and of course technical diving. Technical diving is a nicely balancing hobby for an agile proponent, as it relies so much in up-front plan:

Plan the dive, dive the plan.

There are very few situations where this rule may be broken. It is the opposite to my beliefs in new product development. I'm a strong believer in:

Plans are useless, planning is everything.

This Friday I'm off to Embedded Systems Conference 2007. We're planning to get together with people from AgileEmbedded Yahoo group during the conference. If you you are planning to be in San Jose keep an eye for that. If you are not planning to be there, it's better get started.

ESC2007

2007-02-27T08:36:00.000+01:00

Embedded Systems Conference 2007 in San Jose, California, is having some interesting events. James Grenning is giving several sessions and people from Atomic Object are coming to tell about their low level system agile testing methods.

The best news is that right after my vacation (still have some three to four weeks to burn) I'm going to attend the conference with a colleague. Talking about nice return to work! I'll try to have energy to write something about the conf'.

Scrum Simulation Rocks

2007-02-12T07:27:00.000+01:00

Last week I gave an introduction to agile development and Scrum for a global project organization in Shanghai, China. After the lunch on first day it was time for 59min Scrum simulation. I have written about bad experiences when trying to achieve too much, so it was back to the basics.

There were participants from Finland, China, and Mexico. We used the product backlog for Family Cookbook. Only addition to original Scrum simulation was the use of relative complexity estimations on product backlog items. This time it worked. Estimating the tasks in numbers gave birth to discussion on whether something is simple, or not, not just dividing an item into tasks. Some items were also soon identified to be too large to fit in one 14 minutes Sprint. Team was immediately forced to seek for collaboration with Product Owner.

On retrospective the main thing experienced was the time pressure. I do not see this as a bad thing, it is just the beginning of understanding the discipline needed for time-boxed development. Time pressure will decrease when the team builds trust and improves estimating.

Everyone valued the exercise and though that it was a good lesson. I might consider adding a separate exercise for planning and estimating before the final Scrum Simulation. With this early experience I do not think that I will ever try meaningful work inside the simulation again. The difference was so clear.

TDD'ing State Machines

2007-02-02T22:39:00.000+01:00

I'm an electrical engineer. Everything in programming seems to be very very hard for me. Last week I spent 40 hours in aeroplanes, so I had some quality reading time. I did read Agile Estimating and Planning and Test-Driven Development from cover-to-cover. I had quickly gone through them earlier, but this was enlightening experience.

First of all, I have told that we have been trying to figure out effective TDD for our FW development. To be honest, we've struggled. We wrote tests for a state machine implementation. We expected the tests to be run in certain sequence, and that the state was always left where it should be after the test. We of course ended up in lot of rework on test code when our state machine design emerged. We thought that we need to design the state machine up-front in order to be able effectively write tests for it. This did not sound like test driven development at all. This required some more thorough thinking.

So back to my flight. Kent Beck writes that coupling is bad also between tests. A test should leave the system as it was. So writing unit tests in the same sequence as you expect your state machine to work does not seem like a right thing to do. You should be able to execute individual tests and to change the order of individual tests. So instead we are starting to recognize a single state as something to test.

Ph.D. Miro Samek writes about state machines being a killer app for function pointers. He further defines a design pattern for state chart implementation. Our state chart interface has become to be something like:

construct()
dispatch(event)

construct is self explaining and dispatch dispatches new event. We have specific event tick that gives resource time to state machine every 10ms (based on our 50Hz mains). We could also implement a timeout as OS service, and not to have local counters for timed events. Anyway, if we implement each state as a function as Mr. Samek proposes, and we hold our state as a function pointer we can write

dispatch( me, event )
{
me.myState(event);
}

We get rid of cryptic switch -structures. This solution helps us to unit test each state as a function with natural feel.

I'm going to write some benchmarking code to see the actual overhead, but at the moment I like this idea.

Driving Embedded Firmware C Project on CruiseControl

2007-01-28T15:09:00.000+01:00

I finally made it. A dedicated page for configuring CruiseControl for embedded firmware C project. It combines some of my earlier posts, but also tries to add detail. Take a look here, and tell me what is missing.

Book: The Tipping Point

2007-01-22T13:43:00.000+01:00

I finally read The Tipping Point, by Malcolm Gladwell. The Tipping Point tells us how little things can make a huge difference whether an idea will success or not. Will it tip or not. Examples and cases presented by Mr. Gladwell range from young men's high suicide rate in Micronesia, via fashion trends, cigarettes, children's tv shows, all the way to warning mechanism in surroundings of Boston about the upcoming attack of Brittish in 1775.

Malcolm explains, in a very understandable way, the law of the few; Mavens, Connectors and Salesmen. These are the three different personalities needed for a new idea to become epidemical. He talks about social epidemics. He then further explains the concept of stickiness factor, and the power of context.

What has this to do with things we are interested in? Graig Larman calls introduction of agile development methods a paradigm shift. When you introduce an idea of this magnitude, what you are doing, is asking people to unlearn much of their current knowledge. We of course know that the old knowledge is by no means useless, we just need to update it with some new knowledge. Never the less, introduction of agile methods and lean thinking is a novel idea in most organizations. The theories and concepts presented in The Tipping Point help us to understand the process of making this change and getting it to last. I have learned a long time ago, that it is not enough that you understand, you need to get others to get it as well.

Let's see what these theories in my opinion tell about agile intervention. In many cases, especially early in the decade, agile methods were introduced to organizations by younger champion developer, usually from position of no power. They just happened to be interested in learning and continuous improvement. They read a lot, they go to conferences, and they talk to fellow craftsmen. Furthermore they like to analyse the new things, and then tell these ideas to others. They are the Mavens. If things go nicely they will meet a Connector at some point. Connector knows everyone inside the organization, and knows who to talk to in order to get the change started. After the initial start, to make the idea of agile development really tip, we need extra ordinary people called Salesmen. They are the ones that can translate the ideas and experience reports so that we can communicate with lots of different people in order to get real structural changes to happen in an organization. Agile coaches and consultants are a good bet at this if you can not find someone in your organization fast enough. On top of that agile is a nice sounding word, and it can help to increase the Stickiness Factor. The first agile pilot project should have importance and wide enough commitment, creating the right Context for the intervention to tip.

Other literature (for example Pinchot, Intrapreneuring: Why You Don't Have to Leave Corporation to Become an Entrapreneuer, 1985) documents the same kind of process. Terms used elsewhere include 'Brain workers' who have the intellectual ability to think in novel ways and are able to argue their decisions. 'Intrapreneuers' are daring in using their vision of innovation regardless of for example political resistance. 'Gate keepers' have strong contacts to the organization's environment and have communicational competencies for making initiatives 'digestible' for the organization. Further a top management mentors, godfathers, or corporate angles are terms used for expressing the need for management commitment in such a venture.

Look at the world around you. It may seem like an immovable, implacable place. It is not. With the slightest push - in just the right place - it can be tipped.

Blogger is out of beta

2007-01-09T07:42:00.000+01:00

Blogger is out of beta and I thought I finally upgrade my template. I'm glad I did, managing the appearance of my blog is now a breeze.

I did some rework on links as well. If you are interested in agile development methods for embedded software and firmware you should join the discussion at Agile Embedded Yahoo Group. There are some brilliant people in the group, but the group could use some additional energy.

I seem to have few regular readers, so if this change messed up your reader - my sincere apologies. This was just something that needed to be done at some point.

Weekly Delivery of Firmware

2006-12-30T07:12:00.000+01:00

Yesterday we finished a six week project developing technology platform for User Interface innovation using some practices familiar from agile methods. The project delivered two different configurations of home automation user interface device; one for wall mounting and one for remote control. Some facts:

Vague or missing requirements
New 8-bit microcontroller
Lots of new sensor technology
New embedded firmware team of 3+1, including one consultant (in house)
Some drivers developed by outsourcing
Final project consisted of 20KLOC of firmware C source (bad meter, but...)

Some findings after initial reflection:

Weekly delivery (not release) of firmware is possible, at least in some cases.
Short term goals and daily Scrum meeting effectively work as motivators, and enforce full learning and getting things done done.
There is not small enough project to not use version control and continuous integration (we had no way to have common version control with outsourced team: large org, IT, you know the drill...)
HW evolved from microcontroller starter kit to final electronics via several bread board mocks and one halfway PCB prototype. This causes extra assembly work when looked at the surface, but this overhead is by far overcome by enabling the concurrent engineering instead of waiting for final version.

The last point holds only if all the disciplines work as One Team.

We did not have cross-functional team as we did in earlier larger experiment. Schematics, PCB layout and mechanics were developed concurrent, but they were not part of the team - nor did they follow any of so called agile practices. Firmware planning was tried to be synchronized with deliveries from other disciplines. In several occasions these deliveries were delayed, and a common goal needed for a team to actually be a team was somewhat missing. This is where I would like to do things differently if I was to do it all over again.

Occasionally we heard sentences like "If everything goes smoothly we will be ready on Tuesday, but we will be ready at least on Thursday". At this year's Agile Business Conference in London David Taylor (The Naked Leader) said "Don't have a plan B because you will always achieve it." Just guess if the delivery was on Tuesday or Thursday - or Friday?

The whole project was started by extremely vague idea. In few hours we worked out initial backlog and a delivery (release) plan based on simple themes. From that point on plan, design and requirements emerged during the 6 weeks nicely synchronized with weekly demonstration of prototype and new planning session. This experiment supports the belief that good people can work this way, and that we truly live the era of enlightened experimentation. We believe that we could not have performed any better with thorough analysis, design, and specification phases - at least not to cover the cost of them. During the development there were several points where it was not possible to proceed as we had thought. A comprehensive plan based on these initial assumptions would have failed miserably, and the time creating them would have been wasted. In this project the whole team worked out an alternative solution, adjusted the next week's plan, and the project was fine...

I will write an experience report and hopefully make it available here.

Skunk Work Is Not A Longterm Solution

2006-12-21T16:47:00.000+01:00

In her "The Agile/Waterfall Cooperative (pdf)" presentation Michele Sliger summaries three modes in which agile team can work in a waterfall organization; SWAT team, Skunk Work, and stealth agile. I agree that applying just some agile practices can benefit your work. This is pretty much what we are doing at the moment, but I do not see this as sufficient long term solution - if it is only in team's sphere or just above. All three forms of team work can of course be used. SWAT teams can go parallel even in more wide spread agile organization. Skunk work probably, and hopefully, always exists in large organization - it's fun. If you are working as stealth agile, and avoiding radars by not communicating to the top management, you are not going to be able to fully harness the power. You need to inform top management that you are doing things differently and communicate your need for changes. They probably will not get it, but what they do get - is success. So it is fine and in most cases necessary to fly low at first, but after certain point it is not satisfying anymore to work as a shadow system, because you know you are constrained by the legitimacy structures, and these structures are holding back you and your learning.

The rest of the presentation has also good ideas, and based on my experience these ideas hold in reality as well.

SWAT team – outside the domain of the process police, brought in on projects that are in trouble. Agility is prized in this situation. They take over however – they don’t work in a cooperative mode.
Skunk Works - “a small group of experts who drop out of the mainstream company operations in order to develop some experimental technology or new application in secrecy or at speed, unhampered by bureaucracy or the strict application of regulations.” Sanctioned and protected by management, these teams are rare. This formation however, prevents the headaches involved in org structure, project approval hoops, portfolio metrics and management, budget calls, and other bureaucratic nightmares.
Stealth Agile – As Jim Highsmith once said in response to a question about how to sell agile to upper management – don’t. “They don’t know what you’re doing anyway.” Do as much agile as you can where you can. Some benefit is better than none at all.

Link: Scrum and XP from the Trenches

2006-12-18T10:45:00.000+01:00

No matter if you are developer, PM, or CIO and you don't know what agile/Scrum/XP is, you are thinking about applying, or you are already practicing - you should check out "Scrum and XP from the Trenches (pdf)" report from Henrik Kniberg. Henrik describes how they applied Scrum and XP practices in real life and presents the adaptation needed in this situation. Excellent Writing!

main_theOneAfterFinal.c

2006-12-14T16:14:00.000+01:00

Today I happened to see a file name with word 'new' on it on a presenter's computer. I was slightly amused because of my history. I can still see the old version naming convention in many "shared team folders" in our server. It goes something like this:

original
new
latest
final

I have been using version control system since my first university years (some fifteen years ago) and throughout my professional life. First it was just an advanced XCOPY method putting released versions of one man projects into the system, but about three years it has been collective code ownership and at least daily update-commit (we use CVS) cycles from the beginning of development. Lately we have adapted continuous integration and automated regression (confirmation) testing as you know. Currently we are looking for ways to add some automated acceptance and system testing as well.

During the past couple of years I have had a pleasure to work with software developed outside our own company. Quality of source varies a lot of course, but it is striking that version control typically is not even mentioned. In some occasions lately the software was delivered as zip'd folder with a version number in folder name - at that point I was happily surprised by that! Embedded firmware domain - based on my biased observations - walks some 10 years behind the mainstream software industry. The wide lack of basic discipline, version control, proves the point. I'm glad that I work in a department where I can say that developers would never go back working without version control.

I found this Yahoo group message - nostalgic.

The Lenghty Debugging Phase

2006-12-09T07:19:00.000+01:00

To give embedded firmware TDD a kick start we have had two sessions with a colleague. During these sessions we have created scripts to automate the unit test structure and build process and discussed how the make as thin wrappers for HW as possible. However we learned also other valuable lessions during these sessions. Lesson number one is the "debugging phase" being so strongly in our domain culture. We often hear a firmware developer saying, "well, it takes couple of days to implement it, but you can never tell how long the debugging is going to take". Of course debugging phase is where all the firmware heros truly shine. That's another unfortunate culture issue in firmware development.

In our first session we ran into problem of not getting one test running. Immediately we started debugging. Printf's fly in here and there (as an advanced host run I/O twist). The change to this mode was so rapid, that neither of us realized what is going on until later. After some 30 minutes(!) we recognized that we are not moving, but just desperarately experimenting with printf's, and it hit us; we have not changed our behavior as we were supposed to. We should have picked a more simple thing to implement. When we realized we can not chew what we had chosen, we should have started all over again with simpler thing.

It is our belief that by doing TDD for firmware we should get rid of, or at least dramatically reduce, the debugging phase and thus create an atmosphere of success, and improve the reliability of estimates.

This type of sessions are great tools for learning. I strongly share Michael Harmer's opinion on The Training Course Scam.

Ken Schwaber on Quality of Code

2006-11-30T19:34:00.000+01:00

Ken's presentation "A Canary in a Coal Mine" at Agile2006 conference is available via InfoQ. Ken talks about quality and the danger of Scrum teams droping quality issues to get higher velocity and to be sure to hit the date. He calls this a re-definition of "done", or "somewhat done". We have manifested an increasing technical debt in our iterative work. So in the last project we were putting more focus to acceptance testing inside each sprint and introduced something we called "cross-testing". This means that other pair needed to test the feature implemented by other pair in order to claim feature "done". Embedded firmware systems need a lot of manual acceptance testing (at least our current practice) and it is easy to get slobby testing your own work over and over again.

Was the technical debt there because of Scrum? Well, no - it was made visible by Scrum and the team was able to decide to do something about it early enough.

----

Ken's another speech about Scrum is available at Google Video. (You will find out that Scrum works with idiots as well)

I posted about Jeff Sutherland's Scrum speech earlier. It is also available via InfoQ.

XP2007 Conference Web Page

2006-11-26T15:28:00.000+01:00

XP2007 web page is open. The conference will be held on June 18-22.2007 in Lake Como, Italy.

In the scope of conference is:

Embedded software (e.g., SW/HW co-design) and Agile SW development

...among of course other interesting stuff. Hopefully we see some papers in this domain as well. I attended the conference this year in Oulu, Finland, and if next year's event is anything like that - I can highly recommend attending.

NPD in General With Agile and DSDM

2006-11-21T11:48:00.000+01:00

Iterative and incremental development ideas have been presented for general new product development (NPD) for years. I'm interested in agile methods (as in software development context) because they give a philosophy behind and lots of ideas how to actually do this. It is of course also because of extremely active and powerfull community. Ideas of using these methods and practices in NPD in general have been presented by Jim Highsmith (Agile Project Management), Preston G. Smith (Developing Products in Half the Time), and many more. Both of above mentioned can be contacted via Cutter Consortium.

Now there is something new coming up. At London Agile Business Conference 2006 few weeks back the DSDM consortium talked about the future of the DSDM framework. DSDM version 4.2 has been open for online viewing since this year. They are going to publish a new version of framework in Spring 2007. This new version is supposed to be more general and not specifically targeted to just software development. It was also presented that DSDM can be seen as a wrapper for models like Scrum and XP for use in enterprise environment. It's a bit too structured to my tasting, but very well worth to check out when available. It may also become more open.

Risky Business

2006-11-17T06:54:00.000+01:00

It has been identified that the biggest risks in project work concern management and marketing. Yet, in most environments we only concentrate on technology risks. Agile, incremental and iterative, development is also often assumed to be risky. By changing the way of thinking about risk, people would see that iterative risk management is a proactive way to tackle all aspects of risk. A very simple way.

1. Management risks - just fund the project for the time zone you are comfortable

2. Marketing risks - use the first possible release to test the waters

3. Technology risks - use the effort early to tackle any technological uncertainties that traditionally are left lurking until the (assumed) end of the project

The status quo practice is quite different. We take huge risks by starting a megaproject, for example estimated to 5 calendar years, based on a business case. Everyone knows that business case targeted five years from now is pure speculation. In the global markets it's a very expensive speculation by the way. Based on this speculation management needs to make a decision for funding a five year project. Their natural reaction to tackle the risk is to review the project mercilessly. We can be sure that changes to project requirements, technology and scope will happen. We have this heavy review practice going on making the project move even more slowly. More time means more changes, more changes mean more reviews - are you starting to get the picture?

Most of us see this everyday. How do we try to fix it? Naturally by reviewing. Because? That's what we allways do.

Refactoring for C

2006-11-10T15:57:00.000+01:00

SlickEdit is a commercial ~~compiler~~ source editor that many people speak highly about. I thought I will give it a spin and downloaded the trial version. Their web page describes "cool features", but for some reason refactoring is not listed. Anyway the product has capabilities for refactoring several languages, and few quick refactorings (based on tagging instead of parsing) even for C; extract method, rename, modify parameter list, and replace literal with constant. At least for simple code they all seem to work. SlickEdit is also available as Eclipse plugin. I only tested the own IDE version.

You can find out the reason why there are not so many refactoring tools for C from MS and doctorate thesis by Alejandro Garrido. They are available via this site.

Web Page for Embedded Unit

2006-11-04T17:09:00.000+01:00

Embedded Unit (embUnit) is an unit test framework for embedded C systems. It now has a simple web page with manual. It is not pretty, but it is something.

One Team, but just a team

2006-10-29T07:01:00.000+01:00

One of the agile development practices is One Team, meaning that team as a whole takes the responsibility. When agile team is formed by a bottom-up initiative, it will soon face the problems of limited sphere. Most of the problems in projects are not related to design, technology or engineering practices. These are the only ones that the team is capable to fix quickly through iterations and reflective retrospectives. After fixing its own sphere, the team hits the wall when working in the shadow of a large organization. This can be a stressfull situation as everything is starting to be crystal clear.

Afterall One Team is just a team.

Last monday at Agile Seminar Helsinki Heimo Laukkanen presented that the whole organization needs to be agile in order to fully harness the power. He talked about IT governance, but even faster you will notice need for change in leadership styles, design review practices, personnel development, project planning (not plan), marketing involvement, and a lot more. Even while the agile development only in technical level may be rewarding, seeing clearly the dysfunction of standard organizational processes (so called best practices) and structures will hammer down the gained team spirit and motivation, eventually forcing it back to old ways of working.

Changing these processes and structures in a large organization may be impossible with bottom up approach. Large organization has a whole process department to take care of the process. It would take a complete paradigm shift to access this process. What makes this hard is that this department is not rewarded based on project success or revenue, but the success of current paradigm - from the administrative perspective. Admitting that a paradigm shift is actually needed might feel like a failure of the department.

This is one of the reasons why agile interventions often fail, the rest of the organization fails to learn fast enough.

Come Again, What's the Cost and Overhead?

2006-10-27T07:51:00.000+02:00

I've been implementing object-based stuff in C for embedded firmware with passing an index parameter to an array of structs, like I tried to explain here. Yesterday I remembered an article about pointer-less C in Embedded System Design and thought I will give it a couple minutes more. The article gives some ideas how to make pointers safer to use and talks about the cost and benefit of using pointers. I'm not saying anything about the value of article itself, but I got curious. I took our simple switch debounce filter routine which is used a lot, and which is typically the only one that actually implements multiple objects at run time. It uses the me index as parameter for two functions. I quickly modified the code so that both functions take in a pointer to a struct instead of simple unsigned char. I compiled both versions for Microchip 16F876A (8-bit uC) and got surprised. The pointer version used over 30% less code memory than the index version, reducing the ROM image from above 300 words to below 200 words. I then compiled the source files to give me the assy listing, and below are the results that I found for a simple function that just sets some members of a struct. Because of werd mechanism of indirect addressing in PIC micros, the pointer is actually passed just as a 8-bit register. Word of caution; we have experienced problems when passing pointers and enabling optimization with Hi-Tech compilar for PIC micros, that is another reason why we have avoided this.

The lesson; while I do believe in intuition for great solutions, it is worth to challenge your beliefs every once in a while.

Pointer version:



    30  0764             _Cbutton_construct
    31                   ;button_ptr.c: 27: me->debounceCounter = 0;
    32                          
    33  0764  0084       movwf fsr
    34                            
    35  0765  1383       bcf 3,7
    36  0766  0180       clrf 0
    37                   ;button_ptr.c: 28: me->level = !1;
    38                          
    39                   ; _me assigned to ?a_Cbutton_construct+0
    40  0000             _Cbutton_construct$me set 
                         ?a_Cbutton_construct+0
    41                   ;_me stored from w
    42  0767  1283       bcf status,5
    43  0768  1303       bcf status,6 ;carry unused
    44  0769  00A8       movwf (((?a_Cbutton_construct+0)))
    45  076A  0A28       incf (((?a_Cbutton_construct+0))),w
    46  076B  0084       movwf fsr
    47                            
    48  076C  1383       bcf 3,7
    49  076D  0180       clrf 0

Index version:



    30  06F6             _Cbutton_construct
    31                   ;button.C: 27: theButton[meIndex].
                         debounceCounter = 0;
    32                          
    33                   global _theButton
    34                   ; _meIndex assigned to ?a_Cbutton_construct+0
    35  0000             _Cbutton_construct$meIndex set ?a_
                         Cbutton_construct+0
    36                   ;_meIndex stored from w
    37  06F6  1283       bcf status,5
    38  06F7  1303       bcf status,6 ;carry unused
    39  06F8  00A8       movwf (((?a_Cbutton_construct+0)))
    40  06F9  0828       movf (((?a_Cbutton_construct+0))),w
    41  06FA  00F0       movwf btemp
    42  06FB  1003       bcf status,0
    43  06FC  0DF0       rlf btemp
    44  06FD  1003       bcf status,0
    45  06FE  0D70       rlf btemp,w
    46  06FF  3E20       addlw ((_theButton))
    47  0700  0084       movwf fsr
    48                            
    49  0701  1383       bcf 3,7
    50  0702  0180       clrf 0
    51                   ;button.C: 28: theButton[meIndex].level = !1;
    52                          
    53  0703  0828       movf (((?a_Cbutton_construct+0))),w
    54  0704  00F0       movwf btemp
    55  0705  1003       bcf status,0
    56  0706  0DF0       rlf btemp
    57  0707  1003       bcf status,0
    58  0708  0D70       rlf btemp,w
    59  0709  00F0       movwf btemp
    60  070A  0A70       incf btemp,w
    61  070B  3E20       addlw ((_theButton))
    62  070C  0084       movwf fsr
    63                            
    64  070D  1383       bcf 3,7
    65  070E  0180       clrf 0

Code smells - just refactor

2006-10-23T15:23:00.000+02:00

As we identified, agile development puts more demand to top quality source code than we (firmware developers) are used to. Actually agile development only makes the bad quality visible to larger crowds. This fact fairly quickly takes us searching for new practices like refactoring. You can find about it in Refactoring site, or better yet from Martin Fowler's book.

There is a nice table by Gene Garcia available as a summary of code smells and refactorings. Take a look, this stuff applies to firmware as well.

We Can See Clearly Now

2006-10-20T07:26:00.000+02:00

The beauty of Scrum, or agile development in general, is that it gets everything laid down in front of everyone. Everything, everyone, good and bad.

1. Developers learn to understand what 'done' really means, and how much effort it takes to get things done. It demands discipline not to have technical, or administrative in large organization, debt increase increment after increment, but actually do all-at-once, in parallel. At the same time the satisfaction and enjoyment from accomplishing meaninfull work is immediately rewarding.

2. Stakeholders outside the team start to understand the unpredictable nature of new product development, but also the true capablities of reliable development. Customers understand that they do not need to ask for everything in order to get at least some. Common understanding of business value (benefit) starts to emerge between the customer and developers. Managers and project managers see the effect of meaningless deliverables and context switching. Managers start to understand just to stay out of the way and remove obstacles.

Trust begins to take form.

Agile Seminar Helsinki, Fall 2006

2006-10-17T16:05:00.000+02:00

Title says it all. Check it out.

Jeff Sutherland on the Roots of Scrum

2006-10-13T17:41:00.000+02:00

A presentation about the roots of Scrum is available via infoQ. A 60-minute presentation held at JAOO conference about everything and some behind the Scrum as we know it today from Dr. Jeff Sutherland, the original creator of the framework. Without the need to travel. Great.

Bubble, Bubble...

2006-10-06T06:42:00.000+02:00

We have had our continuous integration server running for a while. At this point we thought that some not so serious step should be taken. It was time to implement the lava lamps.

Being a firmware shop we obviously could not go the normal way with X10 publisher. We used our own shutter relay prototype to drive the two lamps. The relay module is normally driven with I2C -bus, but we simply used the data and clock signals to transfer pulses from pc's serial bus. The I2C bus is running at +3VDC level, so we needed to shift the 12v levels found on serial port to that level. Don't worry, the +3VDC level is isolated, so 'it should be fairly safe'. If you want help with similar hw setup, feel free to contact us.

All we did from that point was a couple of simple python scripts to drive a short pulse on DTR or RTS according the build result. This was easy with a little help from pySerial module. Shutter relay connected the lamps according the last detected pulse. These scripts are called via <execute>-publisher in CruiseControl's config.xml.


<onfailure>
  <execute command="python pulse_rts.py">
  </execute>
</onfailure>

<onsuccess>
  <execute command="python pulse_dtr.py">
  </execute>
</onsuccess>

Next steps:

Write an actual publisher for DTR/RTS signals in Java
Use our rf comm devices to transfer the build status
Hook another lamp to indicate the ongoing build process
Write a LonWorks publisher similar to X10

Mark Michaelis described their similar effort. They are a bit ahead of us.

Immediate Payoff

2006-10-01T06:15:00.000+02:00

Ken Hughes shared his story about their first three months with Scrum development. Findings are all too familiar. There is also a nice example of excel sheet working as backlog.

Certified Scrum Master

2006-09-29T06:54:00.000+02:00

I had been thinking about taking the 2-day Certified Scrum Master (CSM) training for couple of years. Last week I finally got to the course with a colleague. My concern was that since I mainly develop firmware, I would not benefit that much from the course. I could not have been more wrong. I highly recommend CSM course to anyone involved with project work. Even if you do not believe in agile development, this course can still give you pointers on where to improve your work.

We got our training from Boris Oestergard and Jens Gloger. They have been giving in-house training in finland earlier, at least at Nokia Networks, Sulake and Accenture. We had also Bas Vadde from Nokia Networks giving an interesting guest speak about scaling Scrum to larger programs of hundreds of developers.

You can see our group photo at www.scrum.dk (photos).

Just Nail That One Thing

2006-09-20T20:44:00.000+02:00

I have become to value that the team should be imposed to only one new thing at the time. For example bringing in team room, pair development or TDD should not happen in a huge bang but a transformation time is needed. However when I had a chance to run an agile day with senior management I forgot this important lesson. I got over ambitios and tried to achieve too many things at the time.

1. I started with presentation covering agile development philosophy, results from our experiment and also the needs for more advanced future adaptation. All this in 1.5 hours with active discussion. Try to keep your presentations focused.

2. We then applied Open Space Technology and went through three sessions. This went fairly well considering we were only 6 people and all new to this way of working. Some difficulties in coming to a conclusion, but time-boxing still helped to cover the issues.

3. At the end I tried to run a 58 minute agile simulation. I combined XP Game and Scrum Simulation and run a preparation phase with vision creation, then two sprints 10min each and finaly a closure phase. This is OK and could be done again with some slight improvements, but where I made a huge mistake, was that I wanted to wrap up the results from Open Space within simulation. Time pressure from the simulation, new practices coming at fast pace, and meaningfull work at the same time. Not likely.

So my lesson; if you have some very important issue at your hand, do not try to be too clever. Just focus on nailing the one task with the highest priority/value and make sure it gets DONE (aka implemented, tested and documented). If you chose the wrong issue, well buhuu, that's tough.

On the bright side it at least wasn't anything like Dave Nicolette described happening during Jon Kern's presentation. So maybe it was ok from the learning point of view, and at least the journey still continues.

Head up. Reflect. Repeat.

We Are not Alone

2006-09-16T13:35:00.000+02:00

This post by Simon Baker really, really, made my day. It's good to know that the situation I'm currently in is by far not unique.

Can our agility bring about a wider culture change, at least in the program office? Maybe. It depends on the willingness of people to move beyond the comfort zone provided by a command and control environment.

Next week is going to been an interesting one.

Not So Naive Change Model

2006-09-14T07:41:00.000+02:00

Agile methodology is not visible. I do not know what my team is doing. These are complaints often heard from project manager. Ops, I of course mean persons that understand that they are project Managers. Managing is translated into command and control. When saying "not visible" they mean "I can not command", "I can not control", or "I do not understand prototypes, show me paper".

What is described above is the impact of change, and it follows the satir change model. After a smooth start of agile pilot enabling more effective team work, all new kinds of problems start to emerge. Team learns about the actual effort needed in disciplined incremental development versus hacking, but also people who have evaluated their status by how much power they have to command and control (project Managers) may fear loosing their status. All of a sudden product gets developed without screaming, hair pulling and jumping up and down like a freaked out kangaroo. There is no room for the blame game, what is going on here? What is my role in this? A paradigm shift from command and control into trusting and facilitating causes fear among this dying specie of command and control type knowledge-work managers.

One needs to be very carefull not to interpret this new state of chaos as failure in agile intervention. This is excatly what was expected, but fear is a powerfull emotion and when in panic we do irrational things. I on my part fear what happens to a project Manager who in panic starts to act even more irrationaly (compared to my understanding of rational). As we remember from the theory of complex adaptive systems this emerging behavior is of course totally unpredictable...

...fingers crossed everyone, stop-think-act, maybe she gets agile.

Taming the Big Animal - Agile Intervention in a Large Organization

2006-09-11T18:11:00.000+02:00

I made an academic paper about our first larger scale agile experiment available. You can find it via this link (pdf), and it can be found also from the sidebar. Also see the analysis that started this experiment.

Abstract— New Product Development (NPD) today is mainly
managed with plan-driven, relay race type, processes. Today’s
dynamic markets and new technology used in NPD result in a
possibility of benefiting from more adaptive, rugby style,
methods. In the software community methods applying this
approach are commonly called agile methods. In this study I
present experiences from a bottom-up initiative to apply agile
methods for embedded system development in a large
organization using the Stage-Gate® project management
framework. The purpose of the study was to find out whether
agile methods have a chance and how much this type of
organization could benefit from agile methods. Study supported
the earlier research showing that implementing incrementally
some agile practices the project organization was able to improve
the team spirit, motivation level, and additionally to increase
productivity. The affect of agile practices to the Stage-Gate®
framework was minimal. Through wider buy-in and
commitment, the agile development increments could be
synchronized seamlessly to the Stage-Gate® milestones. However
as described earlier in literature, top-down commitment and
changes in organization culture are needed in order to succeed in
process change. The philosophy of agile development is quite
contrary to so called traditional NPD approaches. These changes
in organization culture are also needed in order to fully harness
the power of agile development.

Can RUP Be Like Switzerland?

2006-09-09T16:46:00.000+02:00

At one point debate on agile and more waterfall like sequential process models was claimed to be like religious war. This impression might still be seen in an large organization when agile methods are piloted for the first time. What if the Rational Unified Process (RUP) is the legitimate model for software development in this organization? RUP, being (in my and some others opinion) extremely abstract framework, could easily adapt to pretty much any methodology.

I do not hold a guru status on RUP. I have read the introduction by Kruchten, browsed the net, some papers and that's about it. Yet I propose that RUP could be seen as neutral territory for getting the advocants from both camps to work on common framework in order to let all the flowers bloom.

I know that if I was running a small start-up, and obviously using agile methods full a head, I would not concern my self with RUP because it would not offer me enough. However in an large global organization some level of legitimate system and standardization can be seen necessary, and thus I do not see RUP as bad option. Especially if RUP is already in the house.

RUP has been adopted to agile methods and vice versa starting from the early days, but some major effort has been put to this just lately. Check out:

1. Agile Unified Process (AUP) by Scott Ambler
2. Eclipse Process Framework (EPF) and OpenUP by Eclipse Community

...About Team Rooms and Pairing

2006-09-02T11:56:00.000+02:00

Mishkin Berteig pointed to this web page about setting up a team room. This page pretty much covers everything I have in mind about the subject, especially with the links it has. Pros and cons of the team room are handled. These guys also share my philosophy about forcing these things. If you should force people to do anything, team room and pairing are definately NOT among such things. I believe these issues are automatically brought in when (if) the team jells and learns to work together.

Our Version of Primetime Emmy Award

2006-09-01T18:30:00.000+02:00

It's been awfully quiet around here. It's just because I've been ~~lazy~~ busy.

During the previous Sprint we experimented a practice we called Primetime. Primetime means that immediately after the daily Scrum work was continued in the team room and by pairing for couple of hours. Of course in embedded system development pairing means pair development, not just pair programming.
Yes, yes, we all know that this should be a full day practice. But ... there is lots of other emergent responsibilities, for example maintenance and things like that. At the moment Primetime is the best we can achieve in our environment.

Tom DeMarco highlights the importance of uninterrupted work in his book. The cost of reimmersion time to flow (state of mind when developer "gets her design") is extremely high, and our experiment manifested this.

In our retrospective we easily identified five observations:

People got enthusiast about the design when pairing, and continued way beyond the agreed Primetime
Two people pairing blocks other people from interrupting
Emails, phone calls etc. got postponed
Learning was accelerated because of osmotic communication, and active collaboration
Defects got detected as in review process, thus improving the quality

Within this experiment the team easily achieved Sprint goals, which were set based on earlier velocity. It is too early to make a conclusion, but we decided to keep this practice.

Plans, Planning and Speculation - There's Time for Everything

2006-08-21T18:36:00.000+02:00

It has been funny to me from the very beginning that agile methods are often claimed to lack the planning activity while the opposite holds. The difference lies in the fact that planning for the unknown becomes speculation. Prolonged speculation period brings very little added value and is thus kept to minimum in agile projects until better knowledge is achieved by active learning. Simon Baker in AGILE IN ACTION reminded about the two great early agilists believing in planning instead of plans.

Life in a Cubicle

2006-08-21T07:18:00.000+02:00

Are you working in a sorry ass company that forces people to work in cubicles in order to foster communication and effectivity? It's sad, but turn on your speakers, pump up the volume, and check out this James Blundt remix.

Hope it helps.

Portfolio Through The Grapewine

2006-08-19T05:57:00.000+02:00

Many agile methods miss to address the issue of portfolio management. This is not such a surprise as many organizations miss to address the issue all together. Agile thinking pretty much assumes that the only intelligent way is to work on a single project at the time, and that projects should face delivery or cancellation as soon as possible. After all having too many projects in the inventory is one of the main sources of waste. While all this is very much true, the life on planet earth is sometimes somewhat different. The impression "I heard it through the grapewine" is made famous by Marvin Gaye. The impression means information traveling through gossip and rumor. In some organizations this seems to be the primary channel for portfolio management.

Team is defined as group of people with shared objectives and goals (term "team" is misused a lot!). Teams become superior over individuals when they share these goals and thus have a common direction. Agile methods take advantage of this phenomenon by planning for short enough increments and gaining commitment by having estimates given by the people doing the actual work. This is called time-boxing.

When a team in large organization is piloting agile methods it is usually working as a shadow system parallel to institutional system in the organization. In this case the above mentioned "I heard it through the grapewine" -effect can be very harmful. Each team member will have several inputs about the future of the current project, as well as information about the future projects. Along the current increment short notice invitations to future project meetings will arrive and so on. This makes agile planning unmotivating and even barking mad. Of course the same applies to any method of planning under these conditions. However time-boxing would be perfect tool against "I heard it through the grapewine" -effect, allowing organizations to harness the power of small teams with shared goals and direction. DeMarco calls these teams Jelled (DeMarco and Lister, 1999). If new announcements and meeting invitations are only made public in the same natural rhythm as the team already operates, their negative effect to Jell would be minimized. Who is responsible for this rhythm? Top management, middle management, project management, line management, team leadership? I think this needs to be considered independently for each case, but it is a joint responsibility and everyone should understand the reasoning.

"Increment duration depends on how long you can keep the change away". Allowing rumors and gossip make this a very short period.

The Uneducated Industry

2006-08-13T19:52:00.000+02:00

In many cases professional embedded developer has never attended a single programming course at any level. Further she may have never opened a book on even the most fundamental programming practices. She does not read professional magazines. Embedded programmers are often electronics graduates. Even more scary is that the same applies to managers of embedded developers. They have never attended a single programming course at any level, NOR any managing course at any level. Embedded development department managers have just become managers because they have earlier been successful in programming (and designing the HW) with those 1kB micros solo virtuoso and ad-hoc.

Imagine that. In the past years we have reached an era where a light switch has more bit crunching power than Commodore 64. In few years a key chain, a pen or your T-shirt may out power the old 64. The complexity of these devices is exploded by the communication capabilities which enables distributed intelligence. These increasingly complex systems need to be developed within ever shorter time-to-market by team(s) of embedded software developers co-designing the system with teams from other disciplines. Members of such teams need to have proper skills for development, but even more challenging is that developers and their managers need to have social skills in order to keep the whole development ecosystem fit.

These skills are now been teached in programming and management departments, books and articles, and then tuned and learned within methodologies like agile development. However getting these next generation people into industry is a slow process.

It is essential to teach the existing people in the industry to learn.

Norm Day at the Office

2006-08-04T14:46:00.000+02:00

Back at work, and running a orientation week before we begin a 3 month release plan, which aims at delivering the first possible release candidate for an embedded product family. We chose a meeting cabinet that we found out to be available.

Fist we checked it up. It seemed to be ok.

We could not find a projector in the premises, so we had to settle for Meeting Charts provided by our friends at 3M. Good news; no .PPT's.

Survey Shows: Agile Champions Live Higher

2006-08-03T20:55:00.000+02:00

There has been survey results published at Agile 2006 conference. Pete Behrens has written a nice summary. The survey was about agile adaptation and was run by VersionOne. As VersionOne makes their business in agile scope, the data sample may be "a bit" biased. Even while I do not put much emphasis on the result that 75% of companies deploy agile methods I do see value in other finding:

"The internal agile champion is moving up in the organization to VP from team leader two years ago."

Agile methods have so far been largely deployed by a younger champion engineer in an organization that has gotten frustrated with the dysfunctioning methods. This data may show that agile development is moving directly higher from the trenches. I just hope that the values do not get twisted on their way up. The original ones work so nicely....

Agile Methods and Firmware Development

2006-08-01T20:01:00.000+02:00

I made my paper "Agile Methods and Firmware Development" available. You can find a link to the paper from the sidebar, under "My Papers". This is my initial review and analysis of different software lifecycle models and their applicability to firmware development as part of Ph.D. studies. Paper is dated a few years back. One might say that the current journey started upon writing this paper.

Abstract— The size and complexity of software continues to grow
at a steady pace. This is also true for software embedded in our
everyday electronics, which we have called simple devices. The
term firmware is used to describe the low-level software in
embedded systems. It may even be hard to divide firmware and
actual hardware. Software development for such a target has
special characteristics such as a culture of hacking, small teams
and multiple hats, co-design issues, one-time designs, correctness
and robustness requirements, lack of tools and unconventional
customers. Software process models have been studied also in
this environment to ease the pain of developing more complex
systems. I introduce four currently used methods to develop
firmware; build-and-fix, waterfall, ROPES and RUP SE.

Agile methods are getting a lot of attention in the software
development community at the moment. I review the agile
methods which are most documented. The suitability of these to
firmware development is evaluated. It is also analyzed whether
firmware development could benefit from agile methods.

It is shown that agile methods are not the new cure-all solution
to firmware development, but they are applicable. Their full use
needs modification and innovative thinking. It is, however, shown
that firmware development can surely benefit from the usage of
agile methods.

embUnit Translators for CruiseControl

2006-07-26T17:20:00.000+02:00

Back home and time to get tech oriented again. Our CruiseControl and unit test practices are proceeding at steady pace, so I thought I will make our XSL translators for embUnit unit test framework public in this forum at this point. You can find them here (unittests.xsl and testdetails.xsl). If you are an embedded programmer like me, and do not currently know much about CruiseControl, and/or XML, I'll try to briefly explain how to set it up. If there is enough interest I will make a more detailed intro to it.

You need at least to have following installed:

embUnit unit test framework and compiled libraries
Java SDK. Remember to set JAVA_HOME environment variable to point to install dir.
CruiseControl, of course
Make tools. I recommend for Windows users the whole Cygwin installation to cut down the hassle

Now you can run the Cruisecontrol Ant built Java and JUnit sample, but before you can start cruising with your embedded C project with Make and embUnit framework you need to at least do following modifications:

Build a simple C project which has makefile for automated build. Greate a CVS project and check your project out to your build server for CruiseControl (use Subversion if you are ahead of us, but you are on your own with this). Look at Driving on CruiseControl articles by Lasse Koskela for help.
Modify our config.xml to suite your project (see my earlier post about it) and replace the file in CruiseControl root
Replace unittests.xsl and testdetails.xsl files in CruiseControl folder ./webapps/cruisecontrol/xsl.
Launch CruiseControl and start cruising...

And here are some screenshots on how it looks.

Here a test fails (purely for demonstration purposes of course). Details of failed test are displayed below.

Details of all tests are also available.

After a brief moment everything is in order again. I used the green bar to signal this.

Battery Charging and Reflection

2006-07-23T15:10:00.000+02:00

Yesterday evening we arrived to Lappeenranta and lake Saimaa. I have to say it is a beautiful scenario for battery charging. Even while the city was offering a festival (in finnish), I decided to take a day off from beer drinking and - well - read about change.

My book of choise, Fearless Change, by Ph.D's Linda Rising and Mary Lynn Manns, is of great value for anyone trying to get innovation diffused in organization. The book describes 48 patterns for driving and sustaining change. Many of the patterns we have found out during our journey. Among them a pattern "Time for Reflection" in which doctors recommend:

To learn from the past, take time at regular intervals to evaluate what is working well and what should be done differently.

This is excatly what we need to do again after the vacation.

Small Countries Seem to be Aggressively Agile

2006-07-15T08:02:00.000+02:00

Google trends has been used to map agile development interest in different countries. Jonathan Cogley started it with a search for 'How Agile is Your Country'. Dave Nicolette went on with revisited search term set, and got this google trend. By choosing the regions you find (at least I did at the time) Finland in 10th place. So it is no wonder agile conferences in Finland are packed.

Prisoners of Our Habits - Act Four

2006-07-14T11:40:00.000+02:00

Prisoners of Our Habits. Act three. Executive manager enters.

Executive manager is reading an email from project manager
"Dear Sir, I overheard this developer, John Smith, saying
his technical solution may cause $1.00 increase
in bill-of-material."

EXEC'MNG: -"Stop the press. This John Smith starts
immediately to investigate alternative solutions,
what ever they are."

Product under development may have hundreds of relationships. Tradeoffs between these relationships are being made in hourly basis. When one of these issues gets taken away from the context, it may seem like a big deal. The big deal in fact is when a bad decision (based on bad information) like one above is made the project environment, a complex adaptive system (CAS), gets an unexpected input. What happens in CAS is totally unpredictable. Typically the system is pushed into chaos and disorder. This is why agile development blocks these inputs from the team for a certain time, and opens the window for these new inputs only frequently enough. Command and control is traditional way of managing projects. Developers have not been encouraged to self-organize traditionally.

And old habits are slow to die.

I know paradigm shift is extremely slow, and so does James Shore (see his Change Diary). I'll be on vacation for couple of weeks and after that it would be perfect time to reflect and adjust the course of this agile journey. I just hope couple of weeks is enough for the old batteries.

Prisoners of Our Habits - Act Three

2006-07-12T11:22:00.000+02:00

Prisoners of Our Habits. Act three. A manager enters.


DEV -"We should try to learn new ways of doing thingies 
      around here. We could participate in action research
      program, you know?"

MNG -"Hmm, That sounds like a lot of money. We could buy 
      a lot of chairs in two-day course for that. Wouldn't 
      that be great?"

...

There is a huge difference between being teached and learning. To start with there is a difference of acquiring existing knowledge versus creating new knowledge. Further there is a huge gap between action learning and a static content course which may have zero contextual relation with developer's current situation. I have seen the affect of these courses, and it does not impress me. It is just like many other mental models in this industry; a pretty model without connection to reality. But hey, traditionally developers have not taken part in action learning.

And old habits are slow to die.

Prisoners of Our Habits - Act Two

2006-07-11T19:28:00.000+02:00

Prisoners of Our Habits. Act two. Project manager enters.


PM:   -"You need to focus on project's
        paper deliverables for the next two months"

DEV1: -"Which papers you mean? For whom they are,
        and for what purpose?"

PM:   -"I do not know, but reviewers know all the
        buzzwords, so my ass needs to be covered."

DEV1: -"Eh, That does not sound so valuable and motivating?"

PM:   -"It's not supposed to be. I fill in dashboards
        which no one reads. You should have no problem
        creating papers with no value."

...

The above is not a Dilbert strip. It is status quo in many development projects. Dashboards illustrating Gannts, PERTTs etc. are what we traditionally use for project management, and paper documents are what we traditionally use for progress monitoring and project quality assurance. So what the hell is wrong with that you ask? Again, it is a nice model, I really do like it. It has only one flaw; it does not hold in reality.

Then again working prototype, incremental release plan, Sprint backlog, or burndown chart are not used traditionally.

And old habits are slow to die.

Prisoners of Our Habits - Act One

2006-07-10T08:17:00.000+02:00

I have been describing continuous integration for an embedded firmware project targeting a low-end microcontoller. Based on discussions I see a need for pointing out something about my view;

I do not mean that you need to apply continuous integration, unit tests running on both the host and target, etc. if you are writing a 200 LOC blinking LED application for 8-pin microcontroller for fun. This is a job for Solo-Virtuoso, and methods should be alike. The project I'm talking about, while targeting the low-end microcontrollers, will develop tens of thousands of lines of C source and will eventually be compiled into ten or so configurations for mass production. The configurations have some special areas where expertise is needed, like power management and wireless RF technology. This type of project cannot fit the Solo-Virtuoso category. Instead it requires a collaborative group of people - a job for an agile team and this team should adapt these practices.

Purely based on my biased participant observation, the trend of firmware projects is towards bigger and more complex so we are going to need these practices. The problem in firmware domain is that former electrical engineers (now programmers, or even worse, managers) are not willing to see this increase in complexity, but remain prisoners of their old habits. They are still in denial. My recent experiences manifest this very thing.

Prisoners of Our Habits. Act one. Programmer enters.

A colleague following our agile journey mentioned that based on his observations of daily Scrum meetings and collaboration we continuously run into problems caused by someone changing an interface and all of a sudden none of the configurations work. His obvious solution for this was; "there is a need for up-front designed interface contracts between modules". Problem solved, eh? We do not have full set of requirements, nor the freezed hardware architecture. For me iron-bound interface contracts seem pretty far fetched objective, but then again, it's just me. Still I would be amazed (but happily so) to find this wizard pulling out a comprehensive up-front architecture for us. Oh, and preferably delivered as a PowerPoint presentation I might add. If you happen to know this guy, ask him to drop me a line. I could use a good laugh.

I see these discussions about interfaces as a solution, not the symptom. It is good to see team members having courage to change things, to refactor when opportunity arises. These conflicts usually get solved easily after the daily Scrum at latest, so to me these are just short architecture meetings. In addition there is a huge difference compared to normal high ceremony architecture meeting: decisions get made and implemented immediately and feedback is available the next day. I have witnessed this working extremely well, and fast, even with programmers with junior skill level. This is just not the way it has been done traditionally.

Old habits die slowly.

Agile Business Conference - Program Draft Available

2006-07-08T08:05:00.000+02:00

There is a program draft available for this year's Agile Business Conference in London. I have attended this conference for two years and I can recommend it. It is organized by DSDM Consortium and the title has word 'business' in it, but the agenda is about agile in general. This years highlight for me seems to be 'Agile and Embedded Systems' (Pekka Abrahamsson) . I'm also very interested in presentation by Mike Criffiths - 'Utilising Agile Methods Alongside the PMBOK Guide'. I see very little value in this balancing, but this is something we currently often need to live with and nevertheless we should always remember and reflect where we are coming from. This helps us to understand where we would like to go next (post-agilism).

Again interesting to see keynotes from Kent Beck, Sean Hanly, David Taylor and Polyanna Pixton. I really hope I'll be able to make it! If you are planning to be there, and are interested in agile [product] development, drop me a line and we will catch up.

Objects and Firmware - Inheritance

2006-07-03T22:12:00.000+02:00

This is the third and last post (see earlier basic encapsulation and polymorphism) about object based firmware programming. This is about single inheritance, another important object oriented principle. This is also pretty much as far as I would go with the HW currently available for our embedded projects, low-end 8-bit microcontrollers. Anything further would require even more use of pointers, including function pointers, and I'm not really that comfortable with their heavy use.

If you can see this, your browser doesn't understand IFRAME. However, we'll still
link
you to the file.

The sample code implements a simple Level object type which has only two members, level and currentState. The first one holds a threshold value, the latter indicates wether the latest sample was below the level (currentState = 0), or equal/above (currentState = 1).

Next we have used Level as a parent for another object type, LevelWithOffset. The new object has the Level type as its first member, named super. This gives us possibility to use pointers to both of the object types in similar way. LevelWithOffset object type has an additional offset member. Value of offset is added to set level. In the example code we have two objects, one simple Level and one LevelWithOffset. Constructor is called for both setting the level to 2. The LevelWithOffset object is then further adjusted by setting the offset also to 2.

The main loop calls the notify method for both objects with values from 0 to 4 and outputs the state of both objects. We can see from the test run that the other object switches the state only after the sample is increased by 2 more steps (effect of additional offset).

You can go a long way with this object stuff in C. If you are interested then check out the papers listes below and start working on your skills and creating your own opinion.

Miro Samek, Portable Inheritance and Polymorphism in C, ESP (pdf)
Ron Kreymborg, Single Inheritance Class in C, Dr. Dobbs Journal
Matthew Curreri, Object-Oriented C: Greating Foundation Classes, ESP (part2)

CruiseControl Configuration for GNU Make

2006-06-28T07:49:00.000+02:00

I promised to share the CruiseControl config.xml file we use. Here you go, it is not much of a contribution, but I hope it helps some of you jumping into cruising mode. Our configuration file uses the Exec builder for GNU Make automated builds. Typical examples use Ant in Java environment.

If you can see this, your browser doesn't understand IFRAME. However, we'll stilllinkyou to the file.

For additional information:

Martin Fowler, article on Continuous Integration
Lasse Koskela, Driving on CruiseControl Part1 and Part2
CruiseControl Configuration Reference

See it in action in my earlier post.

Agile Methodology is not a Religion, it's Just a Passion

2006-06-23T07:18:00.000+02:00

I spent three days in Oulu, Finland, attending the XP2006 conference. I have attended other agile conferences, but this was the first time I attended this conference, which was the 7th in series of XP conferences. I have to say I was impressed. The conference is about practitioners to the degree that you really get to sense the belonging in a community of practice. Authors of best selling books, key note speakers, most successful agile consultants in the world, signers of the original agile manifesto, other practitioners from all around the world, all open for discussion on this shared passion, without trying to close a contract before telling more.

I saw that strong passion of doing things right as the combining force of this community.

(Some) People are passionate about religion, too. This is why I think agile thinking is sometimes compared to religion, and comparison of agile and waterfall like methods is said to be a religious war. I saw nothing like religion (at least in a bad sense) in this conference. Instead lots of presentations and hallway talks alike concentrated on balancing methods, applying best from the both worlds, taking existing organization culture into count etc. I do not see this resembling fundamentalist religion at all, do you?

I have to say that in the early days of agile movement I sensed that strong black and white positioning as well. I participated in that. This I believe was the result of so strong survival anxiety after being so fed up with the situation and just wanting to get rid of the old ways of doing stuff all at once. The brilliance of agile methods is manifested in what we are experiencing now. We use continuous retrospectives to improve the method and to adapt it to changing situation. This is possible because practitioners are passionate and enthusiast and put lot of effort into keeping the method fit and healthy.

Sean Hanly from exoftware pointed out in his key note that we must be careful not the end up in the process graveyard with the famous "failures" like RUP, CMM(I), Waterfall and numerous others... By saying this Sean meant that the interpretation of original methodology is in danger to get twisted when number of interpreters increase. When agile development is moving into mainstream we are going to see a lot more practitioners without this passion, but just making the use of a new tool.

There are photos from the conference...

Comprehensive Documentation for Riding a Bike is Difficult

2006-06-14T12:07:00.000+02:00

Nonaka and Takeuchi (1995) defined knowledge as explicit and tacit. This view is of course shared with other researchers in organizational learning.

I posted about agile development emphasizing Enable Future Work -documents (as described by James Shore). That holds, but it still does not make creating quality documents easy.

Day after day engineers will arrive at work and launch MS Word alone because they need to document their work for future use. Unfortunately some will open the same tool for documenting what they are planning to do, but this is Get Work Done- documentation (as describ ed by Jim Shore), and we are not talking about that now. At the exact same time same number of engineers will get a copy of such a document to start working based on it. By the lunch time most of them have realized the same old thing; "Just words, not even grasp of what I would have needed, half a day wasted, thank you very much."

I'm reading the Communities of Practice, by Etienne Wegner, for the second time. This time I understand a lot more, but of course I am still not fully getting it. However Etienne gives good example of two sides of knowledge, explicit and tacit. Mr. Wenger used skill of riding a bike as an example. Most of us master this skill, and we can explain it - even fairly well - in paper;

"Push some initial speed, start pedaling and just steer where ever you want to go. "

This is the explicit part, but there is one fundamental ingredient missing in the above document. How do you keep the balance? This is the tacit knowledge part, and it is extremely hard to put on paper. I give you few minutes to think about it...

OK, I tried it also while reading the book, and it reminded me about something we all have experienced.

Agile methods promote simple design to the degree that documentation is not necessary. In embedded world where you want to minimize the material cost you often end up in software and hardware design which are based on intuition. This tacit information is difficult to put on paper, just like the balancing bit of riding a bike. How many times you have been explaining your design to a peer until a bit of detail which has no meaning to you opens the design to your colleague? These obvious facts seldom get identified and reported in paper document created by the original designer individually. As a consequence the document has little or no value. This was manifested with our three-day faceoffs recently. This has nothing to do with agile development methods, since they promote simple design and light documentation?

Quite the contrary. Agile teams work well in preserving and sharing tacit knowledge among members. Daily meetings, retrospective and pair programming(co-design/debugging) are natural techniques for this. Additionally the team commits to deliver work (software/product) which is "done". Done means delivered with just enough and barely sufficient documentation. Agile team should have also shared commitment to deliver necessary documentation, which would enable them to create the documentation in collaborative way. This type of activity is valued in RaPiD7 (Ph.D. pdf), an agile method for creating documents developed at Nokia. Of course the best thing would be to have the "customer" of a document present during the creation work (or that the "customer" writes the document as Vasco Duarte proposes), but this customer for Enable Future Work -documentation is often not available at that point.

The next best thing might just be a collective team responsibility with other project stakeholders and to give it your best shot from multiple perspectives instead of single unmotivated view.

embUnit has some additional features

2006-06-06T18:15:00.000+02:00

I posted about my options for proceeding with CruiseControl. A while ago I already wrote a python script which ran the unit tests, and interpreted the results into XML following the JUnit schema. This seemed to be working, but there was one major problem. embUnit output did not include any detail about successfull tests. So I started looking my options again. I took a look at CUnit. It has XML output for its own translator, but this of course does not follow JUnit schema either, so I was not getting any closer.

Then while browsing through embUnit source I made an interesting discovery. The code for embUnit library does not require standard C libraries. This I believe is the embedded part of it. In the distribution there is however additional source code for two different user interfaces; 1) text and 2) XML. These can be compiled into another library (the basic libary comes as binary with distribution), which can be used for host run unit tests. Two libraries can be tied together with nice function pointer usage, similar to what I tried to explain earlier. Everything is coming together beatifully. Example of generated XML report is below. This again does not of course follow the JUnit schema, so I will continue by modifying the CruiseControl XSL translator for it.

If you can see this, your browser doesn't understand IFRAME. However, we'll still
link
you to the file.

To be cont'd...