Weekly Delivery of Firmware

Yesterday we finished a six week project developing technology platform for User Interface innovation using some practices familiar from agile methods. The project delivered two different configurations of home automation user interface device; one for wall mounting and one for remote control. Some facts:

• Vague or missing requirements
• New 8-bit microcontroller
• Lots of new sensor technology
• New embedded firmware team of 3+1, including one consultant (in house)
• Some drivers developed by outsourcing
• Final project consisted of 20KLOC of firmware C source (bad meter, but...)

Some findings after initial reflection:

• Weekly delivery (not release) of firmware is possible, at least in some cases.
• Short term goals and daily Scrum meeting effectively work as motivators, and enforce full learning and getting things done done.
• There is not small enough project to not use version control and continuous integration (we had no way to have common version control with outsourced team: large org, IT, you know the drill...)
• HW evolved from microcontroller starter kit to final electronics via several bread board mocks and one halfway PCB prototype. This causes extra assembly work when looked at the surface, but this overhead is by far overcome by enabling the concurrent engineering instead of waiting for final version.

The last point holds only if all the disciplines work as One Team.

We did not have cross-functional team as we did in earlier larger experiment. Schematics, PCB layout and mechanics were developed concurrent, but they were not part of the team - nor did they follow any of so called agile practices. Firmware planning was tried to be synchronized with deliveries from other disciplines. In several occasions these deliveries were delayed, and a common goal needed for a team to actually be a team was somewhat missing. This is where I would like to do things differently if I was to do it all over again.

Occasionally we heard sentences like "If everything goes smoothly we will be ready on Tuesday, but we will be ready at least on Thursday". At this year's Agile Business Conference in London David Taylor (The Naked Leader) said "Don't have a plan B because you will always achieve it." Just guess if the delivery was on Tuesday or Thursday - or Friday?

The whole project was started by extremely vague idea. In few hours we worked out initial backlog and a delivery (release) plan based on simple themes. From that point on plan, design and requirements emerged during the 6 weeks nicely synchronized with weekly demonstration of prototype and new planning session. This experiment supports the belief that good people can work this way, and that we truly live the era of enlightened experimentation. We believe that we could not have performed any better with thorough analysis, design, and specification phases - at least not to cover the cost of them. During the development there were several points where it was not possible to proceed as we had thought. A comprehensive plan based on these initial assumptions would have failed miserably, and the time creating them would have been wasted. In this project the whole team worked out an alternative solution, adjusted the next week's plan, and the project was fine...

I will write an experience report and hopefully make it available here.

Skunk Work Is Not A Longterm Solution

In her "The Agile/Waterfall Cooperative (pdf)" presentation Michele Sliger summaries three modes in which agile team can work in a waterfall organization; SWAT team, Skunk Work, and stealth agile. I agree that applying just some agile practices can benefit your work. This is pretty much what we are doing at the moment, but I do not see this as sufficient long term solution - if it is only in team's sphere or just above. All three forms of team work can of course be used. SWAT teams can go parallel even in more wide spread agile organization. Skunk work probably, and hopefully, always exists in large organization - it's fun. If you are working as stealth agile, and avoiding radars by not communicating to the top management, you are not going to be able to fully harness the power. You need to inform top management that you are doing things differently and communicate your need for changes. They probably will not get it, but what they do get - is success. So it is fine and in most cases necessary to fly low at first, but after certain point it is not satisfying anymore to work as a shadow system, because you know you are constrained by the legitimacy structures, and these structures are holding back you and your learning.

The rest of the presentation has also good ideas, and based on my experience these ideas hold in reality as well.

SWAT team – outside the domain of the process police, brought in on projects that are in trouble. Agility is prized in this situation. They take over however – they don’t work in a cooperative mode.

Skunk Works - “a small group of experts who drop out of the mainstream company operations in order to develop some experimental technology or new application in secrecy or at speed, unhampered by bureaucracy or the strict application of regulations.” Sanctioned and protected by management, these teams are rare. This formation however, prevents the headaches involved in org structure, project approval hoops, portfolio metrics and management, budget calls, and other bureaucratic nightmares.

Stealth Agile – As Jim Highsmith once said in response to a question about how to sell agile to upper management – don’t. “They don’t know what you’re doing anyway.” Do as much agile as you can where you can. Some benefit is better than none at all.

Link: Scrum and XP from the Trenches

No matter if you are developer, PM, or CIO and you don't know what agile/Scrum/XP is, you are thinking about applying, or you are already practicing - you should check out "Scrum and XP from the Trenches (pdf)" report from Henrik Kniberg. Henrik describes how they applied Scrum and XP practices in real life and presents the adaptation needed in this situation. Excellent Writing!

main_theOneAfterFinal.c

Today I happened to see a file name with word 'new' on it on a presenter's computer. I was slightly amused because of my history. I can still see the old version naming convention in many "shared team folders" in our server. It goes something like this:

original
new
latest
final

I have been using version control system since my first university years (some fifteen years ago) and throughout my professional life. First it was just an advanced XCOPY method putting released versions of one man projects into the system, but about three years it has been collective code ownership and at least daily update-commit (we use CVS) cycles from the beginning of development. Lately we have adapted continuous integration and automated regression (confirmation) testing as you know. Currently we are looking for ways to add some automated acceptance and system testing as well.

During the past couple of years I have had a pleasure to work with software developed outside our own company. Quality of source varies a lot of course, but it is striking that version control typically is not even mentioned. In some occasions lately the software was delivered as zip'd folder with a version number in folder name - at that point I was happily surprised by that! Embedded firmware domain - based on my biased observations - walks some 10 years behind the mainstream software industry. The wide lack of basic discipline, version control, proves the point. I'm glad that I work in a department where I can say that developers would never go back working without version control.

I found this Yahoo group message - nostalgic.

The Lenghty Debugging Phase

To give embedded firmware TDD a kick start we have had two sessions with a colleague. During these sessions we have created scripts to automate the unit test structure and build process and discussed how the make as thin wrappers for HW as possible. However we learned also other valuable lessions during these sessions. Lesson number one is the "debugging phase" being so strongly in our domain culture. We often hear a firmware developer saying, "well, it takes couple of days to implement it, but you can never tell how long the debugging is going to take". Of course debugging phase is where all the firmware heros truly shine. That's another unfortunate culture issue in firmware development.

In our first session we ran into problem of not getting one test running. Immediately we started debugging. Printf's fly in here and there (as an advanced host run I/O twist). The change to this mode was so rapid, that neither of us realized what is going on until later. After some 30 minutes(!) we recognized that we are not moving, but just desperarately experimenting with printf's, and it hit us; we have not changed our behavior as we were supposed to. We should have picked a more simple thing to implement. When we realized we can not chew what we had chosen, we should have started all over again with simpler thing.

It is our belief that by doing TDD for firmware we should get rid of, or at least dramatically reduce, the debugging phase and thus create an atmosphere of success, and improve the reliability of estimates.

This type of sessions are great tools for learning. I strongly share Michael Harmer's opinion on The Training Course Scam.

Ken Schwaber on Quality of Code

Ken's presentation "A Canary in a Coal Mine" at Agile2006 conference is available via InfoQ. Ken talks about quality and the danger of Scrum teams droping quality issues to get higher velocity and to be sure to hit the date. He calls this a re-definition of "done", or "somewhat done". We have manifested an increasing technical debt in our iterative work. So in the last project we were putting more focus to acceptance testing inside each sprint and introduced something we called "cross-testing". This means that other pair needed to test the feature implemented by other pair in order to claim feature "done". Embedded firmware systems need a lot of manual acceptance testing (at least our current practice) and it is easy to get slobby testing your own work over and over again.

Was the technical debt there because of Scrum? Well, no - it was made visible by Scrum and the team was able to decide to do something about it early enough.

----

Ken's another speech about Scrum is available at Google Video. (You will find out that Scrum works with idiots as well)

I posted about Jeff Sutherland's Scrum speech earlier. It is also available via InfoQ.

XP2007 Conference Web Page

XP2007 web page is open. The conference will be held on June 18-22.2007 in Lake Como, Italy.

In the scope of conference is:

• Embedded software (e.g., SW/HW co-design) and Agile SW development

...among of course other interesting stuff. Hopefully we see some papers in this domain as well. I attended the conference this year in Oulu, Finland, and if next year's event is anything like that - I can highly recommend attending.

NPD in General With Agile and DSDM

Iterative and incremental development ideas have been presented for general new product development (NPD) for years. I'm interested in agile methods (as in software development context) because they give a philosophy behind and lots of ideas how to actually do this. It is of course also because of extremely active and powerfull community. Ideas of using these methods and practices in NPD in general have been presented by Jim Highsmith (Agile Project Management), Preston G. Smith (Developing Products in Half the Time), and many more. Both of above mentioned can be contacted via Cutter Consortium.

Now there is something new coming up. At London Agile Business Conference 2006 few weeks back the DSDM consortium talked about the future of the DSDM framework. DSDM version 4.2 has been open for online viewing since this year. They are going to publish a new version of framework in Spring 2007. This new version is supposed to be more general and not specifically targeted to just software development. It was also presented that DSDM can be seen as a wrapper for models like Scrum and XP for use in enterprise environment. It's a bit too structured to my tasting, but very well worth to check out when available. It may also become more open.

Risky Business

It has been identified that the biggest risks in project work concern management and marketing. Yet, in most environments we only concentrate on technology risks. Agile, incremental and iterative, development is also often assumed to be risky. By changing the way of thinking about risk, people would see that iterative risk management is a proactive way to tackle all aspects of risk. A very simple way.

1. Management risks - just fund the project for the time zone you are comfortable

2. Marketing risks - use the first possible release to test the waters

3. Technology risks - use the effort early to tackle any technological uncertainties that traditionally are left lurking until the (assumed) end of the project

The status quo practice is quite different. We take huge risks by starting a megaproject, for example estimated to 5 calendar years, based on a business case. Everyone knows that business case targeted five years from now is pure speculation. In the global markets it's a very expensive speculation by the way. Based on this speculation management needs to make a decision for funding a five year project. Their natural reaction to tackle the risk is to review the project mercilessly. We can be sure that changes to project requirements, technology and scope will happen. We have this heavy review practice going on making the project move even more slowly. More time means more changes, more changes mean more reviews - are you starting to get the picture?

Most of us see this everyday. How do we try to fix it? Naturally by reviewing. Because? That's what we allways do.

Refactoring for C

SlickEdit is a commercial compiler source editor that many people speak highly about. I thought I will give it a spin and downloaded the trial version. Their web page describes "cool features", but for some reason refactoring is not listed. Anyway the product has capabilities for refactoring several languages, and few quick refactorings (based on tagging instead of parsing) even for C; extract method, rename, modify parameter list, and replace literal with constant. At least for simple code they all seem to work. SlickEdit is also available as Eclipse plugin. I only tested the own IDE version.

You can find out the reason why there are not so many refactoring tools for C from MS and doctorate thesis by Alejandro Garrido. They are available via this site.

Web Page for Embedded Unit

Embedded Unit (embUnit) is an unit test framework for embedded C systems. It now has a simple web page with manual. It is not pretty, but it is something.

One Team, but just a team

One of the agile development practices is One Team, meaning that team as a whole takes the responsibility. When agile team is formed by a bottom-up initiative, it will soon face the problems of limited sphere. Most of the problems in projects are not related to design, technology or engineering practices. These are the only ones that the team is capable to fix quickly through iterations and reflective retrospectives. After fixing its own sphere, the team hits the wall when working in the shadow of a large organization. This can be a stressfull situation as everything is starting to be crystal clear.

Afterall One Team is just a team.

Last monday at Agile Seminar Helsinki Heimo Laukkanen presented that the whole organization needs to be agile in order to fully harness the power. He talked about IT governance, but even faster you will notice need for change in leadership styles, design review practices, personnel development, project planning (not plan), marketing involvement, and a lot more. Even while the agile development only in technical level may be rewarding, seeing clearly the dysfunction of standard organizational processes (so called best practices) and structures will hammer down the gained team spirit and motivation, eventually forcing it back to old ways of working.

Changing these processes and structures in a large organization may be impossible with bottom up approach. Large organization has a whole process department to take care of the process. It would take a complete paradigm shift to access this process. What makes this hard is that this department is not rewarded based on project success or revenue, but the success of current paradigm - from the administrative perspective. Admitting that a paradigm shift is actually needed might feel like a failure of the department.

This is one of the reasons why agile interventions often fail, the rest of the organization fails to learn fast enough.

Come Again, What's the Cost and Overhead?

I've been implementing object-based stuff in C for embedded firmware with passing an index parameter to an array of structs, like I tried to explain here. Yesterday I remembered an article about pointer-less C in Embedded System Design and thought I will give it a couple minutes more. The article gives some ideas how to make pointers safer to use and talks about the cost and benefit of using pointers. I'm not saying anything about the value of article itself, but I got curious. I took our simple switch debounce filter routine which is used a lot, and which is typically the only one that actually implements multiple objects at run time. It uses the me index as parameter for two functions. I quickly modified the code so that both functions take in a pointer to a struct instead of simple unsigned char. I compiled both versions for Microchip 16F876A (8-bit uC) and got surprised. The pointer version used over 30% less code memory than the index version, reducing the ROM image from above 300 words to below 200 words. I then compiled the source files to give me the assy listing, and below are the results that I found for a simple function that just sets some members of a struct. Because of werd mechanism of indirect addressing in PIC micros, the pointer is actually passed just as a 8-bit register. Word of caution; we have experienced problems when passing pointers and enabling optimization with Hi-Tech compilar for PIC micros, that is another reason why we have avoided this.

The lesson; while I do believe in intuition for great solutions, it is worth to challenge your beliefs every once in a while.

Pointer version:

    30  0764             _Cbutton_construct
    31                   ;button_ptr.c: 27: me->debounceCounter = 0;
    32                          
    33  0764  0084       movwf fsr
    34                            
    35  0765  1383       bcf 3,7
    36  0766  0180       clrf 0
    37                   ;button_ptr.c: 28: me->level = !1;
    38                          
    39                   ; _me assigned to ?a_Cbutton_construct+0
    40  0000             _Cbutton_construct$me set 
                         ?a_Cbutton_construct+0
    41                   ;_me stored from w
    42  0767  1283       bcf status,5
    43  0768  1303       bcf status,6 ;carry unused
    44  0769  00A8       movwf (((?a_Cbutton_construct+0)))
    45  076A  0A28       incf (((?a_Cbutton_construct+0))),w
    46  076B  0084       movwf fsr
    47                            
    48  076C  1383       bcf 3,7
    49  076D  0180       clrf 0

Index version:

    30  06F6             _Cbutton_construct
    31                   ;button.C: 27: theButton[meIndex].
                         debounceCounter = 0;
    32                          
    33                   global _theButton
    34                   ; _meIndex assigned to ?a_Cbutton_construct+0
    35  0000             _Cbutton_construct$meIndex set ?a_
                         Cbutton_construct+0
    36                   ;_meIndex stored from w
    37  06F6  1283       bcf status,5
    38  06F7  1303       bcf status,6 ;carry unused
    39  06F8  00A8       movwf (((?a_Cbutton_construct+0)))
    40  06F9  0828       movf (((?a_Cbutton_construct+0))),w
    41  06FA  00F0       movwf btemp
    42  06FB  1003       bcf status,0
    43  06FC  0DF0       rlf btemp
    44  06FD  1003       bcf status,0
    45  06FE  0D70       rlf btemp,w
    46  06FF  3E20       addlw ((_theButton))
    47  0700  0084       movwf fsr
    48                            
    49  0701  1383       bcf 3,7
    50  0702  0180       clrf 0
    51                   ;button.C: 28: theButton[meIndex].level = !1;
    52                          
    53  0703  0828       movf (((?a_Cbutton_construct+0))),w
    54  0704  00F0       movwf btemp
    55  0705  1003       bcf status,0
    56  0706  0DF0       rlf btemp
    57  0707  1003       bcf status,0
    58  0708  0D70       rlf btemp,w
    59  0709  00F0       movwf btemp
    60  070A  0A70       incf btemp,w
    61  070B  3E20       addlw ((_theButton))
    62  070C  0084       movwf fsr
    63                            
    64  070D  1383       bcf 3,7
    65  070E  0180       clrf 0

Code smells - just refactor

As we identified, agile development puts more demand to top quality source code than we (firmware developers) are used to. Actually agile development only makes the bad quality visible to larger crowds. This fact fairly quickly takes us searching for new practices like refactoring. You can find about it in Refactoring site, or better yet from Martin Fowler's book.

There is a nice table by Gene Garcia available as a summary of code smells and refactorings. Take a look, this stuff applies to firmware as well.

We Can See Clearly Now

The beauty of Scrum, or agile development in general, is that it gets everything laid down in front of everyone. Everything, everyone, good and bad.





1. Developers learn to understand what 'done' really means, and how much effort it takes to get things done. It demands discipline not to have technical, or administrative in large organization, debt increase increment after increment, but actually do all-at-once, in parallel. At the same time the satisfaction and enjoyment from accomplishing meaninfull work is immediately rewarding.

2. Stakeholders outside the team start to understand the unpredictable nature of new product development, but also the true capablities of reliable development. Customers understand that they do not need to ask for everything in order to get at least some. Common understanding of business value (benefit) starts to emerge between the customer and developers. Managers and project managers see the effect of meaningless deliverables and context switching. Managers start to understand just to stay out of the way and remove obstacles.

Trust begins to take form.

Agile Seminar Helsinki, Fall 2006

Title says it all. Check it out.

Jeff Sutherland on the Roots of Scrum

A presentation about the roots of Scrum is available via infoQ. A 60-minute presentation held at JAOO conference about everything and some behind the Scrum as we know it today from Dr. Jeff Sutherland, the original creator of the framework. Without the need to travel. Great.

Bubble, Bubble...

We have had our continuous integration server running for a while. At this point we thought that some not so serious step should be taken. It was time to implement the lava lamps.

Being a firmware shop we obviously could not go the normal way with X10 publisher. We used our own shutter relay prototype to drive the two lamps. The relay module is normally driven with I2C -bus, but we simply used the data and clock signals to transfer pulses from pc's serial bus. The I2C bus is running at +3VDC level, so we needed to shift the 12v levels found on serial port to that level. Don't worry, the +3VDC level is isolated, so 'it should be fairly safe'. If you want help with similar hw setup, feel free to contact us.




All we did from that point was a couple of simple python scripts to drive a short pulse on DTR or RTS according the build result. This was easy with a little help from pySerial module. Shutter relay connected the lamps according the last detected pulse. These scripts are called via <execute>-publisher in CruiseControl's config.xml.

<onfailure>
  <execute command="python pulse_rts.py">
  </execute>
</onfailure>

<onsuccess>
  <execute command="python pulse_dtr.py">
  </execute>
</onsuccess>

Next steps:

1. Write an actual publisher for DTR/RTS signals in Java
2. Use our rf comm devices to transfer the build status
3. Hook another lamp to indicate the ongoing build process
4. Write a LonWorks publisher similar to X10

Mark Michaelis described their similar effort. They are a bit ahead of us.

Immediate Payoff

Ken Hughes shared his story about their first three months with Scrum development. Findings are all too familiar. There is also a nice example of excel sheet working as backlog.

Certified Scrum Master

I had been thinking about taking the 2-day Certified Scrum Master (CSM) training for couple of years. Last week I finally got to the course with a colleague. My concern was that since I mainly develop firmware, I would not benefit that much from the course. I could not have been more wrong. I highly recommend CSM course to anyone involved with project work. Even if you do not believe in agile development, this course can still give you pointers on where to improve your work.

We got our training from Boris Oestergard and Jens Gloger. They have been giving in-house training in finland earlier, at least at Nokia Networks, Sulake and Accenture. We had also Bas Vadde from Nokia Networks giving an interesting guest speak about scaling Scrum to larger programs of hundreds of developers.

You can see our group photo at www.scrum.dk (photos).

Just Nail That One Thing

I have become to value that the team should be imposed to only one new thing at the time. For example bringing in team room, pair development or TDD should not happen in a huge bang but a transformation time is needed. However when I had a chance to run an agile day with senior management I forgot this important lesson. I got over ambitios and tried to achieve too many things at the time.

1. I started with presentation covering agile development philosophy, results from our experiment and also the needs for more advanced future adaptation. All this in 1.5 hours with active discussion. Try to keep your presentations focused.

2. We then applied Open Space Technology and went through three sessions. This went fairly well considering we were only 6 people and all new to this way of working. Some difficulties in coming to a conclusion, but time-boxing still helped to cover the issues.

3. At the end I tried to run a 58 minute agile simulation. I combined XP Game and Scrum Simulation and run a preparation phase with vision creation, then two sprints 10min each and finaly a closure phase. This is OK and could be done again with some slight improvements, but where I made a huge mistake, was that I wanted to wrap up the results from Open Space within simulation. Time pressure from the simulation, new practices coming at fast pace, and meaningfull work at the same time. Not likely.

So my lesson; if you have some very important issue at your hand, do not try to be too clever. Just focus on nailing the one task with the highest priority/value and make sure it gets DONE (aka implemented, tested and documented). If you chose the wrong issue, well buhuu, that's tough.

On the bright side it at least wasn't anything like Dave Nicolette described happening during Jon Kern's presentation. So maybe it was ok from the learning point of view, and at least the journey still continues.

Head up. Reflect. Repeat.

We Are not Alone

This post by Simon Baker really, really, made my day. It's good to know that the situation I'm currently in is by far not unique.

Can our agility bring about a wider culture change, at least in the program office? Maybe. It depends on the willingness of people to move beyond the comfort zone provided by a command and control environment.

Next week is going to been an interesting one.

Not So Naive Change Model

Agile methodology is not visible. I do not know what my team is doing. These are complaints often heard from project manager. Ops, I of course mean persons that understand that they are project Managers. Managing is translated into command and control. When saying "not visible" they mean "I can not command", "I can not control", or "I do not understand prototypes, show me paper".

What is described above is the impact of change, and it follows the satir change model. After a smooth start of agile pilot enabling more effective team work, all new kinds of problems start to emerge. Team learns about the actual effort needed in disciplined incremental development versus hacking, but also people who have evaluated their status by how much power they have to command and control (project Managers) may fear loosing their status. All of a sudden product gets developed without screaming, hair pulling and jumping up and down like a freaked out kangaroo. There is no room for the blame game, what is going on here? What is my role in this? A paradigm shift from command and control into trusting and facilitating causes fear among this dying specie of command and control type knowledge-work managers.

One needs to be very carefull not to interpret this new state of chaos as failure in agile intervention. This is excatly what was expected, but fear is a powerfull emotion and when in panic we do irrational things. I on my part fear what happens to a project Manager who in panic starts to act even more irrationaly (compared to my understanding of rational). As we remember from the theory of complex adaptive systems this emerging behavior is of course totally unpredictable...

...fingers crossed everyone, stop-think-act, maybe she gets agile.

Taming the Big Animal - Agile Intervention in a Large Organization

I made an academic paper about our first larger scale agile experiment available. You can find it via this link (pdf), and it can be found also from the sidebar. Also see the analysis that started this experiment.

Abstract— New Product Development (NPD) today is mainly
managed with plan-driven, relay race type, processes. Today’s
dynamic markets and new technology used in NPD result in a
possibility of benefiting from more adaptive, rugby style,
methods. In the software community methods applying this
approach are commonly called agile methods. In this study I
present experiences from a bottom-up initiative to apply agile
methods for embedded system development in a large
organization using the Stage-Gate® project management
framework. The purpose of the study was to find out whether
agile methods have a chance and how much this type of
organization could benefit from agile methods. Study supported
the earlier research showing that implementing incrementally
some agile practices the project organization was able to improve
the team spirit, motivation level, and additionally to increase
productivity. The affect of agile practices to the Stage-Gate®
framework was minimal. Through wider buy-in and
commitment, the agile development increments could be
synchronized seamlessly to the Stage-Gate® milestones. However
as described earlier in literature, top-down commitment and
changes in organization culture are needed in order to succeed in
process change. The philosophy of agile development is quite
contrary to so called traditional NPD approaches. These changes
in organization culture are also needed in order to fully harness
the power of agile development.

Can RUP Be Like Switzerland?

At one point debate on agile and more waterfall like sequential process models was claimed to be like religious war. This impression might still be seen in an large organization when agile methods are piloted for the first time. What if the Rational Unified Process (RUP) is the legitimate model for software development in this organization? RUP, being (in my and some others opinion) extremely abstract framework, could easily adapt to pretty much any methodology.

I do not hold a guru status on RUP. I have read the introduction by Kruchten, browsed the net, some papers and that's about it. Yet I propose that RUP could be seen as neutral territory for getting the advocants from both camps to work on common framework in order to let all the flowers bloom.

I know that if I was running a small start-up, and obviously using agile methods full a head, I would not concern my self with RUP because it would not offer me enough. However in an large global organization some level of legitimate system and standardization can be seen necessary, and thus I do not see RUP as bad option. Especially if RUP is already in the house.

RUP has been adopted to agile methods and vice versa starting from the early days, but some major effort has been put to this just lately. Check out:

1. Agile Unified Process (AUP) by Scott Ambler
2. Eclipse Process Framework (EPF) and OpenUP by Eclipse Community

...About Team Rooms and Pairing

Mishkin Berteig pointed to this web page about setting up a team room. This page pretty much covers everything I have in mind about the subject, especially with the links it has. Pros and cons of the team room are handled. These guys also share my philosophy about forcing these things. If you should force people to do anything, team room and pairing are definately NOT among such things. I believe these issues are automatically brought in when (if) the team jells and learns to work together.

Our Version of Primetime Emmy Award

It's been awfully quiet around here. It's just because I've been lazy busy.

During the previous Sprint we experimented a practice we called Primetime. Primetime means that immediately after the daily Scrum work was continued in the team room and by pairing for couple of hours. Of course in embedded system development pairing means pair development, not just pair programming.
Yes, yes, we all know that this should be a full day practice. But ... there is lots of other emergent responsibilities, for example maintenance and things like that. At the moment Primetime is the best we can achieve in our environment.

Tom DeMarco highlights the importance of uninterrupted work in his book. The cost of reimmersion time to flow (state of mind when developer "gets her design") is extremely high, and our experiment manifested this.

In our retrospective we easily identified five observations:

1. People got enthusiast about the design when pairing, and continued way beyond the agreed Primetime
2. Two people pairing blocks other people from interrupting
3. Emails, phone calls etc. got postponed
4. Learning was accelerated because of osmotic communication, and active collaboration
5. Defects got detected as in review process, thus improving the quality

Within this experiment the team easily achieved Sprint goals, which were set based on earlier velocity. It is too early to make a conclusion, but we decided to keep this practice.

Plans, Planning and Speculation - There's Time for Everything

It has been funny to me from the very beginning that agile methods are often claimed to lack the planning activity while the opposite holds. The difference lies in the fact that planning for the unknown becomes speculation. Prolonged speculation period brings very little added value and is thus kept to minimum in agile projects until better knowledge is achieved by active learning. Simon Baker in AGILE IN ACTION reminded about the two great early agilists believing in planning instead of plans.

Life in a Cubicle

Are you working in a sorry ass company that forces people to work in cubicles in order to foster communication and effectivity? It's sad, but turn on your speakers, pump up the volume, and check out this James Blundt remix.

Hope it helps.

Portfolio Through The Grapewine

Many agile methods miss to address the issue of portfolio management. This is not such a surprise as many organizations miss to address the issue all together. Agile thinking pretty much assumes that the only intelligent way is to work on a single project at the time, and that projects should face delivery or cancellation as soon as possible. After all having too many projects in the inventory is one of the main sources of waste. While all this is very much true, the life on planet earth is sometimes somewhat different. The impression "I heard it through the grapewine" is made famous by Marvin Gaye. The impression means information traveling through gossip and rumor. In some organizations this seems to be the primary channel for portfolio management.

Team is defined as group of people with shared objectives and goals (term "team" is misused a lot!). Teams become superior over individuals when they share these goals and thus have a common direction. Agile methods take advantage of this phenomenon by planning for short enough increments and gaining commitment by having estimates given by the people doing the actual work. This is called time-boxing.

When a team in large organization is piloting agile methods it is usually working as a shadow system parallel to institutional system in the organization. In this case the above mentioned "I heard it through the grapewine" -effect can be very harmful. Each team member will have several inputs about the future of the current project, as well as information about the future projects. Along the current increment short notice invitations to future project meetings will arrive and so on. This makes agile planning unmotivating and even barking mad. Of course the same applies to any method of planning under these conditions. However time-boxing would be perfect tool against "I heard it through the grapewine" -effect, allowing organizations to harness the power of small teams with shared goals and direction. DeMarco calls these teams Jelled (DeMarco and Lister, 1999). If new announcements and meeting invitations are only made public in the same natural rhythm as the team already operates, their negative effect to Jell would be minimized. Who is responsible for this rhythm? Top management, middle management, project management, line management, team leadership? I think this needs to be considered independently for each case, but it is a joint responsibility and everyone should understand the reasoning.

"Increment duration depends on how long you can keep the change away". Allowing rumors and gossip make this a very short period.

The Uneducated Industry

In many cases professional embedded developer has never attended a single programming course at any level. Further she may have never opened a book on even the most fundamental programming practices. She does not read professional magazines. Embedded programmers are often electronics graduates. Even more scary is that the same applies to managers of embedded developers. They have never attended a single programming course at any level, NOR any managing course at any level. Embedded development department managers have just become managers because they have earlier been successful in programming (and designing the HW) with those 1kB micros solo virtuoso and ad-hoc.

Imagine that. In the past years we have reached an era where a light switch has more bit crunching power than Commodore 64. In few years a key chain, a pen or your T-shirt may out power the old 64. The complexity of these devices is exploded by the communication capabilities which enables distributed intelligence. These increasingly complex systems need to be developed within ever shorter time-to-market by team(s) of embedded software developers co-designing the system with teams from other disciplines. Members of such teams need to have proper skills for development, but even more challenging is that developers and their managers need to have social skills in order to keep the whole development ecosystem fit.

These skills are now been teached in programming and management departments, books and articles, and then tuned and learned within methodologies like agile development. However getting these next generation people into industry is a slow process.

It is essential to teach the existing people in the industry to learn.

Norm Day at the Office

Back at work, and running a orientation week before we begin a 3 month release plan, which aims at delivering the first possible release candidate for an embedded product family. We chose a meeting cabinet that we found out to be available.






Fist we checked it up. It seemed to be ok.









We could not find a projector in the premises, so we had to settle for Meeting Charts provided by our friends at 3M. Good news; no .PPT's.

Survey Shows: Agile Champions Live Higher

There has been survey results published at Agile 2006 conference. Pete Behrens has written a nice summary. The survey was about agile adaptation and was run by VersionOne. As VersionOne makes their business in agile scope, the data sample may be "a bit" biased. Even while I do not put much emphasis on the result that 75% of companies deploy agile methods I do see value in other finding:

"The internal agile champion is moving up in the organization to VP from team leader two years ago."

Agile methods have so far been largely deployed by a younger champion engineer in an organization that has gotten frustrated with the dysfunctioning methods. This data may show that agile development is moving directly higher from the trenches. I just hope that the values do not get twisted on their way up. The original ones work so nicely....

Agile Methods and Firmware Development

I made my paper "Agile Methods and Firmware Development" available. You can find a link to the paper from the sidebar, under "My Papers". This is my initial review and analysis of different software lifecycle models and their applicability to firmware development as part of Ph.D. studies. Paper is dated a few years back. One might say that the current journey started upon writing this paper.

Abstract— The size and complexity of software continues to grow
at a steady pace. This is also true for software embedded in our
everyday electronics, which we have called simple devices. The
term firmware is used to describe the low-level software in
embedded systems. It may even be hard to divide firmware and
actual hardware. Software development for such a target has
special characteristics such as a culture of hacking, small teams
and multiple hats, co-design issues, one-time designs, correctness
and robustness requirements, lack of tools and unconventional
customers. Software process models have been studied also in
this environment to ease the pain of developing more complex
systems. I introduce four currently used methods to develop
firmware; build-and-fix, waterfall, ROPES and RUP SE.

Agile methods are getting a lot of attention in the software
development community at the moment. I review the agile
methods which are most documented. The suitability of these to
firmware development is evaluated. It is also analyzed whether
firmware development could benefit from agile methods.

It is shown that agile methods are not the new cure-all solution
to firmware development, but they are applicable. Their full use
needs modification and innovative thinking. It is, however, shown
that firmware development can surely benefit from the usage of
agile methods.

embUnit Translators for CruiseControl

Back home and time to get tech oriented again. Our CruiseControl and unit test practices are proceeding at steady pace, so I thought I will make our XSL translators for embUnit unit test framework public in this forum at this point. You can find them here (unittests.xsl and testdetails.xsl). If you are an embedded programmer like me, and do not currently know much about CruiseControl, and/or XML, I'll try to briefly explain how to set it up. If there is enough interest I will make a more detailed intro to it.

You need at least to have following installed:

1. embUnit unit test framework and compiled libraries
2. Java SDK. Remember to set JAVA_HOME environment variable to point to install dir.
3. CruiseControl, of course
4. Make tools. I recommend for Windows users the whole Cygwin installation to cut down the hassle

Now you can run the Cruisecontrol Ant built Java and JUnit sample, but before you can start cruising with your embedded C project with Make and embUnit framework you need to at least do following modifications:

1. Build a simple C project which has makefile for automated build. Greate a CVS project and check your project out to your build server for CruiseControl (use Subversion if you are ahead of us, but you are on your own with this). Look at Driving on CruiseControl articles by Lasse Koskela for help.
2. Modify our config.xml to suite your project (see my earlier post about it) and replace the file in CruiseControl root
3. Replace unittests.xsl and testdetails.xsl files in CruiseControl folder ./webapps/cruisecontrol/xsl.
4. Launch CruiseControl and start cruising...

And here are some screenshots on how it looks.

Here a test fails (purely for demonstration purposes of course). Details of failed test are displayed below.








Details of all tests are also available.









After a brief moment everything is in order again. I used the green bar to signal this.

Battery Charging and Reflection

Yesterday evening we arrived to Lappeenranta and lake Saimaa. I have to say it is a beautiful scenario for battery charging. Even while the city was offering a festival (in finnish), I decided to take a day off from beer drinking and - well - read about change.

My book of choise, Fearless Change, by Ph.D's Linda Rising and Mary Lynn Manns, is of great value for anyone trying to get innovation diffused in organization. The book describes 48 patterns for driving and sustaining change. Many of the patterns we have found out during our journey. Among them a pattern "Time for Reflection" in which doctors recommend:

To learn from the past, take time at regular intervals to evaluate what is working well and what should be done differently.

This is excatly what we need to do again after the vacation.

Small Countries Seem to be Aggressively Agile

Google trends has been used to map agile development interest in different countries. Jonathan Cogley started it with a search for 'How Agile is Your Country'. Dave Nicolette went on with revisited search term set, and got this google trend. By choosing the regions you find (at least I did at the time) Finland in 10th place. So it is no wonder agile conferences in Finland are packed.

Prisoners of Our Habits - Act Four

Prisoners of Our Habits. Act three. Executive manager enters.

Executive manager is reading an email from project manager
"Dear Sir, I overheard this developer, John Smith, saying
his technical solution may cause $1.00 increase
in bill-of-material."

EXEC'MNG: -"Stop the press. This John Smith starts
immediately to investigate alternative solutions,
what ever they are."

Product under development may have hundreds of relationships. Tradeoffs between these relationships are being made in hourly basis. When one of these issues gets taken away from the context, it may seem like a big deal. The big deal in fact is when a bad decision (based on bad information) like one above is made the project environment, a complex adaptive system (CAS), gets an unexpected input. What happens in CAS is totally unpredictable. Typically the system is pushed into chaos and disorder. This is why agile development blocks these inputs from the team for a certain time, and opens the window for these new inputs only frequently enough. Command and control is traditional way of managing projects. Developers have not been encouraged to self-organize traditionally.

And old habits are slow to die.

I know paradigm shift is extremely slow, and so does James Shore (see his Change Diary). I'll be on vacation for couple of weeks and after that it would be perfect time to reflect and adjust the course of this agile journey. I just hope couple of weeks is enough for the old batteries.

Prisoners of Our Habits - Act Three

Prisoners of Our Habits. Act three. A manager enters.

DEV -"We should try to learn new ways of doing thingies 
      around here. We could participate in action research
      program, you know?"

MNG -"Hmm, That sounds like a lot of money. We could buy 
      a lot of chairs in two-day course for that. Wouldn't 
      that be great?"

...

There is a huge difference between being teached and learning. To start with there is a difference of acquiring existing knowledge versus creating new knowledge. Further there is a huge gap between action learning and a static content course which may have zero contextual relation with developer's current situation. I have seen the affect of these courses, and it does not impress me. It is just like many other mental models in this industry; a pretty model without connection to reality. But hey, traditionally developers have not taken part in action learning.

And old habits are slow to die.

Prisoners of Our Habits - Act Two

Prisoners of Our Habits. Act two. Project manager enters.

PM:   -"You need to focus on project's
        paper deliverables for the next two months"

DEV1: -"Which papers you mean? For whom they are,
        and for what purpose?"

PM:   -"I do not know, but reviewers know all the
        buzzwords, so my ass needs to be covered."

DEV1: -"Eh, That does not sound so valuable and motivating?"

PM:   -"It's not supposed to be. I fill in dashboards
        which no one reads. You should have no problem
        creating papers with no value."

...

The above is not a Dilbert strip. It is status quo in many development projects. Dashboards illustrating Gannts, PERTTs etc. are what we traditionally use for project management, and paper documents are what we traditionally use for progress monitoring and project quality assurance. So what the hell is wrong with that you ask? Again, it is a nice model, I really do like it. It has only one flaw; it does not hold in reality.

Then again working prototype, incremental release plan, Sprint backlog, or burndown chart are not used traditionally.

And old habits are slow to die.

Prisoners of Our Habits - Act One

I have been describing continuous integration for an embedded firmware project targeting a low-end microcontoller. Based on discussions I see a need for pointing out something about my view;

I do not mean that you need to apply continuous integration, unit tests running on both the host and target, etc. if you are writing a 200 LOC blinking LED application for 8-pin microcontroller for fun. This is a job for Solo-Virtuoso, and methods should be alike. The project I'm talking about, while targeting the low-end microcontrollers, will develop tens of thousands of lines of C source and will eventually be compiled into ten or so configurations for mass production. The configurations have some special areas where expertise is needed, like power management and wireless RF technology. This type of project cannot fit the Solo-Virtuoso category. Instead it requires a collaborative group of people - a job for an agile team and this team should adapt these practices.

Purely based on my biased participant observation, the trend of firmware projects is towards bigger and more complex so we are going to need these practices. The problem in firmware domain is that former electrical engineers (now programmers, or even worse, managers) are not willing to see this increase in complexity, but remain prisoners of their old habits. They are still in denial. My recent experiences manifest this very thing.

Prisoners of Our Habits. Act one. Programmer enters.

A colleague following our agile journey mentioned that based on his observations of daily Scrum meetings and collaboration we continuously run into problems caused by someone changing an interface and all of a sudden none of the configurations work. His obvious solution for this was; "there is a need for up-front designed interface contracts between modules". Problem solved, eh? We do not have full set of requirements, nor the freezed hardware architecture. For me iron-bound interface contracts seem pretty far fetched objective, but then again, it's just me. Still I would be amazed (but happily so) to find this wizard pulling out a comprehensive up-front architecture for us. Oh, and preferably delivered as a PowerPoint presentation I might add. If you happen to know this guy, ask him to drop me a line. I could use a good laugh.

I see these discussions about interfaces as a solution, not the symptom. It is good to see team members having courage to change things, to refactor when opportunity arises. These conflicts usually get solved easily after the daily Scrum at latest, so to me these are just short architecture meetings. In addition there is a huge difference compared to normal high ceremony architecture meeting: decisions get made and implemented immediately and feedback is available the next day. I have witnessed this working extremely well, and fast, even with programmers with junior skill level. This is just not the way it has been done traditionally.

Old habits die slowly.

Agile Business Conference - Program Draft Available

There is a program draft available for this year's Agile Business Conference in London. I have attended this conference for two years and I can recommend it. It is organized by DSDM Consortium and the title has word 'business' in it, but the agenda is about agile in general. This years highlight for me seems to be 'Agile and Embedded Systems' (Pekka Abrahamsson) . I'm also very interested in presentation by Mike Criffiths - 'Utilising Agile Methods Alongside the PMBOK Guide'. I see very little value in this balancing, but this is something we currently often need to live with and nevertheless we should always remember and reflect where we are coming from. This helps us to understand where we would like to go next (post-agilism).

Again interesting to see keynotes from Kent Beck, Sean Hanly, David Taylor and Polyanna Pixton. I really hope I'll be able to make it! If you are planning to be there, and are interested in agile [product] development, drop me a line and we will catch up.

Objects and Firmware - Inheritance

This is the third and last post (see earlier basic encapsulation and polymorphism) about object based firmware programming. This is about single inheritance, another important object oriented principle. This is also pretty much as far as I would go with the HW currently available for our embedded projects, low-end 8-bit microcontrollers. Anything further would require even more use of pointers, including function pointers, and I'm not really that comfortable with their heavy use.



The sample code implements a simple Level object type which has only two members, level and currentState. The first one holds a threshold value, the latter indicates wether the latest sample was below the level (currentState = 0), or equal/above (currentState = 1).

Next we have used Level as a parent for another object type, LevelWithOffset. The new object has the Level type as its first member, named super. This gives us possibility to use pointers to both of the object types in similar way. LevelWithOffset object type has an additional offset member. Value of offset is added to set level. In the example code we have two objects, one simple Level and one LevelWithOffset. Constructor is called for both setting the level to 2. The LevelWithOffset object is then further adjusted by setting the offset also to 2.


The main loop calls the notify method for both objects with values from 0 to 4 and outputs the state of both objects. We can see from the test run that the other object switches the state only after the sample is increased by 2 more steps (effect of additional offset).

You can go a long way with this object stuff in C. If you are interested then check out the papers listes below and start working on your skills and creating your own opinion.



Miro Samek, Portable Inheritance and Polymorphism in C, ESP (pdf)
Ron Kreymborg, Single Inheritance Class in C, Dr. Dobbs Journal
Matthew Curreri, Object-Oriented C: Greating Foundation Classes, ESP (part2)

CruiseControl Configuration for GNU Make

I promised to share the CruiseControl config.xml file we use. Here you go, it is not much of a contribution, but I hope it helps some of you jumping into cruising mode. Our configuration file uses the Exec builder for GNU Make automated builds. Typical examples use Ant in Java environment.



For additional information:

• Martin Fowler, article on Continuous Integration
• Lasse Koskela, Driving on CruiseControl Part1 and Part2
• CruiseControl Configuration Reference

See it in action in my earlier post.

Agile Methodology is not a Religion, it's Just a Passion

I spent three days in Oulu, Finland, attending the XP2006 conference. I have attended other agile conferences, but this was the first time I attended this conference, which was the 7th in series of XP conferences. I have to say I was impressed. The conference is about practitioners to the degree that you really get to sense the belonging in a community of practice. Authors of best selling books, key note speakers, most successful agile consultants in the world, signers of the original agile manifesto, other practitioners from all around the world, all open for discussion on this shared passion, without trying to close a contract before telling more.


I saw that strong passion of doing things right as the combining force of this community.

(Some) People are passionate about religion, too. This is why I think agile thinking is sometimes compared to religion, and comparison of agile and waterfall like methods is said to be a religious war. I saw nothing like religion (at least in a bad sense) in this conference. Instead lots of presentations and hallway talks alike concentrated on balancing methods, applying best from the both worlds, taking existing organization culture into count etc. I do not see this resembling fundamentalist religion at all, do you?



I have to say that in the early days of agile movement I sensed that strong black and white positioning as well. I participated in that. This I believe was the result of so strong survival anxiety after being so fed up with the situation and just wanting to get rid of the old ways of doing stuff all at once. The brilliance of agile methods is manifested in what we are experiencing now. We use continuous retrospectives to improve the method and to adapt it to changing situation. This is possible because practitioners are passionate and enthusiast and put lot of effort into keeping the method fit and healthy.

Sean Hanly from exoftware pointed out in his key note that we must be careful not the end up in the process graveyard with the famous "failures" like RUP, CMM(I), Waterfall and numerous others... By saying this Sean meant that the interpretation of original methodology is in danger to get twisted when number of interpreters increase. When agile development is moving into mainstream we are going to see a lot more practitioners without this passion, but just making the use of a new tool.

There are photos from the conference...

Comprehensive Documentation for Riding a Bike is Difficult

Nonaka and Takeuchi (1995) defined knowledge as explicit and tacit. This view is of course shared with other researchers in organizational learning.

I posted about agile development emphasizing Enable Future Work -documents (as described by James Shore). That holds, but it still does not make creating quality documents easy.

Day after day engineers will arrive at work and launch MS Word alone because they need to document their work for future use. Unfortunately some will open the same tool for documenting what they are planning to do, but this is Get Work Done- documentation (as described by Jim Shore), and we are not talking about that now. At the exact same time same number of engineers will get a copy of such a document to start working based on it. By the lunch time most of them have realized the same old thing; "Just words, not even grasp of what I would have needed, half a day wasted, thank you very much."

I'm reading the Communities of Practice, by Etienne Wegner, for the second time. This time I understand a lot more, but of course I am still not fully getting it. However Etienne gives good example of two sides of knowledge, explicit and tacit. Mr. Wenger used skill of riding a bike as an example. Most of us master this skill, and we can explain it - even fairly well - in paper;






"Push some initial speed, start pedaling and just steer where ever you want to go. "






This is the explicit part, but there is one fundamental ingredient missing in the above document. How do you keep the balance? This is the tacit knowledge part, and it is extremely hard to put on paper. I give you few minutes to think about it...

OK, I tried it also while reading the book, and it reminded me about something we all have experienced.

Agile methods promote simple design to the degree that documentation is not necessary. In embedded world where you want to minimize the material cost you often end up in software and hardware design which are based on intuition. This tacit information is difficult to put on paper, just like the balancing bit of riding a bike. How many times you have been explaining your design to a peer until a bit of detail which has no meaning to you opens the design to your colleague? These obvious facts seldom get identified and reported in paper document created by the original designer individually. As a consequence the document has little or no value. This was manifested with our three-day faceoffs recently. This has nothing to do with agile development methods, since they promote simple design and light documentation?

Quite the contrary. Agile teams work well in preserving and sharing tacit knowledge among members. Daily meetings, retrospective and pair programming(co-design/debugging) are natural techniques for this. Additionally the team commits to deliver work (software/product) which is "done". Done means delivered with just enough and barely sufficient documentation. Agile team should have also shared commitment to deliver necessary documentation, which would enable them to create the documentation in collaborative way. This type of activity is valued in RaPiD7 (Ph.D. pdf), an agile method for creating documents developed at Nokia. Of course the best thing would be to have the "customer" of a document present during the creation work (or that the "customer" writes the document as Vasco Duarte proposes), but this customer for Enable Future Work -documentation is often not available at that point.

The next best thing might just be a collective team responsibility with other project stakeholders and to give it your best shot from multiple perspectives instead of single unmotivated view.

embUnit has some additional features

I posted about my options for proceeding with CruiseControl. A while ago I already wrote a python script which ran the unit tests, and interpreted the results into XML following the JUnit schema. This seemed to be working, but there was one major problem. embUnit output did not include any detail about successfull tests. So I started looking my options again. I took a look at CUnit. It has XML output for its own translator, but this of course does not follow JUnit schema either, so I was not getting any closer.

Then while browsing through embUnit source I made an interesting discovery. The code for embUnit library does not require standard C libraries. This I believe is the embedded part of it. In the distribution there is however additional source code for two different user interfaces; 1) text and 2) XML. These can be compiled into another library (the basic libary comes as binary with distribution), which can be used for host run unit tests. Two libraries can be tied together with nice function pointer usage, similar to what I tried to explain earlier. Everything is coming together beatifully. Example of generated XML report is below. This again does not of course follow the JUnit schema, so I will continue by modifying the CruiseControl XSL translator for it.



To be cont'd...

Celebrating the Complexity - of Life

I just finished reading Stephen Denning book 'The Springboard: How Storytelling Ignites Action in Knowledge-Era Organizations". Easy to read, entertaining material, explaining the power of light touch of storytelling having huge impact. Of course one has to admit that just by reading a book, one won't become the most successful change agent.

I liked the phrasing in the last page:

'...Living instrumentally to achieve explicit fixed objectives is less important than living moment by moment, day by day, appreciating difficulties as much as success. It is a matter of letting go of the urge to control, and the fear that goes with it - learning that the world has the capacity to organize itself, recognizing that managing includes catalyzing this capacity, as well as sparking, creating, unifying, generating emergent truths, celebrating the complexity, the fuzziness and the messiness of living, all the time relishing the sense that almost everything one thinks or knows about the world has turned out to be fake." (Denning, 2001)

I wonder if storytelling is really all it takes. We all know what storyteller Kent Beck achieved with his book.

Original '51 is going to retire

Intel will stop producing embedded processors, including 8051, in 2007. Read what Jack Ganssle wrote about it.

Reserving the Right for Technical Excellence

"...and then the administravite party decided it's time to spend a month repairing and polishing documents, as they seem to be demanded. " I had a nice discussion with two researchers a week ago who have experience in studying NPD processes, also globally. The above phrase is still common. They shared this with a weird smile on their faces. I feel panic closing.

Which documents? For whom? For what purpose? These in my opinion (Vasco Duarte seems to agree) eligible questions all remain unanswered to the date. Of course the idea behind this madness is to offer a project steering committee possibility to control the project (once a year, I'm not even going to get into this). Most of us know that - at best - this is only an illusion of control. Yet, repeatedly the rare resources of a project are streered towards creating this illusion. At the same time people on this same planet are talking about innovative knowledge creating organizations, rapid time to markets etc. Yet some New Product Development can afford spending a month putting 50% obvious facts and 50% of nonsense into form of Word document at the early stages of development?

What do they teach you in courses dealing with panic?
Right, STOP-THINK-ACT

I felt panic, I stopped and I been doing some thinking. What is wrong with this environment? What is there to do? I could apply my (and pretty much majority of developers) favorite "Yes, it is 80% done, and will be complete in - yhm- couple of weeks" -continuity of answers (of course not having a clue what the task in question is). This is not making me laugh anymore, so I propably won't.

Projects that succeed under this type of project management typically have developers which are in "Don't Ask, Don't Tell" mode. I did this for a while. I'm not amused anymore. I do not want to keep up the smoke cover-up. **I want it visible**
After all I think I really need to reserve my right for technical excellence without cover-ups. Always.

Dr. Cooper has acknowledged this and updated his Stage-Gate with seven F's and other guidelines for adapting the model to be more flexible. Phased process models, like waterfall, and early Stage-Gate, survived for 50 years. Transition is ongoing, but all too many organizations remain in denial.

Suddenly at 35 I feel pretty tired, weak and old...

The Seven Principles to Success Revealed

Dr. Robert G. Cooper, the author who introduced the Stage-Gate process model first in 1988, published a Working Paper No.23 recently. In the paper he lists seven New Product Development (NPD) principles that high productivity businesses practice. Data shows that productivity in top organizations can be five times what it is in average company.

The seven principles:

1. Customer focused
2. Front-end loading
3. Spiral development
4. A holistic approach
5. Metrics, accountability, and continuous improvement
6. Focus and effective portfolio management
7. A lean, scalable, and adaptable process

Where have I heard this before?

Get Work Done -Documentation Concidered Harmfull

James Shore writes about two kinds of documentation and agile development.

All of us who have tried to convince someone to value agile methods over plan-driven processes have encountered the phrase "but we need documentation". I really saw the light while reading Mr. Shore's article. It has been clear to me, but division of documentation into these two gatecories gave me the simple structure I have been looking for;

1. Get Work Done
2. Enable Future Work

The following situation is not too uncommon. An embedded system project is getting closer to a process gate. All of a sudden it is time to write documentation because documents are "demanded by the process". This is ridiculous! Process does not demand or want anything, even less likely it is going to NEED anything. Dominating parts in a project system are the people, not the process. What do they NEED in order to progress effectively? That's correct from the back row - communication. Everyone agrees with that, but why is communication so strongly associated with paper document in engineering, or project management? Are we still paying the price of the stereo type created decades ago, that engineers can not communicate? If so, what makes project managers think that engineers could write?

So back to the project. Project has built working prototypes with incremental development and short time-boxing in cross-disciplined engineering teams. Electronic schematics, part lists, price estimates, firmware, mechanical drawings, thermal management results, power handling results, EMC data etc. are available as natural outcome of experimenting. What is missing if we need to COMMUNICATE the project, its status, the risk etc.?

I heard that! "Documentation" you whispered. This is what the process demands, and for some werd reason this is interpreted as Word documentation. So, the team gets into this unproductive mode and writes Word ducumentation at velocity of 2pg/month. Gate passed, everyone happy?

Dead wrong. The first downside is that now the documentation is done. We proceed with prototypes and create more new knowledge on detailed level. This is the spearhead of new knowledge created in this project. Unfortunately since time is wasted at early stages we will be in a hurry towards the end of the project. This means documentation at this crucial stage, at the end, has less focus. Not to mention the frustrated engineers that have already documented a lot without any feedback (the dinosaur process just swallows the papers). It is harder to convince them about the importance of documentation at this later stage. This documentation, which James Shore called Enable Future Work -documentation, remains missing in many so called traditional projects.

The second downside comes from the fact that we actually now have the Get Work Done documentation, but not the high quality Enable Future Work -documentation. This documentation is passed to the next project team as a starting point for their work. This documentation contains obvious facts, which are worthless. More importantly it has flaws and speculations which were found wrong during further experimentation. Unfortunately this new information typically never gets updated into these documents. So there is lots of information that is not true and does not go one-to-one with the actual final design. The new team now spends more time figuring out whether to believe the document or the design. That is if they are not experienced enough to always go with the actual design.

My Three Day Faceoffs -post describes the power of face-to-face communication over paper document in practice.

Get Work Done -(Word) documentation is concidered harmfull.

Agile methods fit firmware as well

David J. Andersson informs us that HP has shown significant productivity gain from agile management techniques in firmware development. There is not much detail available, but here is the short story...

Agile Boot Camp

Pygmalion effect (or self-fullfilling prophecy) is a phenomenon according which the behaviour of group is what is expected. The phenomenon was demonstrated in a study of military training in (Eden, D. Pygmalion without Interpersonal Contrast Effects: Whole Group Gain from Raising Management Expectations, Journal of Applied Psychology, 75, 1990). In the study two teams were observed during their training period. Platoon leaders of one group were told that their men were above averige, while leaders of another group received no information. In reality the two groups of men were equal in potential. However the group that was said to be above average outperformed the other group at the end of the training period.

This supports my belief and observations that to some degree a person has the potential that the leader is willing to see in her.

Based on these ideas I challenge the need for above average (or level 3-5) developers in agile development. Agile framework works also very well as a training camp for junior developers. After all if we said that all developers in agile environment need to be above average, where would all 50,0001% of developers at and below the average go? To marketing? No can do, we need agile customers as well! In these settings the scrummaster's (or XP coach's, or...) role just needs to extend to facilitate this type of learning. Maybe she needs to be more technical than with senior developers, being able to guide with simple design, unit testing, refactoring etc., but it is essential to avoid microleading and accept failure as a method of learning. It is also important to have eyes open for the existing knowledge of junior developers. In most cases it's there enabling two-way learning experience.

Scrum Gives More Time to Surf

"It [Scrum] gives me more time to surf...they bought it...it's great". I think I'm still missing some of the fundamental aspects of Scrum framework. Check if you got it right!

The Firmware "Hassle" with TDD - #3

This post will end theoretical justification of firmware TDD.

Why it would be impossible, or worthless?
"In embedded system we may not have anything to signal the test results with."

Why would it still be worth it, and what kind of hassle is there?
We have to, I seem to repeat myself, distinguish two things in discussion of unit tests in embedded (firmware) system projetcs:

1. host run
2. target run

1. I have been mostly writing about running the tests on host (PC). We are using Cygwin environment. We compile our unit tests with gcc and tests are written on top of embUnit framework. Reporting is done both on screen and as XML.

This article at Object Mentor by James Grenning defines an embedded TDD cycle. This demonstrates the fact that embedded programming is the extreme end of programming. We not only have to worry about all the normal problems in software engineering, but we need to fight these challenges with limited processor resources, poor tools, hard real-time deadlines, stakeholders from other engineering disciplines, usually without formal computer science education etc. This means that having unit test suite alone is not enough for testing the embedded system. The other arrows in embedded TDD cycle illustrate this.

As can be seen in Grenning's article unit testing on host is an important corner piece in the puzzle. Doing TDD has so many other benefits that come as a side products that it is worth the hassle. Unit testing the modules will help you to write clear consistent interfaces and enforce good programming practices.

2. Keeping the modules small and less complex it should not be too much work to port the tests to be run at target environment as well. Nancy has done some work on running the tests in target. In microcontrollers today it is easy to find some kind of serial port to report the results back to PC to be further formatted. Even if you do not have HW pheripheral on board, it's not a big deal to write serial communication driver with bit-banging. This however always needs some arrangements (getting the code to target, resetting the target etc.) In my (limited) experience these little arrangements may just be too much of a barrier to cause the tests not to be run by developers. At least not regularly enough for TDD. Fully automating this at high level is a bit difficult. It needs to be remembered that this would still only partly resolve the problem since the unit test code seldom fits the target memory with production code.

This said, when I get the unit testing on host really going, I will focus on running the same tests on target - and automated. There just is no rush, you cannot get to nirvana over night.

Nancy Van Schooenderwoert will give a presentation at ESC Boston 2006 about their CATS C unittest framework. That should be interesting.