Weekly Delivery of Firmware

Yesterday we finished a six week project developing technology platform for User Interface innovation using some practices familiar from agile methods. The project delivered two different configurations of home automation user interface device; one for wall mounting and one for remote control. Some facts:

• Vague or missing requirements
• New 8-bit microcontroller
• Lots of new sensor technology
• New embedded firmware team of 3+1, including one consultant (in house)
• Some drivers developed by outsourcing
• Final project consisted of 20KLOC of firmware C source (bad meter, but...)

Some findings after initial reflection:

• Weekly delivery (not release) of firmware is possible, at least in some cases.
• Short term goals and daily Scrum meeting effectively work as motivators, and enforce full learning and getting things done done.
• There is not small enough project to not use version control and continuous integration (we had no way to have common version control with outsourced team: large org, IT, you know the drill...)
• HW evolved from microcontroller starter kit to final electronics via several bread board mocks and one halfway PCB prototype. This causes extra assembly work when looked at the surface, but this overhead is by far overcome by enabling the concurrent engineering instead of waiting for final version.

The last point holds only if all the disciplines work as One Team.

We did not have cross-functional team as we did in earlier larger experiment. Schematics, PCB layout and mechanics were developed concurrent, but they were not part of the team - nor did they follow any of so called agile practices. Firmware planning was tried to be synchronized with deliveries from other disciplines. In several occasions these deliveries were delayed, and a common goal needed for a team to actually be a team was somewhat missing. This is where I would like to do things differently if I was to do it all over again.

Occasionally we heard sentences like "If everything goes smoothly we will be ready on Tuesday, but we will be ready at least on Thursday". At this year's Agile Business Conference in London David Taylor (The Naked Leader) said "Don't have a plan B because you will always achieve it." Just guess if the delivery was on Tuesday or Thursday - or Friday?

The whole project was started by extremely vague idea. In few hours we worked out initial backlog and a delivery (release) plan based on simple themes. From that point on plan, design and requirements emerged during the 6 weeks nicely synchronized with weekly demonstration of prototype and new planning session. This experiment supports the belief that good people can work this way, and that we truly live the era of enlightened experimentation. We believe that we could not have performed any better with thorough analysis, design, and specification phases - at least not to cover the cost of them. During the development there were several points where it was not possible to proceed as we had thought. A comprehensive plan based on these initial assumptions would have failed miserably, and the time creating them would have been wasted. In this project the whole team worked out an alternative solution, adjusted the next week's plan, and the project was fine...

I will write an experience report and hopefully make it available here.

Skunk Work Is Not A Longterm Solution

In her "The Agile/Waterfall Cooperative (pdf)" presentation Michele Sliger summaries three modes in which agile team can work in a waterfall organization; SWAT team, Skunk Work, and stealth agile. I agree that applying just some agile practices can benefit your work. This is pretty much what we are doing at the moment, but I do not see this as sufficient long term solution - if it is only in team's sphere or just above. All three forms of team work can of course be used. SWAT teams can go parallel even in more wide spread agile organization. Skunk work probably, and hopefully, always exists in large organization - it's fun. If you are working as stealth agile, and avoiding radars by not communicating to the top management, you are not going to be able to fully harness the power. You need to inform top management that you are doing things differently and communicate your need for changes. They probably will not get it, but what they do get - is success. So it is fine and in most cases necessary to fly low at first, but after certain point it is not satisfying anymore to work as a shadow system, because you know you are constrained by the legitimacy structures, and these structures are holding back you and your learning.

The rest of the presentation has also good ideas, and based on my experience these ideas hold in reality as well.

SWAT team – outside the domain of the process police, brought in on projects that are in trouble. Agility is prized in this situation. They take over however – they don’t work in a cooperative mode.

Skunk Works - “a small group of experts who drop out of the mainstream company operations in order to develop some experimental technology or new application in secrecy or at speed, unhampered by bureaucracy or the strict application of regulations.” Sanctioned and protected by management, these teams are rare. This formation however, prevents the headaches involved in org structure, project approval hoops, portfolio metrics and management, budget calls, and other bureaucratic nightmares.

Stealth Agile – As Jim Highsmith once said in response to a question about how to sell agile to upper management – don’t. “They don’t know what you’re doing anyway.” Do as much agile as you can where you can. Some benefit is better than none at all.

Link: Scrum and XP from the Trenches

No matter if you are developer, PM, or CIO and you don't know what agile/Scrum/XP is, you are thinking about applying, or you are already practicing - you should check out "Scrum and XP from the Trenches (pdf)" report from Henrik Kniberg. Henrik describes how they applied Scrum and XP practices in real life and presents the adaptation needed in this situation. Excellent Writing!

main_theOneAfterFinal.c

Today I happened to see a file name with word 'new' on it on a presenter's computer. I was slightly amused because of my history. I can still see the old version naming convention in many "shared team folders" in our server. It goes something like this:

original
new
latest
final

I have been using version control system since my first university years (some fifteen years ago) and throughout my professional life. First it was just an advanced XCOPY method putting released versions of one man projects into the system, but about three years it has been collective code ownership and at least daily update-commit (we use CVS) cycles from the beginning of development. Lately we have adapted continuous integration and automated regression (confirmation) testing as you know. Currently we are looking for ways to add some automated acceptance and system testing as well.

During the past couple of years I have had a pleasure to work with software developed outside our own company. Quality of source varies a lot of course, but it is striking that version control typically is not even mentioned. In some occasions lately the software was delivered as zip'd folder with a version number in folder name - at that point I was happily surprised by that! Embedded firmware domain - based on my biased observations - walks some 10 years behind the mainstream software industry. The wide lack of basic discipline, version control, proves the point. I'm glad that I work in a department where I can say that developers would never go back working without version control.

I found this Yahoo group message - nostalgic.

The Lenghty Debugging Phase

To give embedded firmware TDD a kick start we have had two sessions with a colleague. During these sessions we have created scripts to automate the unit test structure and build process and discussed how the make as thin wrappers for HW as possible. However we learned also other valuable lessions during these sessions. Lesson number one is the "debugging phase" being so strongly in our domain culture. We often hear a firmware developer saying, "well, it takes couple of days to implement it, but you can never tell how long the debugging is going to take". Of course debugging phase is where all the firmware heros truly shine. That's another unfortunate culture issue in firmware development.

In our first session we ran into problem of not getting one test running. Immediately we started debugging. Printf's fly in here and there (as an advanced host run I/O twist). The change to this mode was so rapid, that neither of us realized what is going on until later. After some 30 minutes(!) we recognized that we are not moving, but just desperarately experimenting with printf's, and it hit us; we have not changed our behavior as we were supposed to. We should have picked a more simple thing to implement. When we realized we can not chew what we had chosen, we should have started all over again with simpler thing.

It is our belief that by doing TDD for firmware we should get rid of, or at least dramatically reduce, the debugging phase and thus create an atmosphere of success, and improve the reliability of estimates.

This type of sessions are great tools for learning. I strongly share Michael Harmer's opinion on The Training Course Scam.