Ken Schwaber on Quality of Code

Ken's presentation "A Canary in a Coal Mine" at Agile2006 conference is available via InfoQ. Ken talks about quality and the danger of Scrum teams droping quality issues to get higher velocity and to be sure to hit the date. He calls this a re-definition of "done", or "somewhat done". We have manifested an increasing technical debt in our iterative work. So in the last project we were putting more focus to acceptance testing inside each sprint and introduced something we called "cross-testing". This means that other pair needed to test the feature implemented by other pair in order to claim feature "done". Embedded firmware systems need a lot of manual acceptance testing (at least our current practice) and it is easy to get slobby testing your own work over and over again.

Was the technical debt there because of Scrum? Well, no - it was made visible by Scrum and the team was able to decide to do something about it early enough.

----

Ken's another speech about Scrum is available at Google Video. (You will find out that Scrum works with idiots as well)

I posted about Jeff Sutherland's Scrum speech earlier. It is also available via InfoQ.

XP2007 Conference Web Page

XP2007 web page is open. The conference will be held on June 18-22.2007 in Lake Como, Italy.

In the scope of conference is:

• Embedded software (e.g., SW/HW co-design) and Agile SW development

...among of course other interesting stuff. Hopefully we see some papers in this domain as well. I attended the conference this year in Oulu, Finland, and if next year's event is anything like that - I can highly recommend attending.

NPD in General With Agile and DSDM

Iterative and incremental development ideas have been presented for general new product development (NPD) for years. I'm interested in agile methods (as in software development context) because they give a philosophy behind and lots of ideas how to actually do this. It is of course also because of extremely active and powerfull community. Ideas of using these methods and practices in NPD in general have been presented by Jim Highsmith (Agile Project Management), Preston G. Smith (Developing Products in Half the Time), and many more. Both of above mentioned can be contacted via Cutter Consortium.

Now there is something new coming up. At London Agile Business Conference 2006 few weeks back the DSDM consortium talked about the future of the DSDM framework. DSDM version 4.2 has been open for online viewing since this year. They are going to publish a new version of framework in Spring 2007. This new version is supposed to be more general and not specifically targeted to just software development. It was also presented that DSDM can be seen as a wrapper for models like Scrum and XP for use in enterprise environment. It's a bit too structured to my tasting, but very well worth to check out when available. It may also become more open.

Risky Business

It has been identified that the biggest risks in project work concern management and marketing. Yet, in most environments we only concentrate on technology risks. Agile, incremental and iterative, development is also often assumed to be risky. By changing the way of thinking about risk, people would see that iterative risk management is a proactive way to tackle all aspects of risk. A very simple way.

1. Management risks - just fund the project for the time zone you are comfortable

2. Marketing risks - use the first possible release to test the waters

3. Technology risks - use the effort early to tackle any technological uncertainties that traditionally are left lurking until the (assumed) end of the project

The status quo practice is quite different. We take huge risks by starting a megaproject, for example estimated to 5 calendar years, based on a business case. Everyone knows that business case targeted five years from now is pure speculation. In the global markets it's a very expensive speculation by the way. Based on this speculation management needs to make a decision for funding a five year project. Their natural reaction to tackle the risk is to review the project mercilessly. We can be sure that changes to project requirements, technology and scope will happen. We have this heavy review practice going on making the project move even more slowly. More time means more changes, more changes mean more reviews - are you starting to get the picture?

Most of us see this everyday. How do we try to fix it? Naturally by reviewing. Because? That's what we allways do.

Refactoring for C

SlickEdit is a commercial compiler source editor that many people speak highly about. I thought I will give it a spin and downloaded the trial version. Their web page describes "cool features", but for some reason refactoring is not listed. Anyway the product has capabilities for refactoring several languages, and few quick refactorings (based on tagging instead of parsing) even for C; extract method, rename, modify parameter list, and replace literal with constant. At least for simple code they all seem to work. SlickEdit is also available as Eclipse plugin. I only tested the own IDE version.

You can find out the reason why there are not so many refactoring tools for C from MS and doctorate thesis by Alejandro Garrido. They are available via this site.

Web Page for Embedded Unit

Embedded Unit (embUnit) is an unit test framework for embedded C systems. It now has a simple web page with manual. It is not pretty, but it is something.

One Team, but just a team

One of the agile development practices is One Team, meaning that team as a whole takes the responsibility. When agile team is formed by a bottom-up initiative, it will soon face the problems of limited sphere. Most of the problems in projects are not related to design, technology or engineering practices. These are the only ones that the team is capable to fix quickly through iterations and reflective retrospectives. After fixing its own sphere, the team hits the wall when working in the shadow of a large organization. This can be a stressfull situation as everything is starting to be crystal clear.

Afterall One Team is just a team.

Last monday at Agile Seminar Helsinki Heimo Laukkanen presented that the whole organization needs to be agile in order to fully harness the power. He talked about IT governance, but even faster you will notice need for change in leadership styles, design review practices, personnel development, project planning (not plan), marketing involvement, and a lot more. Even while the agile development only in technical level may be rewarding, seeing clearly the dysfunction of standard organizational processes (so called best practices) and structures will hammer down the gained team spirit and motivation, eventually forcing it back to old ways of working.

Changing these processes and structures in a large organization may be impossible with bottom up approach. Large organization has a whole process department to take care of the process. It would take a complete paradigm shift to access this process. What makes this hard is that this department is not rewarded based on project success or revenue, but the success of current paradigm - from the administrative perspective. Admitting that a paradigm shift is actually needed might feel like a failure of the department.

This is one of the reasons why agile interventions often fail, the rest of the organization fails to learn fast enough.

Come Again, What's the Cost and Overhead?

I've been implementing object-based stuff in C for embedded firmware with passing an index parameter to an array of structs, like I tried to explain here. Yesterday I remembered an article about pointer-less C in Embedded System Design and thought I will give it a couple minutes more. The article gives some ideas how to make pointers safer to use and talks about the cost and benefit of using pointers. I'm not saying anything about the value of article itself, but I got curious. I took our simple switch debounce filter routine which is used a lot, and which is typically the only one that actually implements multiple objects at run time. It uses the me index as parameter for two functions. I quickly modified the code so that both functions take in a pointer to a struct instead of simple unsigned char. I compiled both versions for Microchip 16F876A (8-bit uC) and got surprised. The pointer version used over 30% less code memory than the index version, reducing the ROM image from above 300 words to below 200 words. I then compiled the source files to give me the assy listing, and below are the results that I found for a simple function that just sets some members of a struct. Because of werd mechanism of indirect addressing in PIC micros, the pointer is actually passed just as a 8-bit register. Word of caution; we have experienced problems when passing pointers and enabling optimization with Hi-Tech compilar for PIC micros, that is another reason why we have avoided this.

The lesson; while I do believe in intuition for great solutions, it is worth to challenge your beliefs every once in a while.

Pointer version:

    30  0764             _Cbutton_construct
    31                   ;button_ptr.c: 27: me->debounceCounter = 0;
    32                          
    33  0764  0084       movwf fsr
    34                            
    35  0765  1383       bcf 3,7
    36  0766  0180       clrf 0
    37                   ;button_ptr.c: 28: me->level = !1;
    38                          
    39                   ; _me assigned to ?a_Cbutton_construct+0
    40  0000             _Cbutton_construct$me set 
                         ?a_Cbutton_construct+0
    41                   ;_me stored from w
    42  0767  1283       bcf status,5
    43  0768  1303       bcf status,6 ;carry unused
    44  0769  00A8       movwf (((?a_Cbutton_construct+0)))
    45  076A  0A28       incf (((?a_Cbutton_construct+0))),w
    46  076B  0084       movwf fsr
    47                            
    48  076C  1383       bcf 3,7
    49  076D  0180       clrf 0

Index version:

    30  06F6             _Cbutton_construct
    31                   ;button.C: 27: theButton[meIndex].
                         debounceCounter = 0;
    32                          
    33                   global _theButton
    34                   ; _meIndex assigned to ?a_Cbutton_construct+0
    35  0000             _Cbutton_construct$meIndex set ?a_
                         Cbutton_construct+0
    36                   ;_meIndex stored from w
    37  06F6  1283       bcf status,5
    38  06F7  1303       bcf status,6 ;carry unused
    39  06F8  00A8       movwf (((?a_Cbutton_construct+0)))
    40  06F9  0828       movf (((?a_Cbutton_construct+0))),w
    41  06FA  00F0       movwf btemp
    42  06FB  1003       bcf status,0
    43  06FC  0DF0       rlf btemp
    44  06FD  1003       bcf status,0
    45  06FE  0D70       rlf btemp,w
    46  06FF  3E20       addlw ((_theButton))
    47  0700  0084       movwf fsr
    48                            
    49  0701  1383       bcf 3,7
    50  0702  0180       clrf 0
    51                   ;button.C: 28: theButton[meIndex].level = !1;
    52                          
    53  0703  0828       movf (((?a_Cbutton_construct+0))),w
    54  0704  00F0       movwf btemp
    55  0705  1003       bcf status,0
    56  0706  0DF0       rlf btemp
    57  0707  1003       bcf status,0
    58  0708  0D70       rlf btemp,w
    59  0709  00F0       movwf btemp
    60  070A  0A70       incf btemp,w
    61  070B  3E20       addlw ((_theButton))
    62  070C  0084       movwf fsr
    63                            
    64  070D  1383       bcf 3,7
    65  070E  0180       clrf 0

Code smells - just refactor

As we identified, agile development puts more demand to top quality source code than we (firmware developers) are used to. Actually agile development only makes the bad quality visible to larger crowds. This fact fairly quickly takes us searching for new practices like refactoring. You can find about it in Refactoring site, or better yet from Martin Fowler's book.

There is a nice table by Gene Garcia available as a summary of code smells and refactorings. Take a look, this stuff applies to firmware as well.

We Can See Clearly Now

The beauty of Scrum, or agile development in general, is that it gets everything laid down in front of everyone. Everything, everyone, good and bad.





1. Developers learn to understand what 'done' really means, and how much effort it takes to get things done. It demands discipline not to have technical, or administrative in large organization, debt increase increment after increment, but actually do all-at-once, in parallel. At the same time the satisfaction and enjoyment from accomplishing meaninfull work is immediately rewarding.

2. Stakeholders outside the team start to understand the unpredictable nature of new product development, but also the true capablities of reliable development. Customers understand that they do not need to ask for everything in order to get at least some. Common understanding of business value (benefit) starts to emerge between the customer and developers. Managers and project managers see the effect of meaningless deliverables and context switching. Managers start to understand just to stay out of the way and remove obstacles.

Trust begins to take form.

Agile Seminar Helsinki, Fall 2006

Title says it all. Check it out.

Jeff Sutherland on the Roots of Scrum

A presentation about the roots of Scrum is available via infoQ. A 60-minute presentation held at JAOO conference about everything and some behind the Scrum as we know it today from Dr. Jeff Sutherland, the original creator of the framework. Without the need to travel. Great.

Bubble, Bubble...

We have had our continuous integration server running for a while. At this point we thought that some not so serious step should be taken. It was time to implement the lava lamps.

Being a firmware shop we obviously could not go the normal way with X10 publisher. We used our own shutter relay prototype to drive the two lamps. The relay module is normally driven with I2C -bus, but we simply used the data and clock signals to transfer pulses from pc's serial bus. The I2C bus is running at +3VDC level, so we needed to shift the 12v levels found on serial port to that level. Don't worry, the +3VDC level is isolated, so 'it should be fairly safe'. If you want help with similar hw setup, feel free to contact us.




All we did from that point was a couple of simple python scripts to drive a short pulse on DTR or RTS according the build result. This was easy with a little help from pySerial module. Shutter relay connected the lamps according the last detected pulse. These scripts are called via <execute>-publisher in CruiseControl's config.xml.

<onfailure>
  <execute command="python pulse_rts.py">
  </execute>
</onfailure>

<onsuccess>
  <execute command="python pulse_dtr.py">
  </execute>
</onsuccess>

Next steps:

1. Write an actual publisher for DTR/RTS signals in Java
2. Use our rf comm devices to transfer the build status
3. Hook another lamp to indicate the ongoing build process
4. Write a LonWorks publisher similar to X10

Mark Michaelis described their similar effort. They are a bit ahead of us.

Immediate Payoff

Ken Hughes shared his story about their first three months with Scrum development. Findings are all too familiar. There is also a nice example of excel sheet working as backlog.

Certified Scrum Master

I had been thinking about taking the 2-day Certified Scrum Master (CSM) training for couple of years. Last week I finally got to the course with a colleague. My concern was that since I mainly develop firmware, I would not benefit that much from the course. I could not have been more wrong. I highly recommend CSM course to anyone involved with project work. Even if you do not believe in agile development, this course can still give you pointers on where to improve your work.

We got our training from Boris Oestergard and Jens Gloger. They have been giving in-house training in finland earlier, at least at Nokia Networks, Sulake and Accenture. We had also Bas Vadde from Nokia Networks giving an interesting guest speak about scaling Scrum to larger programs of hundreds of developers.

You can see our group photo at www.scrum.dk (photos).