Embedded Systems Conference 2007 in San Jose, California, is having some interesting events. James Grenning is giving several sessions and people from Atomic Object are coming to tell about their low level system agile testing methods.
The best news is that right after my vacation (still have some three to four weeks to burn) I'm going to attend the conference with a colleague. Talking about nice return to work! I'll try to have energy to write something about the conf'.
2/27/2007
2/12/2007
Scrum Simulation Rocks
Last week I gave an introduction to agile development and Scrum for a global project organization in Shanghai, China. After the lunch on first day it was time for 59min Scrum simulation. I have written about bad experiences when trying to achieve too much, so it was back to the basics.
There were participants from Finland, China, and Mexico. We used the product backlog for Family Cookbook. Only addition to original Scrum simulation was the use of relative complexity estimations on product backlog items. This time it worked. Estimating the tasks in numbers gave birth to discussion on whether something is simple, or not, not just dividing an item into tasks. Some items were also soon identified to be too large to fit in one 14 minutes Sprint. Team was immediately forced to seek for collaboration with Product Owner.
On retrospective the main thing experienced was the time pressure. I do not see this as a bad thing, it is just the beginning of understanding the discipline needed for time-boxed development. Time pressure will decrease when the team builds trust and improves estimating.
Everyone valued the exercise and though that it was a good lesson. I might consider adding a separate exercise for planning and estimating before the final Scrum Simulation. With this early experience I do not think that I will ever try meaningful work inside the simulation again. The difference was so clear.
There were participants from Finland, China, and Mexico. We used the product backlog for Family Cookbook. Only addition to original Scrum simulation was the use of relative complexity estimations on product backlog items. This time it worked. Estimating the tasks in numbers gave birth to discussion on whether something is simple, or not, not just dividing an item into tasks. Some items were also soon identified to be too large to fit in one 14 minutes Sprint. Team was immediately forced to seek for collaboration with Product Owner.
On retrospective the main thing experienced was the time pressure. I do not see this as a bad thing, it is just the beginning of understanding the discipline needed for time-boxed development. Time pressure will decrease when the team builds trust and improves estimating.
Everyone valued the exercise and though that it was a good lesson. I might consider adding a separate exercise for planning and estimating before the final Scrum Simulation. With this early experience I do not think that I will ever try meaningful work inside the simulation again. The difference was so clear.
2/02/2007
TDD'ing State Machines
I'm an electrical engineer. Everything in programming seems to be very very hard for me. Last week I spent 40 hours in aeroplanes, so I had some quality reading time. I did read Agile Estimating and Planning and Test-Driven Development from cover-to-cover. I had quickly gone through them earlier, but this was enlightening experience.
First of all, I have told that we have been trying to figure out effective TDD for our FW development. To be honest, we've struggled. We wrote tests for a state machine implementation. We expected the tests to be run in certain sequence, and that the state was always left where it should be after the test. We of course ended up in lot of rework on test code when our state machine design emerged. We thought that we need to design the state machine up-front in order to be able effectively write tests for it. This did not sound like test driven development at all. This required some more thorough thinking.
So back to my flight. Kent Beck writes that coupling is bad also between tests. A test should leave the system as it was. So writing unit tests in the same sequence as you expect your state machine to work does not seem like a right thing to do. You should be able to execute individual tests and to change the order of individual tests. So instead we are starting to recognize a single state as something to test.
Ph.D. Miro Samek writes about state machines being a killer app for function pointers. He further defines a design pattern for state chart implementation. Our state chart interface has become to be something like:
construct()
dispatch(event)
construct is self explaining and dispatch dispatches new event. We have specific event tick that gives resource time to state machine every 10ms (based on our 50Hz mains). We could also implement a timeout as OS service, and not to have local counters for timed events. Anyway, if we implement each state as a function as Mr. Samek proposes, and we hold our state as a function pointer we can write
dispatch( me, event )
{
me.myState(event);
}
We get rid of cryptic switch -structures. This solution helps us to unit test each state as a function with natural feel.
I'm going to write some benchmarking code to see the actual overhead, but at the moment I like this idea.
First of all, I have told that we have been trying to figure out effective TDD for our FW development. To be honest, we've struggled. We wrote tests for a state machine implementation. We expected the tests to be run in certain sequence, and that the state was always left where it should be after the test. We of course ended up in lot of rework on test code when our state machine design emerged. We thought that we need to design the state machine up-front in order to be able effectively write tests for it. This did not sound like test driven development at all. This required some more thorough thinking.
So back to my flight. Kent Beck writes that coupling is bad also between tests. A test should leave the system as it was. So writing unit tests in the same sequence as you expect your state machine to work does not seem like a right thing to do. You should be able to execute individual tests and to change the order of individual tests. So instead we are starting to recognize a single state as something to test.
Ph.D. Miro Samek writes about state machines being a killer app for function pointers. He further defines a design pattern for state chart implementation. Our state chart interface has become to be something like:
construct()
dispatch(event)
construct is self explaining and dispatch dispatches new event. We have specific event tick that gives resource time to state machine every 10ms (based on our 50Hz mains). We could also implement a timeout as OS service, and not to have local counters for timed events. Anyway, if we implement each state as a function as Mr. Samek proposes, and we hold our state as a function pointer we can write
dispatch( me, event )
{
me.myState(event);
}
We get rid of cryptic switch -structures. This solution helps us to unit test each state as a function with natural feel.
I'm going to write some benchmarking code to see the actual overhead, but at the moment I like this idea.
1/28/2007
Driving Embedded Firmware C Project on CruiseControl
I finally made it. A dedicated page for configuring CruiseControl for embedded firmware C project. It combines some of my earlier posts, but also tries to add detail. Take a look here, and tell me what is missing.
1/22/2007
Book: The Tipping Point
I finally read The Tipping Point, by Malcolm Gladwell. The Tipping Point tells us how little things can make a huge difference whether an idea will success or not. Will it tip or not. Examples and cases presented by Mr. Gladwell range from young men's high suicide rate in Micronesia, via fashion trends, cigarettes, children's tv shows, all the way to warning mechanism in surroundings of Boston about the upcoming attack of Brittish in 1775.
Malcolm explains, in a very understandable way, the law of the few; Mavens, Connectors and Salesmen. These are the three different personalities needed for a new idea to become epidemical. He talks about social epidemics. He then further explains the concept of stickiness factor, and the power of context.
What has this to do with things we are interested in? Graig Larman calls introduction of agile development methods a paradigm shift. When you introduce an idea of this magnitude, what you are doing, is asking people to unlearn much of their current knowledge. We of course know that the old knowledge is by no means useless, we just need to update it with some new knowledge. Never the less, introduction of agile methods and lean thinking is a novel idea in most organizations. The theories and concepts presented in The Tipping Point help us to understand the process of making this change and getting it to last. I have learned a long time ago, that it is not enough that you understand, you need to get others to get it as well.
Let's see what these theories in my opinion tell about agile intervention. In many cases, especially early in the decade, agile methods were introduced to organizations by younger champion developer, usually from position of no power. They just happened to be interested in learning and continuous improvement. They read a lot, they go to conferences, and they talk to fellow craftsmen. Furthermore they like to analyse the new things, and then tell these ideas to others. They are the Mavens. If things go nicely they will meet a Connector at some point. Connector knows everyone inside the organization, and knows who to talk to in order to get the change started. After the initial start, to make the idea of agile development really tip, we need extra ordinary people called Salesmen. They are the ones that can translate the ideas and experience reports so that we can communicate with lots of different people in order to get real structural changes to happen in an organization. Agile coaches and consultants are a good bet at this if you can not find someone in your organization fast enough. On top of that agile is a nice sounding word, and it can help to increase the Stickiness Factor. The first agile pilot project should have importance and wide enough commitment, creating the right Context for the intervention to tip.
Other literature (for example Pinchot, Intrapreneuring: Why You Don't Have to Leave Corporation to Become an Entrapreneuer, 1985) documents the same kind of process. Terms used elsewhere include 'Brain workers' who have the intellectual ability to think in novel ways and are able to argue their decisions. 'Intrapreneuers' are daring in using their vision of innovation regardless of for example political resistance. 'Gate keepers' have strong contacts to the organization's environment and have communicational competencies for making initiatives 'digestible' for the organization. Further a top management mentors, godfathers, or corporate angles are terms used for expressing the need for management commitment in such a venture.
Malcolm explains, in a very understandable way, the law of the few; Mavens, Connectors and Salesmen. These are the three different personalities needed for a new idea to become epidemical. He talks about social epidemics. He then further explains the concept of stickiness factor, and the power of context.
What has this to do with things we are interested in? Graig Larman calls introduction of agile development methods a paradigm shift. When you introduce an idea of this magnitude, what you are doing, is asking people to unlearn much of their current knowledge. We of course know that the old knowledge is by no means useless, we just need to update it with some new knowledge. Never the less, introduction of agile methods and lean thinking is a novel idea in most organizations. The theories and concepts presented in The Tipping Point help us to understand the process of making this change and getting it to last. I have learned a long time ago, that it is not enough that you understand, you need to get others to get it as well.
Let's see what these theories in my opinion tell about agile intervention. In many cases, especially early in the decade, agile methods were introduced to organizations by younger champion developer, usually from position of no power. They just happened to be interested in learning and continuous improvement. They read a lot, they go to conferences, and they talk to fellow craftsmen. Furthermore they like to analyse the new things, and then tell these ideas to others. They are the Mavens. If things go nicely they will meet a Connector at some point. Connector knows everyone inside the organization, and knows who to talk to in order to get the change started. After the initial start, to make the idea of agile development really tip, we need extra ordinary people called Salesmen. They are the ones that can translate the ideas and experience reports so that we can communicate with lots of different people in order to get real structural changes to happen in an organization. Agile coaches and consultants are a good bet at this if you can not find someone in your organization fast enough. On top of that agile is a nice sounding word, and it can help to increase the Stickiness Factor. The first agile pilot project should have importance and wide enough commitment, creating the right Context for the intervention to tip.
Other literature (for example Pinchot, Intrapreneuring: Why You Don't Have to Leave Corporation to Become an Entrapreneuer, 1985) documents the same kind of process. Terms used elsewhere include 'Brain workers' who have the intellectual ability to think in novel ways and are able to argue their decisions. 'Intrapreneuers' are daring in using their vision of innovation regardless of for example political resistance. 'Gate keepers' have strong contacts to the organization's environment and have communicational competencies for making initiatives 'digestible' for the organization. Further a top management mentors, godfathers, or corporate angles are terms used for expressing the need for management commitment in such a venture.
Look at the world around you. It may seem like an immovable, implacable place. It is not. With the slightest push - in just the right place - it can be tipped.
1/09/2007
Blogger is out of beta
Blogger is out of beta and I thought I finally upgrade my template. I'm glad I did, managing the appearance of my blog is now a breeze.
I did some rework on links as well. If you are interested in agile development methods for embedded software and firmware you should join the discussion at Agile Embedded Yahoo Group. There are some brilliant people in the group, but the group could use some additional energy.
I seem to have few regular readers, so if this change messed up your reader - my sincere apologies. This was just something that needed to be done at some point.
I did some rework on links as well. If you are interested in agile development methods for embedded software and firmware you should join the discussion at Agile Embedded Yahoo Group. There are some brilliant people in the group, but the group could use some additional energy.
I seem to have few regular readers, so if this change messed up your reader - my sincere apologies. This was just something that needed to be done at some point.
12/30/2006
Weekly Delivery of Firmware
Yesterday we finished a six week project developing technology platform for User Interface innovation using some practices familiar from agile methods. The project delivered two different configurations of home automation user interface device; one for wall mounting and one for remote control. Some facts:
- Vague or missing requirements
- New 8-bit microcontroller
- Lots of new sensor technology
- New embedded firmware team of 3+1, including one consultant (in house)
- Some drivers developed by outsourcing
- Final project consisted of 20KLOC of firmware C source (bad meter, but...)
Some findings after initial reflection:
- Weekly delivery (not release) of firmware is possible, at least in some cases.
- Short term goals and daily Scrum meeting effectively work as motivators, and enforce full learning and getting things done done.
- There is not small enough project to not use version control and continuous integration (we had no way to have common version control with outsourced team: large org, IT, you know the drill...)
- HW evolved from microcontroller starter kit to final electronics via several bread board mocks and one halfway PCB prototype. This causes extra assembly work when looked at the surface, but this overhead is by far overcome by enabling the concurrent engineering instead of waiting for final version.
The last point holds only if all the disciplines work as One Team.
We did not have cross-functional team as we did in earlier larger experiment. Schematics, PCB layout and mechanics were developed concurrent, but they were not part of the team - nor did they follow any of so called agile practices. Firmware planning was tried to be synchronized with deliveries from other disciplines. In several occasions these deliveries were delayed, and a common goal needed for a team to actually be a team was somewhat missing. This is where I would like to do things differently if I was to do it all over again.
Occasionally we heard sentences like "If everything goes smoothly we will be ready on Tuesday, but we will be ready at least on Thursday". At this year's Agile Business Conference in London David Taylor (The Naked Leader) said "Don't have a plan B because you will always achieve it." Just guess if the delivery was on Tuesday or Thursday - or Friday?
The whole project was started by extremely vague idea. In few hours we worked out initial backlog and a delivery (release) plan based on simple themes. From that point on plan, design and requirements emerged during the 6 weeks nicely synchronized with weekly demonstration of prototype and new planning session. This experiment supports the belief that good people can work this way, and that we truly live the era of enlightened experimentation. We believe that we could not have performed any better with thorough analysis, design, and specification phases - at least not to cover the cost of them. During the development there were several points where it was not possible to proceed as we had thought. A comprehensive plan based on these initial assumptions would have failed miserably, and the time creating them would have been wasted. In this project the whole team worked out an alternative solution, adjusted the next week's plan, and the project was fine...
I will write an experience report and hopefully make it available here.
12/21/2006
Skunk Work Is Not A Longterm Solution
In her "The Agile/Waterfall Cooperative (pdf)" presentation Michele Sliger summaries three modes in which agile team can work in a waterfall organization; SWAT team, Skunk Work, and stealth agile. I agree that applying just some agile practices can benefit your work. This is pretty much what we are doing at the moment, but I do not see this as sufficient long term solution - if it is only in team's sphere or just above. All three forms of team work can of course be used. SWAT teams can go parallel even in more wide spread agile organization. Skunk work probably, and hopefully, always exists in large organization - it's fun. If you are working as stealth agile, and avoiding radars by not communicating to the top management, you are not going to be able to fully harness the power. You need to inform top management that you are doing things differently and communicate your need for changes. They probably will not get it, but what they do get - is success. So it is fine and in most cases necessary to fly low at first, but after certain point it is not satisfying anymore to work as a shadow system, because you know you are constrained by the legitimacy structures, and these structures are holding back you and your learning. The rest of the presentation has also good ideas, and based on my experience these ideas hold in reality as well.
SWAT team – outside the domain of the process police, brought in on projects that are in trouble. Agility is prized in this situation. They take over however – they don’t work in a cooperative mode.
Skunk Works - “a small group of experts who drop out of the mainstream company operations in order to develop some experimental technology or new application in secrecy or at speed, unhampered by bureaucracy or the strict application of regulations.” Sanctioned and protected by management, these teams are rare. This formation however, prevents the headaches involved in org structure, project approval hoops, portfolio metrics and management, budget calls, and other bureaucratic nightmares.Stealth Agile – As Jim Highsmith once said in response to a question about how to sell agile to upper management – don’t. “They don’t know what you’re doing anyway.” Do as much agile as you can where you can. Some benefit is better than none at all.
12/18/2006
Link: Scrum and XP from the Trenches
No matter if you are developer, PM, or CIO and you don't know what agile/Scrum/XP is, you are thinking about applying, or you are already practicing - you should check out "Scrum and XP from the Trenches (pdf)" report from Henrik Kniberg. Henrik describes how they applied Scrum and XP practices in real life and presents the adaptation needed in this situation. Excellent Writing!
12/14/2006
main_theOneAfterFinal.c
Today I happened to see a file name with word 'new' on it on a presenter's computer. I was slightly amused because of my history. I can still see the old version naming convention in many "shared team folders" in our server. It goes something like this:
original
new
latest
final
I have been using version control system since my first university years (some fifteen years ago) and throughout my professional life. First it was just an advanced XCOPY method putting released versions of one man projects into the system, but about three years it has been collective code ownership and at least daily update-commit (we use CVS) cycles from the beginning of development. Lately we have adapted continuous integration and automated regression (confirmation) testing as you know. Currently we are looking for ways to add some automated acceptance and system testing as well.
During the past couple of years I have had a pleasure to work with software developed outside our own company. Quality of source varies a lot of course, but it is striking that version control typically is not even mentioned. In some occasions lately the software was delivered as zip'd folder with a version number in folder name - at that point I was happily surprised by that! Embedded firmware domain - based on my biased observations - walks some 10 years behind the mainstream software industry. The wide lack of basic discipline, version control, proves the point. I'm glad that I work in a department where I can say that developers would never go back working without version control.
I found this Yahoo group message - nostalgic.
original
new
latest
final
I have been using version control system since my first university years (some fifteen years ago) and throughout my professional life. First it was just an advanced XCOPY method putting released versions of one man projects into the system, but about three years it has been collective code ownership and at least daily update-commit (we use CVS) cycles from the beginning of development. Lately we have adapted continuous integration and automated regression (confirmation) testing as you know. Currently we are looking for ways to add some automated acceptance and system testing as well.
During the past couple of years I have had a pleasure to work with software developed outside our own company. Quality of source varies a lot of course, but it is striking that version control typically is not even mentioned. In some occasions lately the software was delivered as zip'd folder with a version number in folder name - at that point I was happily surprised by that! Embedded firmware domain - based on my biased observations - walks some 10 years behind the mainstream software industry. The wide lack of basic discipline, version control, proves the point. I'm glad that I work in a department where I can say that developers would never go back working without version control.
I found this Yahoo group message - nostalgic.
12/09/2006
The Lenghty Debugging Phase
To give embedded firmware TDD a kick start we have had two sessions with a colleague. During these sessions we have created scripts to automate the unit test structure and build process and discussed how the make as thin wrappers for HW as possible. However we learned also other valuable lessions during these sessions. Lesson number one is the "debugging phase" being so strongly in our domain culture. We often hear a firmware developer saying, "well, it takes couple of days to implement it, but you can never tell how long the debugging is going to take". Of course debugging phase is where all the firmware heros truly shine. That's another unfortunate culture issue in firmware development.In our first session we ran into problem of not getting one test running. Immediately we started debugging. Printf's fly in here and there (as an advanced host run I/O twist). The change to this mode was so rapid, that neither of us realized what is going on until later. After some 30 minutes(!) we recognized that we are not moving, but just desperarately experimenting with printf's, and it hit us; we have not changed our behavior as we were supposed to. We should have picked a more simple thing to implement. When we realized we can not chew what we had chosen, we should have started all over again with simpler thing.
It is our belief that by doing TDD for firmware we should get rid of, or at least dramatically reduce, the debugging phase and thus create an atmosphere of success, and improve the reliability of estimates.
This type of sessions are great tools for learning. I strongly share Michael Harmer's opinion on The Training Course Scam.
11/30/2006
Ken Schwaber on Quality of Code
Ken's presentation "A Canary in a Coal Mine" at Agile2006 conference is available via InfoQ. Ken talks about quality and the danger of Scrum teams droping quality issues to get higher velocity and to be sure to hit the date. He calls this a re-definition of "done", or "somewhat done". We have manifested an increasing technical debt in our iterative work. So in the last project we were putting more focus to acceptance testing inside each sprint and introduced something we called "cross-testing". This means that other pair needed to test the feature implemented by other pair in order to claim feature "done". Embedded firmware systems need a lot of manual acceptance testing (at least our current practice) and it is easy to get slobby testing your own work over and over again.
Was the technical debt there because of Scrum? Well, no - it was made visible by Scrum and the team was able to decide to do something about it early enough.
----
Ken's another speech about Scrum is available at Google Video. (You will find out that Scrum works with idiots as well)
I posted about Jeff Sutherland's Scrum speech earlier. It is also available via InfoQ.
Was the technical debt there because of Scrum? Well, no - it was made visible by Scrum and the team was able to decide to do something about it early enough.
----
Ken's another speech about Scrum is available at Google Video. (You will find out that Scrum works with idiots as well)
I posted about Jeff Sutherland's Scrum speech earlier. It is also available via InfoQ.
11/26/2006
XP2007 Conference Web Page
XP2007 web page is open. The conference will be held on June 18-22.2007 in Lake Como, Italy.
In the scope of conference is:
In the scope of conference is:
- Embedded software (e.g., SW/HW co-design) and Agile SW development
...among of course other interesting stuff. Hopefully we see some papers in this domain as well. I attended the conference this year in Oulu, Finland, and if next year's event is anything like that - I can highly recommend attending.
11/21/2006
NPD in General With Agile and DSDM
Iterative and incremental development ideas have been presented for general new product development (NPD) for years. I'm interested in agile methods (as in software development context) because they give a philosophy behind and lots of ideas how to actually do this. It is of course also because of extremely active and powerfull community. Ideas of using these methods and practices in NPD in general have been presented by Jim Highsmith (Agile Project Management), Preston G. Smith (Developing Products in Half the Time), and many more. Both of above mentioned can be contacted via Cutter Consortium.
Now there is something new coming up. At London Agile Business Conference 2006 few weeks back the DSDM consortium talked about the future of the DSDM framework. DSDM version 4.2 has been open for online viewing since this year. They are going to publish a new version of framework in Spring 2007. This new version is supposed to be more general and not specifically targeted to just software development. It was also presented that DSDM can be seen as a wrapper for models like Scrum and XP for use in enterprise environment. It's a bit too structured to my tasting, but very well worth to check out when available. It may also become more open.
Now there is something new coming up. At London Agile Business Conference 2006 few weeks back the DSDM consortium talked about the future of the DSDM framework. DSDM version 4.2 has been open for online viewing since this year. They are going to publish a new version of framework in Spring 2007. This new version is supposed to be more general and not specifically targeted to just software development. It was also presented that DSDM can be seen as a wrapper for models like Scrum and XP for use in enterprise environment. It's a bit too structured to my tasting, but very well worth to check out when available. It may also become more open.
11/17/2006
Risky Business
It has been identified that the biggest risks in project work concern management and marketing. Yet, in most environments we only concentrate on technology risks. Agile, incremental and iterative, development is also often assumed to be risky. By changing the way of thinking about risk, people would see that iterative risk management is a proactive way to tackle all aspects of risk. A very simple way.
1. Management risks - just fund the project for the time zone you are comfortable
2. Marketing risks - use the first possible release to test the waters
3. Technology risks - use the effort early to tackle any technological uncertainties that traditionally are left lurking until the (assumed) end of the project
The status quo practice is quite different. We take huge risks by starting a megaproject, for example estimated to 5 calendar years, based on a business case. Everyone knows that business case targeted five years from now is pure speculation. In the global markets it's a very expensive speculation by the way. Based on this speculation management needs to make a decision for funding a five year project. Their natural reaction to tackle the risk is to review the project mercilessly. We can be sure that changes to project requirements, technology and scope will happen. We have this heavy review practice going on making the project move even more slowly. More time means more changes, more changes mean more reviews - are you starting to get the picture?
Most of us see this everyday. How do we try to fix it? Naturally by reviewing. Because? That's what we allways do.
1. Management risks - just fund the project for the time zone you are comfortable
2. Marketing risks - use the first possible release to test the waters
3. Technology risks - use the effort early to tackle any technological uncertainties that traditionally are left lurking until the (assumed) end of the project
The status quo practice is quite different. We take huge risks by starting a megaproject, for example estimated to 5 calendar years, based on a business case. Everyone knows that business case targeted five years from now is pure speculation. In the global markets it's a very expensive speculation by the way. Based on this speculation management needs to make a decision for funding a five year project. Their natural reaction to tackle the risk is to review the project mercilessly. We can be sure that changes to project requirements, technology and scope will happen. We have this heavy review practice going on making the project move even more slowly. More time means more changes, more changes mean more reviews - are you starting to get the picture?Most of us see this everyday. How do we try to fix it? Naturally by reviewing. Because? That's what we allways do.
Subscribe to:
Posts (Atom)