This said, no, we have not solved the issue of maintenance and other urgent work disrupting normal work. We are struggling with bouncing velocity. This is for several reasons, but it also woke me to think about true teams again. Anyway, I'd like to pay more attention to this and give true teams a chance to grow.
1. You remember wrapping thinly with hexagonal architecture, right?
2. Company that I work for has a long history of using embedded PC simulation system called E-SIM from Mentor Graphics. This product has evolved, but we are using the old version.
3. E-SIM simulation tool has a Python API wrapper
A continuum is to develop own embedded acceptance test framework using Python, right? Well, we did it anyway. It is fairly simple, really. E-SIM communicates with the application interfaces using the mechanism called channels. There is a Windows service running called Communication Manager. And this fellow is responsible for gathering and sharing values in these channels to various C applications and our test driver written in Python. Indeed, we can have several applications running on one PC and simulate multi-device/processor/board systems as we wish. A channel can be a package in communication, a binary input as push button, binary output as LED, and so on.
Now, wanting to be able to write acceptance tests in a domain language that could be more readily understood by the business, we created a script parser using Python's reflection. This test driver parses the test script file and launches all the application .exe's as needed and then stimulates and senses the system using of E-SIM channel accessors. Sounds simple? The nice part is that it really is simple.
These tests run automatically on our continuous integration server, but we also have a GUI which simulates UI of a real system. Now we can watch the UI in action during the test sequence scripts.
Here is a snippet of .test acceptance test script written in (almost) domain language
sendFire 0,1,2,3,Delay Fire
checkRow 1,FIRE in zone 00021/1
Here is the reflection part of the parser in Python. I have no idea how good of a Python code this is being an average embedded C programmer, but it looks simple to me.
for line in tp.getTestLines():
self.currentScriptLine = line
self.currentScriptLineNumber += 1
if len(line) > 1:
elements = line.split(' ',1)
if len(elements) > 1:
getattr(self, elements)(elements.lstrip(' '))
self.warning("Could not execute line: %s" % line)
And finally here is a simple method in the TestDriver class. You can see that the method's name matches the action word in the test script snippet above. The rest of the test script line is passed as an argument to the method.
def checkRow(self, parameters):
elements = parameters.split(',')
row = int(elements)
expected = string.replace(elements, "\n", "")
actual = self.currentLCD_Rows(row)
self.checkEqual( expected, actual )
Similar approaches are Atomic Object's Systir, and Exoftware's Exactor. These can be labeled as script frameworks in contrast to table frameworks like FIT/Fitnesse.
"it is impossible to say, I have never done that before"
Well, firstly, in embedded software that's the situation pretty much allways, yet agile planning has proven to be efficient enough over and over again.
Secondly, we are living the era of enlightened experimentation. Prototyping hardware is drop dead cheap and fast nowadays. Buying yourself a status of "I have done that once" instead of "haven't done that before" is as easy as deciding to do so.
Thirdly, schematics can be estimated with relative complexity.
Fourthly, Parkinson's law works everywhere where humans are involved. Timeboxing early prototypes will help making the progress reliable and visible. This is however understandably very counter intuitive to the "get it right the first time" camp.
This is what is called up-front prototyping. In contrast to traditional hardware prototyping trying to validate something at the end, up-front prototyping focuses on learning. We may even know that the prototype will not work on most parts, but we just want a reliable measure of how far are we. It is even advisable to design a prototype to prove just one thing. More than one uncertanty will make the work unnecessarily complex.
This distinction between the goals of traditional validating and new era learning prototyping has gotten me concidering a new word to replace "prototype". So far the best candidate is 'product'. Using a word product throughout the lifecycle we would realise that this is aiming to production quality, but the maturity of the desing is evolving.
Of course I'm talking about prototypes that can be assembled within reasonable cycle cost. The bar however is getting lower and lower every day. Interesting exercise is to take your latest embedded development project, draw a bar of development salary cost and a second bar illustrating your prototype cycle cost. You may be surprised how cheap "expensive unnecessary prototyping" is in the big picture.
We also teach engineers to try out practices before they judge anything as "yak".
"Just try it, if it does not work - ditch it".
TDD (test-driven development) is probably the most difficult technical skill associated with so called agile development especially in constrained embedded environment. It is also probably the most rewarding when in use.
The problem with TDD is that it has a learning curve. To make things worse the earlier debug-later development model has an unlearning curve. Most of the true benefits can concretely be seen after a fairly long period of time. It is not enough just to have a taste before judging, you actually need a prolonged tryout period. You most likely need some external help in the beginning. Your short term productivity suffers because of learning new skills and solving new kinds of problems.
You need someone in customer's role who is willing to talk about long term productivity as well.
Never the less I recommend to just have a taste.
During these sessions I have seen a lot of nodding people. I have though that "wow, I'm pretty good at explaining this". I have left with a smile on my face after a huge sense of agreement. I know better now. It's a much longer journey than one day seminar.
The challenge, or one of them, as I realize now, is that if you give a basic introduction with mickey mouse stuff examples to experienced embedded developers you will be immediatelly shot down with "that won't work in our environment, because [choose your favorite item identified on Embedded Agile Yahoo group]".
On the other hand if you rush straight into embedded TDD specific stuff, people are going to just nod.
And this is the other end of the same challenge. They are nodding so that you would go away faster. Not understanding fully is scary for a seasoned engineer who has not opened a book in the past 15 years or so. This nodding experience alone is not going to make anyone start practicing TDD. The best is to give a hands-on experience on TDD itself, show the benefit, and not on mickey mouse stuff but on real deal.
Advice that I'm going to follow is to try to remember when I was also just nodding and how many years, books, articles, conferences, talks, discussions, trials, failures, small successes, and so on, it took for me to get it. And I'm still only beginning to understand TDD.
While tempting at the first look, there are several problems in the above scenario. First is the customer’s role. It is difficult enough to get a functioning product owner for one project. Running multiple projects at the same time is ridiculously challenging. If we get one person to act as a customer proxy for all these small projects, this person soon becomes a bottleneck for all these projects. A project needs collaboration with the customer representative, and the sooner we get the feedback the better. The customer needs to be available at all times, and this is easier to achieve for one project at the time.
Second problem follows from the fact that it is rare that all these projects are completely independent. While we think that no one works on more than one project at the time, we actually have created a highly complex hidden multi-project environment. Typically one, or few, developers know something about everything. This means that people that are supposed to be working for their own project need to consult other projects continuously. These people become the bottlenecks for other projects, and their own project is impossible to estimate. As a net result we push all our projects to be more, and more, late without any control to what is going on.
Thirdly, assigning a developer to her previous pet part of the system further develops the specialization in a narrow part of the overall system, creating small islands of expertice. More and more closed silos of expertice emerge in the department. This is a high risk path as in an unfortunate case of loosing that person for one reason, or another we halt the development of that part for unpredictable period of time. Yes, I have heard that documentation should be there for allowing anyone to work on new part. Well, having been in the development game for nearly 15 years I kind of have lost faith to that approach. Maybe if the previous developers have focused their energy mainly on enabling future work –type of documentation. Never the less, safer path is to several developers to know the work.
The fourth problem is the lack of possibility to gain from the superior performance of true teams. Assigning each person to a single small project will at its best only accomplish the sum of each developer in development department. If we are successful in getting a true team to jell we can accomplish a lot higher performance that the sum of team members. This is important especially when the work we are engaged with gets more complex, and needs higher problem solving capability.
Changing to work in teams and in the same part of the system may feel difficult and weird at the beginning. Progress may also appear slow as most of the team members are unfamiliar with the parts of the system they need to work with. It may also take old school engineers by the surprise that they also need the softer skills, interaction and communication. It is crucial that collective ownership is still enforced. Teams have a remarkable capability in finding the ways to work. The practices supporting the work of team will merge rapidly. You don’t have to know everything to start working this way. Just follow the basics;
- do fast paced iterative development,
- have reflective retrospectives, and
- create open, learning-driven culture.
What comes to that draft, I hope to find time to finish writing about Preston Smith's latest book "Flexible Product Development" and agile in non-software development soon.
September 2007 issue of Embedded System Design Journal revealed the results of reader's vote. Parts of the results add recent data to above observation. According to reader's vote the fraction of "new to the world; a new project from the scratch" compared to "an upgrade or improvement to an earlier or existing product" products has decreased from 48% to 39% in the past three years. 79% of those upgrades or improvements included "New or different software features".
I'm now engaged with much larger systems.
Application changed from lighting control to fire alarm system.
Resource budget has also increased quite a bit. Basic setup now contains:
2M program flash
As a result I'm learning a bunch of new stuff every day. I have a good feeling about this year.