If you're not afraid, you haven't been paying attention

Musings on Software Development

First published on the PARSE Software Devices website May 15th, 2007 © Copyright 2007 by Robert Krten, all rights reserved.

Synopsis

This article presents an analogy of software for managers, and highlights the author's experience over the last 20+ years of software development.

Understanding Software, an Analogy for Managers

I've always been on the lookout for a good analogy for the software development process. When I first started out, software was "like a car". My boss used this to explain the software's operation to himself and his marketing staff. "When we turn on the windshield wipers, the trunk pops open," he'd say, "so we still have a few things to work out." This was his way of equating tens of thousands of lines of code with concrete physical items. I guess in his mind, the software was really just a more complicated car; a lever would pull on a wire, which in turn would cause a mechanical lock to disengage. It was conceivable to him that if we (the software developers) somehow got a wire or two mixed up, then the lever could activate the trunk, instead of the wipers.

I'm not sure how deeply he believed the analogy, but it seemed to work for him. To play along, we'd make it more and more complicated. So, instead of having just a simple lever pulling on a wire, the lever would instead interact with a geared mechanism with a motor driven cam interface and a poorly welded connecting rod. When the connecting rod and the cam intersected at a particular moment in time, it would snag a different wire and start an unexpected interaction. We would then claim "and so, that's why the inbox would have an item from the outbox in it", and he'd buy it.

I never understood the requirement to concretize the software; any physical, "mechanical," representation seemed to be such a poor shadow of the actual complexities involved that I saw no real use for it — it didn't help me in any way.

Recently, working on another contract, a manager put it more succinctly for me. He said, "You draw all these objects in the air, and make all these noises and actions, because you're trying to express a very complicated structure that has no natural analogs in the real world."

Bingo. I did have a habit of waving my arms around while explaining interactions between objects that existed in a netherworld, somewhere between source code and execution. It's hard to describe a multi-process interaction with concurrent asynchronous timing without waving your arms a bit.

Having this much of a disconnect between the real world and the murky reaches of software, it's no wonder that most management that hasn't written code is completely clueless about how software works, or, more importantly, how to schedule and plan it.

Frederick P Brook's excellent book, "The Mythical Man Month", goes a long way towards educating both software and non-software people about how to go about managing software projects -- but it could use a good, fundamental analogy.

I've finally come up with one.

Designing software is like shooting a movie

From my simplistic point of view, a movie consists of plot, locations, actors, dialog, action, makeup, and props, and involves such tasks as continuity, sound synchronization, dubbing, re-recording, optimizing shooting sequences, lighting, special effects, stunts, etc.

The movie world's equivalent of the "Hello, world" program is a (very) short film shot in one take with one actor in one scene with one line of dialogue using a clip-on mike and a 100W light bulb.

Who can make this movie? Anyone. Who can write "Hello, world"? Anyone. (Probably even the same person.)

It's when things get more complicated that our analogy gets more interesting.

How is it that Hollywood (and Bollywood, and Toronto, and a host of other cities) can make a movie and have it delivered more-or-less on time and on-budget, over and over and over, whereas Silicon Valley (and any other city you care to mention) can almost never deliver on time nor on budget, over and over and over again?

I believe it's because the software managers and developers have the wrong mindset.

Today's software development, if it was to be expressed in movie terms, would go like this.

The marketing department puts together the marquis posters. Something bold, brash, and attention grabbing, like "The Accounting Solution from Planet Logical". They'd put in colourful screen shots, like "the one where Mary from Accounting delivers payroll on time", and "Harry from Sales easily inputs his expenses", or the classic manager lure "Sue saves the company money using Planet Logical and is honoured for being a Team Player".

And there'd be all these happy, smiling, airbrush-enhanced prime specimens of male and female anatomy, just about bursting out of their clothing (which would be the minimum allowed, of course). Special effects would include such things as laser-like beams forming reports in mid-air, servers the size of iPods running the software, and so on.

Towards the bottom of the marquis would be little details, like "System Requirements" and "Performance Numbers", but they'd be in small type and buried in amongst the corpses of defeated aliens.

This poster, or perhaps a few variations of it (for different markets), would be handed to Engineering, with an accompanying vague hand-motion indicating "make it so and don't bother us further".

Of course, the first thing that Engineering looks at is the "Coming in May 2007 to a theatre near you" deadline looming out of the corner, and starts panicking.

Now Engineering's job is to make this "movie." So, they gather the managers in a room, and try and figure out how, and who, and when, and how much.

Usually, but not always, Finance comes in and says "You have $X and not a penny more." Sometimes, they'll give Engineering a week of looking at the poster before they ask for an estimate, which they then pare down and make everyone sign.

Now Engineering is ready to make the movie. They've been locked in a room for a week or two, and now know that they will need 3 actors, 2 cameras, a sound effects guy, a stunt double, and a really nice office to shoot the movie in. Several experts from Engineering spent the week figuring out what kind of camera will be required, and looking online to find the cheapest price. Gayle from human factors determined that two of the actors should be female, one of whom should be a visible minority, and that the male actor can be fat and bald because he'll resonate with the geek male in the audience. John figured out the best material for the tables and chairs, and studied the colours used and the sizes and shapes of the background paintings and made sure that the camera would be able to correctly capture all of the nuances of the image. The sound effects guy is dropped because Bill can make excellent sound effects using just his mouth, so there's a cost savings right there! The team is ready to rock and roll!

They start the camera rolling, and find out that nobody has any lines. So, they go back to the room, and try and figure out lines for everyone. "Can't you just ad lib something?" is heard from Mark, a long time veteran of the software biz. "Just say the usual stuff about how you feel" comes out next. "Be sure to mention the product!" (apparently, someone from Marketing had snuck in to the meeting, but luckily they were kicked out almost immediately). 7 slides are hurriedly entered into PowerPoint, and the dialogue is ready.

So, back to the location. The cameras start rolling, and the actors stumble through their lines.

The senior VP of R&D pokes his head in, "How's it going?"

"Oh great, we're just about done!" is the response.

Everyone nods their heads, "all we need to do is a bit of editing, and then print it."

The VP sagely nods his head too, leaving with, "Great work!" and smiles all round. This is going to be a terrific success!

The next morning, the film is developed, and the editing staff start piecing together the movie. "Oops, this scene is too dark, we'll have to re-shoot it; file a PR." Some more editing happens. "Hey, how come the plant moved from the left side of the desk to the right?" Someone had obviously eaten their lunch on set, and carelessly moved the plant. "File a PR."

As each bug is discovered, the PR database grows in size.

Management begins to start worrying.

Finally, an edit is released. The editing staff are not really very happy with it, but it's the best they can do. "Apart from the outstanding PRs, this is 100% complete," they say.

Marketing is presented with a "beta" version of the film, with a promise that "just a few scenes need to be fixed up here and there, and it's all done."

Now, depending on the company, one of several different things may happen. Marketing may scream bloody murder, and demand that they fire everyone and reshoot from the top, this time with experts who know what they are doing. Or, they may blithely smile and nod, assuming that Engineering will fix what they said and everyone will be happy. Or something in between (or, they may decide that the best thing to do is to send the development off-shore [1]).

Generally, Marketing will notice things that Engineering totally missed. "How come Sam's speaking in a British accent at the beginning and end, but switches to a French accent in the middle?" or, "Why does the clock say 12:30 and then 4:30 5 minutes later?" "Where did Jimmy get the shiner? How come it comes and goes?" Minor details, more PRs, and more rework.

On the next release, someone notices that there's no plot — it's just a bunch of scenes stuck together! An expert is called in. "Mr. Spielberg, how do we add an air of 'competence' to this film? We want it to look more engaging, like it can take on the world!" Usually, but not always, the expert's suggestion is ignored, "It'll be too expensive to start over from scratch. We've already invested all this time and energy into it." And it gets patched. The original actors are long gone, working on other films, but a look-alike is found for one of the female leads, and a new scene is added to "bridge" the gap in the movie and give it some plot.

And on an on it goes.

How does a real film get made?

It always struck me as fascinating the amount of work that goes into the planning of a movie. You can get a good idea of some of it by watching the "Making of..." special features included with most big-budget movie DVDs. I'm sure this is just a tiny "tip of the iceberg" view of what actually happens.

The film studio doesn't just assemble a team of actors and start filming. They plan the crap out of the movie, sometimes taking years. Yes, the "Y" word. They know when every actor will be needed, for how long, what lines will issue from their lips, what makeup they need to be wearing, what position the sun will be in, what the condition of the set is, the lighting, the special effects, the sound effects, the props — in short, they've "done" the complete movie before they start shooting! [2]

Typically, you'll see sketches of the entire movie from start to end. The commentary usually goes something like "And in the opening scene, we see Mr. Anderson looking for the androids. Next, he spots one atop the pier. He yells at him to come down. We zoom in on the android's smooth flat metallic face. The one red eye moves, and stops to look at Mr. Anderson. Cut to ..." and so on.

In the world of software development, this is called the "requirements analysis phase" and the "detailed design phase." All too often, these phases are completely neglected, or are relegated to a "we'll complete that document when the software is done" role. [3]

In software development, the ad-hoc "just start shooting" model is exemplified by requirements documents that say "And the software shall provide a recovery mechanism." This would be like the movie commentary saying "And then we shoot the fight scene," without giving any details of who is in the fight scene, the lighting required to get the mood and context, the props necessary, the stunts, sound effects, and so on, let alone the complicated choreography of each move.

Unfortunately, when pressed in meetings to "explain in detail what the recovery mechanism does," there will typically be either a bunch of handwaving and vague motherhood statements (like, "It makes sure the software can recover from a fault") or worse, the person asking the question is viewed as a moron (along the lines of, "why can't you figure it out for yourself, it's obvious!").

When pressed, the embarrassing little secret usually comes out — nobody actually knows exactly what the recovery mechanism does! Everyone has an "of course it does X" assumption in their brains, but for every person at the meeting, there is a different value of "X".

Even worse, just because one such glaring omission is pointed out, nobody gets the fact that the "specification" is littered with these kinds of problems. Anyone caught going through the specification in such a nit-picking manner is usually labelled a troublemaker and silenced ("Well, those are all good points. Tell you what, we'll take that discussion off-line.").

That's because the entrenched attitude is one of "we're wasting our time with these boring and bothersome specifications when we could be coding. All this is going to do is delay our schedule!" Alas, this attitude is shared by both management and software developers alike. Management doesn't see the need for it, because they don't necessarily understand the complexity of the interactions, and software developers, well, at times it's like herding cats. (Unfortunately, management that has prior software experience is already brainwashed with this attitude, having seen it "succeed" so many times in the past.)

In the previous example, the "recovery mechanism" requirement would be more usefully expanded to a blow-by-blow description along the lines of the commentary in the film's "making of" documentary. Something along the lines of "the software calculates the checksums of the control files and verifies them against the master control file. If there is a checksum mismatch, then the secondary database is similarly checksummed and compared. If the secondary database is valid, then the secondary database is used, and the primary database is copied from the secondary database and a logfile entry is written. Then ..."

This ensures that everyone's assumptions about what the "recovery mechanism" actually does are identical (or at least on the same page).

But, just like in a movie, this requires constant updating. If in the movie it turns out that the androids don't have eyes, that they "see" via radar, then someone needs to go back over each and every scene and verify that it is consistent, and if not, change it, and then verify that the change itself is consistent, and so on. This takes time, and is a pain in the ass. But it saves time in the end. Needless to say, time is money.

Orders of Magnitude

When I say "management doesn't necessarily understand the complexity" of software, I'm not just being dramatic.

Software adds orders of magnitude of additional complexity. In a movie, the viewer typically watches it from beginning to end. They don't jump around and expect it to make sense (unless it's a Quentin Tarantino movie, of course). Software, on the other hand, consists of components that run concurrently with each other, and the user is the one who controls (to one extent or another) when a particular section of code runs, and can re-run the same piece over and over again with small or large changes incorporated into each subsequent run.

In the movie analogy, this would be like expecting each of the sub-plots to be living 3-dimensional snippets. So, if the viewer decides that the lead character breaks his arm in a daring rescue, that broken arm needs to propagate through the entire scene and ripple its way into other scenes. Or, the viewer may have decided that the lead character didn't break his arm but instead got a bloody nose; that fact needs to be reflected into all the other scenes. Of course, there would be limits to what the viewer could control — for example, the viewer wouldn't be able to kill off the lead character, because then the plot wouldn't be satisfied, and several scenes would have to be totally removed or heavily altered, and that's not what the movie is about. Similarly with software. The user can't control a program to the extent where an accounting package becomes a web browser — it's just not in the "plot."

But the amount of control a user has over the program is phenomenal, compared to a movie.

Which explains why the specifications need to be solid. They need to state exactly what can, cannot, shall and shall not happen. Can the user abort a particular operation? If so, what is the impact? If not, then how does the user stop something from happening? Are they aware that they can do that? When they do stop something from happening, what is the impact on other components? Can they resume? Do they need to restart? If they restart, does any of their previously entered data stay in the system? If so, what parts? And on and on it goes.

Yes, very boring stuff — but critical.

Which leads me to my next topic.

Testing

When Engineering does bother to write requirements, often they are full of vague statements. In a strict view of a requirements document, every statement must be testable and implementable. If it's not, then it has no place in the requirements document.

I've seen requirements that say "The indication shall be visible." Bravo for using the word "shall," but this is not testable. "Visible" by whom? How far away? In what lighting conditions: full sunlight, "standard" office lighting, or total darkness?

If it's not testable, it's useless. All that happens is that the so-called "requirement" becomes a bone of contention between the developer and the tester. The tester says "I can't see it" and the developer says "Well, I can" and you have a deadlock.

So why all the documentation? Because without it, you will certainly produce something very quickly, but it will not be testable, maintainable, or, ultimately, what the customer wanted.

Which is not to say that throwing money at a problem naturally leads to quality. Say "Water World" with me.

And speaking of quality...

What would a good software rant be without mention of ISO 9000 and related standards?

ISO 9000 is an insidious "boy's club" that does nothing to contribute to quality (in the traditional and normal sense of the word). It seems to me that every place I've seen ISO 9000 deployed, it's been deployed because "someone else insisted that we be ISO 9000 compliant". Basically, what I think happened is that one company got suckered into compliance, and seeing what a phenomenal waste of time, money and resources it was, decided that all of its suppliers should be equally crippled so as to not give them an unfair competitive advantage.

In case you think I'm joking, go and compare the definition of "quality" in a) the dictionary, and b) the ISO 9000 world. The dictionary definition includes such phrases as "degree of excellence."

What does ISO say?

Quality is conformance to requirements

We've already seen how bad the requirements process is in software development. So, substituting for "requirements" in the quality statement for ISO, we get the following:

Quality is conformance to an afterthought.

Which reduces to:

Quality is an afterthought.

Go ahead. Convince me that's useful and somehow desirable.

Footnotes: