Learning from failure
This has been difficult. Writing and publishing encourages me to carefully examine my ideas and helps to clarify them. But looking closely at my own failure hasn't been fun, even if it has been rewarding. Hopefully others can spare themselves the pain by learning from my mistakes.
Back in April, May, and June I had an embarrassingly bad time with the import problem I mentioned. I took three months banging my head against various aspects of the problem. In spite of my post about the importance of knowing when to stop I chased one wrong turn after another for three months. When Rob returned from his vacation he asked Paul to take over who finished it in four days.
Three months vs. four days. That is embarrassing.
It would be easier if I could claim to have solved most of the problem. But Paul's solution little resembled what I had started and addressed aspects of the problem I hadn't even considered. I know Paul is a better programmer. I know he is fast. But I was not prepared for the internal shock and dismay of so glaring a comparison.
With a few months hindsight I can see that I made my most fundamental and most drastic mistake right before I started working on the problem. I assumed it would be easy, but tedious. I underestimated my opponent.
I then failed to notice just how much I was actually struggling. This is especially hard to accept, even in hindsight. How on earth could I go on for weeks, let alone months, without noticing how seriously I had underestimated?
When presented with evidence that conflicts with our beliefs, we dismiss the evidence rather than adjust our beliefs. Getting my butt kicked by an "easy, but tedious" problem is inconsistent with my hot-shot-programmer self-image. I continued to believe the problem was easy and that I was on the verge of the solution, in spite of mounting evidence to the contrary.
The last dimension of failure grew from the same poisoned soil. I couldn't bring myself to ask for help on something "so easy." I didn't want to interrupt Rob and Paul who seemed quite busy on more important matters. I didn't want to admit that this busy-work was kicking my butt. I flailed and struggled and silently suffered along.
As in aikido, getting properly slammed into the mat can really focus your attention. A number of important lessons came hard and fast in July and August.
Sean McGrath's description of the breakthrough syndrome offered some solace, if not a solution:
The syndrome affects all walks of IT professional life but is most acute in software developers. ... The developer "wraps their head" around a problem. ... From that moment of immersion onwards, ideas for solving the problem come thick and fast [and] the developer gets a sense that a triumphant breakthrough is not far away.... That person is likely to feel - really and truthfully - that a breakthrough is just around the corner.
It may be, but it probably isn't.
Paul says "there's no busy-work in programming." I haven't heard that anywhere else, and it's still hard to believe. But there's no question that I would have called for help much sooner if I hadn't thought of it as busy-work.
My failure exposed the underlying importance of a collection of extreme programming practices. For starters, my silence violated three of the four XP values, specifically communication, feedback, and courage. Silence is not an effective form of communication. I was cowardly in concealing my failure rather than courageously admitting that Easy and Tedious had beaten me. I broke the feedback loop and we suffered for it.
By contrast, once my failure was out in the open, I was immediately back into the communication and tight feedback loops that are typical for bivio. Rob offered a couple rules of thumb to measure future progress: most tasks should take less than a day and the hard ones should be done in a week. These rules are hidden in continuous integration. In order to integrate continually, you have to break the tasks into day-sized chunks or smaller. Alarms should be going off if you can't integrate your code at the end of the day.
David called for the planning game -- something we hadn't been doing. For this particular problem, the split would have been especially valuable. I had misjudged the problem and taken on a task that was too big for me to properly get my head around. It would have been useful to get help early in slicing the problem into smaller pieces. Also, regularly reporting and adjusting scope and tasks would cause a failure like mine to fail fast. I would only have gone one week before the alarms started going off. If it went as long as two weeks, serious interventions would have been put in place and a months-long failure would have been averted.
I had to get over my hesitation to interrupt. This is about courage and communication. I had two fears holding me back. First, I was afraid of interrupting Rob and Paul. Second I was afraid of looking stupid. Courage is taking action in spite of fear. I need to trust Rob and Paul to manage their own interrupts. The fear of looking stupid is both simple and incredibly difficult. I simply need to get over it. That is substantially harder than it sounds, and also a story for another day.
Update: We only occasionally pair program. Doing that more consistently would obviously prevent a mistake like mine. If we were always pair programming, it would be hard for me to have even developed a fear of interrupting.
Perhaps the most important lesson of this whole experience is about my coworkers at bivio. No one berated me for my failure -- perhaps it was clear that I had that job well in hand. Instead they offered encouragement and stories of their own failures. In some way we bonded through my failure. We looked at the problem as a team and brainstormed various measures to learn from the experience. My mistake became an authentic opportunity to learn and grow, not to judge and condemn. That's not just a slogan on a motivational poster, but what really happened in action.