For a century the world has experienced an unprecedented exponential and continual economic growth. This growth is driven through replacing manual work with machines. In order to maintain growth while facing demand for increasingly flexible products to lower prices, businesses need to apply more sophisticated software systems. Traditional rigid, black-box software is expensive and forces adaption to severe restrictions. Outsourcing yield little control over results with high risks of total failure and leaves business vulnerable. The only reasonable way ahead is to use best-in-class open architecture software and excel in-house at software adaption. The successful business of the 21st century can no longer specialize in production only, but need to excel at software and algorithms as well.
Creating great software is a serious challenge and unfortunately many ambitious software ventures fail because traditional factory or office management principles are transferred over to software production. Traditional management principles and tightly controlling employee time and task allocation does not translate well to software development.
Traditional management principles for software production
Imagine programmers as great painters of classic art. A true artist is inspired and challenged by what he can accomplish and he needs artistic freedom to be creative and engaged. Precisely as with classic art, programmers come at very different skill levels. There are programmers with basic skill and programmers with great skill. The difference is very large and there is no way 10 or even 100 basic programmers can replace a great one. 100 painters could not paint DaVinci's The Last Supper. The best programming is produced when focus, flow, inspiration and enthusiasm is present. A great programmer forced to work on tiny issues or forced to navigate and decipher a complex mess will not be enthusiastic or inspired. His great talent and ability is wasted.
If we accept the view of programmers as artists - creative, engaged, and driven by a strong internal desire to produce quality results - it becomes abundantly clear that traditional tightly controlling management principles do not fit. It is also clear that the programmer is a valuable asset you want to stay onboard long-term. It is difficult and expensive to replace good programmers. Just make sure that a programmer is not valued for the wrong reason - that he is the only one that understands some complicated module. Continuous cleanup and improvement counteracts such unhealthy relationships.
Complexity is the enemy
Computers and humans interact through a user interface (UI). The UI is much more than buttons, sliders and windows - the UI presents an intuitive model for manipulation and feedback. Great software with a poor UI is useless. UI needs design, careful consideration and effort. It is the management's job to make sure sufficient time and resources exist for this important task. In real life management typically lack involvement and understanding and UI design is low priority or even ignored.
Program code is analogous to UI. The code is the UI between programmers and the computer. Poorly designed code becomes a puzzle to suck in the unaware to waste time and money. The clever programmer simplifies the code until the bugs disappear. The simplification process may introduce a few new bugs, but these are trivial to find and eliminate through good Q&A. A great manager continuously encourages cleanup, simplification and improvement.
Initial understanding of a software development project is always incomplete. Requirements keep changing from (a) improved understanding and (b) external sources. When the shift created by (a) and (b) is sufficiently large, the early conceptual design must be reworked, redesigned or even rewritten. Failure to acknowledge change and failure to allow and allocate sufficient time for change paves the road to failure. Everything ends up as an expensive mess which barely fits its purpose. It gets exceedingly difficult to back out of. And at this point there will be no time available - it is all consumed by patching and bug fixing.
Complexity is the number one enemy in the face of millions of lines of code and potentially gigabytes of data for any sufficiently advanced solution. A software effort will fail from unmanaged complexity. Complexity is an enemy that creeps in from many sources even with the best of intentions. Programmers are super bright and often fail to understand that code is complex. It takes great effort to turn a solution around and lower its complexity and it is more fun to add new features. Constantly adding to a software project over months and years creates a monster of complexity. The end result is a workload where the majority of costly programmer time is spent on debugging to keep the system alive.
Programmers are clever people that keep exercising and improving their brains throughout their career. The high-level of intelligence comes at a price. Programmers are stimulated by complexity and do not even perceive it as hard. On top of this, pressure from management results in constant additions to a software system leading to a complex mess that no one, even with super high IQ, can understand. In the end, such a system becomes slow, cumbersome, bug-prone, costly to maintain, and results in a terrible user-experience.
Programmers are generally weak on (1) simplification, (2) communication and (3) exercise. A great working environment encourages good looking code, simplification, cleanup, refactoring, code improvement and rewrites, freeing up room for invention and progress. A great working environment encourages communication at all levels. A great working environment encourages exercise for a long-term healthy workforce.
In general the (a) programmers and managers collaborate to create a complex, incomprehensible mess, (b) written in a programming language that poorly supports the problem domain, and (c) through a process that slows down to a crawl as the project grows. The software coagulates and adapts badly to new features and requirements. Expensive programmer time is spent on understanding the mess, fixing bugs and waiting for slow build times.
Current development methods take advantage of bug tracking software to efficiently sort through the constant overload of tasks, feature requests and bug reports. Bug tracking software, however, tend to chop up tasks and requirements into tiny pieces. This encourages small local fixes instead of taking in the bigger picture. Large numbers of small fixes creates a messy patchwork of high complexity. Management typically encourage this simply through measuring the number of closed issues.
Managers push programmers to add more and close more issues with tough time constraints and zero slack for fixing and improving code. UI Design and reducing complexity is not perceived as a value adding and actively discouraged. The necessary refactoring and cleanup of code is regarded as a source of errors. Yes - simplification may introduce a few new bugs, but these are trivial to find and eliminate through good Q&A when the code is clear, while fixing a bug in complex code is costly and most likely will introduce new hard to understand bugs.
A manager must create a culture of low tolerance for complexity, bugs, disruptive pauses and strong encouragement for simplification, fundamental requirements and direct design and Q&A involvement should come directly from involved managers and senior programmers. Create a culture of low tolerance for complexity, bugs, disruptive pauses and strong encouragement for simplification, design, creativity, and flow.