Variously attributed to Benjamin Disraeli, or Mark Twain, I have long been a fan of the quote; “There are three kinds of lies: Lies, damn lies, and statistics”. For me, this quote has always rung particularly true in the world of software development. As a developer myself, I always cringed when I heard management types harp naively on LOC counts, or Function Point analysis results, or, egad! Attempts at quantifying software development productivity. Software metrics, as used historically, show pieces of the picture, but are of dubious value overall to software developers or people managing the software development process.
Wikipedia has a very good discussion of the shortcomings of software metrics, including a comprehensive and, dare I say — a tad defensive — list of challenges with defining and applying them:
Unethical: It is said to be unethical to reduce a person’s performance to a small number of numerical variables and then judge him/her by that measure. A supervisor may assign the most talented programmer to the hardest tasks on a project; which means it may take the longest time to develop the task and may generate the most defects due to the difficulty of the task. Uninformed managers overseeing the project might then judge the programmer as performing poorly without consulting the supervisor who has the full picture.
Demeaning: “Management by numbers” without regard to the quality of experience of the employees, instead of “managing people.”
Skewing: The measurement process is biased by the act of measurement by employees seeking to maximize management’s perception of their performances. For example, if lines of code are used to judge performance, then employees will write as many separate lines of code as possible, and if they find a way to shorten their code, they may not use it.
Inaccurate: No known metrics are both meaningful and accurate. Lines of code measure exactly what is typed, but not of the difficulty of the problem. Function points were developed to better measure the complexity of the code or specification, but they require personal judgment to use well. Different estimators will produce different results. This makes function points hard to use fairly and unlikely to be used well by everyone. Big Apple~2 Bites (by Arunabha Sengupta) is a fictional work that contains a detailed and humorous discussion on the theoretical and impractical approach of the Software Community towards measurements.
Uneconomical/Suboptimal: It has been argued that when the economic value of measurements are computed using proven methods from decision theory, measuring software developer performance turns out to be a much lower priority than measuring uncertain benefits and risks.
Yikes. So why the heck are we even bothering to gather metrics with DevCreek? I mean, Intelliware is not generally supportive of unethical, demeaning, skewing, inaccurate, and uneconomical/suboptimal initiatives. Why is DevCreek different?
You know what? Despite the kvetching about the naïve or malicious use of metrics, the act of objectively measuring an activity is important. CRUCIALLY important, actually. For those who have been paying attention, all of science is based on measuring stuff. Software development is not immune from measuring, it is just a difficult, special case.
We believe that the problem with metrics to-date is that the philosophy behind gathering and using the metrics is misguided. To-date, Taylorism has ruled. Developers are thought of (and treated) as perfectly interchangeable units of production. They are not, and it is wrong to assume otherwise.
DevCreek thinks that Deming’s perspective on metrics is the correct one. He used numbers to objectively and accurately monitor processes with transparency to teams responsible for the process. These measures, in turn, provide feedback to the team to fuel discussion, learning and the development of solutions. Rather than Tayloristically break down a process into tiny steps and measure “productivity”, Deming strove to find key, statistically valid measures that could provide concrete feedback to process operators, to allow them to make changes before problems get out of hand.
We see Deming’s perspective on metrics as an extension to the basic principles of Agile software development. Transparency. Communication. Community. Responsibility. Learning. Feedback. Courage. Sharing. Lastly, keeping it simple and lean.
DevCreek aims to embrace these principles and see if we can’t be leaders in objectively measuring success and quality in software development.
Time for the obligatory recruiting request – we have lots of work to do to meet our objectives – we are looking for folks to help! Drop us a note if you have any questions or ideas. Better yet – if you haven’t already, sign up for DevCreek, learn more, or just watch some of the Open projects using DevCreek already.
Internally, here at Intelliware, we will be having a Lunch and Learn on DevCreek in the next couple of weeks. We are also actively looking for Metrics “Evangelists” — developers with an interest in a particular metric, and a desire to build out the data gathering tools and DevCreek API’s. Talk to me, David Jones or Gordon Cameron if you are interested!