In the good old days before the Internet vanquished time and distance, software developer teams were autocratic organizations as shown in Figure 1a. Face-to-face teams developed in a top-down linear fashion under the auspices of a chief programmer. This required central planning and control, and strict adherence to sequential “waterfall” methods. The time and effort to develop a program of m lines of code grew exponentially, mt, largely because of this organizational structure. The resulting program structure mirrored the top-down structure of the organization according to “Conway’s Law”—that the structure of the system mirrors the structure of the organization that designed it [1 , 2]. For the most part, software systems mimicked the structure in Figure 1a.
But then something unusual happened. Software structures underwent a radical change as their development teams became distributed across time and space, as shown in Figure 1b. Instead of a centrally planned, top-down, linear structure, code structure became network-like. Shortage of programmers, lower labor costs in emerging countries, and lifestyle preferences of the extreme programmer all contributed to this tectonic shift. As a consequence of socio-economic pressures and the Internet, a different kind of development process emerged. And in keeping with Conway’s Law, the structure of code followed suit.
The code base has become the central actor in both organizational and program structure. Instead of a top-down, hierachical structure, code now mirrors an organizational structure whereby the code base itself is central, and programmers (actors) are distributed. But, the distributed team of actors made additions, deletions, and modifications based on a set of distributed rules instead of a centrally planned schedule of behaviors.
The resulting distributed network-like organizational structure and its corresponding program structure have more in common with ants than humans. Like ants, bees, and termites, software development has become stigmergic [3]. The emergence of stigmergy among programming teams has gone largely unnoticed. But once we get it, software development teams can improve quality and lower costs even more by duplicating ants, bees, and other insects that have perfected stigmery over the past million years. Here is where stigmergy fits in the new organizational structure of Figure 1b, which dictates the organization of both code and people.
Stigmergy
The French entomologist Pierre-Paul Grassé described a mechanism of insect coordination he called “stigmergy”—work performed by an actor stimulates subsequent work by the same or other actors [4 , 5]. That is, the state of a building, code base, highway, or other physical construction determines what needs to be done next without central planning or autocratic rule. The actors—insects or programmers—know what to do next, based on what has already been done. This intuitive urge to extend the work of others becomes the organizing principle of modern software development.
Stigmergy ensures tasks are executed in the right order, without any need for central planning, control, or direct interaction between the actors performing the work. Stigmergy solves the coordination paradox problem of how insects of very limited intelligence, without apparent communication, manage to collaboratively tackle complex projects, such as building a nest or writing a program. For example, ants find the shortest route to food in optimal time, and simple-minded termites build hives without central control mechanisms or intelligence oversight. Programmers know what to do by looking at the code base and email messages from fellow programmers. A chief programmer is not needed to dictate the next state of the code base
Grassé coined the word stigmergy from the Greek word, stigma, which means “mark or puncture” and ergon, which means “work, action, or the product of work.” According to Gassee, stigmergy is “the stimulation of workers by the very performances they have achieved.” Modern programmers build software by stimulating each other to add, modify, or delete code segments based on the current state of “work, action, or the product of work.” That is, they modify code based on the state of the code base [6]. Tesseract is a tool for facilitating stigmergic code development in a distributed team organized according to Figure 1b [7].
The organizational structure of Figure 1b is completely compatible with stigmergic software development. At its core is the code base—the object to be constructed—and the distributed, loosely managed programmers spread throughout space and time, but coordinated by a combination of stigmergic intellect and markers embedded with the code base—markers may be as simple as email or comments, or as sophisticated as code reviews.
Ants use a special type of chemical marker—pheromones—to highlight the results of their activity. This type of indirect stimulation, not by the work itself but by special markers, is called “marker-based stigmergy.” Similarly, distributed programmer teams involved in cooperative code development across the Internet lay down analogous pheromones for their unseen and virtual collaborators when they augment code development by email interaction, comments, phone calls, documentation, and webcast meetings [8]. In fact, the stronger the marker system, the better the resulting code [9].
The essence of modern software development is stigmergic intellect and markers embedded within the code base. Markers make stigmergy more efficient, by more reliably focusing a programmer’s attention on the most relevant aspects of the work that needs to be done. However, it incurs a cost because programmers must perform the task of manufacturing markers in addition to the work itself. But, in the Internet age of global software development, this cost is more than compensated for by speed, agility, and lower costs.
If we want to advance the art and science of software development, we should direct our attention to the insect world of stigmergy. This means fully comprehending the value of decentralized collaboration and pheromones or markets embedded in the code, itself. If we practice what ants and termites already know, we can build systems that any individual ant, termite, or human cannot fully comprehend. This is the true meaning of collaboration.
References
[1] M. Conway, “How Do Committees Invent?” Datamation, vol. 14, no. 4, 1968, pp. 28–31.
[2] J.D. Herbsleb, and R.E. Grinter, “Architectures, Coordination, and Distance: Conway’s Law and Beyond,” IEEE Software, vol. 16, no. 5, 1999, pp. 63–70.
[3] Francesco Bolici, James Howison, and Kevin Crowston, Stigmergic coordination in FLOSS development teams: Integrating explicit and implicit mechanisms. bolici@unicas.it. OrgLab: Department Economics and Law, UniCLAM, Italy.
[4] Grassé, P. P. (1959). La reconstruction du nid et les coordinations interindividuelles chez Bellicositermes natalensis et Cubitermes sp. La théorie de la stigmergie: Essai d’interprétation du comportement des termites constructeurs. Insectes Sociaux, 6(1), 41–80.
[5] Heylighen, F. Stigmergy as a Universal Coordination Mechanism II: Varieties and Evolution. Evolution, Complexity and Cognition Group, Vrije Universiteit, Brussels.
[6] Dabbish, L., Stuart, C., Tsay, J. and Herbsleb, J. . Social Coding in GitHub: Transparency and Collaboration in an Open Software Repository. In Proceedings, Computer-Supported Cooperative Work, Seattle, WA, 2012, 1277-1286.
[7] Sarma, A., Maccherone, L., Wagstrom, P., and Herbsleb, J. Tesseract: Interactive Visual Exploration of Socio-Technical Relationships in Software Development. In Proceedings, International Conference on Software Engineering. Vancouver, Canada, May 16-24, 2009, 23-33.
[8] Kwan, I., Cataldo, M., and Damian, D. Conway’s Law Revisited: The Evidence for a Task-Based Perspective. IEEE Software 29, 1 (2012), 90-93. doi:10.1109/MS.2012.3
[9] Cataldo, M., and Herbsleb, J. D. Coordination Breakdowns and Their Impact on Development Productivity and Software Failures. IEEE Transactions on Software Engineering 39, 3 (2013), 343-360.
Comments