The Secret of Why Open Source Works

Self-Organization through Shared Goals

Open-source projects often have hundreds or even thousands of people participating in their development. Yet there is no formal management structure to coordinate their many and varied activities.

The traditional approach to managing a large group of workers has been to establish a strict hierarchy of managers controlling the activities of the people below them. Such large command-and-control organizations have been the norm in business and the military from the Roman legions to General Motors. Such organizations pay a big cost in the number of people needed as managers, the amount of communications they require for coordination, and the rigidity of their response to novel situations. A big problem is that people lower down in the organization rarely have any idea how what they are told to do corresponds to the larger strategy. In fact, according to studies reported in the book The Strategy-Focused Organization by Robert Kaplan and David Norton, over 90% of companies fail to implement their strategy largely because the strategy is not communicated properly to the people in the company.

At the second Jini community meeting, Lieutenant General (Retired) Paul Van Riper, the former commanding general of the Marine Corps Combat Development Command, spoke on how the Marine Corps had made a shift from top-down control to policies based on self-organization for use in combat. For example, instead of a platoon being commanded to capture a hilltop, they might now be instructed to set up an observation post on a hilltop in order to view enemy troop movements along an important roadway. If the initial hill is defended or inaccessible, the platoon can choose a neighboring hilltop that also satisfies the goal instead. Including the strategic purpose as part of each order results in a much more flexible and responsive organization. A similar organizational shift in the business world was presented by Tom Petzinger at the first Jini community meeting--and is the topic of his book The New Pioneers .

Open source takes this self-organization process several steps further: Rather than someone high up in the organization setting out the goals, the actual people using and developing the software discuss on public mailing lists or newsgroups what improvements are needed. People propose features or capabilities that they want to see added--usually because they want to use them themselves. If enough other users and developers agree that the feature would be useful, then some of them may volunteer to write the code to implement it. However, if interest is too low, nothing may happen.

Sometimes a single developer will care enough about an idea to code up a prototype of it, and, if enough people try the prototype and like it, it may become part of the official project or at least be kept as an option for those who want to use it. Note that this type of exploratory development is a common aspect of open source. It is a way of introducing new goals into the community through demonstration rather than just conversation. Some may complain that this is not efficient--the developer's effort is wasted if the innovation is rejected--but, as with evolution, efficiency is not the point. That developer may have had no interest in working on anything else aside from the innovation that was rejected and would not have worked on other parts of the project. So the work did not take away from existing efforts--although if the work had been adopted it might have meant that some existing modules would have been replaced. Also, even if a particular contribution is not incorporated into the project directly, the ideas behind it may very well find their way into later contributions.

Although the overall open-source project may have hundreds of participants, there always is a basic core team, usually consisting of fewer than 10 developers. With that small a group, informal communication suffices to coordinate development activities. In fact, the team works pretty much like any proprietary software development team does. Two main differences are that the open-source core team members are almost always geographically dispersed--so they communicate mainly via email and chat--and that they get immediate feedback from the rest of the community on their decisions.

For those larger open-source efforts (e.g., Linux, Mozilla, NetBeans, and Apache) that have many more dedicated developers, the work is divided into separate modules, each with its own small core team. This is a good example of Conway's Law--the architecture of the code follows the structure of the organization. This tendency to keep the code modular gives open-source software a flexibility and adaptability that is often lost in the more monolithic code produced by proprietary companies.

This pressure for modularization also can be seen in the evolution of project mailing lists: Whenever the traffic on a list or newsgroup gets too great, the community breaks the discussion into subtopics. The separation of the code and the associated discussions makes it possible for people with limited time to follow and actively participate in the project.

The project mailing lists also show self-organization in how questions get answered. For conventional products, companies employ people whose job is to answer customer questions. Open-source projects do not have such people, and at first glance we might expect that the main developers would be overwhelmed trying to answer questions from the thousands of users. This isn't what happens, however. Instead, the more advanced users answer questions from the intermediate users, who in turn answer questions from the novices, leaving only the extremely sophisticated or difficult queries for the developers to reply to.

Another way open source self-organizes is seen in how bugs are identified and fixed. The entire community is constantly using, and therefore testing, the software. Each member shares the goal of wanting to improve it and so will report problems, often locating the code causing the bug and including a possible fix. This full testing coverage and repair takes place with no coordination required and minimal core-team overhead.

Having common goals brings people into the project and through group discussion these goals are refined and developed. However, as the project goes forward, if part of the community has goals not shared with the rest, then the project may fork. For example, when some members of the NetBSD community decided that security was a major goal that wasn't being given enough emphasis, that group decided to create a new project, OpenBSD, that had as its main goal the building of the most secure Unix-like operating system.

Trust Enables Cooperation

The sharing of goals creates a reason for people to participate in the open-source project. To create a true community requires trust--trust that your contributions will not be scorned and that you won't be made a fool of or taken advantage of. When participants show each other mutual respect, it becomes possible for them to cooperate. If discussions degenerate into flaming and put-downs, if suggestions and contributions are laughed at, if decisions are made in an arbitrary way, then most folks will go elsewhere and not put any energy into the project.

A classic example of how many open-source projects work to build trust is shown by how a new developer acquires the right to modify the project's source code. When new developers join a project, they need to earn the respect of the current developers. They do this by submitting bug fixes and modifications and by participating in group discussions. Only after demonstrating that they can act responsibly will they be granted check-in privileges. Thus, developers know that the code they write can be modified only by people they trust; their code is safe from anyone who would arbitrarily or ignorantly change it.

Another concern people have is that they be treated fairly. Before investing effort into a project, most people want to be assured that they will have a voice in any major decisions. When decisions flow out of community discussions and represent consensus, that process reinforces the sense of community, people will be invested in carrying out the necessary work. If decisions seem arbitrary or go against their interests, then people have no motivation to participate and are apt to leave. Many open-source developers worry that a company will decide to sponsor an open-source project because it plans to take the community's work, package it up in a proprietary product, and deny the community access to it. If these fears are not laid to rest and the company doesn't earn the community's trust, then it will be hard to attract outside developers to work on the project and it is even likely that a competing open-source project will be created--possibly starting with the very code that the company donated.

It is important to remember that open source is based on a gift economy: Participants trust that the gifts they give to the project will be returned in kind by other community members. It's more like a family than a consortium--individuals create relationships with other participants through continuing interactions; the relationships that hold the project together are person-to-person, not company-to-company.

Achieving Critical Mass Via the Internet

The Internet is a major enabler of open-source projects. It helps people living all over the globe to find others with common interests. Through email, newsgroups, and the Web, these people can easily communicate, share code, and coordinate their activities. Without the Internet, it would be vastly more difficult, if not impossible, to run a successful open-source project.

An open-source project requires people, both users and developers. Many open-source projects start with a developer or two who have a program that they feel others will find useful. They can advertise their application through newsgroups and mailing lists, and they can make it available on relevant websites. Through the use of search engines and special-interest lists, people anywhere in the world can then locate and try out programs they might be interested in. People who like a program will spread the word. This makes it possible for groups to form based on very specialized interests--with the whole, networked world to draw on, it's hard to think of a topic that won't draw a quorum.

After finding a program they like, people are able to provide feedback to its creator via email and they can also discuss issues of using it with other users. The ease with which new mailing lists, newsgroups, and websites can be created makes electronic discussions both quick and simple. Email discussions among a large group of people can be easier both to follow and to separate by topics compared to face-to-face discussion. Mailing lists also allow the group to coordinate future development.

While the Internet makes it easy for new individuals to find an open-source project, whether or not they stay and participate will be determined in large part by the project's culture--is there a sense of community and trust, or instead is it cold, cliquish, and unfriendly? Unlike the business world where people assigned to a project are forced to work together, with open source, interested potential contributors who do not engage with the core developers--because of having a different vision or due to incompatible work styles--will probably go elsewhere.

Open Source Is an Evolutionary Process

The open-source process involves small, incremental improvements to some artifact, usually the source code for a computer program. It is essential that the starting point be a useful artifact, such as a working computer program. If the program is incomplete and does not do anything, then a small improvement will result in a similarly incomplete, broken program--so there is little motivation or satisfaction in making a small change to it. To finish the program and make it useful would require a large amount of work and initially only the originator has such a strong commitment.

A direct consequence of this is that an open-source project that starts around a great idea, but with no code written, should not expect the community to create a working application. The community can only be expected to help develop and refine the idea through discussion. If there are one or more developers already committed to writing the initial version, then this feedback on what the application should do can be invaluable--although without a working prototype it can also be ungrounded and too blue sky to be practical. The main point is that if you start an open-source project without a working prototype, do not expect the community to write the initial code for you. It just won't happen.

It may help to think about this in terms of biological evolution: Dead matter does not evolve;1 only living organisms can evolve. They do so as a result of random small changes. Some of the changes are improvements, whereas others are neutral or harmful to the organism. The environment then selects which of these changes are retained and which are forgotten, and these selections are passed on through reproduction.

For open source, every new idea posted to the project's mailing lists and each contribution of code is a potential change. With new ideas, the community acts to select which are to be discussed further and possibly become part of the project's goals. For code, the module owners are the ones to select which changes are accepted and added to the project's code base. Note that the module owners cannot dictate what other developers work on--in some sense, incoming contributions really are random.

Most changes will be small bug fixes or minor improvements, but occasionally someone will contribute ideas or code that is a major change, possibly taking the project in a new direction. As with biological evolution, these large jumps cannot be predicted. They are also experiments that may or may not succeed. Just like life, open-source projects must embrace these opportunities to remain vibrant.

Evolutionary processes tend to be profligate in their use of resources--they are not what a manager would consider efficient. However, through large-scale parallel exploration of possibilities, evolution discovers wonderful innovative solutions. Open source combines the directed efforts of proprietary software development, in the focused work done by the core team, with the open-ended contributions made by volunteers.

Co-Evolution of Software and Community

In an open-source project, software building and community building are intertwined. As the software matures, the community needs to keep up with it--the principle of slowly dawning insights applies to both activities.

An example is the Apache Incubator Project, which was created in October 2002. It provides an entry path into the Apache Software Foundation for projects and code bases whose owners wish to become part of Apache. The Incubator also provides documentation on how the Foundation works and how to get things done within its framework.

The Incubator is a community structure that was developed after the code base became large, sophisticated, variegated, and widely adopted. This required creating a controlled and self-explanatory mechanism for joining the community.

The Apache Software Foundation has apparently discovered the work-in-progress effect and is using it explicitly. At the top of the home page for the Incubator it says the following:

Notice: This document is a WIP (Work In Progress).2