My book, Bending the Law of Unintended Consequences, was inspired in part by Robert K. Merton (1910-2003), a leading American sociologist who was active during the middle and late twentieth century. Merton’s 1936 paper “The Unanticipated Consequences of Purposive Social Action” was the first academic analysis of the Law of Unintended Consequences (LUC). It offers a powerful lens for understanding why critical decisions go awry, and points the way to alleviating LUC’s ill effects. This post summarizes relevant aspects of Merton’s research and career.
After earning his doctorate at Harvard (1936), Merton spent most of his career teaching at Columbia University, where he co-directed the Bureau of Applied Social Research. In 1994, Merton became the first sociologist to be awarded the National Medal of Science. He received an impressing collection of other honors and awards, including Guggenheim and MacArthur Prize Fellowships, twenty-nine honorary degrees from universities around the world, and dozens of lectureships and memberships in honorary societies. (See Merton’s Curriculum Vitae at http://www.garfield.library.upenn.edu/merton/list.html.)
Merton laid the foundations for the sociology of science. His doctoral thesis demonstrated the positive influence that religion exerted on the development of science in the seventeenth century. This was a paradoxical result that ran contrary to the prevailing view at the time that religion opposed and sought to undermine the legitimacy of emerging sciences such as astronomy and physics. Using a precursor to sentiment analysis, Merton showed how the Puritan ethos (pursuit of facts and the truth) drove acceptance of the scientific method by influential thinkers in Europe (despite the fact that Puritans were concurrently opposing astronomical theories advanced by Copernicus, Kepler, and Newton with great vigor.) Merton published his paper on LUC the same year he earned his doctorate. Merton later noted that he wrote the paper to provide a theoretical framework to support his thesis, an empirical study of religious beliefs that produced unintended social consequences.
Prior to Merton’s work, sociologists treated viewed scientific inquiry as an idealized and dispassionate human endeavor. Merton’s thesis catalyzed a dramatic shift in perspective, generating interest in the psychological and social dimensions of science. Merton later defined a set of core values that ground scientific inquiry, and investigated practices such as professional recognition and priority of discovery where observed social behaviors can conflict with these norms. For example, he found that famous scholars tend to accumulate ever more citations, funding, and awards, while less well-known academics become increasingly disadvantaged. This behavior deviates from the norm of universality, which holds that scientific work should be evaluated objectively, irrespective of source. He dubbed this the “Matthew Effect,” after the biblical passage about the rich getting richer while the poor get poorer. Merton later studied how advantage and disadvantage accumulate more generally in society at large. Decades later, Merton’s work in this area continues to inspire research in economics, business, and social networks.
Merton produced numerous other theories and insights that are widely recognized in the social sciences and popular culture. For example, Merton and his long-time colleague Paul Lazarsfeld investigated how ideas are spread through social communities back in the 1940s. They found that dissemination was driven by a limited number of well-respected and persuasive individuals at diverse social levels, or influential as Merton called them. Sixty years later, researchers that study how ideas propagate through social networks find the same pattern.
Merton also developed the concepts of role models and self-fulfilling prophecies. He investigated and popularized the concept of serendipity in scientific discovery (i.e., unanticipated observations that prove to be fortuitous). During the course of his empirical studies on mass media, Merton originated the concept of focus group, an indispensable method for modern market and political research. Merton even investigated the tendency for the originators of ideas that become mainstays of popular culture to fade into obscurity. Ironically, Merton is the poster child victim for this social pattern, which he called “obliteration by incorporation.”
Throughout his career, Merton worked to bridge a major split that developed in American sociology between theoreticians who followed in the footsteps of the 19th century European pioneers, and field sociologists, who gather and analyze empirical social data using polls and surveys. Merton continually stressed the importance of linking theory with empirical research: without theoretical grounding, empirical social research is fragmentary, inconsistent, and blind, whereas lacking supporting observational data, social theory tends to be sterile, abstract, and speculative. Merton also believed that sociology was too immature a scientific discipline (e.g., compared to physics) to support “grand theories” that attempted to define the complete structure and function of societies. Instead, he advocated for what he called “middle-range” theories that addressed more bounded and tractable topics and were firmly backed by empirical evidence.
One of Merton’s early successes was to develop just such a middle range theory of social anomie. Anomie refers to a breakdown of social standards leading to a lack of cohesion and solidarity within a society. Merton proposed that anomie could arise when individuals lacked sufficient means to achieve the cultural goals that societies encouraged, such as power, wealth, fame, and intellectual accomplishment. He then identified five types of social adaptations to this conflict – conformists, innovators, ritualists, retreaters, and rebels. Merton also investigated social structures such as schools, churches, bureaucracies, and organized crime. He asserted that such groups have intended or manifest functions. For example, schools are expected to educate children with the knowledge and skills that they will need to succeed in society. He argued that social institutions also develop latent functions over time. For example, schools provide supervised recreational and other social activities in addition to education. Latent functions are not only unintended, but often irrelevant or opposed to manifest functions. The manifest function of, civil service regulations is ensure a competent dedicated staff of workers to make government more efficient. But they also have the latent function of producing a rigid bureaucracy. In short, Merton showed that social institutions spawn unintended consequences, similar to purposive social actions such as decisions.
Merton’s conception of social groups, culture, and individual relationships to these structures was both complex and dynamic. He argued that the behaviors, beliefs, attitudes, and motivations of individuals are shaped by their roles in various groups. But Merton also recognized that individual behaviors and attitudes, taken collectively, can drive social groups to evolve over time as well, both incrementally and more radically (e.g., the shift in public opinion and Supreme Court ruling in support of gay marriage). Merton’s insights into the feedback processes that enable these social interactions and shifts are central to his analysis of LUC.
Merton possessed just the right intellectual background, research interests, perspective, and methods, and perspective to appreciate LUC’s full scope and explore all of its primary causes. He was knowledgeable about a broad combination of fields, including social theory, the history and methodology of science, and individual and social psychology. For example, one of his preferred research methods was functional analysis, which identifies the consequences of social phenomena for the various social structures in which they occur. Finally, Merton possessed an ironic sensibility; he relished uncovering and exploring the surprising conflicts, contradictions, paradoxes in outcomes of purposive social action. Thus, it was no accident that it was Robert Merton who christened and analyzed LUC.
The psychologists who followed Merton and worked out the science behind cognitive biases and bounded rationality lack his keen sensitivity to the social and cultural contexts for critical decisions. This has led to a bifurcation in modern thinking about critical decision-making: psychologically-minded management consultants now champion debiasing methods, while decision scientists advance analytic modeling and simulation methods to push back against bounded rationality. Both of these approaches to improving critical decision-making are parochial; by neglecting the nuances of social dynamics that Merton savored, they invite unnecessary and painful encounters with LUC.
Comments