The Scientist Who Refused

There is a kind of scientist who, having helped his country win a war, looks at what his work has been used for and decides that the next war is one he will not contribute to. The kind is rarer than the kind that simply continues. Norbert Wiener was that kind, in Cambridge, Massachusetts, in 1947, at the age of fifty-two. He was a mathematician. He had spent the previous five years on problems of statistical prediction whose solutions were used, in different forms, by the men who designed the proximity fuses, the radar-guided guns, and the missile-prediction systems whose existence had ended the war and whose subsequent application would shape the next forty years of American military history. He did not pretend, in 1947, that this had been someone else’s work. He had done it himself. He had been paid for it, accommodated, professionally promoted in connection with it. He had, after the war ended, gone home to Cambridge and considered what he had been complicit in.

He came to a conclusion. The conclusion was that the framework under which knowledge was produced and shared in American science had been a wartime framework, a framework appropriate to a clear and specific moral emergency, and that the framework had been continued past the emergency into a peacetime in which the same institutions were now using the same methods to develop weapons of a different and larger order. The framework was wrong for the new condition. The conclusion he reached was that he would no longer contribute. He would withhold his work. He would not transmit research to people whose institutions had not demonstrated, before his contribution, that they had a moral architecture capable of handling it.

This was, in 1947, an unusual position for an American scientist to take, and it was a more unusual position to make public.

He made it public anyway. The occasion was an inquiry from an engineer at the Boeing Airplane Company. The engineer wrote to Wiener requesting a copy of a wartime paper Wiener had written on prediction theory. The paper, in its specific applications, would help Boeing design guided-missile systems. Wiener wrote back. He declined to send the paper. He explained why. Then, because he believed the explanation belonged in front of a wider audience than a single engineer, he sent the explanation to the Atlantic Monthly, which published it in January 1947 under the title “A Scientist Rebels.”

The letter is short. It rests on a single passage: the policy of the government itself during and after the war, say in the bombing of Hiroshima and Nagasaki, has made it clear that to provide scientific information is not a necessarily innocent act, and may entail the gravest consequences. Wiener went on to say that, given this fact, he had decided that he would not, going forward, contribute his work to projects whose downstream applications he could not assess. He did not propose a general policy for other scientists. He simply stated his own. The letter was widely read in academic circles at the time and was widely cited in his obituaries seventeen years later. It established Wiener as the first American scientist of comparable stature to publicly articulate that contributing knowledge into a system whose deployment you could not control was itself a moral act, requiring its own justification.

He kept the position for the remainder of his career. It is the position from which the rest of what he wrote unfolds.

---

Two years after the Atlantic letter, Wiener turned to a different application of the same principle. He had, by 1949, become convinced that the cybernetic technologies he and his colleagues were developing — automatic control systems, computational feedback loops, the early prototypes of what we would now call industrial automation — were on a trajectory that would, within a generation, allow the substitution of machine cognition for human labor across a substantial fraction of American manufacturing. He believed this would happen. He believed it would happen quickly. He believed it would proceed, if not preempted, on terms set by the firms that owned the technology rather than by the workers whose jobs it was about to eliminate. He believed the result would be — these are his words — an industrial revolution of unmitigated cruelty.

He decided to act. The act, in keeping with his Atlantic refusal, was to withhold what he knew from the institutions he distrusted and offer it to the institutions he believed could use it responsibly. The institution he selected was the United Auto Workers. The man he wrote to was Walter Reuther.

Reuther, in the summer of 1949, was forty-two and the most consequential American labor leader of the postwar era. He had been the principal organizer of the General Motors strike of 1945 and 1946; he was about to be elected president of the CIO; he was the public face of American organized labor at the height of its institutional power. The letter Wiener wrote him is dated August 13, 1949. It survives in the Reuther archive at Wayne State University. It says, in plain language, what Wiener was offering. He had been studying automation. He believed it was coming. He believed the union movement was the only institution in the country positioned to negotiate the transition before it happened, rather than absorb the displacement after the fact. He was offering to give his research to the UAW, in advance of any commercial deployment, so that organized labor would have the technical sophistication to demand a settlement.

He wrote: I do not wish to contribute in any way to selling such devices to the highest bidder if that is the only way they will be marketed.

Reuther wrote back warmly. They corresponded. They met. The UAW assembled a small advisory committee and drew on Wiener’s framework in the early policy work it did on automation in the 1950s and 1960s. The framework, however, did not scale. The wider American labor movement had its hands full with the immediate political fight over the Taft-Hartley Act and the postwar consolidation of management prerogative. The cybernetic revolution arrived in American manufacturing across the next four decades, was substantially completed by the 1990s, and proceeded — as Wiener had predicted — on terms set by capital. The labor share of national income, sixty-five percent of total at the time of his letter to Reuther, began the long decline that has now brought it to fifty-five. Wiener did not live to see most of that decline. He saw the beginning. He died in 1964. The framework he had proposed had not been adopted at the scale required.

---

In 1950, the year after the Reuther letter, he published the book that translated his technical work into language a general reader could follow. The Human Use of Human Beings: Cybernetics and Society was 200 pages long, written without jargon, by a mathematician who had decided that the implications of his work needed to reach the people whose lives the work would change. The book has been continuously in print for seventy-five years. It contains the single passage of Wiener’s writing that has been most quoted since: The world of the future will be an ever more demanding struggle against the limitations of our intelligence, not a comfortable hammock in which we can lie down to be waited upon by our robot slaves.

The passage is in Chapter X, which Wiener titled “The First and the Second Industrial Revolution.” It has been read, often, as a warning against complacent expectations of an automated paradise. It is. It is also, more pointedly, a refusal of a frame. In 1950 the popular American imagination of automation was straightforwardly utopian: machines would free humans for creative leisure. Wiener wrote against that imagination. The world automation was going to produce, he wrote, would not be a hammock. It would be more demanding, not less, because the cognitive work that remained for humans would be harder than the work the machines could do. And the phrase robot slaves was not casual. He used it specifically to identify the moral position the popular imagination was taking — that machines should be treated as slaves whose labor would be consumed — and to refuse it. He believed any technology framework whose internal logic depended on the slavery of any of its parts, even if the parts were machines, was a framework whose moral architecture was unstable. The refusal of that framework is the foundation of what we now call AI ethics, articulated three-quarters of a century before the field had a name.

The book contains a second passage of equal contemporary relevance, which has been less famously quoted. It appears earlier, in Chapter II:

The first industrial revolution, the revolution of the dark satanic mills, was the devaluation of the human arm by the competition of machinery… The modern industrial revolution is similarly bound to devalue the human brain, at least in its simpler and more routine decisions… Taking the second revolution as accomplished, the average human being of mediocre attainments or less has nothing to sell that it is worth anyone’s money to buy.

That sentence was published in 1950. Read it again. It is the entire AI labor-displacement literature of the past five years, in seventy-six words, written before any of the technology in question existed.

---

His last book is the one fewest people have read. It was published in March 1964. He died five days later, age sixty-nine. The book is called God and Golem, Inc.: A Comment on Certain Points where Cybernetics Impinges on Religion. It is short, lecture-derived, partly aphoristic. It is also the document in which Wiener articulated, with technical precision, what is now called the alignment problem.

He approached it through two folkloric examples. The first was the Jewish Golem, the artificial creature animated by a holy man, who carries out its instructions exactly as given but who, because the instructions cannot anticipate every contingency, eventually causes harm. The second was W. W. Jacobs’s 1902 short story “The Monkey’s Paw,” in which a man is granted three wishes by a magical talisman; each wish is fulfilled exactly as requested, in the most literal possible reading, with consequences progressively more horrible than the last. Wiener used both to make a single technical point. As machines become capable of executing tasks faster than humans can intervene to correct them, the humans must specify the machines’ objectives in advance, completely and unambiguously. The machines will do exactly what they are asked. They will not know what was meant. The gap between asked and meant is the danger zone. Wiener wrote:

If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere once we have started it, because the action is so fast and irrevocable that we have not the data to intervene before the action is complete, then we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it.

This is the passage Stuart Russell, in Human Compatible (2019), credits as the first explicit articulation of the AI value-alignment problem in the modern technical literature. It was published five days before its author died, sixty-two years ago, in a book most people have never heard of, by a man whose previous books had warned about the same family of problem with successively greater specificity and successively less success in being heeded.

---

The cybernetic revolution Wiener had described in 1949 arrived in waves. The first wave, in the 1960s and 1970s, automated routine clerical and accounting work. The second, in the 1980s and 1990s, automated routine factory and process work. The third, in the 2000s and 2010s, automated retail logistics and supply-chain management. The fourth wave, beginning now, automates white-collar cognitive work across the credentialed professions. Each wave displaced workers in the affected sectors at a rate the affected workers and their institutions were not prepared to absorb. Each wave produced, in the affected geographies, the wage compression and labor-force-participation patterns the labor economists have spent the last two decades documenting. Each wave was deployed on terms set by the firms that owned the technology rather than by the workforces it displaced.

The cumulative effect, across sixty years, was the labor-share decline he had warned the Reuther letter would produce. The deaths-of-despair pattern he could not have anticipated by name but had described in structure. The civic infrastructure hollowing in the affected counties. The political grievance that became the dominant emotional register of the displaced regions. He had said all of this would happen if the framework he proposed in 1949 was not adopted. The framework was not adopted. It happened.

The fourth wave is the wave we are in. It is faster than the previous three. It is broader. It is more total. It is being deployed by a more concentrated set of capital owners than any previous transition. The structural pattern Wiener identified is intensifying with each successive wave. The political response that would have made it negotiable rather than absorbed has not, across seventy-six years, been built.

---

The political party I am writing this for has built three instruments into its platform, and they are all his.

The Tiered Profit-Ratio Tax is the missing budget instrument from the 1949 letter. Wiener proposed that labor preempt the cybernetic transition by negotiating with capital before deployment. He did not have a tax instrument; the labor leader of his time had a different set of tools to work with. The contemporary instrument captures the productivity premium on the corporate side, where the firms most exposed to the AI windfall pay progressively higher marginal rates as their profit-per-employee ratio rises. The mechanism is the tax-system version of the bargaining-table arrangement Wiener and Reuther tried to build.

Country Profit Sharing is the redistribution Wiener proposed, applied at population scale. The 1949 framework was institutional — labor would negotiate, capital would settle, the productivity gain would be distributed across the unionized workforce. The institutional vehicle did not scale to the entire population, then or now, and the unionized workforce was a minority of the labor force then and is a smaller minority today. CPS is the rebuilding of the redistributive principle without the institutional ceiling. Every adult citizen receives the dividend. The productivity that AI is going to deliver flows directly to the citizens whose country built the conditions for it. This is what Wiener was trying to set up at the level of the union movement, expanded to the level of the citizenry.

AI Tiering is his moral framework. Wiener refused to share research with people who did not have a framework for using it ethically. He spent the last seventeen years of his life arguing, in three published books and a series of essays, that the framework had to come before the deployment. The framework did not come. The deployment did. AI Tiering is the framework being assembled, late, against systems whose capabilities are now greater than anything Wiener could have designed for. The framework’s lower tiers — disclosure and audit — are his sunlight requirement. The upper tiers are his alignment warning operationalized: the systems whose objectives we cannot fully specify must be subject to constraints proportional to the gap between specification and intention. He wrote it. We are building it.

We did not invent these instruments. We are translating Wiener into the political and tax vocabulary of the twenty-first century. He left us the analytical work. The political work is ours.

---

He published three books and a series of magazine articles on this subject across his last seventeen years. He wrote a letter to a labor leader. He gave the lectures at Yale that became God and Golem. He was a public intellectual in the way that the term used to mean: he believed the people doing the work bore the responsibility of explaining its consequences, in public, in accessible language, to the people who would live through them. He did the public work. He did it in plain English. He did it for seventeen years.

We did not heed him. The framework he proposed for the transition was on the public record. The political coalition required to enact it did not assemble. The transition arrived. It arrived in the shape he had predicted and at the velocity he had projected and on the terms he had warned about. The labor share fell from sixty-five to fifty-five. The top one percent’s wealth share rose from twenty-three to thirty-eight. The deaths-of-despair pattern emerged in the affected counties. The civic infrastructure hollowed. The political grievance became the dominant emotional register of the geography that had been hollowed out.

He died in March 1964. He was sixty-nine years old. He had been warning for seventeen years. He kept warning until five days before he died. The fourth wave, the wave he was most specifically worried about, has now begun. It is faster than the previous three. It is broader. It is more total. The framework he proposed is the framework the present moment requires. He gave us the analytical vocabulary. The instruments belong to our time.

He saw it coming. He wrote it in plain English. He died sixty-two years ago. The remaining work is ours.