Newly unsealed court documents revealed that Microsoft and its close partner OpenAI were concerned over OpenAI’s use of millions of news articles to develop artificial intelligence (AI) systems, and the threat AI products could pose to the publishers whose work helped train them.

The documents include stark internal assessments. Microsoft director of applied science Brent Hecht described large-scale copying of online content as an “astonishing theft of unprecedented proportions” and perhaps the “largest theft of labor in human history”. OpenAI executives, meanwhile, discussed how products such as ChatGPT could increasingly substitute for visits to publishers’ websites.

The findings are the latest development in The New York Times’ case against OpenAI and Microsoft. In December 2023, NYT sued both companies, alleging they had used copyrighted articles without permission to train AI models, and that products such as ChatGPT could reproduce or closely mimic its journalism.

Story continues below.
Subscribe to see fewer ads.

Microsoft and OpenAI are both defendants because of their close partnership, which included Microsoft providing computing infrastructure and data access to OpenAI and incorporating its AI technology into products such as Bing and Copilot.

Both companies have argued that using copyrighted material for AI training can qualify as fair use under US copyright law. Fair use permits copyrighted material to be used without permission in certain circumstances.

US courts consider four factors in deciding whether a use is fair — the purpose and character of the use, the nature of the copyrighted work, how much of a copyrighted work was used, and the effect of the use on the potential market for the original. The dispute therefore rests in part on whether training is sufficiently transformative, and whether the resulting products harm the market for the original works. Courts weigh these factors on a case-by-case basis.

The newly unsealed material could bear directly on this analysis, revealing what employees and executives at the two companies understood about the provenance of training data, the substitutive effect of AI products, and their impact on referral traffic to publishers.

Here are four takeaways from the filings:

Story continues below this ad

1 – Microsoft employees raised concerns about the extent of copying

Stark concerns about widespread copying were raised by Microsoft director of applied science Brent Hecht in a January 2023 internal memo. Hecht described large-scale copying for AI training as an “astonishing theft of unprecedented proportions” and perhaps the “largest theft of labor in human history”.

A subsequent Microsoft memo raised a similar concern, saying the company’s AI content strategy had “started a ‘doom loop’” that relied on materials available on the internet, but could simultaneously undermine economic incentives to produce such materials.

Microsoft has dismissed this contention, saying that Hecht’s comments were made in his personal capacity and did not represent the company’s views.

“Microsoft’s position is set out in its court filings, which explain why these transformative uses are consistent with copyright law and why Copilot is not a substitute for publishers’ journalism,” a Microsoft spokesman said about its AI assistant. A separate company filing described Hecht as a research academic who also worked at Northwestern University, and said he was employed by Microsoft “to present divergent and asymmetric perspectives”, not to be a decision-maker.

Story continues below this ad

However, an employee’s description of the copying as “theft” does not determine whether it constitutes copyright infringement. Whether the use qualifies as fair use is a legal question for the court, which weighs the statutory factors.

2 – Both companies acknowledged that AI could eventually substitute for publishers

The filing notes that both companies had considered the possibility of AI eventually substituting for the sources they relied on for information. OpenAI’s head of ChatGPT, Nick Turley, wrote that the company’s products were “largely substitutive, period”, and predicted that they would become more substitutive as they improved.

Microsoft CEO Satya Nadella also acknowledged in a deposition that chatbot answers could substitute for visiting the underlying website.

Story continues below this ad

Microsoft’s own data, as presented in the filing, also appears to support this contention. Click-through rates to NYT websites from Bing’s AI product were between 87% and 93% lower than from conventional Bing Search. Other publisher groups cited in the litigation also saw substantial declines.

This could become important to the publishers’ fair-use argument. The fourth factor in the US fair-use test considers the effect of a use on the potential market for, or value of, a copyrighted work. The lower click-through figures do not by themselves establish copyright infringement, but could support the publishers’ argument that AI products substitute for and harm the market for the works used to build them.

3 – Through ‘Project Taxi’ and ‘Project Mango’, Microsoft helped OpenAI access copyrighted data

The filings also provide insights about how the companies acquired materials to train OpenAI’s models.

Story continues below this ad

According to the filings, Microsoft supplied data from its Bing database, the “Bing Index” to OpenAI over three years between 2019 and 2022. Microsoft has described this as a compilation of “billions” of webpages it gathered to support its Bing search engine. Codenamed Project Taxi, the resulting data transfer included NYT content which Microsoft sold to OpenAI in an undisclosed transaction.

According to the filings, Microsoft subsequently launched “Project Mango” to help OpenAI “collect as many of the documents as possible for…training the large language models.” In the process, Microsoft designed the “Mango” web crawler, which it used to copy web content on OpenAI’s behalf. The resulting Project Mango dataset contained copies of at least 160,903 unique works belonging to the news publishers involved in the litigation, according to the filing.

The documents also contain allegations that OpenAI employees found ways to access paywalled material. In one exchange cited in the filing, an OpenAI researcher told company president Greg Brockman about a “hack to get around” the NYT paywall; Brockman replied, “ah nice”.

The financial details of both transactions remain undisclosed. The publishers allege that their copyrighted works were copied and used for AI training without their permission.

Story continues below this ad

In a deposition, Nadella said that “anything that is paywalled should be licensed by anyone who wants to use it.” He also said that if he “had been made aware that OpenAI had scraped and trained on information that was behind a paywall,” he would have invoked Microsoft’s right to require OpenAI to retrain its models.

Whether copyrighted material was lawfully obtained and whether its subsequent use qualifies as fair use can raise separate legal questions.

4 – These findings alone do not prove that Microsoft and OpenAI infringed copyright

None of the internal comments amount to a legal finding that the companies infringed copyright. The court will decide whether their use of the publishers’ works was fair by weighing the four factors and the circumstances of the case.

Story continues below this ad

But the publishers can now point to internal discussions and company data to support their argument that AI products can substitute for the journalism used to build them, and harm the market for that journalism.

For users, the issue is fairly straightforward: AI tools can increasingly answer questions without requiring us to visit the websites where that information came from. The filings show that OpenAI and Microsoft were aware of what this could mean for the publishers whose work these systems rely on.