The Opaque, Incomplete Corpus Linguistics Analysis in the Mask Mandate Ruling

As virtually all news outlets have reported, Judge Kathryn Kimball Mizelle of the United States District Court, Middle District of Florida, recently struck down the CDC's requirement that people wear masks in certain settings, including plains, train stations, busses, and other public transit settings. The ruling is here. The case is Health Freedom Defense Fund, Inc. v. Biden.

Reaction to the ruling has been swift. Most of the commentary is an early phase, with initial reactions widespread on Twitter. For some initial, detailed discussion, Ilya Somin writes about the opinion here, suggesting that it is more defensible than critics claim, but that it may still be vulnerable to being overturned on appeal. The Wall Street Journal's editorial page writes favorably of the ruling here. The Washington Post has dueling takes for and against the ruling. 

More undoubtedly will be written as the days go on and as commentators parse the 59-page decision. Initial reactions suggest that the textualist methodology employed by the court is lacking. (See, e.g., here and here). I want to focus on one portion of the analysis: the court's use of corpus linguistics methodology in support of its conclusion.

I won't delve into the intricacies of the dispute and all of the arguments made by the parties and addressed in the ruling. In brief, the court addressed whether 42 U.S.C. § 264(a) was a sufficient basis for the CDC's mask requirement. The court concluded it was not, finding, among other things, that section 264(a)'s grant of power to provide for "sanitation" did not apply to requiring masks.

In reaching this conclusion, one method the court employed was corpus linguistics--a method in which databases of documents and texts are searched for instances in which words and phrases are used. In theory, one trying to determine the meaning of a word or phrase can type that word or phrase into a corpus linguistics database and examine the instances in which that word or phrase is used across a wide variety of texts. In doing so, patterns may emerge demonstrating multiple usages, common trends in meaning, and other information that may aid in determining the definition (or definitions) of a term. This method of interpretation has gained steam in recent years, particularly in originalists circles where it is hailed as a groundbreaking method for determining the original public meaning of constitutional provisions. (See articles by Lawrence Solum and Thomas Lee and James Phillips advocating the use of corpus linguistics in the originalist context). As I've noted in recent work coauthored with Alexander Hiland, this methodology raises a fair share of concerns, including a lack of transparency as to how a judge undertook the corpus linguistics analysis. The corpus linguistics analysis in Judge Mizelle's opinion demonstrates that this concern is well-founded.

Here's the excerpt of the ruling addressing corpus linguistics (from pages 17-18 of the ruling):

Customary usage at the time agrees. One method to assess the ordinary meaning of a term is to search a database of naturally occurring language. A search returns the desired word as well as its context and, with a sufficient sample size, search results permit inferences on how a word was used. This method is known as corpus linguistics.[FN 2] The Court here searched the Corpus of Historical American English (COHA) [FN 3] to find uses of "sanitation" between 1930 and 1944. Of the 507 results, the most frequent usage of sanitation fit the primary sense described above: a positive act to make a thing or place clean. Common examples referred to sanitation in the context of garbage disposal, sewage and plumbing, or direct cleaning of a dirty or contaminated object. In contrast, by far the least common usage-hovering around 5% of the data set-was of sanitation as a measure to maintain a status of cleanliness, or as a barrier to keep something clean. And so, the COHA search results are consistent with the contextual clues of the active words surrounding sanitation in § 264(a).

[FN 2]: "Corpus linguistics is an empirical approach to the study of language that uses large, electronic databases" of language gathered from sources such as books, magazines, and newspapers. Thomas R. Lee & Stephen C. Mouritsen, Judging Ordinary Meaning, 127 YALE L.J. 788, 828 (2018) (footnote omitted) (describing this tool).

[FN 3]: The COHA corpus is publicly available. See CORPUS OF HISTORICAL AMERICAN ENGLISH, https://www.english-corpora.org/coha/ (last visited Apr. 12, 2022). It is "the largest structured corpus of historical English." Id. Because Congress enacted the PHSA in 1944, the Court searched for uses of the word "sanitation" and variants like "sanitary'' and "sanitize" between 1930 and 1944. The search returned 507 hits, or "concordance lines."

This description of the analysis that the court undertook and the conclusions drawn from the analysis lack transparency and raise a number of questions.

To start, it is unclear what search (or searches) the court undertook. In footnote 3, the court indicates that it searched for the word "sanitation," and also "variants like 'sanitary' and 'sanitize'." The court does not specify how many variants of "sanitation" it searched for--only providing two examples of such variants. It appears that the court conducted a single search, although this is unclear as well, given the acknowledgment that the court searched for variations on "sanitation." This suggests that there was a single search for "sanitation," along with some of its variants (say: "sanitation OR sanitary OR sanitize"). The court's failure to specify the precise terms of its search, however, leave the reader to speculate as to how the court conducted its search of the database.

The court's analysis also lacks transparency regarding how it coded the results of its search. The court gives two apparent examples of its coding: results reflecting the meaning, "a positive act to make a thing or place clean" and results reflecting "a measure to maintain a status of cleanliness, or as a barrier to keep something clean." The court does not specify whether its search uncovered alternate meanings, and the percentage of results that fell into these meanings. Indeed, the only percentage specified in the opinion is that five percent of results were consistent with the "measure to maintain a status of cleanliness" meaning. The court is silent as to the percentage of results that fell into the other meaning it identifies, stating only that the "most frequent usage of sanitation" fit that sense. It is unclear whether sanitation as "a positive act to make a thing or place clean" was a majority of the results or a plurality of the results. Even if this sense was a majority of the results, there is still room for the possibility of a frequently used, third meaning that the court does not identify here. Without a breakdown of how results were coded and the frequency of hits for each definition that was code one cannot know how the court reached its decision or evaluate the significance of its conclusions regarding the frequency of definitions.

The court also fails to list sufficient examples detailing how it decided to categorize the search results. The court notes that it one of its definitions was "a positive act to make a thing or place clean," but goes on to provide purported "examples" that list little more than characterizations of the context of the terms. Some of these terms are likely loaded: "sewage" and "garbage disposal," for instance, suggest that some of these results may have been uses of the term, "sanitation," in a specialized sense to describe a "Department of Sanitation." The court does not specify how it classified search results like this, or whether it treated such results differently from the use of "sanitation" in non-departmental contexts. Departments of Sanitation likely carry out a variety of activities, including taking positive actions to make places clean and to maintain a state of cleanliness. Accordingly, it is likely that the court's analysis included hits that, themselves, could have been interpreted in multiple ways. The court does not specify whether, in these cases, it selected one of the alternate definitions, whether it discounted the result from its analysis, or whether it coded the result as including multiple definitions.

Additionally, this analysis illustrates an overall issue with corpus linguistics analysis: the methodology does not contain a basis for selecting among multiple meanings that a search of the relevant corpus uncovers, or whether choosing a single meaning is appropriate. Here, the court notes that the results of its analysis indicated that there were at least two potential meanings of "sanitation." The court ultimately concludes that it should use the variation that appears most frequently, but does not state why the most frequent usage is the only usage that should be employed (for a critique of this assumption, see Donald Drakeman's essay on corpus linguistics; although see Neal Goldfarb's arguments to the contrary). Moreover, selecting a particular meaning of "sanitation" is, itself, a judgment call as to how broadly to read the statute. If one takes a wide view of the CDC's power, one may be inclined to read "sanitation" to cover all of its potential meanings, so as to allow the CDC to take a broad range of actions. Limiting the reading of "sanitation" to only one of its alternate definitions is, itself, a judgment call as to the appropriate breadth of the agency's power--yet no basis or justification for this assumption is set forth.

Advocates of corpus linguistics claim that it can bring a level of empirical rigor to legal interpretation. But as the court's ruling in this case demonstrates, corpus linguistics can backfire and lead to conclusions based on methodology that is impossible to examine or verify. Advocates of corpus linguistics will undoubtedly argue that misapplications of the methodology should not count against the method itself. But in a world where attorneys may increasingly seek to use corpus linguistics in a one-sided manner to convince judges that their position is correct, or where judges themselves employ corpus linguistics without the necessary transparency, the costs of this method to judicial transparency may outweigh the benefits. A judge or attorney may abuse dictionary definitions by selecting a particular dictionary or one particular definition among alternate, plausible definitions. But these abuses can be identified and critiqued. This is not the case with incomplete corpus linguistics analysis, in which a failure to disclose search terms, coding methods, and percentages of coded results makes it impossible to evaluate the interpretive methods employed. The court's decision in Health Freedom Defense Fund illustrates how this opaque, incomplete methodology can impact the lives of millions.