Search Results on Westlaw Next Return Unrelated Material; Vary Between Searches

A few days ago, I was struck by a question: how often do courts use the word, "balderdash" in their opinions, in what context do they use this word, and are there any trends that can be uncovered from a systematic review of cases that mention "balderdash?" I then wondered whether anybody had written about this -- a quick Google search revealed a close call from a January post by John Browning (inspired by Scalia's use of "argle-bargle" in United States v. Windsor) -- but nothing on "balderdash." In light of this, I thought that it might be worthwhile to write a paper that would answer my questions.

This, after all, is the kind of important legal question that keeps me up at night.

While I was researching this question, I found that Westlaw Next was behaving strangely. In two separate searches for "balderdash," the database returned quite a few cases that did not mention the term at all. Even more strangely, two separate searches returned varying numbers of results because they returned two different sets of these unrelated cases.

I've seen authors demonstrate the prevalence or absence of a term or phrase in the legal world by running a search of a term on Westlaw and counting the results. It appears that Westlaw Next's way of generating search results means that this is no longer a reliable method.

After the break, I have included the portion of my paper where I discuss this phenomenon in more detail -- listing the specific cases and numbers of results in each search:

As an aside from my overall project, the research for this paper revealed a strange tendency for Westlaw Next to return varying, unrelated results. My survey of state cases took place over the course of two Westlaw Next sessions in one day. I accomplished my survey of cases by simply typing “balderdash” as a search query, and then narrowing my results to the cases the query returned.  

My first search returned 62 results. My list of results included State v. Craig, 793 P.2d 215 (Idaho 1990), Haugen v. Swanson, 16 N.W.2d 900 (Minn. 1944), State v. Bever, 794 P.2d 1136 (Idaho 1990), Jones v. Fox, 2 So. 853 (Fla. 1887), Gallagher v. Com., 139 S.E.2d 37 (Va. 1964), Miller v. Miller, 366 S.E.2d 682 (Ga. 1988), Mikesell v. Mikesell, 436 N.E.2d 95 (Ind. Ct. App. 1982), Sherwood v. State, 111 So.2d 96 (Fl. App. 1959), and In re State ex rel. Attorney General, 64 So. 310 (Al. 1914). None of these cases mentioned the word, “balderdash.”  

My second search returned 61 results. The list of results included Jones v. Fox, 2 So. 853 (Fla. 1887), Gallagher v. Com., 139 S.E.2d 37 (Va. 1964), Miller v. Miller, 366 S.E.2d 682 (Ga. 1988), Mikesell v. Mikesell, 436 N.E.2d 95 (Ind. Ct. App. 1982), Tracey v. State, 350 S.W.2d 563 (Tex. Ct. Crim. App. 1961), State v. Daly, 313 A.2d 194 (N.J. 1973), Sherwood v. State, 111 So.2d 96 (Fl. App. 1959), In re State ex rel. Attorney General, 64 So. 310 (Al. 1914), and Town of Canaan v. Avery, 58 A. 509 (N.H. 1904), all of which do not contain “balderdash.” 

It is unclear why each of my searches turned up cases that did not contain my search query. It is also unclear why the unrelated results varied between the two searches that I undertook. While these results were unsettling, both searches consistently returned the 52 cases that contained “balderdash,” which is why I have decided to use that number throughout this paper.

It is somewhat troubling to receive differing results to repeated instances of the same search query on Westlaw Next. In particular, it is notable that the number of total results varied with each search that I did. Had I been seeking a quick indication of how many times “balderdash” appears in state court opinions, I would have reached a different conclusion depending on which search I used had I simply pulled the number of results my query returned. And both of these numbers would have been incorrect, since many of the results did not contain the word I was searching for. This should serve as a cautionary tale for those who put too much faith in the number of results a query uncovers.