Wall Street Journal Praises Judicial Ignorance of Statistical Methods

From yesterday's Wall Street Journal editorial page covering yesterday's oral arguments in Gill v. Whitford, a case that involved a challenge to alleged gerrymandering:

[Chief Justice John Roberts] zeroed in on risks for the credibility of the judiciary if the Supreme Court invalidates a state electoral map on purely political grounds for the first time. His reference to EG is to a political science standard offered by the plaintiffs as a test of when an electoral map is too partisan.

. . .

Chief Justice: “It is just not, it seems, a palatable answer to say the ruling was based on the fact that EG was greater than 7%. That doesn’t sound like language in the Constitution.” . . .

[Plaintiffs' Attorney, Paul] Smith: “If you let this go, if you say this is—we’re not going to have a judicial remedy for this problem, in 2020, you’re going to have a festival of copycat gerrymandering the likes of which this country has never seen.

And it may be that you can protect the Court from seeming political, but the country is going to lose faith in democracy big time because voters are going to be like—everywhere are going to be like the voters in Wisconsin and, no, it really doesn’t matter whether I vote.”

Chief Justice: “No, but you’re going to take this—the whole point is you’re taking these issues away from democracy and you’re throwing them into the courts pursuant to, and it may be simply my educational background, but I can only describe as sociological gobbledygook.”

Gerrymanders are unsightly, but worse would be the sight of federal judges becoming political arbiters of every electoral map based on evidence that voters are likely to conclude is itself partisan.

An eye-opening Twitter thread by Steven Mazie illustrating how Chief Justice Roberts and Justice Gorsuch appeared to lack knowledge of the statistical methodology involved paints Chief Justice Roberts' comment as more disturbing than the quippy Editorial Board lets on. While "gobbledygook" is a fun word, reminiscent of "argle bargle" and "pure applesauce" of old, the implications of the comment are that the Court -- the last resort for challenging (often-complex) gerrymandering schemes -- should decline to do so because the methods involved are too complicated and potentially subject to partisan manipulation. Jesse Wegman at the New York Times' editorial page covers Smith's response, which summarizes this problem:

“The problem in this area is if you don’t do it, it is locked up,” Mr. Smith said. “You are the only institution in the United States that can solve this problem just as democracy is about to get worse because of the way gerrymandering is getting so much worse.”

Gerrymanders are more than "unsightly" as the Journal describes. The contorted and bizarre-looking voting districts that gerrymandering creates may result in disenfranchisement of voters along party and racial lines. They threaten representative democracy. The Journal's and Chief Justice's implication that a lack of understanding of statistical methods is enough of a barrier to let this threat go unchallenged is concerning.

The Use of Colons in the Titles of Law Review Articles and Comments

I didn't have much to do yesterday evening, so I decided to test a theory that I have believed for years but never examined: that law review articles tend to avoid using colons, while most student-authored notes and comments use colons.

I have both written and reviewed articles for publication and I have read numerous law review articles and comments in the course of writing my own comments and articles. In doing so, I have seen many titles. From my general experience, it seems that the titles of professor and practitioner-written articles tend to avoid the use of colons, while student-authored work continues to embrace that punctuation mark.

With my free evening, I ignored all of the possible activities that the vibrant city of Los Angeles had to offer and set to work to explore the frequency of colon-use in law review titles. My plan was to look at the most recent issue of the top 16 law journals and law reviews I could find (as ranked by Washington & Lee) and to compare the colon vs. no colon ratio in professor/practitioner-authored articles with the colon vs. no colon ratio in the titles of student-authored notes and comments.

The bright idea of determining whether this had ever been done before did not cross my mind until I was well into my project. I did some searching and found this excellent 2006 article in the best law journal written by Joshua Deahl and Bernard Eskandari. Deahl and Eskandari analyze the titles of a volume of law review articles in ten journals every five years from 1948 through 2003. The authors selected five "elite" law journals and five "second-tier" law journals.

Here is Deahl and Eskandari's graph of colon use in law journal titles:

From their summary of findings comparing titular colon use between articles and notes in elite and second-tier journals:

Since we are most interested in the current state of legal scholarship, the trends over the past decade are especially informative. According to our statistics, the four plotted categories rank in the following order, from lowest to highest percentage of colonized titles: (1) articles in elite journals, (2) articles in second-tier journals, (3) notes in elite journals, and (4) notes in second-tier journals. We suspect most people would rank the expected quality of legal scholarship by category in this same order. This lends further support to the idea that the worse the piece of scholarship, the more likely it is to have a titular colon. Only the "elite articles" clearly stand apart in 2003, with approximately thirty to forty percent fewer articles with colonized titles than each of the other categories.

Even though Deahl and Eskandari's methodology is far more rigorous than my spur of the moment project, I decided that a current snapshot of the state of law review titles could serve as a useful comparison and update to Deahl and Eskandari's work (also, I had already written most of the table below and did not want my work to go to waste). I added the "second-tier" journals that Deahl and Eskandary had examined to my sample, although I replaced the Dickinson Law Review (which is no longer being published) with the Penn State Law Review and Denver University Law Review (Dickinson was ranked 112 at the time Deahl and Eskandari published their article, and Penn State and Denver are tied for 111). This gave me a sample size of the most recent issue of 22 journals.

Here are the numbers I found. For professor/practitioner-authored work, I included both articles and essays in my totals. As noted below, I did not include book reviews. Additionally, for the Harvard Law Review, I used the second most recent issue, because the current issue is a Supreme Court 2015 examination and the "In Memoriam:" article, the foreword, and the cases included did not fit cleanly into my article/comment framework. For each publication, I have linked to the page where I found the articles and comments. Take note, however, that some journals only allowed me to link to a "most current issue" page rather than a specific page for a specific issue, so several of these links will be out of date as time goes on.

Recent Research on Attorney Happiness and Stress

High levels of stress and depression have plagued the legal community for a long time. I have blogged about this problem previously, and today I want to flag two recent articles regarding the happiness and stress of attorneys.

First is a recently-published article by Lawrence Krieger and Kennon Sheldon in the George Washington Law Review. The full version of the article is available here. The New York Times covers the article here. From the Times' summary:

Researchers who surveyed 6,200 lawyers about their jobs and health found that the factors most frequently associated with success in the legal field, such as high income or a partner-track job at a prestigious firm, had almost zero correlation with happiness and well-being. However, lawyers in public-service jobs who made the least money, like public defenders or Legal Aid attorneys, were most likely to report being happy. 

Lawyers in public-service jobs also drank less alcohol than their higher-income peers. And, despite the large gap in affluence, the two groups reported about equal overall satisfaction with their lives.

The next article I want to highlight is a recently-released report commissioned by the Bar Council in the United Kingdom that explores, among other issues, the prevalence of stress in the legal field. The full report is available here. Legal Cheek provides concise (if not overly-dramatic) coverage here.

Both of these articles provide important contributions. The Krieger and Sheldon article is notable for breaking down various levels of stress in different areas of the legal profession. The Bar Council's report highlights the often-neglected problem of stigma against stress itself. Attorneys' jobs are often stressful, and this stress may be compounded by a culture that stigmatizes those who display stress or who seek counseling and treatment for work-related stress.

Hopefully these articles will help highlight some of the problems facing the legal community, and perhaps provide some insight into how attorneys in the most stressful fields may draw on the experiences of those in other fields in order to combat stress and oppression.

The Limits of Judicial Knowledge and the Importance of Briefs

Linda Greenhouse has this very interesting op-ed in the October 30th New York Times where she asks:

How do judges — especially appellate judges, who don’t hear witnesses or take testimony but must rely on the record compiled in the courts below — learn what they need to know? And, of course, how do they — or any of us — choose what to make of the knowledge they have?

Greenhouse surveys a number of cases, ranging from the Supreme Court's determination in Kennedy v. Louisiana that there was a national consensus against child rape -- citing Congress's failure to add child rape to a list of federal capital crimes over a decade earlier (and missing Congress's two-year old decision to make child-rape a capital crime in military cases), Judge Richard Posner's admission that he lacked full knowledge of the consequences of his decision in the voting rights case, Crawford v. Marion County Election Board, and Justice Scalia's admission that he lacked the knowledge of genetics necessary for him to join the Court's statement of facts in Association for Molecular Pathology v. Myriad Genetics.

I will admit that what drew me to the opinion was Greenhouse's mention of Kennedy v. Louisiana. The oral argument this case was portrayed in the fictional case, Serra v. Louisiana, in one of my favorite Boston Legal episodes, "The Court Supreme" (Season 4, Episode 17; transcript available here).

But as I continued past the initial Kennedy hook, I found that I was reminded of an article I read a while back that is forthcoming in the Emory Law Review. That article is Scott Moss's, Bad Briefs, Bad Law, Bad Markets: Documenting the Poor Quality of Plaintiffs’ Briefs, Its Impact on the Law, and the Market Failure It Reflects (Lexis seems to indicate that the article has been released, giving the citation, 63 Emory L.J. 59, though the article is not yet available on the article's website).  Here is the abstract (from the version I found on SelectedWorks):

A Questionable Study on Digital Sampling's Impact

At Volokh Conspiracy, Stewart Baker has a nicely-titled post that discusses a recently-posted paper by W. Michael Schuster on artists that create mashups that digitally sample copyrighted works.  I've posted previously about these mashups, specifically as to whether mashup creators could copyright the mashup and sue others who create similar mashups.

Here is the (lengthy) abstract of Schuster's paper:

This Article presents an empirical study on the effect that digital sampling has on sales of copyrighted songs and how this effect should influence the fair use analysis. To conduct this research, a group of previously sampled songs had to be identified and sales information for these songs collected. The over 350 songs sampled in musician Gregg Gillis’s (AKA Girl Talk’s) most recent album presents an ideal dataset because the album’s instantaneous popularity allows for its influence to be analyzed through a comparison of the sampled songs’ sales immediately before and after release. Collecting and comparing sales information for these songs found that — to a 92.5% degree of statistical significance — the copyrighted songs sold better in the year after being sampled relative to the year before. To the extent that the Copyright Act instructs courts to analyze (among other considerations) the effect that an alleged fair use has on the potential market for the original work, these findings favor the conclusion that digital sampling is a fair use (though each statutory fair use consideration should still be reviewed).  

Additionally, the songs sampled in the subject album were evaluated to ascertain the length of each sample and to what degree each sampled song had experienced prior commercial success. This collected data was used to test the hypothesis that sampled songs which were more recognizable to listeners (e.g., songs that were commercial hits or songs that were sampled for a relatively longer period) would see a greater sales increase after being sampled. The collected data did not find a correlation in post-sampling sales increases and sample length or prior commercial success, but further study may be warranted. 

Beyond supporting the premise that digital sampling may constitute fair use, the results of this study raise several notable issues and subjects for future study. One such issue is that courts only address an alleged fair use’s effect on the market for the original as a binary system, wherein the only options are harm to the market (disfavoring fair use) or no harm to the market (favoring fair use). There is no accepted rule on how to treat a market benefit (such as the one evidenced here). The failure to address this issue is questionable because a market benefit actually furthers the utilitarian goal of copyright by incentivizing the creation of new works through economic gain. The current research makes clear the need for precedent on how the fair use analysis should treat actions (e.g., digital sampling) that may increase sales of the original work. Additionally, this study sets the ground work for an objective financial review of fair use and market effect, which would yield needed predictability and stability to the fair use doctrine (at least, with regard to digital sampling).

I'm not one to hesitate to criticize the empirical methodology of studies, but in this case, Baker got there first.  He notes:

Actually, though, I think the article is a little too comforting. I am always skeptical of scholarly research that reinforces academic prejudices, since scholars tend adjust their standards of proof to fit their prejudices. Hostility to copyright is pretty much the norm in academic circles, and if you read the article skeptically, it loses much of its persuasiveness. Schuster achieves his results by playing with the sample, dropping nine songs from a sample of about 200 because they completely wreck his argument. His reason for dropping the songs is that they were hits in the 30 months prior to the release of Girl Talk’s album, and hits by definition suffer declining sales after topping out. If he didn’t drop those songs, Schuster’s data would show a 50% drop in sales of the songs that Girl Talk samples. 

Schuster says he’s just correcting for noise in the data, and it isn’t appropriate to charge Girl Talk with the natural rhythm of pop music sales. Maybe so, but once you start making big after-the-fact adjustments to a sample of 200, you can prove pretty much anything. At best, Schuster has developed an interesting hypothesis that ought to be tested by a new experiment untainted by data cherry-picking.

The only point that I would add to Baker's reaction is that I was already suspicious of the study by the time I read the abstract, due to Schuster's note that his study arrived at a conclusion "to a 92.5% degree of statistical significance."  This is an oddly specific way of framing the results.  While I am not an expert on statistics, I think that this phrasing is a way of avoiding an admission that the level of statistical significance falls below the typically-accepted levels of P=.05 or P=.01 (for the really strong claims).  Those P-values would translate into levels of statistical significance of 95% and 99%, respectively, and it would seem that Schuster's study falls short of these widely accepted levels of significance.  For more on statistical significance thresholds, see these posts about the .05 threshold at the Empirical Legal Studies blog here and here, and a post cautioning overreliance on these thresholds here.

In the spirit of that last post I mentioned above, I don't think that this worry about significance should be fatal to Schuster's study, but I think that it, combined with Schuster's selection methods, should raise some doubts.  At the same time, however, I think that Schuster is on to something interesting, and a wider study may well lead to more solid results in favor of his thesis.

UCLA Law Review is Not Ranked in the Top Ten...

...of this list compiled by Roger Alford at Opinion Juris.

Noting that Strunk and White's Elements of Style calls for writers to eschew unnecessary words, Alford decided to see how law journals fared at following this rule.  Strunk and White's classic example of an unnecessary phrase is "the fact that," so Alford decided to survey how many times this phrase appeared in the top law reviews over the past ten years.  The top ten (mis)users of this phrase were:

Harvard Law Review: 869
Michigan Law Review: 496
Yale Law Journal: 459
Columbia Law Review: 436
Chicago Law Review: 431
NYU Law Review: 428
Penn Law Review: 408
California Law Review: 406
Stanford Law Review: 388
Virginia Law Review: 364

Looks like Harvard just has to rank at the top of everything.

(H/T: Paul Caron at TaxProf Blog)

Strange Empirical Claims About Mass Shootings and Stock Prices

I have a few "alerts" set up on Google Scholar, with one of them set to notify me whenever something containing the phrase "Second Amendment" is published.  The system has a few kinks to work out (my "Second Amendment" alert emailed me the link to this robotics article this evening...) but the system is an interesting way to stumble across things I might otherwise miss.

This evening's alert clued me in on an article by Benjamin W. Cross and Stephen W. Pruitt entitled Dark Knights Rising: The Aurora Theater and Newtown School Massacres and Shareholder Wealth.  The Article appears in Volume 41 of the Journal of Criminal Justice.  Here is the abstract:

Purpose 

This study analyzes the stock price impact of the Aurora theater and Newtown (Sandy Hook) school massacres on both domestic (US) and foreign theater operators and US gun manufacturers in an effort to document the economic effects of these tragedies. 

Methods 

The well-established “event study” methodology from the fields of economics and finance is employed to assess the impact of the shootings on the affected companies after controlling for risk and overall market movements. 

Results 

The Aurora theater shooting resulted in striking declines for Cinemark (the targeted theater) as well as major US competitors, but had no impact on overseas theater chains. Smith & Wesson (maker of the gun used in Aurora) showed no response, whereas Ruger (a competitor) exhibited large gains. Both Smith & Wesson and Ruger plunged after the Newtown shooting, although neither made the weapons used in the shooting. 

Conclusions 

Contrary to prior research on workplace homicides, the results show that random mass shootings have profound effects on targeted companies. In addition, the results suggest the presence of a very strong “contagion effect” (where negative events affecting one company impact others in the same industry). The negative responses of both publicly-traded US firearms manufacturers to the Newtown shooting suggests a “sea-change” in the debate over gun ownership in the US.

Normally I would write off an article like this as involving subject matter that is typically outside of my area of expertise -- but the abstract's last claim about a "sea-change" in the gun ownership debate caught my eye.  I decided to read through the article to see what the authors had found.

As it turns out, the article is blatantly flawed and highly misleading.

Law School: Two Years or Three Years?

Back in August, President Obama remarked that law school should be two years instead of three, noting that such an approach would be more cost-effective for students.  This is hardly a new idea -- papers and books have been written on the subject, and anybody who has even considered the notion of going to law school has heard the old saying, "The first year scares you to death, the second year works you to death, and the third year bores you to death."

Bruce Ackerman recently responded to Obama's statement with an op-ed in the Washington Post and a follow up today at Balkinization.  Ackerman argues that two years will "lobotomize the profession" and lead to a class of lawyers who are unable to verify the testimony of experts due to sub-par training in "social science" and "statistics."  Ackerman makes this point in various ways.  Here's part of what he said in the Post:

The predictable outcome will be massive professional retreat. Increasingly, lawyers will become secondary figures who prepare the way for “experts” to present the crucial arguments before administrative agencies, courts and legislatures. Decision-makers with two-year law degrees will proceed to rubber-stamp the expert testimony that seems most impressive because they aren’t prepared to test it in a serious way.

In contrast, if law schools redeem the promise of a three-year curriculum, their graduates will have something valuable to contribute to the larger conversation. They will never rival experts in their command of statistics and social science, but so long as they understand the basics they will be in a position to integrate technical insights into a broader understanding of the fundamental values of the American legal tradition.

And here is the similar portion of his post at Balkinization:

America’s law schools must train their students in the use and abuse of statistics, economics, and other social sciences or face increasing irrelevance in the formulation and implementation of public policy. The challenge is to integrate these technocratic skills into an historically informed, and philosophically sophisticated, understanding of the legal tradition. Otherwise, the profession will be pushed to the sidelines in an increasingly technocratic age.

Such an education requires a full three years of study.  It is too easy to deflect this point by claiming that most lawyers will not require such training, since they will be engaged in more humdrum professional activities. First, consider that these skills will increasingly be required by city councilman and leaders of state and local agencies, not only lawyers engaged in high-powered practice in Washington or Wall Street.  Even more important, it is simply impossible to determine which law students will take on leadership roles in the next generation.  Many lawyers from “elite” law schools will turn out to be drones; many  from “second” or “third” tier places will turn out to be leaders.  A two year curriculum promises to lobomotize the profession by 2050.

I do not think that Ackerman has as strong of a case as he seems to claim, mainly because many law schools do not systematically teach or require classes in social science or statistics.  This should not be a surprise to anybody, as it is very much in vogue to disparage students' lack of understanding of important issues when it comes to criticizing law journals.  I have previously remarked on this phenomenon here.  Judge Richard Posner, James Lindgren, and Arthur Austin are some others in the legal academy who have claimed that students lack the competence to discern quality legal scholarship and review the accuracy of submitted papers.  (See footnote 1 of Natalie Cotton's comment on this issue for citations and summaries of this criticism).

As somebody interested in legal academia, I think that I will be best served by three years in law school.  My third year gives me the opportunity to attend more lecture courses so that I have a knowledge base in more areas of the law.  It also gives me the opportunity to write more.  I think that my reason for favoring the third year is fairly particular, however.  For students who were not summer associates in the summer after their second year, the third year of law school is typically devoted to finding jobs and taking easy classes or clinical courses.  For students who were summer associates, the third year of law school is typically devoted to taking easy classes or clinical courses.  As far as required classes are concerned, many law schools only go so far as to meet the ABA's requirements that students take an advanced writing course and a class in professional responsibility.  UCLA School of Law is an example -- it does not even require students to take a course in evidence law.

Do I think that a third year of law school helps students learn the basics of social science and statistics?  No, because most law schools do not encourage the study of these subjects in their curricula and many students do not take advantages of the opportunities when they do exist.  Do I think that a third year of law school will allow students to take courses where they systematically learn the doctrine in new areas of the law?  Yes.  Do I think that students use the third year of law school to take these courses?  Only sometimes.  Many students grow weary of lectures and turn to clinical courses for practical experience.  Moreover, with large law firms typically extending offers before the beginning of the third year of law school, there is no reason for these students to exert a great deal of effort in the third year.  Some students with offers do.  I am friends with several of them.  But I think that Ackerman is a little too optimistic in expecting students to use the third year of law school to become knowledgeable in statistics and social science.

Lessons on Empirical Legal Research Through a Twombly/Iqbal Lens

Through the Civil Procedure and Federal Courts Blog, I learned of the existence of this article by David Freeman Engstrom a while back, but due to the hectic schedule of wrapping up work I was not able to read it until my flight back to Iowa.  Once I was finally able to read it, I found the article to be very informative and interesting, and I recommend that anybody who is interested in Civil Procedure, the ramifications of Twombly and Iqbal, or empirical legal research in general to read it.  The full citation is: David Freeman Engstrom, The Twiqbal Puzzle and Empirical Study of Civil Procedure, 65 Stan. L. Rev. 1203 (2013).

Here is the (mercifully short) abstract:

This essay, written for a Stanford Law Review issue exploring “The Empirical Revolution in Law,” offers a critical assessment of the large body of empirical scholarship examining the effect of the Supreme Court’s decisions in Bell Atlantic Corp. v. Twombly and Ashcroft v. Iqbal on judicial and litigant behavior and then uses the critique to make some broader observations about the past, present, and future of empirical study of civil procedure.

The article offers some good examples of empirical legal studies of the impact of Twombly and Iqbal -- the two cases that raised the standard of proof required in civil pleadings.  While many commentators theorize and lament that these cases would create a sea change in civil procedure, the satisfactory studies apparently show that while some changes have occurred, they are not as substantial as the changes forecasted or supposedly demonstrated by less-satisfactory studies.

I found the most informative portion of the article to be that beginning on page 1213 which detailed the missteps of commentators discussing Twombly and Iqbal's impact.  Among other problems, the article discusses sampling bias, failure to include control groups, and the danger of using Westlaw or LexisNexis in generating samples of cases for legal research.  I mentioned in this earlier post that I had come across some scholarship that appeared to make these errors, but Engstrom's article uncovers the depth and widespread nature of these errors in many of the published studies of these cases.

I think that this article, or in the very least, the portions of the article discussing the systematic errors made by scholars studying Twombly and Iqbal are very helpful for law journals' articles editors who may not be entirely familiar with what good empirical legal research looks like.  Engstrom tells a good cautionary tale, and provides an approachable list of missteps that should be identified and avoided, both by authors and by law journals.