Lessons on Empirical Legal Research Through a Twombly/Iqbal Lens

Through the Civil Procedure and Federal Courts Blog, I learned of the existence of this article by David Freeman Engstrom a while back, but due to the hectic schedule of wrapping up work I was not able to read it until my flight back to Iowa.  Once I was finally able to read it, I found the article to be very informative and interesting, and I recommend that anybody who is interested in Civil Procedure, the ramifications of Twombly and Iqbal, or empirical legal research in general to read it.  The full citation is: David Freeman Engstrom, The Twiqbal Puzzle and Empirical Study of Civil Procedure, 65 Stan. L. Rev. 1203 (2013).

Here is the (mercifully short) abstract:

This essay, written for a Stanford Law Review issue exploring “The Empirical Revolution in Law,” offers a critical assessment of the large body of empirical scholarship examining the effect of the Supreme Court’s decisions in Bell Atlantic Corp. v. Twombly and Ashcroft v. Iqbal on judicial and litigant behavior and then uses the critique to make some broader observations about the past, present, and future of empirical study of civil procedure.

The article offers some good examples of empirical legal studies of the impact of Twombly and Iqbal -- the two cases that raised the standard of proof required in civil pleadings.  While many commentators theorize and lament that these cases would create a sea change in civil procedure, the satisfactory studies apparently show that while some changes have occurred, they are not as substantial as the changes forecasted or supposedly demonstrated by less-satisfactory studies.

I found the most informative portion of the article to be that beginning on page 1213 which detailed the missteps of commentators discussing Twombly and Iqbal's impact.  Among other problems, the article discusses sampling bias, failure to include control groups, and the danger of using Westlaw or LexisNexis in generating samples of cases for legal research.  I mentioned in this earlier post that I had come across some scholarship that appeared to make these errors, but Engstrom's article uncovers the depth and widespread nature of these errors in many of the published studies of these cases.

I think that this article, or in the very least, the portions of the article discussing the systematic errors made by scholars studying Twombly and Iqbal are very helpful for law journals' articles editors who may not be entirely familiar with what good empirical legal research looks like.  Engstrom tells a good cautionary tale, and provides an approachable list of missteps that should be identified and avoided, both by authors and by law journals.

Twombly, Iqbal, and Civil Rights: An Interesting Thesis with a Flawed Methodology

Raymond H. Brescia and Edward J. Ohanian have posted their article, The Politics of Procedure: An Empirical Analysis of Motion Practice in Civil Rights Litigation Under the New Plausibility Standard on SSRN.  Here is the abstract (buckle down, it's three paragraphs)  To assist the busy reader, I have emphasized the most important parts of the abstract:

Is civil procedure political? In May of 2009, the Supreme Court issued its decision in Ashcroft v. Iqbal, which explicitly extended the “plausibility standard,” first articulated in Bell Atlantic v. Twombly two years earlier, to all civil pleadings. That standard requires that pleadings, in order to satisfy Rule 8(a) of the Federal Rules of Civil Procedure, must state a plausible claim for relief. For many, these rulings represented a sea change in civil pleading standards. Where prior Supreme Court precedent had provided that a pleading should not be dismissed “unless it appears beyond doubt that the plaintiff can prove no set of facts in support of his claim,” the new standard requires that judges utilize their own “judicial experience and common sense” to determine whether claimants have set forth facts sufficient to “nudge their claims across the line from conceivable to plausible.” In the years since their issuance, this standard has provoked many questions. One such question, which lurks behind all otherwise neutral rules of procedure is the following: could this apparently neutral principle of procedure be subject to political manipulation?  

After Twombly, and again after Iqbal, many expressed fears that the new plausibility standard offered judges too much discretion; a judge could dismiss a case where a plaintiff’s claims did not comport with that judge’s experience and common sense. There was a particular fear that this discretion would have a disparate and adverse impact on civil rights cases: i.e., if members of the federal bench were predisposed to disfavor such claims, they might use these precedents to dismiss civil rights cases too readily. Several years have now passed since the Court issued these decisions, and the district courts have compiled a body of thousands of decisions citing these precedents. As a result, it is now possible to assess the impact of these decisions on practice in the lower courts, particularly their effect on civil rights cases. The study described here attempted to do just that by looking at outcomes and trends in motions challenging the specificity of the pleadings in over 500 employment and housing discrimination cases over a period of six years (including decisions issued both before and after Twombly and Iqbal). This research reviewed the outcomes in such cases based on a number of metrics, including, most importantly, the political affiliation of the president who appointed the judge issuing each decision reviewed.  

The study revealed a statistically significant relationship between the outcomes in civil rights cases and time period (i.e. pre-Twombly, post-Twombly but pre-Iqbal, and post-Iqbal) where the political affiliation of the president who appointed the judge reaching the decision in each case was Republican. For cases decided by judges appointed by Democrat-affiliated presidents, no such relationship was observed. This paper reports on the findings of this study and discusses their implications.

Okay, back to my post.

For those still with me, here are the two most telling sentences of the article:

"One shortcoming of this study is that it did not compare outcomes in civil rights cases to outcomes in other types of cases. While other studies have done that, as stated earlier, those studies had their limitations."

This is a big deal, and it is a much bigger qualification on this article than the authors admit.  The authors are advancing a conclusion that specifies civil rights cases and claims to reach significant results.  I contend that the authors' failure to compare outcomes in civil rights cases to other types of cases amounts to a failure to include a control group.

While Republican-appointed judges may bar more civil rights suits, if these judges are as political as the author hypothesizes, it would not be surprising if these judges were more hostile towards litigation in general.  It has long been a part of Republican political platforms that more litigation is a bad thing, so we might expect Republican-appointed judges to turn down more complaints in general with the discretion they get from Twombly and Iqbal.  While this language was noticeably absent in the most recent presidential campaign (a phenomenon explored by Professor Stephen C. Yeazell in an article that will eventually be available here), Republicans still espouse this position and many of the Republican judges involved in this study were probably appointed by earlier presidents (although this is unclear).

So are the higher rates of dismissal of civil rights cases by Republican-appointed judges due to the nature of the cases?  Or can they be explained by a hostility that Republican-appointed judges hold towards litigation in general?  I think that these questions can only be answered by a study that controls for hostility to general litigation by including a randomized set of cases as a control group.  Without this control group, I have serious doubts that any claim of significance can be made.

This is a problem that deserves more attention than two sentences.