Search This Blog

Monday, May 27, 2024

Long Live U.S. News

Brian Frye has posted a recent essay entitled, "The Gray Lady's Guide to Legal Education." Here's the abstract:

For better or worse, the U.S. News & World Report law school ranking has dominated legal education for decades, by giving prospective law students information about the relative prestige of law schools. But in recent years, it has become largely irrelevant, as prospective law students have begun to look elsewhere when deciding which law school to attend. Apparently, we need a new law school ranking to provide useful information about prestige. This essay provides such a ranking.

Frye briefly summarizes the history of the U.S. News rankings, their trend toward fluctuation, and growing irrelevance. Sophisticated readers will find much to enjoy in this discussion.

Like all good attorneys (and a few law professors), Frye doesn't stop with identifying a problem. He also presents a solution: rank law schools using the "NYT ranking" instead. What does this mean?

The methodology of the NYT ranking is simple, but elegant. I created a list of 196 United States law schools by copying it from the U.S. News & World Report “Best Law Schools” 2024 ranking. I then searched for the official name of each law school in the New York Times archive and recorded how many times the law school had been mentioned by name in the past 12 months. I then ranked all 196 law schools according to the number of times they were mentioned by the New York Times in the last 12 months. The law school with the highest number of mentions was ranked number one, the law schools with no mentions were ranked last, and the rest of the law schools came in between.

Frye goes on to address and dismiss potential objections and concerns with the method--including worries over typos (won't happen / if so, disrespectful), bad publicity (doesn't exist), and regional bias (the NYT is Everyone's Paper). Plus, Frye notes the comparative benefits of the NYT ranking, as it is "much funnier than other law school ranking methodologies." Count me convinced!

But Frye goes even further, comparing his present project to the task of "writ[ing] the New York Times obituary for the U.S. News law school rankings." Frye concludes that U.S. News "must cede the vessel to another"--the New York Times.

Here, I think, Frye goes too far (something he's never done before).

Frye's "Conclusion" section is followed by a further section (which Frye should probably label an "Appendix" if he wants to appeal to the fancier publications), entitled "The New York Times 'Best Law Schools' 2024 Rankings." Here, Frye lays out the New York Times rankings for all to see. 

In doing so, though, Frye reveals that a ranking system is an idea that must ultimately be expressed in some visual form--typically a list. And in laying out the list for readers, one faces the question: how to display those schools that end up tying in the rankings? In the list, one school must still come first, and one must come last. As the overall list corresponds to the ranking system, a physically higher placement among tied entries leads to the appearance of more prestige, even if this prestige is simply illusory. But for readers in a busy world increasingly dominated by images, video, and charts rather than dense text, illusory prestige is the coin of the realm.

As it turns out, ties pervade the NYT rankings. Five schools are tied in 18th place, eight in 24th, 10(?) in 32nd, 17 in 42nd, 44 at 59th, and 94(!) at 103rd. This leaves a lot of schools to list, despite having the same ranking.

And how does the New York Times rankings proceed to list these tied entries? By resorting to the very thing this list was supposed to replace--if not actively destroy: the U.S. News Rankings of each of the tied institutions.

One could have resorted to all manner of alternate ordering mechanisms. Perhaps list tied schools in alphabetical order--resulting in dramatic movement for schools like Ave Maria and Appalachian School of Law. Perhaps use Excel's random-number function to change the ordering each year, giving Microsoft a well-deserved, outsized influence in law schools' prestige appearances. Perhaps one can write the names of law schools onto a set of uniform balls that are deposited in a box and drawn out by lot.

But by listing tied schools in order of their U.S. News rankings, the NYT ranking list perpetuates that which it was meant to replace, as schools with higher U.S. News rankings appear higher--sometimes much higher--on the list than their purportedly equally ranked counterparts. The result is a blow to the appearance of prestige that corresponds to U.S. News rankings, bringing us back to a situation not unlike the status quo. The U.S. News rankings are dead. Long live U.S. News.

To be sure, there are plenty of fixes for those who wish to see U.S. News truly buried. I've listed a few above--though one must not count out the addition of a secondary set of tie-breaking newspaper references to add onto New York Times references. I think the San Antonio Express News is a good choice, as it adds geographic and political diversity to the list.

Thursday, May 23, 2024

Law Review Submissions: Editors' Perspectives, Ideal Articles, and the Summer Submission Cycle

It's that time of the year: Scholastica has released their "Advice from Outgoing Law Review Editors" post, featuring thoughts and feedback from law review editors on submission criteria and strategy, as well as the state and future of legal scholarship.

Much earlier in the year, the chief articles editors* of the Harvard Law Review, Yale Law Journal, and Stanford Law review held an online presentation addressing similar points, including issues of article length, timing of submissions, and peer review procedures.

Below are some thoughts and reactions to the post and the presentation--with particular attention to points that I found surprising and illuminating. 

*A note: my notes from the Harvard/Yale/Stanford call (which took place in January 2024) do not include the names or exact titles of the editors who were speaking, although it's my recollection that they were all the chief articles editors (or that equivalent role) at the respective journals. To the extent that I refer to the "editors" at these journals, I'm referring to those participants on the call.

Law Review Submissions: The Basics

Some advice from law review editors on ideal submissions should go without saying. A number of comments in Scholastica's survey of editors reflect the desire that authors substantially research, revise, and footnote their pieces before submitting them.

This appears to be of particular concern for smaller journals with limited editorial staffing. One anonymous editor in the Scholastica survey noted that these journals "giv[e] weight to whether or not the article needs significant or challenging changes before publication outside the normal scope of editing." Another editor notes the importance of proper citation to "help ensure a smooth editing process."

What about article length? Law professors (and others) love to lament the length of law review articles, critiquing their overall length, over-attention to background materials, and extensive footnoting. Nearly two decades ago, a number of high-profile journals issued a joint statement that persists to this day acknowledging concerns over the length of law review articles and suggesting that "[t]he vast majority of law review articles can effectively convey their arguments within the range of 40-70 law review pages." While I agree this is true, I have doubts over law reviews' and authors' efforts to achieve this ideal, as lengthy articles continue to pervade law journals at all levels (though I do think that there's been some improvement over the past few decades).

Unfortunately, the Scholastica survey is relatively quiet on the subject. Last year's survey also didn't have much to contribute on the issue of length, though one anonymous editor predicted that "submissions and published articles will continue to shorten in length." I applaud the optimism!

When asked how long articles should be, the editors at Harvard, Yale, and Stanford gave some numbers--though much remained waffly. Yale's editor said that they look for articles that are around 25,000 words, including footnotes. This is decidedly less helpful than a suggested range for articles, although this same editor noted that they may take "Essays" of around 15,000 words. Stanford's editor suggested a range of 20,000-30,000 words, with a cap of around 30,000 words, although she made room for the possibility of going lower than 20,000. Harvard's editor threw out a 30,000 word figure, though she noted it was not a strict cutoff. Longer lengths would count against the probability of acceptance. Given Harvard's selectivity, one would think that such a negative weight would spell doom for an article. But a quick (admittedly limited) check of the one article in the HLR's most recent issue suggests that the editors are perfectly willing to go well beyond the 30,000 word mark. 

Letterhead Bias and Anonymity

Interestingly, none of the editors Scholastica surveyed mentioned anonymous submission practices. One editor in the 2023 survey did predict that "more law reviews will move towards anonymized submissions."

Concerns over letterhead bias--that is, editors' tendency to more heavily weight submissions from authors employed by, or who have graduated from, elite institutions--do seem to be on at least some editors' minds. One of the recent respondents urged that articles be selected "for their quality rather than based on the perceived prestige of the author." One hopes that editors generally strive for this, but without anonymous submission procedures, this ideal remains largely a matter of hope and faith.

Yale, Harvard, and Stanford incorporate anonymity to at least some degree. All of these journals' initial reviewers read the piece without any information regarding the author. Things get a little fuzzier, however, when it comes to the "peer review" process that these journals employ. Harvard's editor stated that faculty review of submissions is double-blind. Yale's editor stated that while its reviewers remain blind to the author of the piece, their chief articles editor is aware of the author's identity at that stage of review.

I'm on record calling for more anonymity in journals' review process. I think that concerns over circumventing anonymous submissions are overstated, and that more anonymity would go a long way to combat letterhead bias. Several elite journals appear to share this stance, and I hope that more continue to follow suit.

Grabbing Attention

Those who author legal scholarship love to tell their readers how important their articles are. Noah Chauvin details and critiques authors' obsession with telling their readers how their work fills a gap in the literature. Other calling cards for this technique include claims that an area of law is "undertheorized," or that an article is "the first" to ever broach a subject.

While exaggerated claims of novelty are easy to criticize, they exist for a reason. One editor responding to this year's Scholastica survey emphasized authors' need to detail their contribution:

Provide a detailed abstract that includes your thesis and how this paper will add to the literature or field. Often, the value of the article isn’t immediately apparent. If the reviewers aren’t familiar with the field, they might overlook something innovative or conclude that it isn’t novel.

This remark highlights editors' continuing use of the novelty benchmark in assessing the value of scholarship. It also urges authors to continue to place their assertions of gap-filling and trailblazing at the top of their work. 

The Yale/Harvard/Stanford editors didn't have much to say on the subject of novelty, although my notes reflect that Yale's editor indicated that novelty (along with staying power and a secret, third thing that I couldn't write down in time) was one of the primary things they look for in a submission. If you are writing in a novel or creative format, you'd do better to submit this as an essay (although, remember, you'll still have up to 15,000 words to devote to the project!)

Submissions and Expedites

Those who first learn about the law review submission process often come away with the impression that the system is bonkers. And that's because it is. Authors submit articles to the Scholastica website (along with a CV and cover letter), and then submit to dozens of journals all at once. Each submission costs money--an amount that seems to creep up each year. Most (but not all) schools provide unlimited funding to authors, which indirectly incentivizes mass submissions to dozens (if not hundreds) of journals for each piece. Law reviews become inundated with these submissions and Scholastica gets rich.

Journals eventually make offers to publish submissions, which often come with a time limit. The authors then notify all other journals in which they'd rather publish that they've received that offer, and that these other journals have until that other journal's deadline to make their own decisions on whether to publish the piece. This is known as "expediting" an article, and authors use it to "trade-up" their article into as prestigious a placement as possible.

This system isn't without its critics. Anthony Kreis has a succinct takedown of the process, arguing that the expedite process exploits the labor of those in less-prestigious journals who end up serving as an initial round of review for higher-ranked journals. Due to the volume of submissions that journals tend to receive, I suspect that many high-ranking journals effectively require expedite notifications as a prerequisite for initial review. 

The editors at Harvard, Yale, and Stanford had little to say on the subject of expediting. And why would they? They have little to worry about when making offers to authors. When one is at the top of the law review pecking order, where else will an author hope to place upon receiving such an offer?

But other editors don't have it so easy. One editor, responding to Scholastica's survey, took a pretty dismal view of the future of law review submissions:

People have preconceived notions of which journals are prestigious or will provide the best editing.
With the fall of the rankings, I am seeing smaller schools do more to get on the map and be seen as reputable and competitive institutions. But unfortunately, I think the way the submission cycles work, with people “gaming” offers, smaller schools will get left behind. They will spend their time reviewing submissions from authors who submit to every journal available. Or the submissions received will not be high enough quality, which can start a vicious cycle of poor articles year after year, leading to the journal’s demise. The submission cycle favors the traditional, elite, and robust.

Another editor has thoughts on how to improve this state of affairs:

I think legal scholarship would ultimately benefit if a cap were imposed on the number of law reviews to which an author could submit a single article. This would eliminate the gamesmanship that authors sometimes engage in that involves submitting to as many journals as possible and using existing offers in an attempt to procure an offer to publish with a higher-ranked journal. Under a system with a cap, journals would waste far less time reviewing hundreds of articles and be able to more efficiently and thoroughly review articles where the authors are truly interested in publishing with that law review.

Authors face a variety of pressures on where to place. For those hoping to break into a career in academia, publishing in prestigious outlets can make the difference in landing a job. Those lucky enough to get onto the tenure track may then have incentives to published in more highly ranked journals--whether doing so is related to grant/bonus/reward payment schemes, or whether doing so is tied to tenure requirements. These incentive structures play a role in contributing to the phenomena of mass submissions and expediting and shouldn't be left out when one is criticizing these tactics.

Other pressures, though, are more illusory. Ezra Rosser describes "[t]he never-ending pursuit of the gold star," that frequently becomes an obsession for law professors. The "intangible quality of 'being or seeming smart'" motivates law professors to write on ephemeral matters of high theory, and to seek out prestige when publishing. This desire to continue attaining the highest placement possible--even after one has a job, tenure, and all manner of lovely titles, may drive many law professors to continue playing the expedite game long after it is of any true use to their career and prestige. 

To be sure, placement prestige does play a role. Readers may give more weight to articles that place more highly. Placements in high-ranking law reviews may garner more citations. 

But authors shouldn't forget that this pursuit of constant validation impacts law students' lives. Editors who put their time into reviewing a piece, only to have it snapped up by a higher-ranking journal, have sunk their time and energy into a lost cause. This becomes all the more concerning when authors submit to journals without the intention of ever actually publishing--instead hoping to game any acceptances into successful expedites, or to leave the piece unpublished until a later submission cycle. This latter practice, I propose, is something that all authors should take great efforts to avoid.

The Summer Cycle

Sorry--back off my soapbox. Law review submissions revolve around two primary "seasons" or "cycles." There's the spring submission season which starts near the beginning of February, and the summer submission season which starts in August. The general wisdom is that the spring submission cycle is preferable because this is the time of year when editorial boards are beginning to fill their volume. In summer, on the other hand, editors are trying to fill what slots remain--which may mean that there are only a few potential openings for new articles compared with the comparatively unlimited potential of the spring cycle. Because of this, the advice you'll typically hear is to publish in the spring whenever possible, and to avoid the comparatively tough summer cycle.

I've tended to take this advice with a grain of salt, as I've submitted and published most of my work during the summer cycle. I typically do so out of a combination of reasons--summer is when I have the most time to write, a fair amount of what I write on tends to be time-sensitive, and I'd rather move onto new projects than to build up a glut of unsubmitted work. This experience leads me to encourage people not to sleep on the summer submission cycle.

But what I heard earlier this year from the Harvard/Yale/Stanford editors may cause me to rethink my way of doing things.

All of these journals assured the audience that they accept pieces submitted during the summer. Harvard claims to have a year-round submission cycle. And yet the Harvard editor stated that for their last issue, they accepted seven articles in the spring and three in the fall. Yale's editor stated that they always leave the summer open, noting that they accepted two (2) submissions during the prior summer cycle. And Stanford's editor stated that they want authors to have a "substantial opportunity" to publish in the summer, noting that their prior volume consisted of ten pieces submitted in the spring, and four submitted in the summer.

While all of these editors asserted that this information demonstrated the utility of submitting in the summer, my takeaway was that the critics of the summer submission season were right--at least with regard to these journals. Getting placed at journals like these is difficult enough in the aggregate, and doing so in a season where one's prospects are placing only two to four articles for the overall volume leaves one with dismal prospects of success. To be sure, these are only three journals--and their prestige gives them the pick of the submissions universe. But perhaps there is something to waiting till the spring, if possible, to submit pieces to law reviews if this disparity is more widespread.

Artificial Intelligence?!

Here's a note from an anonymous editor in the Scholastica survey that raised at least one professor's concern:

It appears that there have been a lot more theoretical/social science and law crossovers lately. I predict that there will also be a significant increase in AI articles, given the number of symposiums that are talking about AI. I’m sure that selecting articles will no doubt use the same technology that we are discussing in our articles to also assist in choosing articles.

The notable portion of this quotation is the un-bolded text of the last sentence, which suggests that editors who select articles for publication will use AI technology "to also assist in choosing articles." What might this mean? Is it not enough that generative AI is going to end the world--or worse: render attorneys obsolete? Will our technological overlords also begin making the publication decisions that determine the trajectory of my and my colleagues' careers?

I'm not sure I'm all that concerned by the remark as some other authors might be--primarily because I don't think it's a fully enough formed idea to pose a discernable threat (or promise?) of substantial change to the way things work. Perhaps there is something to be said about running preemption checks using technology--and maybe AI will play a role in the process (though I suspect that the technology at issue here would more likely resemble existing search algorithms rather than the language models powering technology like ChatGPT). Indeed, when we start thinking of the varieties of technologies that exist under the "AI" umbrella, one might begin to see just how broad a notion editorial use of AI might be.

Law Review "Policies"

A theme I notice--particularly in thinking over the remarks of the Harvard/Yale/Stanford editors is the distance between written policies and how these policies work out in practice.

The Harvard Law Review, for example, claims to review submissions year-round. Yet the Harvard editor repeatedly made remarks about following submission cycles--at one point noting that they accepted seven submissions in the preceding spring cycle, and three in the summer. To me, this sounds like a journal that--like the others--follows the submission cycles. To be sure, this may result from submission patterns rather than the journal's own review procedures. But if this is the case and Harvard's review practices end up mirroring the summer and spring submission cycles, I wonder how much it means for Harvard to claim to review submissions year round.

There are other examples of discrepancies between policies and practices. Overly long articles are less likely to be accepted--except for the ones that are indeed accepted and published. The journals strive to maintain robust summer submission cycles--sometimes publishing two, even four!, articles submitted in the summer months. We accept "essays" that are around 15,000 words in length (Yale); or we accept essays (well--maybe 1-2 per year) (Harvard).

Perhaps this is an effort to educate authors who lack practice experience to how the theory and letter of the law breaks down in practice. A legal safeguard or requirement that is clear as day in the statute books can be of little use in a crowded courtroom where the judge has their own view of the law and the client lacks the funds to appeal. Law journals' willingness to bend or break their stated rules offer legal academics a taste of this all-too-frequent frustration practitioners face.