Eira Tansey

Peer review for archivists (or, WTF is going on with this SAA pre-print)

One of the many things that library school did not prepare me for was how to do effective peer review. The economics of peer review is that the more you write, the more you get asked to read other people’s work, either informally or formally (I define informally as when a friend or acquaintance asks me to look at something, formally is when a third-party like an editor asks me). As I’ve been reviewing more and stepping into some temporary editorial roles, it’s made me wish there was better guidance for archivists how to do peer review. I’ve often thought about how I could turn one of my reviewer or editor’s reports into a tutorial, but that would be a major breach of confidentiality. However, a timely event that shows how important peer review is just presented itself days before the Society of American Archivists annual meeting in Austin.

As I was packing for the trip to Texas, a friend tipped me off that the big chatter on Archivist Twitter was Frank Boles’ pre-print in American Archivist. The pre-print will be the subject of a lunch time discussion forum at the annual meeting. Having some skepticism about Twitter in general, I decided to print off Boles’ article and tuck it into my luggage to read and draw my own conclusions on the road down.

As full disclosure, the only thing I’ve published so far in American Archivist (AA) is a book review. I’ve been emailing with the current managing editor Cal Lee about something I’m thinking of submitting. And on the other side of the process, I have done peer review for the journal. I was recently asked to review another article a few days ago but turned it down only because I couldn’t make the timeline with other obligations. I know first hand how difficult it is to get good reviewers (“good” meaning both competent and reliable for meeting deadlines for reviewer reports), so I informed Cal right away. I do not know much about the internal workings of the AA editorial board.

In brief: if I were a peer reviewer for this piece, I would have recommended such drastic changes that I would have hoped the submission would not have gotten any further through the editorial process in its current state. Boles’ writing has serious issues and some alarming conclusions, but equally vexing is what happened with the editorial process and how this piece got this far.

I pulled my reviewer report from when I last reviewed for AA summer 2018. I am not sure if the reviewer form is still the same, but here is the report I would have written if the Boles piece were sent to me for review. I hope this helps those who aren’t familiar with the peer review process understand how a good peer review should prevent things like Boles’ article from ever getting this far. I might do a follow-up blog post talking about tips for peer review – if you want me to address anything specific in a follow-up post, let me know.

Reviewer comments for “To Everything There Is a Season”

Statement of Problem of Purpose (the theoretical or practical problem or challenge):
This articles proposes that there are three interlocking ideas that have predominated archival discourse in recent years: universal documentation, the role of social justice in archival appraisal, and the construction of archival power. The author argues that these ideas are counterproductive and proposes that archivists should answer first to the needs of their institution.

Relevance of the Topic (to the mission and purpose of the journal):
The theory and application of social justice to archives has been a topic of intense archival discourse, both within this journal and elsewhere within the larger archival profession. Topics of documentation methodologies and archival power have also been present in the journal.

Importance of the Topic (advancing thought on archival principles and practices):
This article represents a sweeping critique of several ideas that the author claims are connected, and are hurting archival practice. Some of these ideas – especially archival power and the application of social justice to appraisal – have been the sources of significant professional arguments over the last several years. The author drew on some of these past critiques in constructing their argument.

“Social responsibility” and other related ideas are part of the SAA Core Values, and a large number of archivists take it for granted that these are implicitly good things. A well constructed counter-argument can serve an important purpose in clarifying the shared norms around implicit values. However, a counter-argument is only as strong as it shows deep familiarity with the material it is critiquing. Counter-arguments must also not lapse into caricatures or superficial treatments of topics that a community deem to be of significant importance. Despite the importance of the topic and the useful role that counter-arguments can play, I do not believe this submission is a successfully constructed counter-argument, for reasons I will articulate through this report.

Contribution to the Literature (original contributions to the literature):
I have significant concerns about whether this article actually advances the point-counterpoint discourse that is an inherent part of any long-standing academic debate. This piece does not demonstrate a comprehensive grasp of either the published literature of archival social justice or the practical applications of it through current archival projects. In order to properly critique something, one needs to be familiar with the norms, literature, and standards of a particular community. My sense from reading this article was that the author has a very superficial understanding of the theory and application of the ideas they critique. Trying to bring together three somewhat related but ultimately independent ideas into one large critique means there is not enough space to engage in a meaningful dissection of each idea.

The author is often overly-reliant on the writings and ideas of others to make their argument for them. This muddies any claims of originality, since as the author points out, they are not the first to raise some of these concerns. While building on the ideas of others is normal, it is not clear what additional original research was added. For example, if the author had reviewed a selection of institutional collection development policies over the last 30 years to determine how social justice had impacted collection decisions, this would be a highly original contribution to the research.

In addition, the author is often reliant on straw man and slippery slope arguments. There are many, but I include two examples here. First, on pages 4-5 the author claims that other archivists have called for a complete and universal “documentary mirror” but have not acknowledged the challenges of doing so. This is not accurate, as many archivists who have called for us to challenge our approaches to collection and appraisal have also acknowledged that (to paraphrase Verne Harris), archives are but a sliver of a sliver. Another example appears on page 11, when the author presents a hypothetical scenario about whether the profession would embrace one set of social beliefs over another, pass a resolution at an SAA annual meeting, and potentially call into question the professional status of archivists who refuse to go along. This is a slippery slope argument because SAA is not a professional regulatory or licensure body similar to a bar or medical association, and has no power to materially sanction archivists who may disagree with prevailing norms.

Perhaps the greatest weakness that obscures how the author situates themselves in the existing discourse is that the author does not provide their own definition of social justice or power. Social justice is a term that is subject to a variety of different interpretations, and I suspect many individuals who are uncomfortable with the connotations of social justice and its historical associations with left-wing activism would be in agreement with at least some claims typically associated with social justice, particularly if they were presented under a different moniker. These include ideas like how societal power is not equally distributed and there are historical factors for this, or that accountability is not meted out equally. Because the author never defines what social justice or power means to them, as a reader I was left wondering if social justice simply means everything the author disagrees with.

Organization (of ideas and supporting points):
There are four main sections of this paper: the critique of universal documentation, the critique of social justice as a factor in appraisal, the critique of archival power, and a conclusion. The conclusion closes with three recommendations for where archivists should shift their attention instead. Many of these recommendations raise their own questions of relevance, application, and morality. However, the author devotes less than a page to these recommendations.

Drawing and Building Upon Relevant Literature (summary of the major points in the relevant literature):
This article does not demonstrate significant familiarity with existing archival literature, as well as practical applications of these ideas currently in existence, particularly those of social justice-influenced archival projects. In the critique of universal documentation, I would have expected to see a meaningful treatment of Helen Samuel’s documentation strategy work, and a consideration of the many projects since then that have built on documentation strategy (particularly those that exist outside of institutional constraints), such as the South Asian American Digital Archive, a People’s Archive of Police Violence, Documenting the Now, or Student Activism Now Documented (STAND). Also, the Levy Report of the 1980s is a touchstone for any discussion about the challenges of funding for archival operations, and the author should consider referencing this.

I also found that the author often made claims about the outlooks of certain authors without demonstrating a meaningful engagement with their work, relying instead on cherry-picked quotes. For example, regarding Christopher Hurley’s work, the author claims “Expressed somewhat differently, the approach to appraisal Hurley suggests is overly oriented toward bureaucracy and records management. In taking a narrow records management approach toward institutional records, Hurley carries forward without nuance the original purpose for which records are created.” This is a very strange interpretation of Christopher Hurley’s 2001 address to the ACA, which specifically delved into the role that accountability plays in government archives, and how governmental records mean different things for different groups. One of the major examples he used in his talk was the existence of British government records concerning Stalinist Russia and how these records were used in a lawsuit, but their disappearance impacted legal proceedings.

In addition to the cherry-picked sources, the author often has fairly questionable sources – for example, a reference to a conversation at a bar several decades ago (endnote #7) might be interesting in the context of a personal essay but is not an appropriate reference for a reviewed publication.

Methodology:
It is not clear to me whether this piece was submitted as a research article, a case study, or a perspective. I assume based on the length it was either a research article or case study. The author does not share a methodology for their critique, nor does the author provide any clear reasoning for why they chose the sources they did. In a well-constructed critique that relies mainly on existing literature, I would have expected to see something like “In reviewing articles published in the American Archivist over the last two decades, [number] of articles have been published concerning social justice and archives. This article will consider those that received the highest citations since 2000.” This would have demonstrated why they chose the sources they did – but without any clues, the reader is left with the impression that the author arbitrarily picked the pieces that most closely matched their pre-existing ideas.

Discussion (develops major points with relevant evidence and solid reasoning):
The discussion is mainly embedded within the three critiques that structure the majority of the submission. The first critique the author offers mainly relies on thinly sourced claims and the assertion that universal documentation is both a widely embraced value and unrealistic due to resource constraints. This is a very short section of the manuscript, particularly given the breadth of work on how many archivists have written about their collection strategies in light of constrained resources. It is curious that the author claims that “archivists have failed to answer fundamental questions” as many archivists have written quite a bit about documentation strategy, and much of the discourse around post-custodial community archives explicitly touches on the creation of these archives in the face of institutional resource challenges.

The second critique concerns social justice and archives. In this section, the author draws on work by Michelle Caswell, Mario Ramirez, Christopher Hurley, Verne Harris, Rand Jimerson, and Mark Greene. The author then considers that norms about what is considered moral are subject to change, by exploring historical and current attitudes to slavery, prohibition of alcohol, and abortion. This section read far more like a distracting digression, because the author never connected how changing social norms around what is considered acceptable or moral should actually influence appraisal decisions. The claim that social norms are constantly changing is not in itself an argument against incorporating social justice into appraisal decisions. The author does not provide any concrete evidence of actual collecting decisions made by archivists to substantiate their claims that social justice is an inappropriate appraisal factor.

The critique of archival power draws largely on Rand Jimerson’s work, and then uses Christine George’s article about archival privilege as a case study. Unlike the first two critiques, the structure of this critique is improved by a) not trying to cover too much material at once and b) actually demonstrating the consequences of the idea that the archivist is critiquing.

Overall, author seems prone to false equivalencies that undermine many of their arguments. For example, equating stereotypes of young black men with older white men ignores the fact that stereotypes of young black men are contributing factors to disproportionate uses of state-sanctioned forms of control (whether from law enforcement or incarceration). While stereotyping may be unfair to older white men, it does not result in the same potential material consequences.

Conclusion (conclusion with justification from evidence presented):
I found the author’s conclusion to raise so many questions and concerns that I think if they are serious about their conclusions they should have led with them first. The claim that archivists should serve their institutions first and foremost, even under questionable circumstances, is an alarming conclusion to draw. As many scholars from other fields have pointed out, simply “following orders” is not consistent with many established legal frameworks. If the institution ordered archivists and records managers to destroy records in violation of state or federal records laws, would the author still make the claim that archivists are to “implement an institutional mission fully and well”?

The author needs to deeply consider the implications of the claim that archivists should take their main directives from an institution’s mission for three reasons. First, in many cases archivists have and exercise far more agency over institutional documentary missions than the author suggests. Archivists are often responsible for implementing institutional records management decisions, and collection development policies. Second, if non-archivist led institutional missions should override the expertise of archivists and professional practice norms, then one might ask – why even have a professional association of archivists? Very few other professions would comport themselves with such total deference to an institution’s needs. Doctors may work in hospitals run by healthcare administrators, but it is not healthcare administrators who are directly responsible for treating patients and exercising medical judgement. Faculty may work in universities managed by higher education administrators, but faculty maintain significant curricular control as part of their disciplinary expertise.

Indeed, institutional missions are often at significant odds with professional standards of practice. To use a current example, federal environmental agency scientists are increasingly finding that their work using standardized scientific norms and practices are being curtailed by political appointees. Rather than ceding ground to non-scientists, many are now resigning rather than compromising scientific integrity that changes with shifts from top management. Those unable to resign due to economic circumstances are finding other routes through union representation, anonymous tips to journalists, or whistleblowing to register their concerns. If archivists are to maintain professional integrity and standards – and most scholars of professions would argue are defined by professional norms, not institutional interests – I would hope archivists put into similar situations by their institutions would either resign or find ways to become a whistleblower or otherwise throw sand in the gears.

Mechanics (errors in usage, spelling, punctuation, and reference format):
The submission is largely free from any punctuation or spelling errors. One area of confusion is the reference to the judge in 1986 on page 15. It seems clear from reading this is a reference to the 1986 court case involving access to civil rights activist Anne Braden’s papers, though the author includes the example in such a way that implies it might have concerned the IRA oral histories case.

Additional blind comments to author:
I recommend narrowing the focus in this paper to just one issue to critique. Currently there is too much going on here to make an effective well-connected counter-argument. The treatments of the three issues seem superficial and the interpretations seem based on personal hunches and slippery slope arguments as opposed to evidence-based findings to support your analysis. Any counter-argument to widely-embraced community values needs to demonstrate a strong grasp of familiarity with how these values came about. I strongly recommend reading additional sources about documentation strategy, and current social justice archives projects such as Documenting the Now. Any claims that widely-shared norms are actually harmful need to be substantiated by evidence, not hypothetical situations.

If you feel strongly about your conclusions, you may wish to reorganize the paper to lead with these first, and show evidence for why you think these should be prevailing interests over the ones you critique.

Additional confidential comments to editor:
While I think there are legitimate counter-arguments to make regarding recent archival social justice discourse, I do not think this article in its current state meets that threshold. For any further consideration, it would need such significant and major revisions that it would be an almost completely different paper.

I found much of the tone of this submission to be needlessly provocative, such as endnotes #10 and #35. In many ways this submission read as a personal diatribe disguised as a journal article than an actual meaningful contribution to define the limitations of archival social justice. The topics the author raises are important ones and deserve to be treated both with care and professional due diligence. It does not seem like either consideration was a priority in the writing of this submission.

Tagged with: Peer review

Categorised as: SAA