Last week on twitter I was asked for advice about how to go about doing your first peer review. I have some experience of being a peer reviewer but not a huge amount so I asked the twitter hive mind and, as ever, lots of fantastic suggestions came pouring in! Below is a selection of that advice but please tweet me or comment below if you have other ideas to add. Thank you very much to everyone who replied and also to Kathryn Maude who tweeted the same question on the same day! It’s an important issue in academia and one that we need to talk about more openly and in more detail, so thank you to everyone who raised the issue and contributed their ideas. (And sorry that I haven’t been able to quote everyone below, this is in part because the new twitter on my desktop is being impossible and some of the comments seem to have disappeared!!)
I think we can all agree that being a ‘good peer reviewer’ means avoiding being the awful stereotype of ‘reviewer 2’ that we all dread (and I’ve written about the emotional impact of negative and unconstructive criticism in another blogpost- click here– which includes practical tips for how approach revising a submission.) Even when criticism is constructive, it can be difficult to confront those revisions as it can bring up all kinds of ‘imposter syndrome’ anxieties, no matter how many successful ‘revise and resubmits’ we may have achieved!
So when we are the peer reviewer, and we know the impact that feedback has, how can we make sure that we produce rigorous, accurate, and scholarly feedback while at the same time being kind, helpful, and showing the author that we have engaged with their work in detail and with serious consideration?
I was interested in the fact that many of the suggestions in response to my tweet were a matter of style as well as content. When I’m marking students’ essays, I’ll often use the ‘feedback sandwich’ method– start with the positives, then the critique (with specific instructions for improvement), and then a reaffirmation of the progress that has been made that week. Psychologically, I think we all respond positively to this kind of approach! So the advice below focuses on content and style, and proceeds from the basic assumption that kindness should be at the heart of what we do. We all know how emotionally entangled we are with our own work so we should do others the courtesy of feedback which respects their efforts and offers genuine suggestions for improvement, so that the work can be the best that it can be. No one really wants to publish work before its time- I am now so glad that my first attempt at my monograph was rejected! (see this post for more on that topic)- but rejection is never easy to cope with. So, do unto others as you would have them do unto you!! Ask yourself, how would I feel if I received this review? (And if you’re unsure you’ve got the tone right, you could try sending it to a colleague to see what they make of it)
Related to this- and it’s always a temptation!- don’t write a review which is really a ‘if I’d written this article, I’d have done it this way’ comment! You haven’t, it’s the author’s article, so judge it on their terms and offer feedback that that helps them to improve it for themselves! So don’t be reviewer 3!!
General thoughts
As @DrFrancisYoung summarised, ‘always be kind, always be constructive- but also be honest.’ Not everything is publishable, of course, and not necessarily in its first form (I had an article out last year that went through many different incarnations before it was accepted but it was better for it in the end!). But there is no need to be unnecessarily cruel to be kind, in my view.
@SarahBlick3 commented that ‘peer review is meant to help fellow scholars and the field as a whole’, so it can helpful to think about your review as part of your ‘service’ to your colleagues and to your field, just as you have been helped by the peer reviewers who have put in time to review your work.
Matthew Weait (@ProfWetpaint) commented that ‘I find it helpful to imagine the author is a friend whose career you want to support. That way you orient your comments in the most constructive way’, even if you do end up concluding that the paper should be rejected because it’s not quite there yet. @lizgloyn also comment that she peer reviews ‘as if I were reading an article for a friend who has asked for pre-submission feedback.’ @schuklenk commented, ‘write the review in such a way that you would have appreciated receiving it, even if highly critical, had you been the author’ and I absolutely agree with this.
Praise the author, critique the work! (‘The author has done an excellent job at… The argument is less effective when…’ etc). This was an excellent tip I picked up from @EllieMackin on twitter a while ago and in terms of the style of the feedback, I think it really helps to minimise how personal it can feel when your work is criticised.
Bear in mind that if you read something that needs a lot of work, it could be someone’s first submission! @lizgloyn commented that she has reviewed materials that do look like a seminar paper with a few tweaks and so ‘clear and encouraging feedback for someone getting their first peer review is SO important.’ There is so much that can feel like a closed shop in academia (which is partly why I write my blogposts to demystify some of these processes!) and not everyone has access to the same support or knowledge, so don’t be cruel!!
When I did my first peer review, I was surprised by how ‘easy’ I found it to list my critiques upon first reading and I had to make a conscious effort to make a list of the positives- there were lots and I was recommending to publish, and I think of myself as a kind and supportive reader! But I think we can be trained to look for problem areas and they often jump out first, so I’m very conscious of this whenever I’m writing feedback for peer review or indeed for students’ essays.
There was some discussion in the light of my tweet about when to decline a peer review invitation. It is important to feel that you can support the work, even if you are critiquing it. If you fundamentally disagree with it, that is not going to make for a constructive peer review! Equally, if you are far too close to the work, you may not be able to be objective. I recently declined a peer review after discussion with the editor because I had published an essay on precisely the same topic and didn’t feel that that would be fair to the author as I felt far too close to the topic (and I knew that I wasn’t the only person with expertise in the broad area so I wasn’t ultimately doing the author a disservice by refusing to review it). Use your common sense and fair judgement on this issue! (And thanks to Diane Watt for raising this point).
There is also an issue here about being professional and responsible about the time it will take you to do the review. @andrewdbuck said that if you can’t realistically fit it into your schedule, say no. I would add that if you do agree and then things happen- as they did to me when I reviewed a larger project the other year- I made sure that I kept the editors informed and gave them a realistic date by which to expect my review (so that they could then pass that on to the author). No one wants to be kept hanging around- especially when there is so much pressure to publish these days!- so do try to get reviews done in a timely fashion. I had to wait nine months for reports on an essay once and it was agony, and I’m sure there are people who have waited longer…
Specific and practical suggestions
When it comes to writing your review, what should it contain and how should you organise it?
@lizgloyn tweeted that her feedback is often about ‘structure; methodology; contribution to the field/debate/conversation; independent contribution’ and that the key contribution that we as peer reviewers are making is ‘advice on how to make [the paper] better, not tear it down.’ In one peer review that I did, I had a helpful ‘checklist’ from the editor with questions to answer, so obviously if you are asked for specific feedback, follow that, but otherwise make your own checklist like this.
As with my ‘feedback sandwich’, @SarahRoseCrook suggested ‘always start with the positives’ and she also suggested a great practical tip which is to number suggested additions / amendments as it makes it easier for authors to comment on their changes (or to refute suggestions) when submitting the amended version. I will definitely be doing this in the future!
@DrKylieMurray also agreed, ‘start with the positives, no matter how difficult to find!’ What are the good things in the paper, what contribution does it make? Before then identifying particular problem areas and the ‘potential’ that could be developed further. I find this comment about potential especially helpful and resonant- I received a peer review that identified that I was trying to run two arguments simultaneously, one of which had already been accepted in the field overall but the second of which was far more interesting and original. Once I recognised that, I could prioritise the second argument and the paper made more sense to me!! Thanks to the reviewer who spotted this!
Be specific!! If you are going to suggest improvements to the paper, be precise about how the author might achieve those improvements. Is there a particular area of the argument that needs sharpening up or needs more evidence? Can you suggest specific secondary reading that the paper should engage with? (Don’t just say, ‘lots of scholars have discussed x’, who exactly?! If the author hasn’t mentioned them, they probably don’t know about their work) Is there an additional piece of primary material that should be analysed? Is there a counter argument that the author might want to raise and argue against?
Be detailed in your feedback. @Yoav_inn_riki commented that a detailed review shows that you are involved in the process and have thought carefully about your feedback. It shows that you care and, as @KingsManorGhost commented, shows that you are on the author’s ‘side’, as well as showing the author precisely where their arguments might need to be clarified.
This relates to @lizgloyn’s suggestion that you begin the review with ‘a summary of what the article is trying to do’ and the ‘extent to which it achieves its goal.’ @LauraMorreale also commented that by starting with a summary of the argument, you communicate to the author what the reader understood it to be about (which is very helpful to the author, ‘because sometimes we lose the thread or main points when we edit and re-edit’- that has definitely happened to me!) It might also be that the author sets up an ambitious goal which is only partially fulfilled in the scope of the paper.
@lizgloyn suggested reading through the article and making lots of notes first, then collating the feedback into a few ‘Big Points’ before listing the minutiae. @rachel_delman similarly described a really useful peer review that she had received in which the reviewer gave feedback on wider conceptual points and then smaller, more detailed comments. The reviewer also posed questions which showed engagement with the work and softened the tone.
@andrewdbuck also made an important comment about not ‘over reading’ a paper. He commented, ‘I usually do a first pass and make notes on the hard copy; then write those up and read again to fine-tune notes; and then read one last time just to be sure.’ This strikes me as excellent advice for how to approach the practical ‘how to’ side of peer review.
I also had some very useful replies from editors on twitter. One comment was that the peer reviewer should state a clear recommendation and justify it. If you are recommending publication, state what contribution the paper makes to the field. If you are recommending revise and resubmit, state the areas that need to be improved in order for the paper to fully succeed. If you are recommending rejection, state clear and detailed reasons why. @AnneOAlbert commented that editors can receive reviews that recommend publication but then focus on the critique and so can leave the editor reading between the lines. @CanaryCaroline also commented that of course it is better to received constructive critique at the peer review stage rather than, say, when a monograph has been published and a book reviewer takes it task for issues which could have been addressed earlier. Peer review should identified these issues and offer potential solutions so that the author can iron them out at this stage.
Importantly, if you are rejecting a paper, as @lizgloyn notes, ‘be encouraging- you know it’s disappointing but here’s stuff to build on’
* * * * * *
Thank to everyone who has contributed to this discussion. I hope it helps those who are new to peer reviewing. It has certainly helped me to remind myself of the important principles to follow and given me lots of practical suggestions to implement when I’m doing my next peer review.
Do check out my page of Resources for Grads and ECRs for similar posts and let me know if you have any other ideas for useful topics that I could write about. And finally…
Dr Laura Varnam 