Improving the reviewing process in Ecology and Evolutionary

Improving the reviewing process in Ecology and Evolutionary Biology.— I discuss current issues in reviewing and editorial practices in ecology and evolutionary biology and suggest possible solutions for current problems. The reviewing crisis is unlikely to change unless steps are taken by journals to provide greater inclusiveness and incentives to reviewers. In addition, both journals and institutions should reduce their emphasis on publication numbers (least publishable units) and impact factors and focus instead on article synthesis and quality which will require longer publications. Academic and research institutions should consider reviewing manuscripts and editorial positions an important part of a researcher’s professional activities and reward them accordingly. Rewarding reviewers either monetarily or via other incentives such as free journal subscriptions may encourage participation in the reviewing process for both profit and non–profit journals. Reviewer performance will likely be improved by measures that increase inclusiveness, such as sending reviews and decision letters to reviewers. Journals may be able to evaluate the efficacy of their reviewing process by comparing citations of rejected but subsequently published papers with those published within the journal at similar times. Finally, constructive reviews: 1) identify important shortcomings and suggest solutions when possible, 2) distinguish trivial from non–trivial problems, and 3) include editor’s evaluations of the reviews including identification of trivial versus substantive comments (i.e., those that must be addressed).


Grossman
Discussion of shortcomings in the peer review process and editorial practices within scientific journals likely started with publication of the first journal and employment of the first editor.Recently, multiple aspects of this topic have been described in publications dealing with ecology and evolutionary biology (EEB) (Hochberg et al., 2009;Mesnard, 2010;Statzner & Resh, 2010;Albuquerque, 2011;Rohr & Martin, 2012a) and the subject has received considerable attention in the biomedical research community (Smith, 2006;Tite & Schroter, 2007).Multiple critical issues of the peer review process have been raised including: 1) the difficulties of finding good reviewers*, 2) the lack of reward for reviewing, 3) the increased number of manuscripts submitted to journals exacerbating issue 1, and 4) negative institutional policies reduce incentives for participating in the editorial process.Although a number of potential solutions to the reviewer crisis have been suggested, there is little consensus regarding what should be done (DeVries et al., 2009;Montesinos, 2012;Rohr & Martin, 2012b;Duffy, 2013) and there appear to be few changes in editorial practices by journals (Grod et al., 2010).In this paper I will discuss additional issues contributing to the reviewer crisis and propose several additional solutions.Much of what I report is based on my own experiences as an author of 110+ papers and also as a reviewer/editorial board member/associate editor for five different journals run by both scientific societies and commercial publishers.

Publication proliferation
There is no doubt that the number of manuscripts submitted for publication in scientific journals has increased substantially in the last few decades, primarily due to an increase in the number of scientists.In addition, the pressures of promotion and high competition for jobs in the last four decades contribute to the pressure to 'slice' publications into what historically has been known as the Least Publishable Unit (LPU) or 'salami tactic'.The combination of increasing publication frequency and decreasing publication length was recognized decades ago (Broad, 1981;Lyman, 2013) and is one of the main factors contributing to the reviewer crises in EEB.There is no doubt that many will judge a scientist's performance based on publication quantity rather than quality, and this is likely true for most scientific fields.The phenomenon itself is most easily observed in discussions of faculty search or tenure/promotion committees.It is clear that overall productivity (i.e., number of publications) should play a role in evaluations, but first assessments (and cuts) typically are made using simple criteria such as 'number of publications in refereed journals'.This criterion is easy, quick and may even be correlated with quality, but it also encourages vita padding.It is easily gamed by dividing larger potential research publications into LPUs, which contribute significantly to the editorial burden of the EEB community.Nonetheless, I doubt that publication frequency will ever disappear as an assessment criterion, but perhaps journal editors and referees should be more stringent in accepting papers that clearly are small slices of a complete pie.
The LPU syndrome has been exacerbated by the proliferation of journals in EEB (Statzner & Resh, 2010); including the explosion of 'Letters' (i.e., short format) and open-access journals (Bohannon, 2013), all of which require enough papers to regularly fill issues.Some researchers appear to think that the publication process is slower than it was 25 years ago (Statzner & Resh, 2010), but recent studies provide surprising answers to that question.For example, there has been no demonstrable increase in average review time for journals in either behavioral sciences or natural history between 1980 and 2012 (Pautasso & Schaefer, 2010;Lyman, 2013).In addition, although there is a positive correlation between impact factor of a journal and the number of manuscripts submitted, there also is a negative trend between impact factor and time to acceptance (Pautasso & Schaefer, 2010).Hence, higher number of submissions does not necessarily result in more extensive editorial delays (Pautasso & Schaefer, 2010).It is possible; however, that the latter result is a consequence of many papers being rejected by journals without review (Pautasso & Schaefer, 2010) as has been the policy of a number of prominent EEB journals.This practice, although providing a quick turn-around for a manuscript, is quite susceptible to bias and cliquishness in publication, as noted in1974 (VanValen & Pitelka, 1974) and still in evidence today (Arnqvist, 2013).Nonetheless, in contrast to the results of Pautasso and Schaefer (2010) a recent survey of EEB editors showed a negative relationship between the number of papers handled and the proportion rejected without review (McPeek et al., 2009).

The referee pool
Given the increasing number of both journals and submissions, coupled with a pool of experienced referees that while increasing, still is insufficient to handle the current load (Hauser & Fehr, 2007;Statzner & Resh, 2010;Arnqvist, 2013;Duffy, 2013), it is obvious that the EEB community has yet to effectively deal with the 'reviewer crises'.Several investigators have suggested ideas for dealing with the decreased willingness of referees to perform reviews, the high number of review requests received by 'good referees', and issues of review quality (Hauser & Fehr, 2007;Fox & Petchey, 2010;Rohr & Martin, 2012a;Duffy, 2013).These suggestions involve punishing slow reviewers and rewarding timely referees who provide thorough reviews, but as all authors admit, these solutions may do little to prevent some scientists employing 'cheater' strategies.Nonetheless, they all are right that changes are necessary to improve the current status of reviewing.
Perhaps referees are no more nor less altruistic than they have been in the past, but what has changed in the last 30 years are the external constraints on a researcher's time.Most university researchers, at least in the United States, are now faced with a plethora of administrative responsibilities from both their own universities and governmental sources (e.g., faculty committees, training sessions for compliance with laws such as the Family Educational Rights and Privacy Act [FERPA], monthly documentation of graduate student performance, Institutional Animal Use and Care Committee (IACUC) requirements and training, federal data accessibility requirements, etc.).Concomitantly, both university and federal research budgets have been slashed in the United States and other countries; consequently researchers must devote much more time to seeking research funding than they have in the past.This is one of the major reasons referees are slow or reluctant to review papers; simply put, there is little time or energy left after performing one's daily research responsibilities (Statzner & Resh, 2010).At the same time, the qualifications needed to obtain a research or faculty position are increasing (Statzner & Resh, 2010).Hence, even if someone is a 'good Samaritan' (McPeek et al., 2009), there are strong selective pressures acting against altruism, even if they are merely perceived rather than real.
There is no solution to this problem until reviewing manuscripts, and editorial work in general, are viewed as normative responsibilities, with appropriate recognition and rewards from administrators.I suspect that in most institutions, editorial board membership or extensive reviewing rarely results in raises, increased release time or help from support staff.My supposition is that administrators resort to claims like 'well everyone does that so we can just assume that it is a constant across faculty' but the current crises suggest that reviewing and editorial work are not constant across faculty.In addition, an erroneous assumption by administrators that reviewing is equal across faculty promotes 'cheaters' who do no reviewing and devote all their time to writing grants or papers instead, especially when promotion decisions are made on a comparative basis.Faculty must become more proactive in demanding that incentives be provided for highly active and competent reviewers and associate editors, and managing editors should support them in this quest.Hopefully, this will result in administrators providing substantive rewards for participation in the editorial process as well as penalties for faculty who do not participate.

How can journals and editors improve the situation
At present, there appear to be few journals that provide incentives for reviewers.A few journals provide free access to online versions of the journal although frequently this only extends over a month or two.Certainly one perquisite for reviewers that could increase referee responsiveness would be to give a free online subscription to the journal after a given number of reviews in a year.Even non-profit scientific societies could employ this incentive because it is not costly.Incentives could be provided on a graduated scale where it might take four reviews in a year to obtain free access for a year, and a single review might earn only three months access.Of course this may penalize members of scientific societies who already receive a journal subscription, but they still might not have online access or they could be rewarded with free access in the next year or access to a journal they do not receive (many scientific societies publish multiple journals).No incentive scheme is perfect but it seems that some experimentation is called for at the present time, given the repeatedly voiced concerns by both editors and authors.
It is possible that paying referees for reviews could improve both referee participation and performance, but its discussion mostly has occurred on online forums.I have found no published evaluation of this practice in EEB, although EEB outside examiners are paid by universities for dissertation reviews in both Australia and New Zealand and likely other countries.In addition, multiple European countries (Ireland, Poland and Spain) pay for proposal reviews, as does at least one commercial publisher for editorial board work.Nonetheless, a study of biomedical reviewers found that reviewers had mixed opinions regarding the positive impacts of financial rewards on the reviewing process (Tite & Schroter, 2007).The biggest objections to payment for services involve the end of volunteerism, and the assumption that financial rewards will bias the reviewing process, or pull referees away from journals that cannot provide incentives.There is a lack of evidence but I suspect this is unlikely.From a philosophical perspective, I deplore the loss of the volunteer ethic in science; however, the current crisis seems immune to philosophical regrets and perhaps represents the triumph of the market economy even in science.One of my goals is to suggest possible approaches leading to data on potential strategies to resolve the reviewer crisis.It would be useful for an EEB journal or society to conduct an experiment in which some reviewers are paid and others not and then compare the quality, timing, and responsiveness of the two reviewer treatment groups.There is no doubt that such an experiment would require a sophisticated design and still likely present logistical hurdles, however, it should aid in determining whether financial rewards would improve the reviewing process in EEB.Finally, it is true that payment for reviewers and editorial work may present logistical and financial difficulties for non-profit journals, however these obstacles are mostly irrelevant for the many journals published by highly profitable commercial publishers or open-access journals with high publication fees.
One of the reasons for the poor performance of reviewers is that too many journals fail to cultivate a culture of inclusion in the editorial process.I suspect reviewer performance would be substantively improved if journals practiced a few simple steps that demonstrated the importance of individual reviews in the overall editorial process.For example, although some journals provide a reviewer with all reviews of a manuscript and editor's decision letter, too many do not.Reading the comments of other reviewers and the editor always is an educational experience and is an excellent me-Grossman chanism for less experienced reviewers to learn from more experienced reviewers.In addition, it would be beneficial for everyone involved if editors explained their reasoning when they overrule a referee.Finally, I wonder how much effort is expended by journals in evaluating whether their editorial practices are efficient and unbiased, or whether the prevailing attitude is one of laissez faire (Grod et al., 2010).Certainly one way that journals could evaluate the accuracy of their reviewing practices would be to compare citation frequencies of a random sample of articles rejected by the journal but subsequently published in other journals with a sample of articles accepted in that same year.Although citation frequencies are not a perfect metric of quality, they are easily obtained and certainly indicative of quality if the citations are positive.Such an analysis should be conducted with historic data, for example volumes published 10, 7 and 4 years previously.If no difference exists between citation frequencies of the two sets of papers, and assuming that the rejected papers were appropriate subject matter for both journals, then it would be cause for examining historic editorial practices, or to determine if specific associate editors were the cause of these rejections.Of course the citations would have to be checked randomly to assure that the citations were comparable (i.e., to avoid the case where total citations are equal but one paper has all positive citations and the other has all negative citations).It also might help identify continuing trends in problematic editorial practices.In addition, recent work has shown that factors such as journal impact factors may affect reviews independent of manuscript quality, and that reviewer ratings of the same manuscript may not be highly correlated (Eyre-Walker & Stoletzki, 2013).It is likely that there is little formal or quantitative evaluation of associate editors for many journals, except where an editor's behavior becomes intolerable, such as failing to act on multiple manuscripts.These issues all call for journals to evaluate the accuracy and precision of their reviewing policies.

Improving reviews and reviewing
There is no doubt that high quality referees and editors are both typically overworked.Nonetheless, if editors believe that reviewers should not use this excuse, then neither should they.My own experience suggests there has been a decline in the quality of review interpretation and decisions made by editors as well as a general decline in review quality.I have already mentioned fostering a sense of inclusion for referees in the editorial process and (Statzner & Resh, 2010) have covered many of the current negative trends in the editorial process.Having published my first paper in 1977, I have seen just about every constructive and inane comment possible, typically with no comments from the editor on inappropriate or obviously erroneous comments.I believe that it is an editor's responsibility to ensure that an editorial decision letter does not come back to an author without commentary on the reviews.At the very least, editors should identify reviewer's comments that must be addressed versus those that are optional.
Nonetheless, the evaluation of reviewer's comments by editors certainly is not general policy for scientific journals.Given the complaints by editors regarding the poor quality of many reviews, this is not a trivial issue, yet most editors provide an author with little guidance other than 'all comments must be addressed, especially revisions that you do not incorporate'.But how much detail must be provided by an author when a comment clearly is erroneous: a not infrequent situation?This can be particularly problematical for young scientists, especially given the many picayunish negative comments written by reviews of today.Frankly, if editors are actually reading reviews closely, as they should, then it does not take much more time to identify which comments need to be addressed and which do not.After all, how can an editor reach an informed decision without evaluating reviews, even when both ratings are reject?Every author deserves at least this much from an editor.An additional problem of today is that the category of 'accepted with revision' seems to have disappeared from many journals and instead the author is told that their manuscript has landed in the large gray category called 'not acceptable in this form'.I have spoken with many researchers, especially young researchers, who have interpreted this as a rejection, when in fact it really is just code for 'significant revision'.Nonetheless, some editors have justified this change by saying that it was difficult to obtain substantive revisions from authors once the term 'accepted' had been used.

What constitutes a good review?
A thorough discussion of the reviewing process is provided by DeVries et al. (2009), an article that is particularly useful for young scientists.An interesting psychological question for both editors and reviewers is whether a paper should be viewed as acceptable until a sufficient number of problems render it unacceptable, or whether papers should be viewed as unacceptable until a sufficient number of positive points are identified so that it becomes acceptable.I favor the first view point, mainly because I believe it leads to more constructive reviewing and hopefully a more positive experience for the authors, even when a paper is rejected.Many journals do not have review templates that ask a reviewer to specifically identify both the strengths and weaknesses of the manuscript but this would lead to more objective reviewing and improved editorial decisions.
For both referees and editors, clearly the criterion for any comment is whether or not it is truly constructive.Probably the most significant improvement would be to require referees to reference their criticisms.I have seen comments ranging from 'this simply is wrong' to the 'literature review was inadequate' without any subsequent explanation of why a given technique was wrong nor any subsequent listing of missing papers.Such comments are completely unhelpful to the author and certainly do not fall under the rubric of 'constructive criticisms'.It is not the reviewer's responsibility to rewrite an author's manuscript; nonetheless, unconstructive comments and reviews help no one and eventually result in a bad reputation for a journal.I know more than one scientist who simply has stopped submitting manuscripts to journals that have persistently poor reviewing policies even when they have high impact factors.Nonetheless, clearly this is a luxury of the full professor, not the untenured assistant professor.A final comment on writing style is warranted, given that many current referees seem to have little tolerance for a style different from their own.I have received reviews stating that a manuscript is poorly written without any description of how this judgment was reached, let alone an 'example' paragraph that was rewritten to demonstrate good writing.In addition it is not uncommon to receive reviews in which one reviewer ranks the paper as well written while another says it is poorly written.Once again, this is the type of comment that should prompt an editor's intervention but this is rare in my experience.Consequently, if you cannot identify specific problems in grammar, clarity or verbosity accompanied by examples of how this can be corrected, then it is likely that you and the author have different writing styles, and it should be left at that.An even more problematical stylistic issue is that of non-native English writers, and the level of grammatical 'stretch' that should be allowed in such manuscripts (Clavero, 2010) As with any large volunteer enterprise, problems exist with the current peer review system and whether or not they will be fixed depends on the EEB community itself.Nonetheless, I hope that the suggestions made in this paper are helpful, even if they only lead to small improvements in the overall EEB editorial system.Most importantly, journals should begin conducting experiments regarding changes in editorial practices that may improve the various aspects of the 'reviewing crisis', and ultimately communicate the results of these experiments to the EEB community.