[AISWorld] On review cycles in our discipline

[Doug Vogel]

I think you also need to include some form of reward system as Kevin Crowston has previously noted

Sent from my iPhone

> On 7 Apr 2015, at 12:28 am, mmora at securenym.net wrote:
> 
> Colleagues,
> Several issues have been commented:
> 
> - lack of acknowledgement in tenure-track careers by reviewing papers
> - lack of time for conducting 5-10 reviews by year
> - lack of good reviewers and willing for doing it in reasonable time frames
> 
> and some facts:
> 
> - a large cycle review period in our discipline compared with other sciences
> - an ad-hoc review process with a large variability on styles and tones
> - a misused or abused blind review mode
> 
> then, some feasible (?) solutions for improving it have been also posited:
> 
> - to eliminate the blind review mode
> - to foster the acknowledgement of the review process
> - to posit some standards review forms (accredited from AIS) with some
> exemplary cases
> 
> We have and live in this problem (junior and mid career Faculty) but
> solution "is in hands" of senior Faculty,
> Thanks,
> Prof. Mora
> Autonomous University of Aguascalientes
> Mexico
> 
> 
> 
> 
> 
> On Sun, Apr 5, 2015 at 9:58 AM, Juhani Iivari <Juhani.Iivari at oulu.fi> wrote:
> 
>    Hi All,
> 
>    Yes, I largely agree with the two comments below. Some time ago I
> wrote my thoughts about peer reviews, based on my experiences as an
> author, reviewer and editor. Perhaps not surprisingly, I have not
> attempted to get it published anywhere, but you can access it form
> ResearchGate
> (https://www.researchgate.net/publication/263766605_How_to_improve_the_quality_of_peer_reviews__Three_suggestions_for_system-level_change),
> if you are interested in.
> 
>    With best regards,
> 
>    Juhani Iivari
>    Professor emeritus
>    University of Oulu
> 
> 
> 
>>    On 05 Apr 2015, at 16:50, Arto Lanamäki <Arto.Lanamaki at oulu.fi> wrote:
>> 
>> Hi,
>> 
>> I think that cycle time reduction is one of many aspects when
> considering peer review process improvement. But it is just one
> aspect, and probably not even the most important aspect. I would
> emphasize the developmental aspect of peer reviewing, in line of a
> recent AMR editorial: http://amr.aom.org/content/40/1/1.extract
>> 
>> With kind regards,
>> Arto Lanamäki
>> University of Oulu
>> 
>> -----Original Message-----
>> From: AISWorld
> [/mail/src/compose.phpaisworld-bounces at lists.aisnet.org] On Behalf
> Of Paul Ralph
>> Sent: 2. huhtikuuta 2015 2:41
>> To: aisworld at lists.aisnet.org
>> Subject: [AISWorld] Practical suggestions for improving journal
> cycle times
>> 
>> Dear all,
>> 
>> Here are some ways to reduce review cycle times:
>> 
>> 1) Give reviewers only two options: reject or accept with minor
> revisions.
>> 2) Limit revisions to one cycle, i.e., manuscript, revision one,
> galle proofs, published. No revisions two and three.
>> 3) Direct reviewers specifically to evaluate methodology and rigour,
> rather than respond to tone (see
> http://www.paulgraham.com/disagree.html).
>> 4) For a 'minor revision' decision, insist on list of specific
> action items rather than a vague discussion.
>> 5) Limit review periods to one month.
>> 6) Officially suspend dysfunctional reviewers from authorship, i.e.,
> if someone, fails to complete a review or does a terrible job, they
> lose the right to submit papers to that outlet for one year. Of
> course, this has to come with a limit, e.g., the right to refuse
> more than two reviews per year. It also has to be transparent;
> silently blacklisting people contributes to nepotism (see
> recommendation 12).
>> 7) Flatten the editorial hierarchy - one paper doesn't need both an
> SE and an AE. One editor per paper is enough.
>> 8) Limit editors to one month of decision time. Dismiss editors who
> can't make these deadlines and suspend their authorship privileges
> (see recommendation 6).
>> 9) Abandon blind review. Blind review is supposed to free junior
> reviewers to reject the papers of their more powerful peers without
> repercussion.
>> This obviously isn't working. It's protecting bad reviewers from
> well-deserved backlash. Knowing your name is on a review encourages
> you to stick to actionable suggestions rather than name calling and
> quibbling about tone.
>> 10) Stop peer-reviewing position papers. Peer review is a system for
> checking the methodological rigour of empirical research, not for
> analyzing essays. Treating a position paper as a "peer reviewed
> contribution" is absurd. Journals are for empirical science. If you
> want to share an opinion, start a blog.
>> 11) Develop a clear set of desk-reject rules that allows more desk
> rejects.
>> Publish them, let them be challenged and continually evolve them. If
> these policies are regularly updated, they'll save everyone time and
> drive up research standards. For example, we might reject any
> interview-only study based on less than 10 hours of interviews.
>> 12) Make no exceptions. Exceptions will inevitably apply more often
> to more powerful academics, increasing nepotism.
>> 
>> None of these suggestions are particularly novel or inventive.
> Common-sense improvements like these are only resisted because of
> the incorrect belief that anything that simplifies review will
> reduce quality. A simpler, more direct review process will encourage
> everyone to focus on key issues – methodology and results rather
> than framing, positioning and tone – increasing quality.
>> 
>> P.S. Long review cycles are not caused by poor reviewer incentives.
> This is a red herring, designed to divert criticism of the
> extraordinarily inefficient way we review papers, and the
> editors-in-chief who have the authority to improve it but choose not
> to.
>> 
>> —
>> Dr. Paul Ralph
>> Lecturer in Computer Science, University of Auckland
> http://paulralph.name
> _______________________________________________
>> AISWorld mailing list
>> AISWorld at lists.aisnet.org
>> _______________________________________________
>> AISWorld mailing list
>> AISWorld at lists.aisnet.org
> 
> 
>    _______________________________________________
>    AISWorld mailing list
>    AISWorld at lists.aisnet.org
> 
> 
> 
> 
> -- 
> --------------------------------------------------------------------------
> Manuel Mora, EngD.
> Full Professor and Researcher Level C
> ACM Senior Member / SNI Level I
> Department of Information Systems
> Autonomous University of Aguascalientes
> Ave. Universidad 940
> Aguascalientes, AGS
> Mexico, 20131
> --------------------------------------------------------------------------
> Attachments:
> untitled-[1]    5.4 k      [ text/plain ]         Download  |  View
> As Attachment
> 
> Bypass Trash
> 
> Move to:
> [Previous | Next]    [Delete & Prev | Delete & Next]    [Message List]
> 
> 
> 
> _______________________________________________
> AISWorld mailing list
> AISWorld at lists.aisnet.org