[AISWorld] On review cycles in our discipline

Wed Apr 8 20:09:28 EDT 2015

Thanks for senior academicians (with more than 30+ years of expertise)
publishing, reviewing and managing high quality journals from your
observations and comments. I would like to add some additional ones
and some inquiries respectfully:

#1 the review process (by being a voluntary, honorary and unfairly not
recognized academic activity) has suffered a performance degradation.

#2 the review process (by being highly variable) produces contrasted
reviews (acceptance and rejection for the same paper from different
reviewers), but the SE relies more on the negative evaluation. I mean,
paper might be conditioned, or SE could evaluate the correctness and
fairness of the review.

#3 SEs should be in a fast overview reading to identify the quality level,
and use only the extra detailed reviews as advises (in about 2-3 months).

Finally, when we review papers from top journals published in the 70's and
80's, it is clear that the relevance of contributions for a specific
context were strongly privileged. At present, the main criteria seems to
be: (i) to use a complex statistical technique, (ii) to collect data from
famous organizations, and (iii) to write with a highly academic style such
that IT practitioners cannot understand it.

Maybe a TASK FORCE from AIS created for addressing this problem and
generating academic recommendations for AIS accredited journals (rather
from ISI Reuters-Thomson company) can be part of the solution.

The permanent critique/inquiry is why in our discipline has happened it?
The seminal paper from Banville and Landry (1989):

Banville, C., & Landry, M. (1989). Can the Field of MIS be Disciplined?.
Communications of the ACM, 32(1), 48-60.

seems to provide reasons for it: we are community conformed with a varied
types of backgrounds and strongly shaped for multiple disciplines. Thus,
the variety of perspectives, styles, tones, values, methods, schools,
approaches produces more disagreements than agreements.

Well, thanks
Prof. Mora
Autonomous University of Aguascalientes
Mexico

On Tue, April 7, 2015 6:08 pm, Carol Saunders wrote:
> The review process in the IS discipline reflects a very complex system.
> That is why we are seeing suggestions in this discussion thread about the
> process, reviewers, reward systems, etc. One aspect of the reviewing
> system that is often not discussed is the role of the Senior Editor (SE).
> Most SEs spend a considerable, if not inordinate, amount of time and
> effort doing a great job. Still, to ensure that they manage a good review
> process, I suggest that our discipline undertake some actions to improve
> the process in regard to their role. I provide a rationale for my
> suggestions and describe them in more detail in an editorial that will be
> published in JAIS in August or September, as part of a set of intriguing
> editorials on the role of SE.
>
> Journals should establish a policy that papers that have made it through
> some number of rounds of review (i.e., two or three) must be published in
> that journal. If the SE does not reject the paper by the stated round,
> then the SE is required to work with the authors to get the paper in a
> form that is acceptable for publication within a few more rounds.  If
> there is that much potential that the SE can hold the authors on the line
> for three or four rounds, then that SE should make it clear what steps
> should be taken to make the article publishable.
>
> Journals should enforce their stated policies.  In particular, the
> journal should determine if it is meeting its time guidelines for
> processing papers under review. Of course, some papers may take longer
> than others to process for understandable reasons. However, the journal
> should try to assess if most papers are being processed within the stated
> desired time intervals and, if not, refine the process or articulate more
> realistic average times for processing papers so the authors can make
> informed decisions about where to send their papers. It would be
> especially helpful if journals could publish their turnaround times in
> their journals, on their websites, and/or on the AIS website.
>
> Journals are in the business of publishing papers, not rejecting them.
> The EICs should establish a culture among the SEs of accepting, not
> rejecting, papers. To strengthen this culture, the names of SEs who have
> accepted papers should be announced, if not publically, at least to the
> other SEs on a regular basis such as at the annual meeting. Moreover, the
> journal should share basic statistics amongst its SEs on how many papers
> each SE has processed, how many were rejected without review, how many
> were rejected after review, and how many eventually got published. Such
> statistics should form the basis for ongoing discussions at the annual
> meetings in which SEs hold each other accountable against the ideal role
> and principles described above. Such continuous discussions will also
> help incoming SEs more quickly form their own appropriate practices in
> their very demanding new roles.
>
> Journals should not place untenured and/or junior faculty in the awkward
> position of being an SE. These individuals have not had the time to
> acquire both an in-depth knowledge of the discipline as well as the
> holistic perspective that is critical for SEs.
>
>
> ________________________________________
> From: AISWorld [aisworld-bounces at lists.aisnet.org] on behalf of Doug Vogel
> [vogel.doug at gmail.com]
> Sent: Tuesday, April 07, 2015 8:47 AM
> To: mmora at securenym.net
> Cc: aisworld at lists.aisnet.org
> Subject: Re: [AISWorld] On review cycles in our discipline
>
>
> I think you also need to include some form of reward system as Kevin
> Crowston has previously noted
>
>
> Sent from my iPhone
>
>
>> On 7 Apr 2015, at 12:28 am, mmora at securenym.net wrote:
>>
>>
>> Colleagues,
>> Several issues have been commented:
>>
>>
>> - lack of acknowledgement in tenure-track careers by reviewing papers
>> - lack of time for conducting 5-10 reviews by year
>> - lack of good reviewers and willing for doing it in reasonable time
>> frames
>>
>> and some facts:
>>
>> - a large cycle review period in our discipline compared with other
>> sciences - an ad-hoc review process with a large variability on styles
>> and tones - a misused or abused blind review mode
>>
>>
>> then, some feasible (?) solutions for improving it have been also
>> posited:
>>
>>
>> - to eliminate the blind review mode
>> - to foster the acknowledgement of the review process
>> - to posit some standards review forms (accredited from AIS) with some
>> exemplary cases
>>
>> We have and live in this problem (junior and mid career Faculty) but
>> solution "is in hands" of senior Faculty, Thanks,
>> Prof. Mora
>> Autonomous University of Aguascalientes
>> Mexico
>>
>>
>>
>>
>>
>>
>> On Sun, Apr 5, 2015 at 9:58 AM, Juhani Iivari <Juhani.Iivari at oulu.fi>
>> wrote:
>>
>>
>> Hi All,
>>
>>
>> Yes, I largely agree with the two comments below. Some time ago I
>> wrote my thoughts about peer reviews, based on my experiences as an
>> author, reviewer and editor. Perhaps not surprisingly, I have not
>> attempted to get it published anywhere, but you can access it form
>> ResearchGate
>> (https://www.researchgate.net/publication/263766605_How_to_improve_the_q
>> uality_of_peer_reviews__Three_suggestions_for_system-level_change), if
>> you are interested in.
>>
>> With best regards,
>>
>>
>> Juhani Iivari
>> Professor emeritus
>> University of Oulu
>>
>>
>>
>>
>>> On 05 Apr 2015, at 16:50, Arto Lanamäki <Arto.Lanamaki at oulu.fi>
>>> wrote:
>>>
>>>
>>> Hi,
>>>
>>>
>>> I think that cycle time reduction is one of many aspects when
>>>
>> considering peer review process improvement. But it is just one aspect,
>> and probably not even the most important aspect. I would emphasize the
>> developmental aspect of peer reviewing, in line of a recent AMR
>> editorial: http://amr.aom.org/content/40/1/1.extract
>>
>>>
>>> With kind regards,
>>> Arto Lanamäki
>>> University of Oulu
>>>
>>>
>>> -----Original Message-----
>>> From: AISWorld
>>>
>> [/mail/src/compose.phpaisworld-bounces at lists.aisnet.org] On Behalf
>> Of Paul Ralph
>>
>>> Sent: 2. huhtikuuta 2015 2:41
>>> To: aisworld at lists.aisnet.org
>>> Subject: [AISWorld] Practical suggestions for improving journal
>>>
>> cycle times
>>>
>>> Dear all,
>>>
>>>
>>> Here are some ways to reduce review cycle times:
>>>
>>>
>>> 1) Give reviewers only two options: reject or accept with minor
>>>
>> revisions.
>>> 2) Limit revisions to one cycle, i.e., manuscript, revision one,
>>>
>> galle proofs, published. No revisions two and three.
>>> 3) Direct reviewers specifically to evaluate methodology and rigour,
>>>
>> rather than respond to tone (see
>> http://www.paulgraham.com/disagree.html).
>>
>>> 4) For a 'minor revision' decision, insist on list of specific
>>>
>> action items rather than a vague discussion.
>>> 5) Limit review periods to one month.
>>> 6) Officially suspend dysfunctional reviewers from authorship, i.e.,
>>>
>> if someone, fails to complete a review or does a terrible job, they lose
>> the right to submit papers to that outlet for one year. Of course, this
>> has to come with a limit, e.g., the right to refuse more than two
>> reviews per year. It also has to be transparent; silently blacklisting
>> people contributes to nepotism (see recommendation 12).
>>> 7) Flatten the editorial hierarchy - one paper doesn't need both an
>>>
>> SE and an AE. One editor per paper is enough.
>>
>>> 8) Limit editors to one month of decision time. Dismiss editors who
>>>
>> can't make these deadlines and suspend their authorship privileges (see
>> recommendation 6).
>>> 9) Abandon blind review. Blind review is supposed to free junior
>>>
>> reviewers to reject the papers of their more powerful peers without
>> repercussion.
>>> This obviously isn't working. It's protecting bad reviewers from
>>>
>> well-deserved backlash. Knowing your name is on a review encourages you
>> to stick to actionable suggestions rather than name calling and
>> quibbling about tone.
>>> 10) Stop peer-reviewing position papers. Peer review is a system for
>>>
>> checking the methodological rigour of empirical research, not for
>> analyzing essays. Treating a position paper as a "peer reviewed
>> contribution" is absurd. Journals are for empirical science. If you want
>> to share an opinion, start a blog.
>>> 11) Develop a clear set of desk-reject rules that allows more desk
>>>
>> rejects.
>>> Publish them, let them be challenged and continually evolve them. If
>>>
>> these policies are regularly updated, they'll save everyone time and
>> drive up research standards. For example, we might reject any
>> interview-only study based on less than 10 hours of interviews.
>>> 12) Make no exceptions. Exceptions will inevitably apply more often
>>>
>> to more powerful academics, increasing nepotism.
>>>
>>> None of these suggestions are particularly novel or inventive.
>>>
>> Common-sense improvements like these are only resisted because of
>> the incorrect belief that anything that simplifies review will reduce
>> quality. A simpler, more direct review process will encourage everyone
>> to focus on key issues – methodology and results rather than framing,
>> positioning and tone – increasing quality.
>>>
>>> P.S. Long review cycles are not caused by poor reviewer incentives.
>>>
>> This is a red herring, designed to divert criticism of the
>> extraordinarily inefficient way we review papers, and the
>> editors-in-chief who have the authority to improve it but choose not to.
>>
>>>
>>>

>>> Dr. Paul Ralph
>>> Lecturer in Computer Science, University of Auckland
>>>
>> http://paulralph.name
>> _______________________________________________
>>
>>> AISWorld mailing list
>>> AISWorld at lists.aisnet.org
>>> _______________________________________________
>>> AISWorld mailing list
>>> AISWorld at lists.aisnet.org
>>>
>>
>>
>> _______________________________________________
>> AISWorld mailing list
>> AISWorld at lists.aisnet.org
>>
>>
>>
>>
>>
>> --
>> ------------------------------------------------------------------------
>> --
>> Manuel Mora, EngD.
>> Full Professor and Researcher Level C
>> ACM Senior Member / SNI Level I
>> Department of Information Systems
>> Autonomous University of Aguascalientes
>> Ave. Universidad 940
>> Aguascalientes, AGS
>> Mexico, 20131
>> ------------------------------------------------------------------------
>> --
>> Attachments:
>> untitled-[1]    5.4 k      [ text/plain ]         Download  |  View As
>> Attachment
>>
>>
>> Bypass Trash
>>
>>
>> Move to:
>> [Previous | Next]    [Delete & Prev | Delete & Next]    [Message List]
>>
>>
>>
>>
>> _______________________________________________
>> AISWorld mailing list
>> AISWorld at lists.aisnet.org
>>
>
> _______________________________________________
> AISWorld mailing list
> AISWorld at lists.aisnet.org
>
>