[AISWorld] Sound conference desk rejection policy

Mon Feb 27 16:15:10 EST 2017

Hi Murray,

I personally value other inputs and the article was fascinating reading.  Please do not take the comments as a rejection of the idea.  I seek elaboration, just as I have sought elaboration from others.

Apologies for taking so long to reply.  I had to read the article first.

You make a number of high level suggestions for addressing the review problem in the article.  I was just wondering if anyone attempted them and if so, how successful were those attempts?

For those who haven't read the article, the suggestions are:

-"Require all authors who submit to journals to participate in reviewing."  I will change this to include conferences as my original question focused on those specifically.  Can we ensure the competence of the authors as reviewers?  I have read articles, where it was pretty clear authors tried to write without any understanding of the existing literature and had no methodological training.  Also, what's to prevent authors from saying no?  Do we blacklist and bar such authors from future submissions until they have done their duty?

Perhaps we combine this with suggestion 3- only if authors pass the screening do we set up an author chargesheet?

-"Reduce the number of reviews by linking conference and journal reviewing."  Will this really reduce work?  Conference papers are often short- 10 pages or less.  Journal papers are long- 40 to 50 pages at times.

I would note someone else (who emailed me personally and so I will not reveal who) suggested that we have a tiered review system.  One submits a paper to a basket of journals, and the reviewers decide not whether to accept or reject, but which journal takes the paper.  This is a similar idea.

-"Reduce the time for reviews by having authors prepare better paper submissions, and focus reviews on the results, not the writing, the literature review, or the methodology."  I could not differentiate this suggestion from a desk rejection.  What is the difference?

-"Support faculty in attending conferences if they review for them and not just because they have a paper in them."  How do we incentivize universities and research institutions to do this?

-"Analyze faculty’s tasks and allocate time for reviewing."  How do we incentivize universities and research institutions to do this?  My experience is universities make this vague claim that we are supposed to have 40-40-20% research-teaching-service time, but no one ever evaluates this.  In reality it becomes 60-50-30% (yes greater than 100%).

I would emphasize that I think all of your ideas are good.  The issue is making them a reality.

Cecil Chua

________________________________
From: MurphJen at aol.com <MurphJen at aol.com>
Sent: Monday, February 27, 2017 7:34 PM
To: Cecil Eng Huang Chua; aisworld at lists.aisnet.org
Cc: murphjen at aol.com
Subject: Re: [AISWorld] Sound conference desk rejection policy

We actually had a discussion on the review process in 2016 including publications in CAIS. The problems with reviews are recognized and many proposals have been made to improve the process, however, reducing conference reviews and reviewing quality is not one of them, including conference desk rejection.  I won't say that conference desk rejection should never be done, but I do suggest it be done rarely, perhaps only in those cases where the submission just does not fit the conference/track/minitrack theme.  Otherwise I can tell you that my suggestion in CAIS is that we put more emphasis on the conference reviews and use the conference reviews to shorten journal review cycles and work load.  We are starting to see journals becoming more accepting of taking articles straight from the conference with little or no modification, this can only be done if we keep the conference review process rigorous.  I know this is not what you want to hear but I really think we have to be judicious with conference desk rejects.  In addition to not fitting the conference perhaps the only other times a desk reject is warranted is for a submission that is not complete, does not meet the conference submission guidelines, or is so bad that anyone looking at it would agree it is not worth sending out for review (a hard thing to quantify though and of course if this is done the conference should publish that desk rejection is possible in the cfp).  My CAIS article is:

Jennex, M.E., (2016) "No Free Lunch: Suggestions for Improving the Quality of the Review Process," Communications of the Association for Information Systems: Vol. 38, Article 17. Available at: http://aisel.aisnet.org/cais/vol38/iss1/17

Note that I am editor in chief of the International Journal of Knowledge Management and co-editor in chief of the International Journal of Information Systems for Crisis Response and Management so of course I do have my biases on the review process (just saying this for full disclosure).  Given the growth in the number of conferences and the pressures of publishing I recognize that there is a big growth in the number of submissions so the below issue of people not being willing to review is one has to be addressed.  I personally believe that you should do two or three reviews for every submission you make and that reviewing should be considered as much of a intellectual contribution as writing.  So basically I do not believe any member of our community who is publishing should refuse to review (and conferences should also publish any expectation of authors reviewing in the cfp).  Just my 2 cents worth....murray jennex

Professor MIS, Fowler College of Business, San Diego State University

In a message dated 2/26/2017 5:48:54 P.M. Pacific Standard Time, aeh.chua at auckland.ac.nz writes:
What's interesting about this post is that I have received several personal emails rather than posts to ISWorld.  As some people have explicitly asked me not to repost their mails, I won't do so.

The correspondence I have received has been of the following forms:

(1) Interesting question. Complex issue.

(2) War stories of how some editor had to ask many people before someone would finally review for them.

(3) Comments about how some people write a lot of papers, but won't review.

(4) I used to oppose desk rejections until the day I became an editor.

(5) The issue of trust- if you appoint editors you need to trust them to desk reject appropriately.

(6) Recommendation that journals adopt fee-based submission policies and actually pay reviewers.

(7) Requests for an update on the conversation.

I do think we should bring the issue out in the open and discuss it.  It affects all of us, and a public discussion would better inform policy.

Cecil Chua

On Fri, Feb 24, 2017 at 1:37 PM -0500, "Cecil Eng Huang Chua" <aeh.chua at auckland.ac.nz<mailto:aeh.chua at auckland.ac.nz>> wrote:

I would like to start a discussion on desk rejections at conferences.  I want to start this, because I am editing/have edited for two conferences now where the editor instructions were “no desk rejects.”  A desk rejection occurs when either the track chair or associate editor rejects a paper without sending it to reviewers.

I suspect a proper policy is not “no desk rejects.”  It isn’t “give editor all the power to desk reject they want” either.  I am hoping that we will make some of the issues transparent so we can develop good policy.

An argument in favor of desk rejection is the total uncompensated manpower required for conference reviewing.  The typical conference structure is track chair, associate editor, and two reviewers.  So, each fully reviewed paper receives 4 man-review units of effort.  If there are 50 papers per track (yes, some tracks have hundreds of papers- bear with me), that’s 200 man-review units in each track.  If there are 15 tracks, that’s 3000 man-review units.  At just 5 major conferences  (ICIS, AMCIS, ECIS, PACIS, ACIS), that’s 15000 man-review units, and we have more than just 5 major conferences.  Desk rejections can shave a lot of uncompensated man-review units from this.

A counter argument is that one can get authors of a track to review for the track.  I would note that if the editor feels a paper is of desk rejection quality, that the authors may not be competent to review.

An argument against desk rejection is conferences are about providing feedback to authors.  However, this requires uncompensated time from reviewers.  In many of our premier conferences, there’s a pre-submission game where the track chairs try to “reserve” as many reviewers/AEs as possible.  We wouldn’t have that game if there wasn’t a shortage of reliable reviewers.  Indeed, there probably aren’t more that 5000 active researchers in the IS field at any one time- see above 15000 man-review units.  Everyone gets involved in the reviews.  It is unfair to favor authors who benefit from reviews rather than reviewers who are harmed (by having to spend uncompensated time) when they have to do reviews.

Frankly, our community seems to underappreciate reviewers.  For the above conservative 15000 man-review units, we maybe give 5 best reviewer awards, which are often paper certificates with no money attached.  Best paper authors receive shiny plaques and often a cheque.  We could argue reviewer competence is recognized because reviewers are invited to become track chairs and editors.  To this, I would note the pyramidal structure of conference/journal organization-many reviewers, few board positions and the fact that appointment to boards is not explicitly based on reviewing competence and is often based on getting papers accepted.  Also, it is not clear to me that being appointed to boards is necessarily a reward.

So, that’s my discussion seed.  I’d like to hear your thoughts.

Cecil Chua
_______________________________________________
AISWorld mailing list
AISWorld at lists.aisnet.org

_______________________________________________
AISWorld mailing list
AISWorld at lists.aisnet.org