<div dir="ltr"><p class="MsoNormal">Dear Colleagues,</p><p class="MsoNormal"><br></p>
<p class="MsoNormal">I’ve been following this discussion with great interest. </p>
<p class="MsoNormal"><br></p><p class="MsoNormal">Personally, I wouldn't go as far as calling the current
review process nonsensical. In fact, I would say that, in many ways, it's a good system based on very noble goals. In my limited experience as an author, virtually
all of the reviews that I received made a lot of sense and were quite valuable
in terms of improving and developing my papers. So I’m very thankful to those anonymous
reviewers and editors who worked with me with no extrinsic motivation for doing
so whatsoever (as one of the participants in this discussion thread pointed
out). I had only one “horror story”.
This story involved an editor telling me to go ahead and
comply with the revisions mandated by the reviewers (which made a lot of sense,
by the way). After we revised the paper, the editor rejected our paper due to
one fundamental flaw that was actually a result of us complying with the
suggestions made by reviewers. But, of course, no system is perfect and I view
this experience as an “accident”. </p>
<p class="MsoNormal"><br></p><p class="MsoNormal">At the same time, I also believe that the review process has
one fundamental flaw that will, eventually, lead to its collapse. This flaw is absence
of any free market forces that decide on what constitutes a “good paper” and proper motivation (compensation) mechanisms for editors and reviewers. As
someone who was born in the Soviet Union and who saw the dark side of the
planned economy, I just don’t believe in a system driven by volunteers (whose only
motivation is to instill research quality and ensure prosperity of the IS field)
who, somehow, know what research scholars should produce and how they should
produce it. If history is of any value for predicting the future, then this
system is bound to collapse, just like the planned economy of the Soviet Union. </p>
<p class="MsoNormal"><br></p><p class="MsoNormal">In fact, the current review system reminds me of the quality
control mechanisms that were instilled in the Soviet Union. At some point, the
Soviet government realized that they could no longer ignore the problem of the Soviet
industry producing consumer products of low quality that, which is even
more problematic, Soviet consumers did not really need or want. The fundamental
problem was that factories and workers within those factories were not
motivated to produce high quality products that meet consumer demand. Workers
were compensated largely based on the quantity of products they produce. The
type of products to be produced was, again, mandated “from the top”. The belief
was that the Communist party, somehow, knew very well what the consumers needed and, overall, was very concerned about well-being and prosperity of Soviet
citizens. To address the quality part of the issue, the government introduced
the so-called “quality inspections”. At first, if I’m not mistaken, these
quality inspections were internal. After some time, the government realized
that internal quality inspectors could be easily pressed by the factory
directors to approve the factory products. Eventually, the government
introduced external inspectors. But again, there was a misalignment of interest
(or lack of proper motivation). Those external inspectors were compensated based
on how much flaws they can find in a factory product. And the fundamental
problem remained – nobody knew what consumers really wanted. I believe this was
happening in the 1980s – at the dusk of the Communist era. </p>
<p class="MsoNormal"><br></p><p class="MsoNormal">With this historical perspective in mind, I predict that it
will take nothing short of a collapse for the system to change. I think we all
understand the flaws of the current review system. Unfortunately, so much has
been invested into the system (which is, I would like to stress, is by no means
nonsensical and is based on very noble ideas, just like the Soviet planned
economy) that only a complete collapse will lead to emergence of a new system
for determining what constitutes a “good paper”. </p>
<p class="MsoNormal"><br></p><p class="MsoNormal">Sincerely,<br>
Vlad Krotov</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Oct 17, 2013 at 7:27 PM, <span dir="ltr"><<a href="mailto:mmora@securenym.net" target="_blank">mmora@securenym.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dear colleagues in AIS:<br>
<br>
Motivated by the recent debate on the quality of journals and the<br>
influence of ISI listed ones, I kindly raised the logical previous<br>
step regarding the quality of reviews. Many interesting comments<br>
have been shared by other colleagues. In particular the recommendations<br>
from A.S.Lee (a past EiC of MIS Quarterly) on 15 key points that<br>
should be addressed by anyone who self-named an ETHICAL REVIEWER<br>
(posted by John Lamp) are excellent. Link is:<br>
<a href="http://www.people.vcu.edu/~aslee/referee.htm" target="_blank">http://www.people.vcu.edu/~aslee/referee.htm</a><br>
<br>
On it, while the problematic of the existence of pseudo-scientific<br>
journals with fee rates for publishing is real, I believe that it is not<br>
our main<br>
concern. I mean, we must be worried about well-trained PhD people<br>
that conducting a rather emotional than rational review, and that<br>
by their novelty in research topics, their reviews are more flawed<br>
that the paper on review. It regrettably happens with just graduated<br>
PhDs (from any Institution to be honest) given by a plausible academic<br>
pressure for being "hard". A PhD training is just a training to know<br>
1 or 2 research methods (from the available 25+) and an expertise<br>
of 1 journal and 2-3 conference papers.<br>
<br>
In summary, the ethic concern that we should formalize/legitimatize<br>
are the reviewers' requirements (e.g. the expected years of experience<br>
(after the PhD completion), the self-evaluation of the topics to be<br>
reviewed, and the willingness to follows -at least- some of the excellent<br>
recommendations provided by A.S. Lee and reminded to us by John Lamp).<br>
<br>
The anecdotal real history on core contributions by Schneirderman,<br>
and Zadeh, on serius journals suggest a real problem. Furthermore,<br>
the awarded 1978 Economy Nobel Prof. Herbert A. Simon reported in<br>
his biographic book, that one of their first papers on statistics<br>
methods was wrongly rejected at first (he gained his PhD in Political<br>
Sciences) but fortunately this mistake was corrected.<br>
<div class="im"><br>
<br>
<br>
On Thu, October 17, 2013 9:37 am, Galletta, Dennis wrote:<br>
> As a journal editor, these stories are always frightening. But to provide<br>
> a little levity, and a few smiles, you MUST read the following blog entry<br>
</div>> so you can see the hilarious clues about the paper’s authenticity, or<br>
<div><div class="h5">> lack thereof:<br>
> <a href="http://retractionwatch.wordpress.com/2013/09/23/a-serbian-sokal-authors-s" target="_blank">http://retractionwatch.wordpress.com/2013/09/23/a-serbian-sokal-authors-s</a><br>
> poof-pub-with-ron-jeremy-and-michael-jackson-references/<br>
><br>
> It will lead you to try and read a little of the paper, only to recoil in<br>
> pain and agony.<br>
><br>
> In that otherwise hilarious blog, some were questioning the actual<br>
> existence of the journal. I am unable to find the journal in the EBSCO<br>
> system under Engineering, so the jury still seems to be out given some of<br>
> their links are to EBSCO that purport to defend its existence. This<br>
> recalls a famous similar hoax in 1996 by Sokal that has been covered<br>
> extensively elsewhere (outlined nicely in Wikipedia at<br>
> <a href="http://en.wikipedia.org/wiki/Sokal_affair" target="_blank">http://en.wikipedia.org/wiki/Sokal_affair</a>).<br>
><br>
><br>
> I believe the editorial process will stand up to these jokes. The<br>
> marginal journals are the most vulnerable. But in any event, reviewers<br>
> and editors obviously should read the papers they agree to review.<br>
><br>
> DG<br>
><br>
><br>
> -------------------------------------------------------------------------<br>
> ---<br>
> Dennis F. Galletta Professor of Business<br>
> Administration<br>
> University of Pittsburgh and Director, Katz Doctoral<br>
> Program<br>
> 282a Mervis Hall Katz Graduate School of<br>
> Business<br>
> Phone <a href="tel:%2B1%20412-648-1699" value="+14126481699">+1 412-648-1699</a> Pittsburgh, PA<br>
> 15260<br>
> E-mail: galletta @ Fax +1<br>
> <a href="tel:412-648-1693" value="+14126481693">412-648-1693</a><br>
> <a href="http://katz.pitt.edu" target="_blank">katz.pitt.edu</a> homepage:<br>
</div></div>> <a href="http://www.pitt.edu/~galletta" target="_blank">www.pitt.edu/~galletta</a><<a href="http://www.pitt.edu/~galletta" target="_blank">http://www.pitt.edu/~galletta</a>><br>
> --------------------------------------------------------------------------<br>
<div class="im">> --<br>
><br>
><br>
> From: <a href="mailto:aisworld-bounces@lists.aisnet.org">aisworld-bounces@lists.aisnet.org</a><br>
> [mailto:<a href="mailto:aisworld-bounces@lists.aisnet.org">aisworld-bounces@lists.aisnet.org</a>] On Behalf Of Ejub Kajan<br>
> Sent: Thursday, October 17, 2013 10:03 AM<br>
> To: <a href="mailto:aisworld@lists.aisnet.org">aisworld@lists.aisnet.org</a><br>
> Cc: <a href="mailto:ekajan@ieee.org">ekajan@ieee.org</a><br>
> Subject: [AISWorld] Fwd: Re: On quality of journals and quality of reviews<br>
> !<br>
><br>
><br>
> Wake up people,<br>
><br>
><br>
><br>
><br>
> ------- Forwarded message -------<br>
</div><div class="im">> From: "Ejub Kajan" <<a href="mailto:ejubkajan@sbb.rs">ejubkajan@sbb.rs</a><mailto:<a href="mailto:ejubkajan@sbb.rs">ejubkajan@sbb.rs</a>>><br>
> To: <a href="mailto:aisworld@lists.aisnet.org">aisworld@lists.aisnet.org</a><mailto:<a href="mailto:aisworld@lists.aisnet.org">aisworld@lists.aisnet.org</a>><br>
> Cc: <a href="mailto:ekajan@ieee.org">ekajan@ieee.org</a><mailto:<a href="mailto:ekajan@ieee.org">ekajan@ieee.org</a>><br>
> Subject: Re: [AISWorld] On quality of journals and quality of reviews !<br>
> Date: Thu, 17 Oct 2013 15:27:25 +0200<br>
> My dear friends and colleges,<br>
><br>
><br>
> Attached is an article that addressed several important issues on papers<br>
> reviews, where the misserable of review process *or not at all) has<br>
> occurred . Authors have proved , with a lot of carriage (the paper has<br>
> made intentionally), that everything is possible if there no control over<br>
> it, that so-called scientific people may reach full professor positions<br>
> due that administration.<br>
><br>
> I would like to pay attention to reference section where Walt Disney<br>
> actor has been cited, Laplas has given his oppinion on the Web and many<br>
> other stpitides,<br>
><br>
> Actually, there is no reviews, authors paid for paper appearance,<br>
> conditions applied are cited articles in that journal and paid fee. WHAT<br>
> ISI (THOMSON REUTERS) yet wait to send this journal into history.<br>
><br>
><br>
> I think that authors of this paper should be reworded by many thanks by<br>
> scientific community.<br>
><br>
> Thanks,<br>
> Ejub<br>
><br>
><br>
><br>
> On Thu, 17 Oct 2013 00:25:57 +0200, John Lamp<br>
</div><div class="im">> <<a href="mailto:john.lamp@deakin.edu.au">john.lamp@deakin.edu.au</a><mailto:<a href="mailto:john.lamp@deakin.edu.au">john.lamp@deakin.edu.au</a>>> wrote:<br>
><br>
><br>
> When dealing with new reviewers, you could not really do better than to<br>
</div>> direct them to Allen Lee’s note on reviewing a manuscript for<br>
<div class="im">> publication. It was originally published in the Journal of Operations<br>
> Management, but is online at:<br>
> <a href="http://www.people.vcu.edu/~aslee/referee.htm" target="_blank">http://www.people.vcu.edu/~aslee/referee.htm</a><br>
><br>
><br>
> Fifteen simple (well maybe) criteria to address to create a decent<br>
> review. I still refresh my mind from it when doing a review.<br>
><br>
> Cheers<br>
> John<br>
><br>
><br>
> From:<br>
> <a href="mailto:aisworld-bounces@lists.aisnet.org">aisworld-bounces@lists.aisnet.org</a><mailto:<a href="mailto:aisworld-bounces@lists.aisnet.or">aisworld-bounces@lists.aisnet.or</a><br>
</div>> g> [mailto:<a href="mailto:aisworld-bounces@lists.aisnet.org">aisworld-bounces@lists.aisnet.org</a>] On Behalf Of William Tastle<br>
<div class="im">> Sent: Thursday, 17 October 2013 12:43 AM<br>
</div>> To: <a href="mailto:aisworld@lists.aisnet.org">aisworld@lists.aisnet.org</a><mailto:<a href="mailto:aisworld@lists.aisnet.org">aisworld@lists.aisnet.org</a>><br>
<div class="im">> Subject: Re: [AISWorld] On quality of journals and quality of reviews !<br>
><br>
><br>
> I, too, am a former journal editor and the difficulties in securing<br>
> proper reviewers is sometimes a bit of a challenge. However, I consider<br>
> it a responsibility of scholars to engage in a mentoring function with<br>
> respect to paper reviewing. In general, the best reviews are from the<br>
> senior researchers and hence, they are thepeople most bombarded with<br>
> review requests. Those reviews are nurturing and guiding as they attempt<br>
> to raise the quality of content (and writing style) such that the paper<br>
> crosses the threshold of acceptability. The worst reviews, again<br>
> speaking very generally and without benefit of a scientific effort to<br>
> confirm my experiential observations, are from the younger, or perhaps<br>
</div>> more insecure, scholars who seemingly engage in a “slap leatherâ€<br>
<div class="im">> attitude towards the review process. Their reviews are more apt (again,<br>
> a generalization) to be vicious in content and of little use to the paper<br>
> author.<br>
><br>
> I recall a story from Ben Schneiderman about how his seminal paper on<br>
> structure diagrams was severely trashed by some reviewers until it was<br>
> finally published, only to become one of the most downloaded papers in<br>
</div>> the journal’s history. I heard a similar story, again first hand, from<br>
<div class="im">> Lotfi Zadeh on his theory of fuzzy sets. They persevered to success. We<br>
> have a responsibility to mentor our colleagues, just as we have<br>
> (hopefully) been mentored ourselves.<br>
><br>
><br>
> However, this does not excuse a poorly written or researched paper. That<br>
> is sloppy work and papers of that sort should simply be returned to the<br>
> authors without any review until the basic threshold of scientific<br>
> literacy is reached.<br>
><br>
> Bill Tastle, Professor of IS, Ithaca College<br>
><br>
><br>
> From:<br>
> <a href="mailto:aisworld-bounces@lists.aisnet.org">aisworld-bounces@lists.aisnet.org</a><mailto:<a href="mailto:aisworld-bounces@lists.aisnet.or">aisworld-bounces@lists.aisnet.or</a><br>
</div>> g> [mailto:<a href="mailto:aisworld-bounces@lists.aisnet.org">aisworld-bounces@lists.aisnet.org</a>] On Behalf Of<br>
> <a href="mailto:MurphJen@aol.com">MurphJen@aol.com</a><mailto:<a href="mailto:MurphJen@aol.com">MurphJen@aol.com</a>><br>
<div class="im">> Sent: Tuesday, October 15, 2013 8:42 PM<br>
</div>> To: <a href="mailto:mmora@securenym.net">mmora@securenym.net</a><mailto:<a href="mailto:mmora@securenym.net">mmora@securenym.net</a>>;<br>
> <a href="mailto:aisworld@lists.aisnet.org">aisworld@lists.aisnet.org</a><mailto:<a href="mailto:aisworld@lists.aisnet.org">aisworld@lists.aisnet.org</a>><br>
<div><div class="h5">> Subject: Re: [AISWorld] On quality of journals and quality of reviews !<br>
><br>
><br>
> I'm not sure what the point of the below is but I'll add my perception as<br>
> a journal editor in chief. Quite frankly authors are lucky to get<br>
> reviews. I have seen a flood of submissions the last 2 years from all<br>
> parts of the world where before I saw mostly US and European submissions.<br>
> I think this is good but, there are a lot of junior level papers and not<br>
> enough senior level reviewers to go around. The trend I do not like is<br>
> that of authors feeling it is their right to get their reviews quick.<br>
> I'm not sure if this is an output of the open source journals that have<br>
> promised fast turnarounds, but to get quality reviews takes time as there<br>
> simply aren't enough quality reviewers who have the time to be almost<br>
> full time reviewers. My gripe (and I admit it is as an EiC) is that it<br>
> seems everyone wants/has to author papers and few have the desire/time to<br>
> review. Also, unfortunately, new authors don't seem to understand the<br>
> concept of a thorough literature review and seem to have taken the debate<br>
> on the ethics of recommending papers from the journal being submitted to,<br>
> or of previous papers from senior reviewers as license to ignore them.<br>
><br>
> This leads to my major question: Why should a journal spend limited<br>
> reviewing resources on doing thorough reviews of papers that do not meet<br>
> the basic standard of scholarly research by grounding themselves in the<br>
> literature.<br>
><br>
> I am tired of the excuse that the authors do not have access to the<br>
> articles so they ignore them. I see so much research that authors<br>
> consider new but is at best a minor extension of something that has been<br>
> published but the authors did not or could not get the article to know<br>
> it.<br>
><br>
> I think we are at a crossroads of scholarly literature. We argue about<br>
> plagiarism but I think we are seeing a greater issue of authors not<br>
> looking at the literature so that they can make the claim "there is<br>
> little research" or that "this is new" as a justification for their<br>
> paper. To give credit we have to recognize the research that has been<br>
> done, to build a body of knowledge we have to build on what has been<br>
> done, not do it over and over again.<br>
><br>
> If authors want quality reviews they need to show the respect to the<br>
> senior scholars by at least looking at their work and building on it<br>
> rather than trying to waste their time by making them tell them what<br>
> literature to look at rather than addressing the quality of the paper.<br>
><br>
> Also, I propose that we establish a new ethic: authors can only submit if<br>
> they serve as reviewers. I am also tired of hearing the cries of needing<br>
> to get published to get tenure or promotion so they don't have time to<br>
> review. We need all universities to start recognizing the intellectual<br>
> contribution of reviewing as being on equal or near equal par with<br>
> authoring. Do this and we will all have quality reviews.<br>
><br>
> sorry to rant but you can see this has touched a nerve that has been<br>
> throbbing for a while.<br>
><br>
> Murray E. Jennex, Professor of MIS, San Diego State University<br>
><br>
><br>
> In a message dated 10/15/2013 5:05:48 P.M. Pacific Daylight Time,<br>
</div></div><div class="im">> <a href="mailto:mmora@securenym.net">mmora@securenym.net</a><mailto:<a href="mailto:mmora@securenym.net">mmora@securenym.net</a>> writes:<br>
> Dear colleagues,<br>
> In past weeks, a strong and relevant debate on the academic influence<br>
> of ISI listed journals was discussed in this forum. Well, on same ideas I<br>
> wonder whether there are studies on the quality of the reviewers (e.g.<br>
> level of seniority, level of expertise in the topic, level of expertise on<br>
> research methods, and in particular on the role played as peers seeking to<br>
> improve the advance of science and suggesting clear insights rather some<br>
> simple elaborations of flaws without any rational justification). In<br>
> summary, are we living in a spiral of hard reviewers and bad researchers?<br>
> or rather the opposite one is the reality? I am sure that Senior<br>
> researchers will have reviewed rare reviews from people with less<br>
> expertise and seniority level, so comments on it are welcome ! Of course,<br>
> this non ethical practice should be eliminated. As a funny real history, a<br>
> Mexican top researcher<br>
> in Education received a strong critique on the null value of paper that he<br>
> wrote, and the suggestion was to use some papers writen by him in the past<br>
> (of course, the reviewers did know it). I know of other cases similar<br>
> cases. Well, sciences is about truthness but wrong reviews maybe are now a<br>
> real headache ! Cheers ! Manuel Mora<br>
> ACM Senior Member<br>
><br>
><br>
><br>
> _______________________________________________<br>
> AISWorld mailing list<br>
</div>> <a href="mailto:AISWorld@lists.aisnet.org">AISWorld@lists.aisnet.org</a><mailto:<a href="mailto:AISWorld@lists.aisnet.org">AISWorld@lists.aisnet.org</a>><br>
<div class="HOEnZb"><div class="h5">><br>
><br>
> Important Notice: The contents of this email are intended solely for the<br>
> named addressee and are confidential; any unauthorised use, reproduction<br>
> or storage of the contents is expressly prohibited. If you have received<br>
> this email in error, please delete it and any attachments immediately and<br>
> advise the sender by return email or telephone.<br>
><br>
> Deakin University does not warrant that this email and any attachments<br>
> are error or virus free.<br>
><br>
><br>
> --<br>
> Ejub Kajan<br>
> State University of Novi Pazar<br>
> Editor of Electronic Business Interoperability: Concepts, Opportunities<br>
> and Challenges Editor of Handbook of Research on E-Business Standards and<br>
> Protocols<br>
><br>
><br>
> Using Opera's revolutionary email client: <a href="http://www.opera.com/mail/" target="_blank">http://www.opera.com/mail/</a><br>
><br>
><br>
><br>
> --<br>
> Ejub Kajan<br>
> State University of Novi Pazar<br>
> Editor of Electronic Business Interoperability: Concepts, Opportunities<br>
> and Challenges Editor of Handbook of Research on E-Business Standards and<br>
> Protocols<br>
><br>
><br>
> Using Opera's revolutionary email client: <a href="http://www.opera.com/mail/" target="_blank">http://www.opera.com/mail/</a><br>
> _______________________________________________<br>
> AISWorld mailing list<br>
> <a href="mailto:AISWorld@lists.aisnet.org">AISWorld@lists.aisnet.org</a><br>
<br>
<br>
<br>
_______________________________________________<br>
AISWorld mailing list<br>
<a href="mailto:AISWorld@lists.aisnet.org">AISWorld@lists.aisnet.org</a><br>
</div></div></blockquote></div><br></div>