[AISWorld] Fwd: Downside of impact factors: Scientists engaging in 'citation stacking'

Thu Sep 26 21:16:36 EDT 2013

Manual

The solution is to put the creation of knowledge back at the centre of 
our academic practices instead of the creation of "numbers".
This is hard to do because other institutional players are using these 
numbers for their own games.

Dr Robert B. Johnston

Mail: 81 Hunter St. Richmond, VIC, Australia. 3121
Email: Robert.Johnston at ucd.ie <Robert.Johnston at ucd.ie>
Mobile: +61 414 493 255
Skype: johno.home

On 27/09/2013 9:49 AM, mmora at securenym.net wrote:
> Dear colleague Lee Giles,
> Your links to DORA on the bias on IF is excellent !
> In particular, some additional links on DORA suggest
> the bias caused by mutually-reinforcing (it can be
> modeled through system dynamics simulation models),
> where higher IFs --> higher rankings from survey studies
> on quality of journals --> higher submission to these
> journals --> lower acceptance rate due to the huge
> number of submitted papers --> higher fame o journals !
> ---> higher IFs. In particular, my critique is to
> "close our eyes" to the essential roots of the problem:
> acceptance of papers based on two evaluations from peers
> is intrinsically biased, and the implicit rule is rather
> to accept papers that are similarly written as reviewers
> could having done it and with similar socio-economic-cultural and
> political worldviews. It is a hard reality in MIS area.
> On solutions ! maybe to cancel the blind-mode, to publish
> in the accepted papers the comments from reviewers, to
> increase the number of reviewers (commentators) adding
> people from the IT industry, etc. In summary, any sophisticated
> metric is undermined while the acceptance/rejection process
> keeps being biased. "Science is not a value free human activity"
> (from several references).
> Manuel Mora
> * IT Researcher in a developing country
>
> * I estimate that over the 70% of academics enrolled in this list
>    we will never have a paper in top-5 journals, indepedently from
>    the quality of the paper by all aforementioned reported reasons.
>
> * Google metrics could be simple but report facts: number of papers
> published by a researcher, number of citations. That is sufficient
> for a single analysis.
>
>
>
>
>
>
> On Thu, September 26, 2013 6:02 pm, Manuel Mora wrote:
>> ---------- Forwarded message ----------
>> From: Lee Giles <giles at ist.psu.edu>
>> Date: Wed, Sep 25, 2013 at 10:51 AM
>> Subject: Re: [AISWorld] Downside of impact factors: Scientists engaging in
>>   'citation stacking'
>> To: John Lamp <john.lamp at deakin.edu.au>, Michael Cuellar <
>> mcuellar at georgiasouthern.edu>, ISWorld <aisworld at lists.aisnet.org>
>>
>>
>> One may be interested in the San Francisco Declaration on Research
>> Assessment's effort to deal with the impact factor.
>>
>>
>> http://am.ascb.org/dora/
>> ________________________________________
>> From: aisworld-bounces at lists.aisnet.org
>> [aisworld-bounces at lists.aisnet.org]
>> on behalf of John Lamp [john.lamp at deakin.edu.au] Sent: Monday, September
>> 23, 2013 7:56 PM
>> To: Michael Cuellar; ISWorld
>> Subject: Re: [AISWorld] Downside of impact factors: Scientists engaging in
>>   'citation stacking'
>>
>>
>> As long as you accept that the h-index, and the multiple variants that
>> follow it, are not a silver bullet, fine. A word of caution is that
>> citation does not imply approval. The most cited paper is the cold fusion
>>   paper – I would not hold my breath looking for a positive article citing
>> it.
>>
>> The next question is why apply it to journals? Why not apply it to
>> articles? The fundamental fact is that articles determine the quality of a
>>   journal. Journals do not determine the quality of an article. I’ve
>> attached the SPARC Primer on Article Level Metrics.
>>
>> Also, journals are an artefact of the paper based information
>> distribution system. We don’t need them. Peer reviewing and the 15 page
>> gold standard are also artefacts of the paper based information
>> distribution system. I know this will create an additional flurry of
>> defensive responses, but just as armies of clerks had to accept that
>> information technology had transformed their job, so we too must accept
>> it. The defensive responses matter as little as those defending the
>> superiority of the quill and manuscript against movable type. We will keep
>> what is still relevant, and develop new things which take advantage of
>> electronic distribution.
>>
>> There is an amazing irony in this that most of you probably only know me
>> from the Index of Information Systems Journals and related work I have
>> been involved with. Ah well, I was a printer once too!
>>
>> BTW if you are interested in h-, g-, and other indices and how they can
>> measure impact, check out “Publish or Perish” from Anne-Wil Harzing (just
>> up the road at the University of Melbourne), an amazing piece of
>> software. http://www.harzing.com/pop.htm
>>
>>
>> Cheers
>> John
>>
>>
>> From: Michael Cuellar [mailto:mcuellar at georgiasouthern.edu]
>> Sent: Tuesday, 24 September 2013 9:28 AM
>> To: John Lamp
>> Cc: Ilia Bider; aisworld at lists.aisnet.org
>> Subject: Re: [AISWorld] Downside of impact factors: Scientists engaging in
>>   'citation stacking'
>>
>>
>> I think we have seen in this post that Impact Factors should not be used
>> for evaluation of scholarly productivity given their volatility and
>> ability to manipulated.
>>
>> That is why we have advocated using the Hirsch indices as a measure of
>> the "scholarly influence" of the scholar in the field. Depending on how
>> you define the field, you can use the Hirsch indices to assess impact on
>> the scholarly community, or the practitioner community or both or the
>> social world.
>>
>> The Hirsch indices do this by assessing both productivity (how many
>> papers did you get somebody to publish) and impact (how many people have
>> cited your work). Thus it gets beyond the viewpoint of a few editors and
>> reviewers and lets the field determine what is important. This is what
>> the field does anyway. Most of the most highly cited articles are not to
>> be found in the "top journals" of the field. See Singh, G., K. M. Haddad,
>> et al. (2007). "Are Articles in "Top" management Journals Necessarily of
>> Higher Quality." Journal of Management Inquiry 16(4): 319-331 for the
>> situation in management. They are also less subject to manipulation since
>> once you get beyond the lowest levels, they are very difficult to
>> manipulate by self-citation. They are also able to account for the
>> influence of recent papers, big hit papers, numbers of co-authors, and
>> number of publications.
>>
>> See these papers for more discussion of influence and the hirsch indices.
>>
>>
>> Truex III, D. P., M. J. Cuellar, et al. (2011). "The Scholarly Influence
>> of Heinz Klein: Ideational and Social Measures of His Impact on IS
>> Research
>> and IS Scholars." European Journal of Information Systems 20(4). Truex III,
>> D. P., M. J. Cuellar, et al. (2009). "Assessing Scholarly
>> Influence: Using the Hirsch Indices to Reframe the Discourse." Journal of
>> the Association of Information Systems 10(7): 560-594. Takeda, H., D. P.
>> Truex III, et al. (2012). "Evaluating Scholarly Influence
>> Through Social Network Analysis: the Next Step in Evaluating Scholarly
>> Influence." The International Journal of Social and Organizational
>> Dynamics
>> in Information Technology 2(1).
>>
>> Best Regards,
>>
>>
>> Mike Cuellar
>>
>>
>> On Sep 23, 2013, at 6:58 PM, John Lamp <john.lamp at deakin.edu.au<mailto:
>> john.lamp at deakin.edu.au>> wrote:
>>
>>
>> I don’t know whether this attachment is too big (<1Mb) but the Australian
>>   Government recently published a discussion paper on the wider definition
>> of research impact. What will come of this following the change of
>> government is moot.
>>
>> Essentially it looks beyond internal impact (within the originating
>> academic discipline) to suggest three dimensions of external impact: •
>> Academic impact (on researchers outside the originating
>> discipline) •         Professional Impact (on practitioners)
>> •         Social impact (broadly defined.
>>
>>
>> If the attachment is stripped, email me directly for a copy.
>>
>>
>> Cheers
>> John
>>
>>
>> From: aisworld-bounces at lists.aisnet.org<mailto:
>> aisworld-bounces at lists.aisnet.org>
>> [mailto:aisworld-bounces at lists.aisnet.org
>> <mailto:bounces at lists.aisnet.org>] On Behalf Of Ilia Bider
>> Sent: Monday, 23 September 2013 5:16 PM
>> To: aisworld at lists.aisnet.org<mailto:aisworld at lists.aisnet.org>
>> Subject: Re: [AISWorld] Downside of impact factors: Scientists engaging in
>>   'citation stacking'
>>
>>
>> Hossein,
>>
>>
>> Sadly enough :-(, no academic measurement, including the Impact Factor,
>> is directed at measuring the impact of research on real life, only its
>> influence on the academic life. Publishing in a  highest ranking journal
>> does not guarantee that a paper will be picked up by the industry, or
>> other real life sector for implementation. Part of the problem here is the
>>   academic papers being written in such a style that a "normal" person,
>> most probably, does not understand. What's more, IMHO, the high ranking
>> journals are among those that promote this style in no lesser degree than
>> the less ranking journals. On the positive side, this problem is started
>> to be understood, and, hopefully, the situation will change in the nearest
>> future :-).
>>
>>
>> Hope, my message will take away some of your frustration :-).
>>
>>
>> Best regards/Ilia
>>
>>
>> On 2013-09-23 04:38, Hossein Seif Zadeh wrote:
>> Not currently employed as an academic, I have just prepared a fresh jar of
>>   coffee and I am embracing for an "interesting" philosophical debate
>> without any tangible, actionable, outcome; but alas here we go again...
>>
>>
>>
>> Just to share one personal experience; I successfully completed a
>> research project a few years ago and I received a letter of commendation
>> (on federal
>> government letterhead) specifically saying my research was of "national
>> importance" and "impact". Later that year when I sought feedback on
>> whether I should include the letter in my upcoming promotion pack, I was
>> told it would contribute "zilch" as it was not peer-reviewed, not a
>> recognized publication, and it was "just one letter by one person"!! Oh,
>> well... Chalked it up as a lesson learnt; original research of national
>> importance has no place in today's academia. Is this really the message we
>> want to convey to junior, aspiring, academics?
>>
>>
>>
>> Coffee mug in hand, feet up, ready for the onslaught of emails.... on
>> second thought, I might need a glass of red instead of coffee :-) Should
>> be an interesting 48 hours or or so before the debate winds down.
>>
>>
>>
>> Cheers,
>> Hossein
>>
>>
>> Sent from my iPhone
>>
>>
>> On 23/09/2013, at 3:22 AM, "Kappelman, Leon"
>> <Leon.Kappelman at unt.edu<mailto:
>> Leon.Kappelman at unt.edu>> wrote:
>>
>>
>>
>>
>> Some of our institutions have adopted or are considering the use of
>> impact factors as a metric for faculty performance.  Certainly our current
>> faculty performance measures are less than perfect so it seems to be a
>> suggestion worthy of consideration.  But impact factors are not without
>> controversy and apparently prone to manipulation and questionable
>> behaviors.  So before embracing impact factors, consider that at least to
>> some extent they are a measure of what some might call “incestuous
>> citation behaviors.”  Not surprising since most all of us understand that
>> when it comes to human behavior, you get what you measure.  The big
>> question is: Do the positives of using impact factors, or any other metric
>> or combination of measure for that matter, sufficiently outweigh the
>> negatives?
>>
>> Here is some of the “food for thought” that raised my eyebrows enough to
>> send this note:
>>
>>
>> http://www.nature.com/news/brazilian-citation-scheme-outed-1.13604
>>
>>
>>
>>
>> http://occamstypewriter.org/scurry/2012/08/13/sick-of-impact-factors/
>>
>>
>>
>>
>> http://retractionwatch.wordpress.com/2013/08/30/journal-retracts-two-pape
>> rs-after-being-caught-manipulating-citations/
>>
>>
>>
>> http://www.naturalnews.com/042152_citation_stacking_scientific_journals_d
>> ishonesty.html
>>
>> Best wishes,
>> Leon Kappelman
>> Only a virtuous people are capable of freedom.” – Benjamin Franklin
>> --------------------------------------------------------------------------
>> --------------------------------------------------------------------
>> Leon A. Kappelman, Ph.D.
>> Professor of Information Systems
>> Director Emeritus, Information Systems Research Center
>> Fellow, Texas Center for Digital Knowledge
>> College of Business, University of North Texas
>> Voice: 940-565-4698<tel:940-565-4698>   Email:kapp at unt.edu<mailto:
>> kapp at unt.edu> Founding Chair, Society for Information Management's
>> Enterprise
>> Architecture Working Group
>> --------------------------------------------------------------------------
>> --------------------------------------------------------------------
>> Please consider the environment before printing this e-mail
>>
>>
>>
>> _______________________________________________
>> AISWorld mailing list
>> AISWorld at lists.aisnet.org<mailto:AISWorld at lists.aisnet.org>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>>
>>
>> AISWorld mailing list
>>
>>
>> AISWorld at lists.aisnet.org<mailto:AISWorld at lists.aisnet.org>
>>
>>
>>
>>
>>
>>
>> --
>>
>>
>> ===============================================
>>
>>
>> Dr. Ilia Bider
>>
>>
>> Process- och systemutvecklingskonsult at ibissoft.se<http://ibissoft.se>
>>
>>
>> Lektor & Forskare at DSV.su.se<http://DSV.su.se>
>>
>>
>> ilia at ibissoft.se<mailto:ilia at ibissoft.se>        +46 (0)8 164998
>>
>>
>> Design science in action ... http://slidesha.re/Uq3RTC
>>
>>
>> Important Notice: The contents of this email are intended solely for the
>> named addressee and are confidential; any unauthorised use, reproduction
>> or storage of the contents is expressly prohibited. If you have received
>> this email in error, please delete it and any attachments immediately and
>> advise the sender by return email or telephone.
>>
>> Deakin University does not warrant that this email and any attachments
>> are error or virus free. <Research Impact Discussion
>> Paper.pdf>_______________________________________________
>> AISWorld mailing list
>> AISWorld at lists.aisnet.org<mailto:AISWorld at lists.aisnet.org>
>>
>>
>>
>> Important Notice: The contents of this email are intended solely for the
>> named addressee and are confidential; any unauthorised use, reproduction
>> or storage of the contents is expressly prohibited. If you have received
>> this email in error, please delete it and any attachments immediately and
>> advise the sender by return email or telephone.
>>
>> Deakin University does not warrant that this email and any attachments
>> are error or virus free. _______________________________________________
>> AISWorld mailing list
>> AISWorld at lists.aisnet.org
>>
>>
>>
>>
>> --
>> --------------------------------------------------------------------------
>>   Manuel Mora, EngD.
>> Full Professor and Researcher Level C
>> ACM Senior Member / SNI Level I
>> Department of Information Systems
>> Autonomous University of Aguascalientes
>> Ave. Universidad 940
>> Aguascalientes, AGS
>> Mexico, 20131
>> --------------------------------------------------------------------------
>>
>>
>
>
> _______________________________________________
> AISWorld mailing list
> AISWorld at lists.aisnet.org