[AISWorld] Downside of impact factors: Scientists engaging in 'citation stacking'

Michael Cuellar mcuellar at georgiasouthern.edu
Mon Sep 23 20:26:25 EDT 2013


Hi John,

 In our papers we argue that a citation connotes influence rather than quality or approval. You have to admit that the cold fusion paper was influential, if only negatively. It sucked up the time and attention of the field that arguably could have been better spent on something else. Also, the very fact that you mention it is an indicator of the widespread influence that it has had! So when we look at citations, what we see are what the field considers to be important enough to discuss, not necessarily what it agrees with or thinks is high quality.

And you are correct, influence in the field is only one aspect of the evaluation of a scholar. We have argued for a "wide net" of methods to evaluate the scholar's influence. You might also consider their mentorship of PhD students, their consulting activities, etc. 

We have not argued to apply it to journals. As you suggest, we argue to that it should be applied to scholars. It can be applied to journals and we have a paper on that but as you indicate, journals are artifacts of the paper based publishing system. To create a fair and open evaluation system of scholars, the measurements on which we evaluate should be open, repeatable and minimally subject to corruption. We believe that the h-indices meet these requirements. 

To open another can of worms, Publish or Perish (POP) is based on the Google Scholar database.  GS has been considered to be the better citation database for Information Systems scholars as opposed to that of Scopus or ISI. However, the GS database is rather rough, and therefore quite a lot of massaging is required to the GS data in order to get an accurate number, see our JAIS paper for the process. Therefore POP, which doesn't do that massaging provides a good rough estimation of the h-indices, but to get an accurate number you really need to make sure that you review the data that goes into the calculation.

Regards,

Mike Cuellar

On Sep 23, 2013, at 7:56 PM, John Lamp <john.lamp at deakin.edu.au> wrote:

> As long as you accept that the h-index, and the multiple variants that follow it, are not a silver bullet, fine. A word of caution is that citation does not imply approval. The most cited paper is the cold fusion paper – I would not hold my breath looking for a positive article citing it.
>  
> The next question is why apply it to journals? Why not apply it to articles? The fundamental fact is that articles determine the quality of a journal. Journals do not determine the quality of an article. I’ve attached the SPARC Primer on Article Level Metrics.
>  
> Also, journals are an artefact of the paper based information distribution system. We don’t need them. Peer reviewing and the 15 page gold standard are also artefacts of the paper based information distribution system. I know this will create an additional flurry of defensive responses, but just as armies of clerks had to accept that information technology had transformed their job, so we too must accept it. The defensive responses matter as little as those defending the superiority of the quill and manuscript against movable type. We will keep what is still relevant, and develop new things which take advantage of electronic distribution.
>  
> There is an amazing irony in this that most of you probably only know me from the Index of Information Systems Journals and related work I have been involved with. Ah well, I was a printer once too!
>  
> BTW if you are interested in h-, g-, and other indices and how they can measure impact, check out “Publish or Perish” from Anne-Wil Harzing (just up the road at the University of Melbourne), an amazing piece of software.
> http://www.harzing.com/pop.htm
>  
> Cheers
> John
>  
> From: Michael Cuellar [mailto:mcuellar at georgiasouthern.edu] 
> Sent: Tuesday, 24 September 2013 9:28 AM
> To: John Lamp
> Cc: Ilia Bider; aisworld at lists.aisnet.org
> Subject: Re: [AISWorld] Downside of impact factors: Scientists engaging in 'citation stacking'
>  
> I think we have seen in this post that Impact Factors should not be used for evaluation of scholarly productivity given their volatility and ability to manipulated.
>  
> That is why we have advocated using the Hirsch indices as a measure of the "scholarly influence" of the scholar in the field. Depending on how you define the field, you can use the Hirsch indices to assess impact on the scholarly community, or the practitioner community or both or the social world. 
>  
> The Hirsch indices do this by assessing both productivity (how many papers did you get somebody to publish) and impact (how many people have cited your work). Thus it gets beyond the viewpoint of a few editors and reviewers and lets the field determine what is important. This is what the field does anyway. Most of the most highly cited articles are not to be found in the "top journals" of the field. See Singh, G., K. M. Haddad, et al. (2007). "Are Articles in "Top" management Journals Necessarily of Higher Quality." Journal of Management Inquiry 16(4): 319-331 for the situation in management. They are also less subject to manipulation since once you get beyond the lowest levels, they are very difficult to manipulate by self-citation. They are also able to account for the influence of recent papers, big hit papers, numbers of co-authors, and number of publications.
>  
> See these papers for more discussion of influence and the hirsch indices.
>  
> Truex III, D. P., M. J. Cuellar, et al. (2011). "The Scholarly Influence of Heinz Klein: Ideational and Social Measures of His Impact on IS Research and IS Scholars." European Journal of Information Systems 20(4).
> Truex III, D. P., M. J. Cuellar, et al. (2009). "Assessing Scholarly Influence: Using the Hirsch Indices to Reframe the Discourse." Journal of the Association of Information Systems 10(7): 560-594.
> Takeda, H., D. P. Truex III, et al. (2012). "Evaluating Scholarly Influence Through Social Network Analysis: the Next Step in Evaluating Scholarly Influence." The International Journal of Social and Organizational Dynamics in Information Technology 2(1).
>  
> Best Regards,
>  
> Mike Cuellar
>  
> On Sep 23, 2013, at 6:58 PM, John Lamp <john.lamp at deakin.edu.au> wrote:
> 
> 
> I don’t know whether this attachment is too big (<1Mb) but the Australian Government recently published a discussion paper on the wider definition of research impact. What will come of this following the change of government is moot.
>  
> Essentially it looks beyond internal impact (within the originating academic discipline) to suggest three dimensions of external impact:
> ·         Academic impact (on researchers outside the originating discipline)
> ·         Professional Impact (on practitioners)
> ·         Social impact (broadly defined.
>  
> If the attachment is stripped, email me directly for a copy.
>  
> Cheers
> John
>  
> From: aisworld-bounces at lists.aisnet.org [mailto:aisworld-bounces at lists.aisnet.org] On Behalf Of Ilia Bider
> Sent: Monday, 23 September 2013 5:16 PM
> To: aisworld at lists.aisnet.org
> Subject: Re: [AISWorld] Downside of impact factors: Scientists engaging in 'citation stacking'
>  
> Hossein,
> 
> Sadly enough :-(, no academic measurement, including the Impact Factor, is directed at measuring the impact of research on real life, only its influence on the academic life. Publishing in a  highest ranking journal does not guarantee that a paper will be picked up by the industry, or other real life sector for implementation. Part of the problem here is the academic papers being written in such a style that a "normal" person, most probably, does not understand. What's more, IMHO, the high ranking journals are among those that promote this style in no lesser degree than the less ranking journals. On the positive side, this problem is started to be understood, and, hopefully, the situation will change in the nearest future :-).
> 
> Hope, my message will take away some of your frustration :-).
> 
> Best regards/Ilia
> 
> On 2013-09-23 04:38, Hossein Seif Zadeh wrote:
> Not currently employed as an academic, I have just prepared a fresh jar of coffee and I am embracing for an "interesting" philosophical debate without any tangible, actionable, outcome; but alas here we go again... 
> 
> 
> 
> Just to share one personal experience; I successfully completed a research project a few years ago and I received a letter of commendation (on federal government letterhead) specifically saying my research was of "national importance" and "impact". Later that year when I sought feedback on whether I should include the letter in my upcoming promotion pack, I was told it would contribute "zilch" as it was not peer-reviewed, not a recognized publication, and it was "just one letter by one person"!! Oh, well... Chalked it up as a lesson learnt; original research of national importance has no place in today's academia. Is this really the message we want to convey to junior, aspiring, academics?
> 
> 
> 
> Coffee mug in hand, feet up, ready for the onslaught of emails.... on second thought, I might need a glass of red instead of coffee :-) Should be an interesting 48 hours or or so before the debate winds down.
> 
> 
> 
> Cheers,
> Hossein 
> 
> Sent from my iPhone
> 
> On 23/09/2013, at 3:22 AM, "Kappelman, Leon" <Leon.Kappelman at unt.edu> wrote:
> 
> 
> 
> Some of our institutions have adopted or are considering the use of impact factors as a metric for faculty performance.  Certainly our current faculty performance measures are less than perfect so it seems to be a suggestion worthy of consideration.  But impact factors are not without controversy and apparently prone to manipulation and questionable behaviors.  So before embracing impact factors, consider that at least to some extent they are a measure of what some might call “incestuous citation behaviors.”  Not surprising since most all of us understand that when it comes to human behavior, you get what you measure.  The big question is: Do the positives of using impact factors, or any other metric or combination of measure for that matter, sufficiently outweigh the negatives? 
>  
> Here is some of the “food for thought” that raised my eyebrows enough to send this note:
>  
> http://www.nature.com/news/brazilian-citation-scheme-outed-1.13604
>  
> http://occamstypewriter.org/scurry/2012/08/13/sick-of-impact-factors/
>  
> http://retractionwatch.wordpress.com/2013/08/30/journal-retracts-two-papers-after-being-caught-manipulating-citations/
>  
> http://www.naturalnews.com/042152_citation_stacking_scientific_journals_dishonesty.html
>  
> Best wishes,
> Leon Kappelman
> 
> “Only a virtuous people are capable of freedom.” – Benjamin Franklin
> ----------------------------------------------------------------------------------------------------------------------------------------------
> Leon A. Kappelman, Ph.D. 
>   Professor of Information Systems 
>   Director Emeritus, Information Systems Research Center
>   Fellow, Texas Center for Digital Knowledge
>     College of Business, University of North Texas
>     Voice: 940-565-4698   Email:kapp at unt.edu
> Founding Chair, Society for Information Management's Enterprise Architecture Working Group
> ----------------------------------------------------------------------------------------------------------------------------------------------
> Please consider the environment before printing this e-mail 
>  
>  
> _______________________________________________
> AISWorld mailing list
> AISWorld at lists.aisnet.org
> 
> 
> 
> 
> _______________________________________________
> AISWorld mailing list
> AISWorld at lists.aisnet.org
> 
> 
> 
> 
> -- 
> ===============================================
> Dr. Ilia Bider  
> Process- och systemutvecklingskonsult at ibissoft.se
> Lektor & Forskare at DSV.su.se
> ilia at ibissoft.se        +46 (0)8 164998
> Design science in action ... http://slidesha.re/Uq3RTC
> 
> Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone.
> 
> Deakin University does not warrant that this email and any attachments are error or virus free. <Research Impact Discussion Paper.pdf>_______________________________________________
> AISWorld mailing list
> AISWorld at lists.aisnet.org
>  
> 
> Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone.
> 
> Deakin University does not warrant that this email and any attachments are error or virus free. <sparc-alm-primer.pdf>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.aisnet.org/pipermail/aisworld_lists.aisnet.org/attachments/20130923/13f4e21c/attachment.html>


More information about the AISWorld mailing list