[AISWorld] Downside of impact factors: Scientists engaging in 'citation stacking'

Sun Sep 22 19:36:26 EDT 2013

Where do I start?

JIF and most other quantitative measures of quality have been shown to have major problems of external influence, the results being largely irreproducible, and methodology. Despite that the bean counters love it!

This is essential reading.
Seglen, P. O. (1997). "Why the impact factor of journals should not be used for evaluating research." British Medical Journal 314 (Feb 15): 498-502.

The JIF is an inverted extension of Bradford's Law
Bradford S C (1934) "Sources of information on specific subjects" Engineering: An Illustrated Weekly Journal Arthur Ernest Maw, London 137(3550) [Reprinted (1985) J Information Science 10,176-180]
which has been shown to be a Gini distribution anyhow.
Burrell, Q. L. (1992). "The Gini Index and the Leimkuhler Curve for Bibliometric Processes." Information Processing & Management 28(1): 19-33.

You can get a summary of that and its development in
Lamp, J. W., S. K. Milton, L. Dawson and J. Fisher (2007) "RQF Publication Quality Measures: Methodological Issues", Proceedings of the Eighteenth Australasian Conference on Information Systems, University of Southern Queensland, 478-486
http://aisel.aisnet.org/cgi/viewcontent.cgi?article=1012&context=acis2007

Do also have a look at http://lamp.infosys.deakin.edu.au/journals/?page=rankanalysis

Specifically IS related sample:
Ferratt, T.W., Gorman, M.F., Kanet, J.J. & Salisbury, W.D. (2007) "IS Journal Quality Assessment Using the Author Affiliation Index "Communications of the Association for Information Systems, 19(34).
Katerattanakul, P., B. Han, et al. (2003). "Objective quality ranking of computing journals." Communications of the ACM 46(10): 111-114.
Katerattanakul, P., M. Razi, et al. (2003). "IS journal rankings versus citation analysis: consistency and concerns". Ninth Americas Conference on Information Systems, Tampa, Florida.
Knight, S.A. (2009)."Journal Self-Citation XVI: Academic Citations - A Question of Ethics?," Communications of the Association for Information Systems 25(1), 130-139.
Lamp, J. W. (2006) "Recognition as a distinguishing characteristic of IS journals" Australasian Journal of Information Systems, 13(2), 7-16 [Click for PDF file] <http://lamp.infosys.deakin.edu.au/pubs/06_ajis_recognition.pdf>
Lamp, J. W. (2007) "Perceptions of Gender Balance of IS Journal Editorial Positions" Communications of the AIS, 20, 124-133 [Click for PDF file] <http://lamp.infosys.deakin.edu.au/pubs/getgb.php>
Lamp, J. W. (2009). "Journal Ranking and the Dreams of Academics." Online Information Review 33(4): 827-30.
Mingers, J, Macri, F & Petrovici, D (2012), "Using the h-index to measure the quality of journals in the field of business and management", Information Processing & Management, 48(2), 234-41.
Mylonopoulos, N. A. and V. Theoharakis (2001). "Global Perceptions of IS Journals: Where is the Best IS Research Published?" Communications of the ACM 44(9): 29-33.
Peffers, K. and Y. Tang (2003). "Identifying and evaluating the universe of outlets for information systems research: Ranking the journals." J Information Technology Theory and Application 5(1): 63-84.
Rainer, R. K. and M. D. Miller (2005). "Examining Differences Across Journal Rankings." Comm of the ACM 48(2): 91-94.

I also have these:
Althouse, B. M., et al. (2008). "Differences in impact factor across fields and over time." Journal of the American Society for Information Science and Technology 60(1): 27-34.
Bjork, B.-C. and D. Solomon (2012). "Open access versus subscription journals: a comparison of scientific impact." BMC Medicine 10(1): 73.
Drew, D. E. (2009). "Journal Self-Citation XXII: On the Journal Impact Factor-A Historical Perspective." Communications of the Association for Information Systems 25(1): 22.
Eugene, G. (2005). The Agony and the Ecstasy-The History and Meaning of the Journal Impact Factor. International Congress on Peer Review And Biomedical Publication.
Garfield, E. (1996). "How can impact factors be improved?" BMJ: British Medical Journal 313(7054): 411.
Huang, M.-H. and W.-Y. C. Lin (2012). "The Influence of Journal Self-citations on Journal Impact Factor and Immediacy Index." Online Information Review 36(5).
Jacsó, P. (2012). "Using Google Scholar for journal impact factors and the h-index in nationwide publishing assessments in academia - siren songs and air-raid sirens." Online Information Review 36(3): 462 - 478.
Jarwal, S. D., et al. (2009). "Measuring research quality using the journal impact factor, citations and 'Ranked Journals': blunt instruments or inspired metrics?" Journal of Higher Education Policy and Management 31(4): 289 - 300.
Lozano, G. A., et al. (2012). "The weakening relationship between the Impact Factor and papers' citations in the digital age." arXiv preprint arXiv:1205.4328.
Seglen, P. O. (1999). "Causal relationship between article citedness and journal impact." Journal of the American Society for Information Science 45(1): 1-11.
Solomon, D. J. and B.-C. Björk (2012). "A study of open access journals using article processing charges." Journal of the American Society for Information Science and Technology 63(8): 1485-1495.

Excuse the mixed reference format! :)

Cheers
John

From: aisworld-bounces at lists.aisnet.org [mailto:aisworld-bounces at lists.aisnet.org] On Behalf Of Kappelman, Leon
Sent: Monday, 23 September 2013 3:23 AM
To: aisworld at lists.aisnet.org
Subject: [AISWorld] Downside of impact factors: Scientists engaging in 'citation stacking'

Some of our institutions have adopted or are considering the use of impact factors as a metric for faculty performance.  Certainly our current faculty performance measures are less than perfect so it seems to be a suggestion worthy of consideration.  But impact factors are not without controversy and apparently prone to manipulation and questionable behaviors.  So before embracing impact factors, consider that at least to some extent they are a measure of what some might call "incestuous citation behaviors."  Not surprising since most all of us understand that when it comes to human behavior, you get what you measure.  The big question is: Do the positives of using impact factors, or any other metric or combination of measure for that matter, sufficiently outweigh the negatives?

Here is some of the "food for thought" that raised my eyebrows enough to send this note:

http://www.nature.com/news/brazilian-citation-scheme-outed-1.13604

http://occamstypewriter.org/scurry/2012/08/13/sick-of-impact-factors/

http://retractionwatch.wordpress.com/2013/08/30/journal-retracts-two-papers-after-being-caught-manipulating-citations/

http://www.naturalnews.com/042152_citation_stacking_scientific_journals_dishonesty.html

Best wishes,
Leon Kappelman
"Only a virtuous people are capable of freedom." - Benjamin Franklin
----------------------------------------------------------------------------------------------------------------------------------------------
Leon A. Kappelman, Ph.D.
  Professor of Information Systems
  Director Emeritus, Information Systems Research Center
  Fellow, Texas Center for Digital Knowledge
    College of Business, University of North Texas
    Voice: 940-565-4698   Email: kapp at unt.edu<mailto:kapp at unt.edu>
Founding Chair, Society for Information Management's Enterprise Architecture Working Group
----------------------------------------------------------------------------------------------------------------------------------------------
Please consider the environment before printing this e-mail

Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone.

Deakin University does not warrant that this email and any attachments are error or virus free.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.aisnet.org/pipermail/aisworld_lists.aisnet.org/attachments/20130922/481efad7/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 905 bytes
Desc: image001.gif
URL: <http://lists.aisnet.org/pipermail/aisworld_lists.aisnet.org/attachments/20130922/481efad7/attachment.gif>