[AISWorld] Downside of impact factors: Scientists engaging in 'citation stacking'

Fri Sep 27 12:36:01 EDT 2013

Hi All,

I admit that citation counts and related indexes have their weaknesses, but I think that it is the case of all attempts to evaluate research quality and productivity.  

Yet, coming from a culture with no tradition of small talk and with a weird language, I have been pleased that there such "neutral" measures to evaluate one's position in the scientific community. "Neutral" in the sense you publish your work and it is up to the scientific community at the large to decide if the work is interesting enough to be read and to be cited. After publishing you cannot do much to increase citations to it, especially if you are a poor speaker (salesman)and cannot travel around to have seminars in the topic. 

Of course, there is a problem that the attention a paper receives depends on the forum where it is published. If you manage to publish in so called top journals (with a high IF), even a mediocre paper likely receives more citations than a much better paper on a less prestigious forum (with a low IF). The reinforcing cycle described in one of the earlier comments is really important to understand. That is why journals are so keen to improve their IF, sometimes using dubious means, imposing an official or unofficial editorial policy that a submission to a journal should cite to earlier articles published in that journal. I do not know if anyone has analyzed how much it really biases the journal impact factors.

Self-citations provide another possibility to manipulate statistics. I have an impression that it is not necessarily a major issue. Personally, I feel that I have referred quite a lot to my own publications. Yet, using some available statistics provided by Web of Knowledge, my self-citations constitute 3.8 % of the total citations. I would guess that most of IS researchers have proportionally less self-citations than I do.

A more serious problem might be that research groups, departments, universities or whatever communities discretely agree to reciprocally refer to publications of its members. I do not know if it has been really practiced anywhere, but if used one may have real impact. At least I do not observe that I would have been favored by any such communities, except by Finnish researchers as whole (citations from Finland constituting 9.2 %, i.e. 5.4 %, if my self-citations are excluded), but not  by researchers from the University of Oulu (only 0.4 %, if my self-citations are excluded).

Of course, all citations do not indicate approval. However, I believe that it is more an exception that the majority of citations to a highly cited article indicate disapproval, and it is still more challenging to build one's career so that one has highly cited papers and most citations to them are critical. At least one must have managed to build a highly ranked reputation so that other researchers consider it meaningful to start to criticize his/her later works just because he/she is a big name. 

So, I am not convinced that the problem lies so much in numbers, but more in the way they are used for various purposes and interests.

Based on experiences from a RAE conducted at the University of Oulu in 2007 I attempted to reflect on pros and cons of expert (peer) assessments and bibliometric assessments. To me they seem to be complementary rather than mutually exclusive  (see Iivari, J., Expert evaluation vs. bibliometric evaluation: experiences from Finland, European Journal of Information Systems, Vol. 17, No. 2, 2008, pp. 169-173).

Best,

Juhani

On Sep 24, 2013, at 1:58 AM, John Lamp wrote:

> I don’t know whether this attachment is too big (<1Mb) but the Australian Government recently published a discussion paper on the wider definition of research impact. What will come of this following the change of government is moot.
>  
> Essentially it looks beyond internal impact (within the originating academic discipline) to suggest three dimensions of external impact:
> ·         Academic impact (on researchers outside the originating discipline)
> ·         Professional Impact (on practitioners)
> ·         Social impact (broadly defined.
>  
> If the attachment is stripped, email me directly for a copy.
>  
> Cheers
> John
>  
> From: aisworld-bounces at lists.aisnet.org [mailto:aisworld-bounces at lists.aisnet.org] On Behalf Of Ilia Bider
> Sent: Monday, 23 September 2013 5:16 PM
> To: aisworld at lists.aisnet.org
> Subject: Re: [AISWorld] Downside of impact factors: Scientists engaging in 'citation stacking'
>  
> Hossein,
> 
> Sadly enough :-(, no academic measurement, including the Impact Factor, is directed at measuring the impact of research on real life, only its influence on the academic life. Publishing in a  highest ranking journal does not guarantee that a paper will be picked up by the industry, or other real life sector for implementation. Part of the problem here is the academic papers being written in such a style that a "normal" person, most probably, does not understand. What's more, IMHO, the high ranking journals are among those that promote this style in no lesser degree than the less ranking journals. On the positive side, this problem is started to be understood, and, hopefully, the situation will change in the nearest future :-).
> 
> Hope, my message will take away some of your frustration :-).
> 
> Best regards/Ilia
> 
> On 2013-09-23 04:38, Hossein Seif Zadeh wrote:
> Not currently employed as an academic, I have just prepared a fresh jar of coffee and I am embracing for an "interesting" philosophical debate without any tangible, actionable, outcome; but alas here we go again... 
> 
> 
> Just to share one personal experience; I successfully completed a research project a few years ago and I received a letter of commendation (on federal government letterhead) specifically saying my research was of "national importance" and "impact". Later that year when I sought feedback on whether I should include the letter in my upcoming promotion pack, I was told it would contribute "zilch" as it was not peer-reviewed, not a recognized publication, and it was "just one letter by one person"!! Oh, well... Chalked it up as a lesson learnt; original research of national importance has no place in today's academia. Is this really the message we want to convey to junior, aspiring, academics?
> 
> 
> Coffee mug in hand, feet up, ready for the onslaught of emails.... on second thought, I might need a glass of red instead of coffee :-) Should be an interesting 48 hours or or so before the debate winds down.
> 
> 
> Cheers,
> Hossein 
> 
> Sent from my iPhone
> 
> On 23/09/2013, at 3:22 AM, "Kappelman, Leon" <Leon.Kappelman at unt.edu> wrote:
> 
> 
> Some of our institutions have adopted or are considering the use of impact factors as a metric for faculty performance.  Certainly our current faculty performance measures are less than perfect so it seems to be a suggestion worthy of consideration.  But impact factors are not without controversy and apparently prone to manipulation and questionable behaviors.  So before embracing impact factors, consider that at least to some extent they are a measure of what some might call “incestuous citation behaviors.”  Not surprising since most all of us understand that when it comes to human behavior, you get what you measure.  The big question is: Do the positives of using impact factors, or any other metric or combination of measure for that matter, sufficiently outweigh the negatives? 
>  
> Here is some of the “food for thought” that raised my eyebrows enough to send this note:
>  
> http://www.nature.com/news/brazilian-citation-scheme-outed-1.13604
>  
> http://occamstypewriter.org/scurry/2012/08/13/sick-of-impact-factors/
>  
> http://retractionwatch.wordpress.com/2013/08/30/journal-retracts-two-papers-after-being-caught-manipulating-citations/
>  
> http://www.naturalnews.com/042152_citation_stacking_scientific_journals_dishonesty.html
>  
> Best wishes,
> Leon Kappelman
> 
> “Only a virtuous people are capable of freedom.” – Benjamin Franklin
> ----------------------------------------------------------------------------------------------------------------------------------------------
> Leon A. Kappelman, Ph.D. 
>   Professor of Information Systems 
>   Director Emeritus, Information Systems Research Center
>   Fellow, Texas Center for Digital Knowledge
>     College of Business, University of North Texas
>     Voice: 940-565-4698   Email:kapp at unt.edu
> Founding Chair, Society for Information Management's Enterprise Architecture Working Group
> ----------------------------------------------------------------------------------------------------------------------------------------------
> Please consider the environment before printing this e-mail 
>  
>  
> _______________________________________________
> AISWorld mailing list
> AISWorld at lists.aisnet.org
> 
> 
> 
> _______________________________________________
> AISWorld mailing list
> AISWorld at lists.aisnet.org
> 
> 
> 
> -- 
> ===============================================
> Dr. Ilia Bider  
> Process- och systemutvecklingskonsult at ibissoft.se
> Lektor & Forskare at DSV.su.se
> ilia at ibissoft.se        +46 (0)8 164998
> Design science in action ... http://slidesha.re/Uq3RTC
> 
> Important Notice: The contents of this email are intended solely for the named addressee and are confidential; any unauthorised use, reproduction or storage of the contents is expressly prohibited. If you have received this email in error, please delete it and any attachments immediately and advise the sender by return email or telephone.
> 
> Deakin University does not warrant that this email and any attachments are error or virus free. <Research Impact Discussion Paper.pdf>_______________________________________________
> AISWorld mailing list
> AISWorld at lists.aisnet.org