[AISWorld] Downside of impact factors: Scientists engaging in 'citation stacking'

Tue Sep 24 15:06:21 EDT 2013

Dear colleagues,
This topic is really relevant and sadly without an acceptable scientific
solution, why ?
before to measure all such metrics, we should think on the mode to accept
or reject
a paper: it totally subjective based on the expertise of two peers !
Experimented EiCs
knows that sometimes papers can received two totally opposite reviews ! In
summary,
we should improve the review and acceptance procedure (more scientific !)
than current
ones which is more sociopolitical and power-biased. Too many instruments
have
developed for measuring constructs in MIS but is there some really useful
for
the overall quality of a paper? Cheers !
Manuel

On Mon, Sep 23, 2013 at 6:56 PM, John Lamp <john.lamp at deakin.edu.au> wrote:

>  As long as you accept that the h-index, and the multiple variants that
> follow it, are not a silver bullet, fine. A word of caution is that
> citation does not imply approval. The most cited paper is the cold fusion
> paper – I would not hold my breath looking for a positive article citing it.
> ****
>
> ** **
>
> The next question is why apply it to journals? Why not apply it to
> articles? The fundamental fact is that articles determine the quality of a
> journal. Journals do not determine the quality of an article. I’ve attached
> the SPARC Primer on Article Level Metrics.****
>
> ** **
>
> Also, journals are an artefact of the paper based information distribution
> system. We don’t need them. Peer reviewing and the 15 page gold standard
> are also artefacts of the paper based information distribution system. I
> know this will create an additional flurry of defensive responses, but just
> as armies of clerks had to accept that information technology had
> transformed their job, so we too must accept it. The defensive responses
> matter as little as those defending the superiority of the quill and
> manuscript against movable type. We will keep what is still relevant, and
> develop new things which take advantage of electronic distribution.****
>
> ** **
>
> There is an amazing irony in this that most of you probably only know me
> from the Index of Information Systems Journals and related work I have been
> involved with. Ah well, I was a printer once too!****
>
> ** **
>
> BTW if you are interested in h-, g-, and other indices and how they can
> measure impact, check out “Publish or Perish” from Anne-Wil Harzing (just
> up the road at the University of Melbourne), an amazing piece of software.
> ****
>
> http://www.harzing.com/pop.htm****
>
> ** **
>
> Cheers****
>
> John****
>
> ** **
>
> *From:* Michael Cuellar [mailto:mcuellar at georgiasouthern.edu]
> *Sent:* Tuesday, 24 September 2013 9:28 AM
> *To:* John Lamp
> *Cc:* Ilia Bider; aisworld at lists.aisnet.org
>
> *Subject:* Re: [AISWorld] Downside of impact factors: Scientists engaging
> in 'citation stacking'****
>
>  ** **
>
> I think we have seen in this post that Impact Factors should not be used
> for evaluation of scholarly productivity given their volatility and ability
> to manipulated.****
>
> ** **
>
> That is why we have advocated using the Hirsch indices as a measure of the
> "scholarly influence" of the scholar in the field. Depending on how you
> define the field, you can use the Hirsch indices to assess impact on the
> scholarly community, or the practitioner community or both or the social
> world. ****
>
> ** **
>
> The Hirsch indices do this by assessing both productivity (how many papers
> did you get somebody to publish) and impact (how many people have cited
> your work). Thus it gets beyond the viewpoint of a few editors and
> reviewers and lets the field determine what is important. This is what the
> field does anyway. Most of the most highly cited articles are not to be
> found in the "top journals" of the field. See Singh, G., K. M. Haddad, et
> al. (2007). "Are Articles in "Top" management Journals Necessarily of
> Higher Quality." *Journal of Management Inquiry* *16*(4): 319-331 for the
> situation in management. They are also less subject to manipulation since
> once you get beyond the lowest levels, they are very difficult to
> manipulate by self-citation. They are also able to account for the
> influence of recent papers, big hit papers, numbers of co-authors,
> and number of publications.****
>
> ** **
>
> See these papers for more discussion of influence and the hirsch indices.*
> ***
>
> ** **
>
> Truex III, D. P., M. J. Cuellar, et al. (2011). "The Scholarly Influence
> of Heinz Klein: Ideational and Social Measures of His Impact on IS Research
> and IS Scholars." *European Journal of Information Systems* *20*(4).****
>
> Truex III, D. P., M. J. Cuellar, et al. (2009). "Assessing Scholarly
> Influence: Using the Hirsch Indices to Reframe the Discourse." *Journal
> of the Association of Information Systems* *10*(7): 560-594.****
>
> Takeda, H., D. P. Truex III, et al. (2012). "Evaluating Scholarly
> Influence Through Social Network Analysis: the Next Step in Evaluating
> Scholarly Influence." *The International Journal of Social and
> Organizational Dynamics in Information Technology* *2*(1).****
>
> ** **
>
> Best Regards,****
>
> ** **
>
> Mike Cuellar****
>
> ** **
>
> On Sep 23, 2013, at 6:58 PM, John Lamp <john.lamp at deakin.edu.au> wrote:***
> *
>
>
>
> ****
>
>  I don’t know whether this attachment is too big (<1Mb) but the
> Australian Government recently published a discussion paper on the wider
> definition of research impact. What will come of this following the change
> of government is moot.****
>
>  ****
>
> Essentially it looks beyond internal impact (within the originating
> academic discipline) to suggest three dimensions of external impact:****
>
> ·         Academic impact (on researchers outside the originating
> discipline)****
>
> ·         Professional Impact (on practitioners)****
>
> ·         Social impact (broadly defined.****
>
>  ****
>
> If the attachment is stripped, email me directly for a copy.****
>
>  ****
>
> Cheers****
>
> John****
>
>  ****
>
> *From:* aisworld-bounces at lists.aisnet.org [mailto:aisworld-
> bounces at lists.aisnet.org] *On Behalf Of *Ilia Bider
> *Sent:* Monday, 23 September 2013 5:16 PM
> *To:* aisworld at lists.aisnet.org
> *Subject:* Re: [AISWorld] Downside of impact factors: Scientists engaging
> in 'citation stacking'****
>
>  ****
>
> Hossein,
>
> Sadly enough :-(, no academic measurement, including the Impact Factor, is
> directed at measuring the impact of research on real life, only its
> influence on the academic life. Publishing in a  highest ranking journal
> does not guarantee that a paper will be picked up by the industry, or other
> real life sector for implementation. Part of the problem here is the
> academic papers being written in such a style that a "normal" person, most
> probably, does not understand. What's more, IMHO, the high ranking journals
> are among those that promote this style in no lesser degree than the less
> ranking journals. On the positive side, this problem is started to be
> understood, and, hopefully, the situation will change in the nearest future
> :-).
>
> Hope, my message will take away some of your frustration :-).
>
> Best regards/Ilia
>
> On 2013-09-23 04:38, Hossein Seif Zadeh wrote:****
>
>  Not currently employed as an academic, I have just prepared a fresh jar
> of coffee and I am embracing for an "interesting" philosophical debate
> without any tangible, actionable, outcome; but alas here we go again... **
> **
>
>
>
>
> ****
>
> Just to share one personal experience; I successfully completed a research
> project a few years ago and I received a letter of commendation (on federal
> government letterhead) specifically saying my research was of "national
> importance" and "impact". Later that year when I sought feedback on whether
> I should include the letter in my upcoming promotion pack, I was told it
> would contribute "zilch" as it was not peer-reviewed, not a recognized
> publication, and it was "just one letter by one person"!! Oh, well...
> Chalked it up as a lesson learnt; original research of national importance
> has no place in today's academia. Is this really the message we want to
> convey to junior, aspiring, academics?****
>
>
>
>
> ****
>
> Coffee mug in hand, feet up, ready for the onslaught of emails.... on
> second thought, I might need a glass of red instead of coffee :-) Should be
> an interesting 48 hours or or so before the debate winds down.****
>
>
>
>
> ****
>
> Cheers,****
>
> Hossein ****
>
>
> Sent from my iPhone****
>
>
> On 23/09/2013, at 3:22 AM, "Kappelman, Leon" <Leon.Kappelman at unt.edu>
> wrote:
>
>
>
> ****
>
>  Some of our institutions have adopted or are considering the use of
> impact factors as a metric for faculty performance.  Certainly our current
> faculty performance measures are less than perfect so it seems to be a
> suggestion worthy of consideration.  But impact factors are not without
> controversy and apparently prone to manipulation and questionable
> behaviors.  So before embracing impact factors, consider that at least to
> some extent they are a measure of what some might call “incestuous citation
> behaviors.”  Not surprising since most all of us understand that when it
> comes to human behavior, you get what you measure.  The big question is: Do
> the positives of using impact factors, or any other metric or combination
> of measure for that matter, sufficiently outweigh the negatives? ****
>
>  ****
>
> Here is some of the “food for thought” that raised my eyebrows enough to
> send this note:****
>
>  ****
>
> http://www.nature.com/news/brazilian-citation-scheme-outed-1.13604****
>
>  ****
>
> http://occamstypewriter.org/scurry/2012/08/13/sick-of-impact-factors/****
>
>  ****
>
>
> http://retractionwatch.wordpress.com/2013/08/30/journal-retracts-two-papers-after-being-caught-manipulating-citations/
> ****
>
>  ****
>
>
> http://www.naturalnews.com/042152_citation_stacking_scientific_journals_dishonesty.html
> ****
>
>  ****
>
> Best wishes,
> Leon Kappelman****
>
> *“Only a virtuous people are capable of freedom.” – Benjamin Franklin*****
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------
> Leon A. Kappelman, Ph.D.
>   Professor of Information Systems
>   Director Emeritus, Information Systems Research Center
>   Fellow, Texas Center for Digital Knowledge
>     College of Business, University of North Texas
>     Voice: 940-565-4698   Email:kapp at unt.edu
> Founding Chair, Society for Information Management's Enterprise
> Architecture Working Group
>
> ----------------------------------------------------------------------------------------------------------------------------------------------
> Please consider the environment before printing this e-mail ****
>
>  ****
>
>  ****
>
>  _______________________________________________
> AISWorld mailing list
> AISWorld at lists.aisnet.org****
>
>
>
>
>
> ****
>
> _______________________________________________****
>
> AISWorld mailing list****
>
> AISWorld at lists.aisnet.org****
>
>
>
>
>
> ****
>
> -- ****
>
> ===============================================****
>
> Dr. Ilia Bider  ****
>
> Process- och systemutvecklingskonsult at ibissoft.se****
>
> Lektor & Forskare at DSV.su.se****
>
> ilia at ibissoft.se        +46 (0)8 164998****
>
> Design science in action ... http://slidesha.re/Uq3RTC****
>
> *
> Important Notice:** **The contents of this email are intended solely for
> the named addressee and are confidential; any unauthorised use,
> reproduction or storage of the contents is expressly prohibited. If you
> have received this email in error, please delete it and any attachments
> immediately and advise the sender by return email or telephone.**
>
> Deakin University does not warrant that this email and any attachments are
> error or virus free.* <Research Impact Discussion
> Paper.pdf>_______________________________________________
> AISWorld mailing list
> AISWorld at lists.aisnet.org****
>
>  ** **
>  *
> Important Notice: The contents of this email are intended solely for the
> named addressee and are confidential; any unauthorised use, reproduction or
> storage of the contents is expressly prohibited. If you have received this
> email in error, please delete it and any attachments immediately and advise
> the sender by return email or telephone.
>
> Deakin University does not warrant that this email and any attachments are
> error or virus free.*
>
> _______________________________________________
> AISWorld mailing list
> AISWorld at lists.aisnet.org
>

-- 
--------------------------------------------------------------------------
Manuel Mora, EngD.
Full Professor and Researcher Level C
ACM Senior Member / SNI Level I
Department of Information Systems
Autonomous University of Aguascalientes
Ave. Universidad 940
Aguascalientes, AGS
Mexico, 20131
 --------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.aisnet.org/pipermail/aisworld_lists.aisnet.org/attachments/20130924/c7d61e3e/attachment.html>