[AISWorld] Fwd: Fwd: Downside of impact factors: Scientists engaging in 'citation stacking'

[mmora at securenym.net]

Robert,
Thanks, I totally agree with you: a research should be happy by helping
in the solution of problems (with direct solutions or plausible pieces
for a future solution) rather by their paper's ranking.

Furthermore, based on the underlying premise of AIS research community
pursues the development of MIS/IT area for improving human interests
(social, economic, political, environmental, etc) rather for its own
sake and of their researchers, we should to re-evaluate and re-valuate
our scientific recognition in MIS/IT area. Two examples:

1. Contribution of E.F. Codd (A relational data model for large shared
data banks, published in CACM, 1970) is relevant not for the IFs of
CACM or the 8000+ citations, but for opening a new billionary industry!
caused by new technology.2.  It supported also by P.Chen's seminal paper
(The entity-relationship model—toward a unified view of data, TODS,
1976).

In top journals MISQ, JMIS, ISR there are excellent contributions
(specially from 1980-1990's period) but our subjective perception
of a great quality distance between these journals (IF's over 4 to 5)
compared with others in the range of 1-3 is a misperception. A
basked of top journals should be rated by their scope to worlwide
problems and lifetime impact. Maybe a better classification of journals
-as initial point- could be something like:

A. Journals with papers well-written and well-conducted on themes with a
worlwide relevance or highly specialized (about 15-20 journals with
similar status)

B. Journals with papers well-written and well-conducted on themes with a
regional/country relevance (about 15-20 journals with similar status)

C. Journals for beginners or short ideas.

A journals should be also to publish long papers (as Technical Reports,
over 25,000 words) or at least through electronic means. Many papers
at present precludes relevant information by lack of space.

Well, again my 5 cents contribution from the worlview from a researcher
in a developing country (Mexico) connected geographically and culturally
with the research country #1 USA.

Manuel



On Fri, September 27, 2013 12:18 pm, Manuel Mora wrote:
> On Thu, Sep 26, 2013 at 8:16 PM, Robert Johnston
> <robert.johnston at ucd.ie>wrote:
>
>
>> Manual
>>
>>
>> The solution is to put the creation of knowledge back at the centre of
>> our academic practices instead of the creation of "numbers". This is hard
>> to do because other institutional players are using these numbers for
>> their own games.
>>
>>
>> Dr Robert B. Johnston
>> Mail: 81 Hunter St. Richmond, VIC, Australia. 3121
>> Email: Robert.Johnston at ucd.ie <Robert.Johnston at ucd.ie>
>> Mobile: +61 414 493 255
>> Skype: johno.home
>>
>>
>> On 27/09/2013 9:49 AM, mmora at securenym.net wrote:
>>
>>
>>> Dear colleague Lee Giles,
>>> Your links to DORA on the bias on IF is excellent !
>>> In particular, some additional links on DORA suggest
>>> the bias caused by mutually-reinforcing (it can be modeled through
>>> system dynamics simulation models), where higher IFs --> higher
>>> rankings from survey studies on quality of journals --> higher
>>> submission to these journals --> lower acceptance rate due to the huge
>>>  number of submitted papers --> higher fame o journals ! ---> higher
>>> IFs. In particular, my critique is to
>>> "close our eyes" to the essential roots of the problem:
>>> acceptance of papers based on two evaluations from peers is
>>> intrinsically biased, and the implicit rule is rather to accept papers
>>> that are similarly written as reviewers could having done it and with
>>> similar socio-economic-cultural and political worldviews. It is a hard
>>> reality in MIS area. On solutions ! maybe to cancel the blind-mode, to
>>> publish in the accepted papers the comments from reviewers, to increase
>>> the number of reviewers (commentators) adding people from the IT
>>> industry, etc. In summary, any sophisticated metric is undermined
>>> while the acceptance/rejection process keeps being biased. "Science is
>>> not a value free human activity" (from several references).
>>> Manuel Mora
>>> * IT Researcher in a developing country
>>>
>>>
>>> * I estimate that over the 70% of academics enrolled in this list
>>> we will never have a paper in top-5 journals, indepedently from the
>>> quality of the paper by all aforementioned reported reasons.
>>>
>>> * Google metrics could be simple but report facts: number of papers
>>> published by a researcher, number of citations. That is sufficient for
>>> a single analysis.
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, September 26, 2013 6:02 pm, Manuel Mora wrote:
>>>
>>>
>>>> ---------- Forwarded message ----------
>>>> From: Lee Giles <giles at ist.psu.edu>
>>>> Date: Wed, Sep 25, 2013 at 10:51 AM
>>>> Subject: Re: [AISWorld] Downside of impact factors: Scientists
>>>> engaging in 'citation stacking'
>>>> To: John Lamp <john.lamp at deakin.edu.au>, Michael Cuellar <
>>>> mcuellar at georgiasouthern.edu>, ISWorld <aisworld at lists.aisnet.org>
>>>>
>>>>
>>>> One may be interested in the San Francisco Declaration on Research
>>>> Assessment's effort to deal with the impact factor.
>>>>
>>>>
>>>>
>>>> http://am.ascb.org/dora/
>>>> ______________________________**__________
>>>> From:
>>>> aisworld-bounces at lists.aisnet.**org<aisworld-bounces at lists.aisnet.o
>>>> rg> [aisworld-bounces at lists.**aisnet.org
>>>> <aisworld-bounces at lists.aisnet.org>
>>>> ]
>>>> on behalf of John Lamp [john.lamp at deakin.edu.au] Sent: Monday,
>>>> September
>>>> 23, 2013 7:56 PM
>>>> To: Michael Cuellar; ISWorld
>>>> Subject: Re: [AISWorld] Downside of impact factors: Scientists
>>>> engaging in 'citation stacking'
>>>>
>>>>
>>>>
>>>> As long as you accept that the h-index, and the multiple variants
>>>> that follow it, are not a silver bullet, fine. A word of caution is
>>>> that citation does not imply approval. The most cited paper is the
>>>> cold fusion paper – I would not hold my breath looking for a
>>>> positive article citing it.
>>>>
>>>> The next question is why apply it to journals? Why not apply it to
>>>> articles? The fundamental fact is that articles determine the
>>>> quality of a journal. Journals do not determine the quality of an
>>>> article. I’ve attached the SPARC Primer on Article Level Metrics.
>>>>
>>>> Also, journals are an artefact of the paper based information
>>>> distribution system. We don’t need them. Peer reviewing and the 15
>>>> page gold standard are also artefacts of the paper based information
>>>>  distribution system. I know this will create an additional flurry
>>>> of defensive responses, but just as armies of clerks had to accept
>>>> that information technology had transformed their job, so we too
>>>> must accept it. The defensive responses matter as little as those
>>>> defending the superiority of the quill and manuscript against
>>>> movable type. We will keep what is still relevant, and develop new
>>>> things which take advantage of electronic distribution.
>>>>
>>>> There is an amazing irony in this that most of you probably only
>>>> know me from the Index of Information Systems Journals and related
>>>> work I have been involved with. Ah well, I was a printer once too!
>>>>
>>>> BTW if you are interested in h-, g-, and other indices and how they
>>>> can measure impact, check out “Publish or Perish” from Anne-Wil
>>>> Harzing (just
>>>> up the road at the University of Melbourne), an amazing piece of
>>>> software. http://www.harzing.com/pop.htm
>>>>
>>>>
>>>> Cheers
>>>> John
>>>>
>>>>
>>>>
>>>> From: Michael Cuellar
>>>> [mailto:mcuellar@**georgiasouthern.edu<mcuellar at georgiasouthern.edu
>>>> >
>>>> ]
>>>> Sent: Tuesday, 24 September 2013 9:28 AM
>>>> To: John Lamp
>>>> Cc: Ilia Bider; aisworld at lists.aisnet.org
>>>> Subject: Re: [AISWorld] Downside of impact factors: Scientists
>>>> engaging in 'citation stacking'
>>>>
>>>>
>>>>
>>>> I think we have seen in this post that Impact Factors should not be
>>>> used for evaluation of scholarly productivity given their volatility
>>>> and ability to manipulated.
>>>>
>>>> That is why we have advocated using the Hirsch indices as a measure
>>>> of the "scholarly influence" of the scholar in the field. Depending
>>>> on how you define the field, you can use the Hirsch indices to
>>>> assess impact on the scholarly community, or the practitioner
>>>> community or both or the social world.
>>>>
>>>> The Hirsch indices do this by assessing both productivity (how many
>>>>  papers did you get somebody to publish) and impact (how many
>>>> people have cited your work). Thus it gets beyond the viewpoint of a
>>>> few editors and reviewers and lets the field determine what is
>>>> important. This is what the field does anyway. Most of the most
>>>> highly cited articles are not to be found in the "top journals" of
>>>> the field. See Singh, G., K. M. Haddad, et al. (2007). "Are Articles
>>>> in "Top" management Journals Necessarily of Higher Quality." Journal
>>>> of Management Inquiry 16(4): 319-331 for the situation in
>>>> management. They are also less subject to manipulation since once
>>>> you get beyond the lowest levels, they are very difficult to
>>>> manipulate by self-citation. They are also able to account for the
>>>> influence of recent papers, big hit papers, numbers of co-authors,
>>>> and number of publications.
>>>>
>>>> See these papers for more discussion of influence and the hirsch
>>>> indices.
>>>>
>>>>
>>>> Truex III, D. P., M. J. Cuellar, et al. (2011). "The Scholarly
>>>> Influence
>>>> of Heinz Klein: Ideational and Social Measures of His Impact on IS
>>>> Research
>>>> and IS Scholars." European Journal of Information Systems 20(4).
>>>> Truex
>>>> III,
>>>> D. P., M. J. Cuellar, et al. (2009). "Assessing Scholarly
>>>> Influence: Using the Hirsch Indices to Reframe the Discourse."
>>>> Journal of
>>>> the Association of Information Systems 10(7): 560-594. Takeda, H.,
>>>> D. P.
>>>> Truex III, et al. (2012). "Evaluating Scholarly Influence
>>>> Through Social Network Analysis: the Next Step in Evaluating
>>>> Scholarly
>>>> Influence." The International Journal of Social and Organizational
>>>> Dynamics
>>>> in Information Technology 2(1).
>>>>
>>>> Best Regards,
>>>>
>>>>
>>>>
>>>> Mike Cuellar
>>>>
>>>>
>>>>
>>>> On Sep 23, 2013, at 6:58 PM, John Lamp <john.lamp at deakin.edu.au<**
>>>> mailto:
>>>> john.lamp at deakin.edu.au>> wrote:
>>>>
>>>>
>>>> I don’t know whether this attachment is too big (<1Mb) but the
>>>> Australian
>>>> Government recently published a discussion paper on the wider
>>>> definition of research impact. What will come of this following the
>>>> change of government is moot.
>>>>
>>>> Essentially it looks beyond internal impact (within the originating
>>>>  academic discipline) to suggest three dimensions of external
>>>> impact:
>>>> Academic impact (on researchers outside the originating
>>>> discipline) •         Professional Impact (on practitioners) Social
>>>> impact (broadly defined.
>>>>
>>>>
>>>> If the attachment is stripped, email me directly for a copy.
>>>>
>>>>
>>>>
>>>> Cheers
>>>> John
>>>>
>>>>
>>>>
>>>> From:
>>>> aisworld-bounces at lists.aisnet.**org<aisworld-bounces at lists.aisnet.o
>>>> rg> <mailto:
>>>> aisworld-bounces at lists.aisnet.**org
>>>> <aisworld-bounces at lists.aisnet.org>>
>>>> [mailto:aisworld-bounces@**lists.aisnet.org<aisworld-bounces at lists.a
>>>> isnet.org> <mailto:bounces at lists.aisnet.**org
>>>> <bounces at lists.aisnet.org>>] On
>>>> Behalf Of Ilia Bider
>>>> Sent: Monday, 23 September 2013 5:16 PM
>>>> To:
>>>> aisworld at lists.aisnet.org<**mailto:aisworld at lists.aisnet.**org<aisw
>>>> orld at lists.aisnet.org>
>>>>>
>>>> Subject: Re: [AISWorld] Downside of impact factors: Scientists
>>>> engaging in 'citation stacking'
>>>>
>>>>
>>>>
>>>> Hossein,
>>>>
>>>>
>>>>
>>>> Sadly enough :-(, no academic measurement, including the Impact
>>>> Factor,
>>>> is directed at measuring the impact of research on real life, only
>>>> its influence on the academic life. Publishing in a  highest ranking
>>>> journal does not guarantee that a paper will be picked up by the
>>>> industry, or other real life sector for implementation. Part of the
>>>> problem here is the academic papers being written in such a style
>>>> that a "normal" person, most probably, does not understand. What's
>>>> more, IMHO, the high ranking journals are among those that promote
>>>> this style in no lesser degree than the less ranking journals. On
>>>> the positive side, this problem is started to be understood, and,
>>>> hopefully, the situation will change in the nearest future :-).
>>>>
>>>>
>>>> Hope, my message will take away some of your frustration :-).
>>>>
>>>>
>>>>
>>>> Best regards/Ilia
>>>>
>>>>
>>>>
>>>> On 2013-09-23 04:38, Hossein Seif Zadeh wrote:
>>>> Not currently employed as an academic, I have just prepared a fresh
>>>> jar of coffee and I am embracing for an "interesting" philosophical
>>>> debate without any tangible, actionable, outcome; but alas here we
>>>> go again...
>>>>
>>>>
>>>>
>>>> Just to share one personal experience; I successfully completed a
>>>> research project a few years ago and I received a letter of
>>>> commendation (on federal
>>>> government letterhead) specifically saying my research was of
>>>> "national
>>>> importance" and "impact". Later that year when I sought feedback on
>>>> whether I should include the letter in my upcoming promotion pack,
>>>> I was
>>>> told it would contribute "zilch" as it was not peer-reviewed, not a
>>>> recognized publication, and it was "just one letter by one
>>>> person"!! Oh, well... Chalked it up as a lesson learnt; original
>>>> research of national importance has no place in today's academia. Is
>>>> this really the message we want to convey to junior, aspiring,
>>>> academics?
>>>>
>>>>
>>>>
>>>> Coffee mug in hand, feet up, ready for the onslaught of emails....
>>>> on second thought, I might need a glass of red instead of coffee :-)
>>>> Should
>>>> be an interesting 48 hours or or so before the debate winds down.
>>>>
>>>>
>>>>
>>>> Cheers,
>>>> Hossein
>>>>
>>>>
>>>>
>>>> Sent from my iPhone
>>>>
>>>>
>>>>
>>>> On 23/09/2013, at 3:22 AM, "Kappelman, Leon"
>>>> <Leon.Kappelman at unt.edu<**mailto:
>>>> Leon.Kappelman at unt.edu>> wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Some of our institutions have adopted or are considering the use of
>>>>  impact factors as a metric for faculty performance.  Certainly our
>>>>  current faculty performance measures are less than perfect so it
>>>> seems to be a suggestion worthy of consideration.  But impact
>>>> factors are not without controversy and apparently prone to
>>>> manipulation and questionable behaviors.  So before embracing impact
>>>> factors, consider that at least to some extent they are a measure of
>>>> what some might call “incestuous citation behaviors.”  Not
>>>> surprising since most all of us understand that when it comes to
>>>> human behavior, you get what you measure.  The big question is: Do
>>>> the positives of using impact factors, or any other metric or
>>>> combination of measure for that matter, sufficiently outweigh the
>>>> negatives?
>>>>
>>>> Here is some of the “food for thought” that raised my eyebrows
>>>> enough to send this note:
>>>>
>>>>
>>>> http://www.nature.com/news/**brazilian-citation-scheme-**outed-1.13
>>>> 604<http://www.nature.com/news/brazilian-citation-scheme-outed-1.13
>>>> 604>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> http://occamstypewriter.org/**scurry/2012/08/13/sick-of-**
>>>> impact-factors/<http://occamstypewriter.org/scurry/2012/08/13/sick-o
>>>> f-impact-factors/>
>>>>
>>>>
>>>>
>>>>
>>>> http://retractionwatch.**wordpress.com/2013/08/30/**
>>>> journal-retracts-two-pape<http://retractionwatch.wordpress.com/2013/
>>>> 08/30/journal-retracts-two-pape>
>>>> rs-after-being-caught-**manipulating-citations/
>>>>
>>>>
>>>>
>>>> http://www.naturalnews.com/**042152_citation_stacking_**
>>>> scientific_journals_d<http://www.naturalnews.com/042152_citation_sta
>>>> cking_scientific_journals_d> ishonesty.html
>>>>
>>>> Best wishes,
>>>> Leon Kappelman
>>>> Only a virtuous people are capable of freedom.” – Benjamin Franklin
>>>> ------------------------------**------------------------------**
>>>> --------------
>>>> ------------------------------**------------------------------**----
>>>> ----
>>>> Leon A. Kappelman, Ph.D.
>>>> Professor of Information Systems
>>>> Director Emeritus, Information Systems Research Center
>>>> Fellow, Texas Center for Digital Knowledge
>>>> College of Business, University of North Texas
>>>> Voice: 940-565-4698<tel:940-565-4698>   Email:kapp at unt.edu<mailto:
>>>> kapp at unt.edu> Founding Chair, Society for Information Management's
>>>> Enterprise
>>>> Architecture Working Group
>>>> ------------------------------**------------------------------**
>>>> --------------
>>>> ------------------------------**------------------------------**----
>>>> ----
>>>> Please consider the environment before printing this e-mail
>>>>
>>>>
>>>>
>>>>
>>>> ______________________________**_________________
>>>> AISWorld mailing list
>>>> AISWorld at lists.aisnet.org<**mailto:AISWorld at lists.aisnet.**org<AISWo
>>>> rld at lists.aisnet.org>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ______________________________**_________________
>>>>
>>>>
>>>>
>>>> AISWorld mailing list
>>>>
>>>>
>>>>
>>>> AISWorld at lists.aisnet.org<**mailto:AISWorld at lists.aisnet.**org<AISW
>>>> orld at lists.aisnet.org>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>>
>>>> ==============================**=================
>>>>
>>>>
>>>>
>>>> Dr. Ilia Bider
>>>>
>>>>
>>>>
>>>> Process- och systemutvecklingskonsult at
>>>> ibissoft.se<http://ibissoft.se*
>>>> *>
>>>>
>>>>
>>>>
>>>> Lektor & Forskare at DSV.su.se<http://DSV.su.se>
>>>>
>>>>
>>>>
>>>> ilia at ibissoft.se<mailto:ilia@**ibissoft.se <ilia at ibissoft.se>>
>>>> +46 (0)8 164998
>>>>
>>>>
>>>>
>>>> Design science in action ... http://slidesha.re/Uq3RTC
>>>>
>>>>
>>>>
>>>> Important Notice: The contents of this email are intended solely
>>>> for the named addressee and are confidential; any unauthorised use,
>>>> reproduction or storage of the contents is expressly prohibited. If
>>>> you have received this email in error, please delete it and any
>>>> attachments immediately and advise the sender by return email or
>>>> telephone.
>>>>
>>>> Deakin University does not warrant that this email and any
>>>> attachments are error or virus free. <Research Impact Discussion
>>>> Paper.pdf>____________________**___________________________
>>>> AISWorld mailing list
>>>> AISWorld at lists.aisnet.org<**mailto:AISWorld at lists.aisnet.**org<AISWo
>>>> rld at lists.aisnet.org>
>>>>>
>>>>
>>>>
>>>>
>>>> Important Notice: The contents of this email are intended solely
>>>> for the named addressee and are confidential; any unauthorised use,
>>>> reproduction or storage of the contents is expressly prohibited. If
>>>> you have received this email in error, please delete it and any
>>>> attachments immediately and advise the sender by return email or
>>>> telephone.
>>>>
>>>> Deakin University does not warrant that this email and any
>>>> attachments are error or virus free.
>>>> ______________________________**
>>>> _________________
>>>> AISWorld mailing list
>>>> AISWorld at lists.aisnet.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> ------------------------------**------------------------------**
>>>> --------------
>>>> Manuel Mora, EngD.
>>>> Full Professor and Researcher Level C
>>>> ACM Senior Member / SNI Level I
>>>> Department of Information Systems
>>>> Autonomous University of Aguascalientes
>>>> Ave. Universidad 940
>>>> Aguascalientes, AGS
>>>> Mexico, 20131
>>>> ------------------------------**------------------------------**
>>>> --------------
>>>>
>>>>
>>>>
>>>>
>>>
>>> ______________________________**_________________
>>> AISWorld mailing list
>>> AISWorld at lists.aisnet.org
>>>
>>>
>>
>>
>> ______________________________**_________________
>> AISWorld mailing list
>> AISWorld at lists.aisnet.org
>>
>>
>
>
>
> --
> --------------------------------------------------------------------------
>  Manuel Mora, EngD.
> Full Professor and Researcher Level C
> ACM Senior Member / SNI Level I
> Department of Information Systems
> Autonomous University of Aguascalientes
> Ave. Universidad 940
> Aguascalientes, AGS
> Mexico, 20131
> --------------------------------------------------------------------------
>
>