Say that an agency introduces a three-level ranking of journals in history and philosophy of science, technology and medicine and that this ranking can influence funding decisions, because it
will help to identify excellence in Humanities scholarship and should prove useful for the aggregate benchmarking of national research systems, for example, in determining the international standing of the research activity carried out in a given field in a particular country.
Then the process will obviously be observed, discussed, analysed.
Which is just what is happening with the European Science Foundation and its attempts to get a bibliometric grip on what European researchers in the humanities are doing. The initial list (this is an ongoing process) produced by the expert group in history and philosophy of science is interesting reading; some journals are rated C - they are more or less national in orientation, so getting out to an international audience is deemed positive -, a small group is rated A, leaving a quite large and heterogeneous group of journals as B.
Besides raising obvious questions about why some journals end up where they do (Social Studies of Science is … B), the methodology behind this exercise can be discussed.
For starters: why only three categories? With so many journals squeezed into the B category, you loose precision.
A comparison with the ISI/Thomson ranking system, specifically their Journal Citation Reports, will lead to interesting questions. The European Science Foundation, in the beginning of the process, found out that the ISI data “is not an appropriate bibliometric for (European) Humanities”.
Fair enough; building new metrics could lead to an improvement, and I guess that ISI:s data is not a perfect map of European humanities. Specifically, if a new metric would include journals not indexed by ISI as well as genres other than journal articles, it would better reflect the quality of the research, since many humanities researchers publish monographs, chapters in edited books and so on.
The results of ESF:s ranking will nevertheless be compared to the ISI statistics. It will also be compared to other, less rigorous but important, metrics, like the ranking each and every humanities scholar carries in the back of his or her head; used by researchers for evaluating publications in their everyday activities and built up by qualitative rather than quantitative means, such an experience-based on-going evaluation of the literature is a vital part of the researcher’s tacit knowledge. Time is limited and the number of publications, even in a small field such as history of science, technology, and medicine, is very large; thus, we must constantly exercise our bibliographic phronetic skills to function as scholars.
I am merely pointing out that such a comparison will be made, not making such a comparison. Let me just give one example to show that the matter is non-trivial: Social Studies of Science, ranked B in ERIH:s exercise, was ranked number 1 out of 29 journals in the History & Philosophy of Science section of JCR Social Science Edition and number 2 out of 37 journals in History & Philosophy of Science (JCR Science Edition) (2007 data).
Now, everything is not doom-and-gloom. At least in theory - and, I believe, contrary to the ISI process -, the ERIH process is open; we can provide feedback to the ERIH group about the initial lists, and it is an open question to what extent the “initial lists” will be finetuned by the feedback.
Another kind of feedback is the editorial to be published by a number of journals in history of science, including several leading ones (as you can see, I am exercising my phronetic bibliometric skills) and circulated, in advance of publication, yesterday on mailing lists in the history of science, technology and medicine community. It was picked up yesterday by Thomas Söderqvist on his blog and, like him, I quote it here in extenso:
Journals under Threat: A Joint Response from HSTM Editors
We live in an age of metrics. All around us, things are being standardized, quantified, measured. Scholars concerned with the work of science and technology must regard this as a fascinating and crucial practical, cultural and intellectual phenomenon. Analysis of the roots and meaning of metrics and metrology has been a preoccupation of much of the best work in our field for the past quarter century at least.
As practitioners of the interconnected disciplines that make up the field of science studies we understand how significant, contingent and uncertain can be the process of rendering nature and society in grades, classes and numbers. We now confront a situation in which our own research work is being subjected to putatively precise accountancy by arbitrary and unaccountable agencies. Some may already be aware of the proposed European Reference Index for the Humanities (ERIH), an initiative originating with the European Science Foundation. The ERIH is an attempt to grade journals in the humanities - including “history and philosophy of science”.
The initiative proposes a league table of academic journals, with premier, second and third divisions. According to the European Science Foundation, ERIH “aims initially to identify, and gain more visibility for, top-quality European Humanities research published in academic journals in, potentially, all European languages”. It is hoped “that ERIH will form the backbone of a fully-fledged research information system for the Humanities”. What is meant, however, is that ERIH will provide funding bodies and other agencies in Europe and elsewhere with an allegedly exact measure of research quality. In short, if research is published in a premier league journal it will be recognized as first rate; if it appears somewhere in the lower divisions, it will be rated (and not funded) accordingly.
This initiative is entirely defective in conception and execution. Consider the major issues of accountability and transparency. The process of producing the graded list of journals in science studies was overseen by a committee of four (the membership is currently
listed at http://www.esf.org/research-areas/humanities/research-infrastructures-including-erih/erih-governance-and-panels/erih-expert-panels.html). This committee cannot be considered representative. It was not selected in consultation with any of the various disciplinary organizations that currently represent our field such as BSHS, HSS, PSA, SHoT or SSSS. Only in June 2008 were journal editors belatedly informed of the process and its relevant criteria or asked to provide any information regarding their publications. No indication has been given of the means through which the list was compiled; nor how it might be maintained in the future.The ERIH depends on a fundamental misunderstanding of conduct and publication of research in our field, and in the humanities in general. Journals’ quality cannot be separated from their contents and their review processes. Great research may be published anywhere and in any language. Truly ground-breaking work may be more likely to appear from marginal, dissident or unexpected sources, rather than from a well-established and entrenched mainstream. Our journals are various, heterogeneous and distinct. Some are aimed at a broad, general and international readership, others are more specialized in their content and implied audience. Their scope and readership say nothing about the quality of their intellectual content. The ERIH, on the other hand, confuses internationality with quality in a way that is particularly prejudicial to specialist and non-English language journals. In a recent report, the British Academy, with judicious understatement, concludes that “the European Reference Index for the Humanities as presently conceived does not represent a reliable way in which metrics of peer-reviewed publications can be constructed.” Such exercises as ERIH can become self-fulfilling prophecies. If such measures as ERIH are adopted as metrics by funding and other agencies, then many in our field will conclude that they have little choice other than to limit their publications to journals in the premier division. We will sustain fewer journals, much less diversity and
impoverish our discipline.Along with many others in our field, this Journal has concluded that we want no part of this illegitimate and misguided exercise. This joint Editorial is being published in journals across the fields of history of science and science studies as an expression of our collective dissent and our refusal to allow our field to be managed and appraised in this fashion. We have asked the compilers of the ERIH to remove our journals’ titles from their lists.
Neil Barton (Transactions of the Newcomen Society)
Robert Fox (Notes & Records of the Royal Society)
Michael Hoskin (Journal for the History of Astronomy)
Nick Jardine (Studies in History and Philosophy of Science)
Trevor Levere (Annals of Science)
Bernie Lightman (Isis)
Michael Lynch (Social Studies of Science)
Peter Morris (Ambix)
Iwan Rhys Morus (History of Science)
Simon Schaffer (British Journal for the History of Science)
