Sanli
12 April 2009
Open discussion on science
Tags: Open discussionPosted in Getting published, Web 2.0
Every scientific journal nowadays has a web-listing with a lot of useful links added to each abstract page, like citing and related articles. This features are among commodities for almost any web-publishing service. Many news websites or other political or economic magazines allow, or even actively solicit, comments from their readers on their websites. This feature is (deliberately) absent for scientific publications.
I am wondering, why? I discussed this issue with a few senior researchers and a publisher. They were all against allowing web-comments. That describes why it has not yet happened, but I am not yet convinced that it is impossible. Here, I list a few of their reasons and some thought of my own.
Quality and Credibility: One major concern about web-comments is their quality. The level of discussion must be kept high, unless a good comment will be hidden among thousands of mere “opinions” of “professional comment writers” who comment on everything. Even on very popular blogs, more than fifty comments is often not followed anymore.
I believe this issue can be avoided by a well-designed moderation system. Of course it should be human-based, but web can make it very simple since it does the book-keeping very efficiently. To every section, judges can be assigned very much like the current peer-review system. These judges do not have much to do except for keeping the level of discussion at the professional-level. They do not have to judge the content since the open nature of the discussion will judge it on its own.
Friend circles: What if scientist A gathers some followers who manage to aggressively outnumber any disagreement with the work of A. This is a common act in some forums and social sites.
Well, a new web feature is not meant to cure immoral acts or solve ethical issues. These circles of friends already exist as citation circles, community conferences, or even for grants and hiring. What the web can well do is to help tracking such acts by making these relations visible to everybody.
Credit: If something new and important comes out of a discussion on web, who will get the credit?
This situation, if arises, is not solvable even in group discussions, let alone the world-wide case. It may be the time for scientists to give up a bit of their ego into public’s interest. If a web-discussion makes something possible, which was otherwise not thinkable by each individual party, then let it happen.
Apparently some mathematicians have done it. Read the story here.
Workload: A complain is the workload that answering comments puts on researcher’s shoulder.
I thinks this is just an excuse. Researchers are competing so hard on the visibility of their articles. They go to conference after conference to advertise their work. I think any single comment will make the authors very happy, unless it is from a “longstanding competitor”. I believe, having open discussions will help solving these (rather personal) issues much faster.
As a final remark, I like to mention that scientific thoughts has been discussed since Socrates. Discussion is what all conferences are about. It polishes the current knowledge and makes new ideas. Web can help it to become faster and more widely accessibile.
Follow
13 Apr 2009 10:32, Martin Heimann
Some of the journals of the European Geophysical Union have this, albeit in a limited form: After an “access review” a manuscript gets published on the discussion webpage, where it remains during a “discussion phase”. Anybody can place a comment; these, the formal reviews by the reviewers (selected by the editor), as well as the response by the author(s) are openly displayed. Afterwards, if accepted, the revised paper gets published in the main journal. It is important, however, that all material, i.e. the discussion comments and replies remain frozen, are accessible through the website and are citable. Thus at least the issue of credit for an idea placed in a comment is solved.
This publishing model has become very popular and the EGU is expanding this system to different science fields in their domain. Practice has shown, however, that the number of posted comments beyond the actual reviews remains limited – apparently it takes time to write well-thought comment that will remain in the literature in the future. And of course: even if citable, the comments remain “gray literature”, since they have not been refereed themselves. In a few controversial cases, however, the discussion has been quite lively.
As it is now, the discussion is closed after the paper has been accepted and published in final form. Perhaps one could think of a publishing model that would allow discussion comments also after the final publishing. This would maybe then resemble what you had in mind?
13 Apr 2009 12:57, Sanli
Thank you Martin,
It is a very exciting news for me. Now I am sure that this feature will be available for all journals, in a couple of years. Once a major player provides this option, other publishers have to follow. It is better for the scientific community to take the initiative, before Google does it for them. Google has already both services: Scholar and Discussions. For Google, it is just a matter of clever combination.
The idea of providing access to referee reports is also great. It motivates them to do a good review and rewards them if they have a major contribution to the quality or scientific content of the paper. I am not surprised that not so many comments are posted in average. The same is for published material. The (perhaps) majority will never be cited (excluding self-citation), at least not in the first year of their publications.
A major class of discussion contributors will be the PhD’s who have to thoroughly read the paper (sometime several years after its publication). Many seminal papers are extremely hard to read, and these discussion will be extremely helpful in understanding the paper.
14 Apr 2009 12:53, jacopo
I would be extremely in favour of a public discussion of papers on the website of the journal (especially if someone take some time to send a message to the authors to tell them that someone posted a message there even after years from the publication).
On the other side I’m a bit sceptic about a discussion at the level of peer-review. Ok, I agree that it would be nice for referee comments to be public, but it happens that the authors (or even the editor) would like to keep the results secret until publication. Stealing of ideas is not something unheard of.
16 Apr 2009 22:03, Ad Lagendijk
Sanli,
I think your ideas are indeed very interesting. And yes I am very much against out of practical reasons. It is not my task to as an author to answer to all possible reactions. Basically I do not see how to keep the crap out. And I do not want my articles being polluted by being commented by crackpots. Just go the website of Scientific American. You will find many comments on a thread started by an idiot saying that Cern could explode if they would find the Higgs boson.
18 Apr 2009 21:43, Sanli
Ad,
I still think it is possible to keep the level of comments high enough, to be considered as a legitimate scientific discussion.In fact, I believe the comments will help, directly and indirectly, in filtering the poorly-written and flawed material, which are already published. Even if it helps a few articles to be more understandable, it is an added value and not just pollution. As Martin has said from his experience, the problem is attracting enough comment writers, rather than filtering them. A you have said also in the survival guide, it is only “apocalypse now” or “global warming”, which draws so much attention that may become unmanageable. I think you are giving too much weight to the negative side.
Patrick Johnson just showed me another journals, British Medical Journal (bmj.org), that has allowed what they call Rapid Responses. You can clearly see that the posted material is not crap. Generally I was very impressed by the very advanced web-features of BMJ.
19 Apr 2009 20:54, Bram van Ginneken
You write “this feature is (deliberately) absent for scientific publications” but I’m not so sure about this.
Many journals have mechanisms to post comments and responses, including journals with high impact factors. You mentioned BMJ, other examples include the PLOS journals, e.g. the Responses of PLOS Biology and the Comments in PLOS Computational Biology. Also there it seems that most articles attract few, if any, responses. The responses you do find are mostly very in-depth, similar to old-fashioned Letters To The Editor, and your suggestion is, if I understand your response to Martin correctly, that the discussion should be more informal as well, with questions and answers from and for other researchers struggling to understand the paper. I have not seen any high impact journal having such discussion threads attached to articles. Anybody? The questions I receive about my papers by e-mail are most often about if I would be willing to make data and software available, but I also get questions for clarifications. For one paper I repeatedly got similar questions, so it must have been pretty unclear what I wrote there!
I generally do my best to answer questions, unless I get the feeling that the person asking questions is never going to understand it anyway. I would not mind to post such responses on the web site of a journal if that was possible. I think such mechanisms will become more widely available. Scientific journals have never been great pioneers in using modern technology, so it just takes a while.
20 Apr 2009 1:01, Peter Binfield
For 3 reviews of the post publication commenting activity on PLoS ONE (one of th first journals to provide this functionality as standard) see: . . . . . . . . .
Adie, E., “Commenting on scientific articles (PLoS edition)”, writing in the Nascent blog, February 11th, 2009 (http://blogs.nature.com/wp/nascent
/2009/02/commenting_on_scientific_artic.html) . . .
. . . . . .
Neylon, C., “Can post publication peer review work? The PLoS ONE report card”, writing in the Science In The Open blog, August 27th, 2008 (http://blog.openwetware.org/scienceintheopen/2008/08/27/can-post-publication-peer-review-work-the-plos-one-report-card/) . . . . . . . . . . . . . . .
Singh, D., “Peering into PLoS ONE Comment Stats”, writing in the bbgm blog, August 27th, 2008 (http://mndoci.com/blog/2008/08/27/peering-into-plos-one-comment-stats/) . . .
. . .Peter Binfield (Managing Editor of PLoS ONE)
20 Apr 2009 9:38, Sanli
Bram,
You have a point about the current comments being old-fashioned. Once a new feature is available, it takes some time from the users to get used to its new applications. In your case, which you got questions for clarification, it would have been helpful to have an open discussion available so that you do not have to answer the same question several times.
I am a bit uncomfortable with the fact that your example (PLoS) is also from life sciences. It reminds me again of my colleagues comment that in life sciences, researches actually care about the “result” while in physics community people are more concerned about the “impact” (and whether or not they will get the next grant). I think that has something to do with the competition over the available funding.
An open discussion can criticize, clarify and thus enhance the result and its degree of reliability, while it may reduce the impact.
20 Apr 2009 11:30, Sanli
Thanks Peter,
These are very interesting reviews. Knowing about all these advances makes me think about how it can be best adapted by other journals, rather than when.