Earth System Science Data

Review Guidelines – Step by Step

For future reuse and reinterpretation it is mandatory for the user to be assured about research data quality. It is the aim of ESSD to provide the quality assessment for datasets which already reside in permanent repositories.

Thus, when reviewing a paper in ESSD we would like you to review not just the manuscript but more importantly, the dataset itself. For your guidance, a step by step review approach is suggested:

1. Read the manuscript: are the data and methods presented new? Is there any potential of the data being useful in the future? Are methods and materials described in sufficient detail? Are any references/citations to other datasets or articles missing or inappropriate?
   
   

   Is the article itself appropriate to support the publication of a dataset?




2. Check the data quality: Is the dataset accessible via the given identifier? Is the dataset complete? Are error estimates and sources of errors given (and discussed in the article)? Is the accuracy, calibration, processing etc. state of the art? Are common standards used for comparison?
   
   

   Is the dataset significant – unique, useful and complete?




3. Consider article and dataset: Are there any inconsistencies within these, implausible assertions or data or noticeable problems which would suggest the data are in error (or worse). If possible, apply tests (e.g. statistics).  Unusual formats or other circumstances which impede such tests as are usual in your discipline may raise suspicion.
   
   

   Is the dataset itself of high quality?




4. Check the presentation quality: Is the dataset usable in its current format and size? Is the formal metadata appropriate? Check the publication: Is the length of the article appropriate? Is the overall structure of the article well structured and clear? Is the language fluent and precise? Are mathematical formulae, symbols, abbreviations and units correctly defined and used? Are figures and tables correct and of high quality?
   
   

   Is the dataset publication, as submitted, of high quality?

Finally: By reading the article and downloading the dataset would you be able to understand and (re-)use the dataset in the future?

Review Criteria – Rating

Reviewers are asked to decide how well the respective datasets presented by an article and the article itself meet the following criteria (rated 1-4, excellent-poor):

Significance

Is there any potential of the data being useful? This is obviously the most problematic decision to take. There are at least three sub-criteria to evaluate:

• Uniqueness: It should not be possible to replicate the experiment or observation on a routine basis. Thus, any dataset on a variable supposed or suspected to reflect changes in the Earth System deserves to be considered unique. This is also the case for cost-intensive data sets which will not be replicated due to financial reasons. A new or improved method should not be trivial or obvious.


• Usefulness: It should be plausible that the data, alone or in combination with other datasets, can be used in future interpretations, for the comparison to model output or to verify other experiments or observations. Other possible uses mentioned by the authors will be considered.


• Completeness: A dataset or collection must not be split artificially, e.g., to increase the possible number of publications. It should contain all data that can be reviewed without undue multiplication of workload and can be re-used in one context by a reader.

Data Quality

The data must be presented readily and accessible to inspection and analysis to make the reviewer's task possible. Even if a dataset submitted is the first ever published (on a parameter, in a region, etc.), its claimed accuracy, the instrumentation employed and methods of processing should reflect the "state of the art" or "best practises". Considering all conditions and influences presented in the article, these claims and factors must be mutually consistent. The reviewer will then apply his expert knowledge and operational experience in the specific field to perform tests, e.g., statistical tests, and make a judgement whether the claimed findings and its factors - individually and as a whole - are plausible and without detectable faults.

Presentation Quality

It is not expected to receive long articles. Regarding the style, it is aimed that a stereotypical wording will develop, so that unambiguous meaning can be expressed and received without much effort. The article should express clearly what has been found, where, when and how. The article text and references should contain all information necessary to evaluate all claims about the dataset or collection, whether the claims are explicitly written down in the article, or implicit, through the data being published or their metadata. The authors should point at suitable software or services for simple visualisation and analysis, keeping in mind that neither the reviewer nor the casual "reader" will install or pay for it.

Access Review, Peer-Review & Interactive Public Discussion (ESSDD)

Manuscripts submitted to ESSD at first undergo a rapid access review by the editor (initial manuscript evaluation), which is not meant to be a full scientific review but to identify and sort out manuscripts with obvious major deficiencies in view of the above principal evaluation criteria.

If they are not immediately rejected, they will be published on the Earth System Science Data Discussions (ESSDD) website, the discussion forum of ESSD, where they are subject to full peer-review and Interactive Public Discussion.

Peer-Review Completion (ESSD)

At the end of the Interactive Public Discussion, the authors may make their final response and submit a revised manuscript. Based on the Referee Comments, other relevant comments, and the authors' response in the public discussion, the revised manuscript is re-evaluated and rated by the editor. If rated 1-2 (excellent-good) in all of the principal criteria and specific aspects listed above, it will normally be accepted for publication in ESSD. Additional advice from the referees in the evaluation and rating of the revised manuscript will be requested by the editor if the public discussion in ESSDD is not sufficiently conclusive.