Monthly Archives: August 2020

Case Study Amberley, Queensland, Australia

Data quality

Part 1: A robust fitness test for maximum temperature data

Dr. Bill Johnston[1]

Synopsis

The first post in this series about the reliability of automatic weather stations (AWS) provides physical justification for using the First Law of thermodynamics as the reference frame against which weather station data may be assessed. While advection warms the air, which increases maximum temperature (Tmax), latent heat removed by evaporation of rainfall is locally cooling. The First Law theorem predicts a dynamic balance between average annual Tmax and annual rainfall, which is evaluated statistically.

Adopting Amberley RAAF as the case study, analysis of Tmax ~ rainfall residuals identified non-rainfall changes that impacted on trend; while statistical significance and variation explained objectively measured overall conformance with the First Law theorem. Analysis of Tmax for six other sites showed the methodology was robust, replicable, widely applicable and therefore useful for benchmarking the operational performance of Australia’s automatic weather station network.

Data are coarse and all sites were poorly documented. Incomplete and misleading metadata, inadequate site control and biased homogenisation methods undermine claims that Australia’s climate has changed or warmed. 

Background

With their current focus on climate warming, climate scientists need robust quality assurance methods for verifying that maximum temperature data (Tmax) used for testing hypothesis about the historic and contemporary climate and its possible changeableness going forward are fit for purpose. Although thousands of datasets are potentially accessible not all are equally useful. The question examined in this series of posts about automatic weather stations is: are their data fit for the purpose of determining change or long-term trends; or are they only useful for monitoring day-to-day weather?

The problem

Since the 1990s Australia’s Bureau of Meteorology has all but abandoned its monitoring network.  Met-offices like at Ceduna, Oodnadatta, Cobar and Mildura sit empty; some like Hobart and Sydney Observatory have been sold or re-purposed, while others like Coffs Harbour, Albany, Canberra, Tindal and Mount Isa have been demolished and removed. Rapid-sampling platinum resistance probes have replaced thermometers, automatic weather stations have replaced observers and 230-litre Stevenson screens used to house instruments, have been replaced by 60-litre ones with that program accelerating over recent years. Due to their restricted capacity to buffer against transient eddies; the 60-litre screens are likely to be biased-high on warm days (Figure 1).

Sensitive instruments housed in 60-litre screens, which are hard-wired to the Bureau’s computer in Melbourne is a potent source of on-going bias. Neighbouring sites that have changed to AWS more or less in unison used to cross-validate and adjust each other’s data in real-time, reinforces rather than adjusts potential errors and embedded anomalies. Its unlikely for example, that data for any sites are truly independent and it is likely that the combined behaviour of infrequently maintained unmanned sites operating with small screens enhances warming. There are other problems too including the propensity for 1-second observations to report random spikes – flurries of warmer air rising from pavements or created by vehicles or people going past. Of the 86,400 one-second values recorded by an AWS each 24-hours, only two of those carry forward as data – the highest is the maximum for the day and the lowest is the minimum.   

Error control using cross validation also requires that neighbouring site data be of acceptable quality and unaffected by parallel non-climate effects, which is impossible to gauge from an office hundreds or thousands of kilometres from the site.

Figure 1. Internal view of the large Stevenson screen at Sydney Observatory in 1947 (Top above, black and white image) and the small screen at Wagga Wagga airport in June 2016 (lower above, colour image). While thermometers in the 230-litre screen are exposed on the same plane, the electronic probe at Wagga Wagga is placed behind the frame about 2 cm closer to the rear of the screen, which faces north to the sun. According to metadata, the 60-litre screen at Wagga Wagga was installed on 10 January 2001 and although thermometers were removed on 28 April 2016 intercomparative data is unavailable.

The burning question is whether data reported by AWS reflect the change in screen size (60-litres vs. 230-litres), behaviour of the electronic instrument (vs. observed thermometers), conversion of electrical resistance to temperature (calibration error), data processing (detection and filtering of erroneous values; averaging; cross-validation); reduced site control and maintenance (grass mowing, cleaning equipment etc.); the climate, or something else.

The First Law Theorem

The First Law of Thermodynamics, which is a universal theorem, is used a reference frame for assessing the fitness of maximum temperature (Tmax) data. Data are expected to behave rationally and not in some random, chaotic way and the methodology outlined in the following posts has been devised to test and evaluate conformance with the theorem. Using Amberley RAAF as a case study, the first paper in this series outlines the physical basis underpinning the approach. Six widely dispersed airport sites were also analysed to replicate and verify that methods are robust and widely applicable. Subsequent posts will evaluate AWS datasets from across Australia and those for sites in the vicinity of capital cities.

As droughts are always hot and dry, and rainy years mild and moist, lack of negative correlation between Tmax and rainfall; low explanatory power (variation explained); confounded signals and weird values – high Tmax in rainy years and vice versa, are causes of concern.

The First Law theorem provides a rational basis for determining if Australia’s automatic weather stations are any good, or if the data they produce consists of incoherent random numbers that bear little relationship to the ‘real’ climate.

An important link – find out more

The page you have just read is the basic cover story for the full paper. If you are stimulated to find out more, please link through to the full paper – a scientific Report in downloadable pdf format. This Report contains far more detail including photographs, diagrams, graphs and data and will make compelling reading for those truly interested in the issue.

Click here to download the full paper including photographs and tables of data

[1] Former Senior Research Scientist. Email: scientistATbomwatch.com.au

On Peer Review

Editorial

by David Mason-Jones

A limited tool at best – never a proof  

Many people may have a mental image of peer review as a process where white-coated scientists re-run the experiment in laboratories or repeat the research out in the real world.

After this they have long meetings in board rooms to discuss the paper and, finally, they confirm that the research and results described in the paper are rock solid and beyond doubt. They then approve the paper for publication in a journal.

The belief in peer review as proof of scientific fact becomes conflated with concepts like truth, beyond doubt, trustworthy, reliable, beyond dispute, the gold standard of science and so on. Sadly, for people who hold these beliefs, peer review is nothing of the sort. 

So, what is peer review?

Peer review is a step in the publishing process where an editor/publisher attempts to weed out papers that are spurious or obviously in error. It is a process where papers are vetted to ensure they present a cogent argument based on recognized scientific analysis. 

At a superficial level peer review can be as simple as a spelling or syntax check, that scientific terms are correctly used and that the paper reads okay. This might sound lightweight for a journal at the forefront of scientific knowledge but presentation is important. 

This part of the process can also include checks on the visual aids used in the paper such as photographs, diagrams, graphs and tables. Are these clear and understandable? Do they support the points made in the body of the paper? Are they relevant or do they just look good?

At a deeper level the peer review process addresses issues like: ‘Is the hypothesis sound and relevant and is it supported by the Introduction?’ Is the logic of the paper sound? Are the methods sound? Do they relate to the hypothesis? Is the argument well constructed, brief and to the point? Is there anything missing in the chain of logic? Does the paper present new information or does it support existing knowledge?

Peer reviewers are appointed by the editor and most scientific journals are quite specialized as to the field of science upon which they are writing, so reviewers need to confirm whether the paper satisfies the scape of the journal.

Given that there are many different journals and editors, it becomes likely across the spectrum that there can be different standards and requirements as to what the editors and publishers require. It is fair to say the term ‘peer review’ has taken on a life of its own, free from real meaning and certainly not a re-running of an experiment.

Not a proof

Peer review is not intended as proof and yet peer review is trotted out all the time as the gold standard of scientific proof.

The ‘proof’ or disproof of the findings of the paper comes when it is put under the blowtorch of criticism of the wider scientific community and, most importantly, when it is put under the blowtorch of Test by Replication. 

Peer reviewers are not claiming they have done the experiment again or made the same observations or done the same maths and obtained the same results. They are not claiming to have replicated the research. This is a really important point.

Behind closed doors

There can be an area of grey when it comes to the transparency of the peer review process because the peer reviewers can choose to do their work anonymously. This conflicts with a characteristic of the scientific method which requires that science be open.

It is true that this anonymity aspect is not always the case and reviewers can choose to be anonymous or open. Despite this discretion, it is common for peer the peer review process to be dome behind closed doors. Where this happens, the peer reviewers are simply put in a position where their background, track record, expertise and even their strongly held opinions cannot openly be taken into account. The peer reviewer may simply be prejudiced against the thesis of the proponent and, by exercising this prejudice discretely and anonymously, can stymie the publication of a paper that is otherwise a valuable contribution.  

Usually a paper would have two or three peer reviewers and the writer – the proponent – has the right to contest the comments coming back to the editor from the reviewers. But where the reviewers remain anonymous, it just makes an open scientific discussion harder. 

Conflating peer review with proof – an example

An example of the conflation of peer review with proof came with an ABC Television Media Watch segment some years ago. Not only were the two concepts conflated, but the high profile Reef scientist, Ove Hoegh-Guldberg, Ph.D., Professor of Marine Science, University of Queensland, gave a ludicrous analogy of the credibility of peer review.

      Hoegh-Guldberg was quoted as asserting that the idea that reef science can’t be trusted because it’s only peer reviewed, was, ‘… just ridiculous.’ 

Hoegh-Guldberg was then cited as saying that peer review was, ‘… the same process we use when we’re studying aeronautics, which produces planes that we travel on …’

On first hearing this it sounds like a compelling analogy. But think it through. There is no way an aeronautics engineer would accept something say, a new alloy for inclusion in an aircraft design, on the basis of peer review alone. The engineers would go for the higher standard of replicability. The alloy would be tested to destruction many times in a process of replication over and again to see if it stood up to the claims made about it.

The engineers would depend on three things; replication, replication, replication.

Replication, not peer review, would be the key to the proof.

Weeding out fraud, spoofing, dumb errors

In its quality control aspect, peer review might be able to weed out totally spurious papers, rants by complete cranks, fraudulent works, mathematical incompetents, mischievous papers by tricksters and even April Fools’ Day jokes. It is not, however, even guaranteed to perform that role very well.   

Vulnerable to spoofing

A recent hoax involving peer review was first reported in ‘The Times’ newspaper in the UK and subsequently reprinted in ‘The Australian’ newspaper on 9th January, 2019. The journalist, Rhys Blakely, is The Times science correspondent and the article in The Australian is headlined, ’Academic faces sack over hoax that fooled academic journals.’ The article outlined the plight of Peter Broghossian, an Associate Professor of Philosophy at Portland State University in Oregon, USA, who is facing the university’s censure over his role in the well-intentioned hoax.

Led by Broghossian, several academic wags spoofed the peer review process and wrote 20 spurious papers, all of which had the trappings of serious scientific papers. They submitted them to academic journals for publication. The papers were meaningless rubbish but that is not the way the peer reviewers saw it. Of the 20 bogus papers put out to peer review, seven were accepted for publication – that is 35% of the total!

Dr. Broghossian and his colleagues were shocked by the ease with which the papers were accepted. Although it may have been a hoax, it confirmed peer review is not the robust gatekeeper of truth that many people believe.

Another example 

This involves papers written by research student, Oona Lonnstedt, who conducted research at James Cook University and gained a Ph.D. at the Australian Research Council Centre of Excellence for Coral Reef Studies. Lonnstedt then went back to Sweden where she did the further research which has tripped her up.

The paper which set the suspicions running was at Uppsala, Sweden. It was about the effect on small fish of ingesting ocean micro plastics and how this affected the ability of the fish to grow, hunt and survive. The paper was published in the high profile journal ‘Science’ in 2016, and was challenged by two concerned scientists within a week of publication. The challenge came after the paper had been peer review.

Lonnstedt’s paper has been examined by Uppsala University and been retracted. The report of the University’s Board for Investigation of Misconduct in Research was published in December, 2017, and found that Lonnstedt had fabricated data. Both Lonnstedt and her supervisor were found to have engaged in research malpractice.

Another body in Sweden, The Central Ethical Review Board, found that Lonnstedt and her supervisor had committed scientific dishonesty.

Again, it is noteworthy that the peer review process did not detect the issue of scientific dishonesty in this case.  

Resplandy et al, 2018

The case of Resplandy et al, 2018, also illustrates the unreliability of peer review. This paper was published in the prestigious journal ‘Nature’ in 2018. (Resplandy et al, Quantification of ocean heat uptake from changes in atmospheric O2 and CO2 composition).

Upon publication it soon became evident to certain readers that there was fundamental flaw in the analysis. The flaw multiplied the degree of uncertainty on the paper so much that, even after a correction had been made, the publishers eventually decided that the paper could not be allowed to stand and was retracted.

It is interesting that the first reader to raise the alarm was not a marine scientist, nor a climate scientist, nor a person with a Ph.D. Rather, he was a analyst in the finance industry and this shows that one should not be intimidated by a host of ‘experts’ who have peer reviewed a paper.

Retraction Watch

Retractions of peer reviewed are not rare and I invite you to visit the Retraction Watch website https://retractionwatch.com which commenced in 2010. At the start, the founders knew that retractions were happening on a regular basis but wondered if there would be enough to sustain a website. They thought they might have been able to identify around 80 cases in the first year but, in the event, they found over two hundred. 

The rate for Retraction Watch has continued unabated. As of January, 2020, the site has reported on 21,792 retractions. All of these had been peer reviewed prior to publication.

Conclusion

The belief in peer review as proof of scientific fact is false. The flip side of this belief – that the lack of peer review shows a paper is untrue – is also false.

Peer review is not a proof of anything and is not intended to be. It is vulnerable to fraud, hoaxes, spoofing and simple errors of maths. Peer review is not a replication of the original experiment or research.