When to trust (and not to trust) peer reviewed science

  • Written by Merlin Crossley, Deputy Vice-Chancellor Academic and Professor of Molecular Biology, UNSW
When to trust (and not to trust) peer reviewed science

The article is part of our occasional long read series Zoom Out, where authors explore key ideas in science and technology in the broader context of society.

The words “published in a peer reviewed journal” are sometimes considered as the gold standard in science. But any professional scientist will tell you that the fact an article has undergone peer review is a long way from an ironclad guarantee of quality.

To know what science you should really trust you need to weigh the subtle indicators that scientists consider.

Read more: Why I disagree with Nobel Laureates when it comes to career advice for scientists

Journal reputation

The standing of the journal in which a paper is published is the first thing.

For every scientific field, broad journals (like Nature, Science and Proceedings of the National Academy of Science) and many more specialist journals (like the Journal of Biological Chemistry) are available. But it is important to recognise that hierarchies exist.

Some journals are considered more prestigious, or frankly, better than others. The “impact factor” (which reflects how many citations papers in the journal attract) is one simple, if controversial measure, of the importance of a journal.

In practice every researcher carries a mental list of the top relevant journals in her or his head. When choosing where to publish, each scientist makes their own judgement on how interesting and how reliable their new results are.

If authors aim too high with their target journal, then the editor will probably reject the paper at once on the basis of “interest” (before even considering scientific quality).

If an author aims too low, then they could be selling themselves short – this could represent a missed opportunity for a trophy paper in a top journal that everyone would recognise as significant (if only because of where it was published).

Read more: Not just available, but also useful: we must keep pushing to improve open access to research

Researchers sometimes talk their paper up in a cover letter to the editor, and aim for a journal one rank above where they expect the manuscript will eventually end up. If their paper is accepted they are happy. If not, they resubmit to a lower ranked, or in the standard euphemism, a “more specialised journal”. This wastes time and effort, but is the reality of life in science.

Neither editors nor authors like to get things wrong. They are weighing up the pressure to break a story with a big headline against the fear of making a mistake. A mistake in this context means publishing a result that becomes quickly embroiled in controversy.

To safeguard against that, three or four peer reviewers (experienced experts in the field) are appointed by the editor to help.

The peer review process

At the time of submitting a paper, the authors may suggest reviewers they believe are appropriately qualified. But the editor will make the final choice, based on their understanding of the field and also on how well and how quickly reviewers respond to the task.

The identity of peer reviewers is usually kept secret so that they can comment freely (but sometimes this means they are quite harsh). The peer reviewers will repeat the job of the editor, and advise on whether the paper is of sufficient interest for the journal. Importantly, they will also evaluate the robustness of the science and whether the conclusions are supported by the evidence.

This is the critical “peer review” step. In practice, though, the level of scrutiny remains connected to the standing of the journal. If the work is being considered for a top journal, the scrutiny will be intense. The top journals seldom accept papers unless they consider them to be not only interesting but also water tight and bullet proof – that is they believe the result is something that will stand the test of time.

If, on the other hand, the work is going into a little-read journal with a low impact factor, then sometimes reviewers will be more forgiving. They will still expect scientific rigour but are likely to accept some data as inconclusive, provided the researchers point out the limitations of their work.

Knowing this is how the process goes, whenever a researcher reads a paper they make a mental note of where the work was published.

Read more: What was missing in Australia's $1.9 billion infrastructure announcement

Journal impact factor

Most journals are reliable. But at the bottom of the list in terms of impact lie two types of journals:

  1. respectable journals that publish peer reviewed results that are solid but of limited interest – since they may represent dead ends or very specialist local topics

  2. so-called “predatory” journals, which are more sinister – in these journals the peer review process is either superficial or non-existent, and editors essentially charge authors for the privilege of publishing.

Professional scientists will distinguish between the two partly based on the publishing house, and even the name of the journal.

The Public Library of Science (PLOS) is a reputable publisher, and offers PLOS ONE for solid science – even if it may only appeal to a limited audience.

Read more: Universities spend millions on accessing results of publicly funded research

Springer Nature has launched a similar journal called Scientific Reports. Other good quality journals with lower impact factors include journals of specialist academic societies in countries with smaller populations – they will never reach a large audience but the work may be rock solid.

Predatory journals on the other hand are often broad in scale, published by online publishers managing many titles, and sometimes have the word “international” in the title. They are seeking to harvest large numbers of papers to maximise profits. So names like “The International Journal of Science” should be treated with caution, whereas the “Journal of the Australian Bee Society” may well be reliable (note, I invented these names just to illustrate the point).

The value of a journal vs a single paper

Impact factors have become controversial because they have been overused as a proxy for the quality of single papers. However, strictly applied they reflect only the interest a journal attracts, and may depend on a few “jackpot” papers that “go viral” in terms of accumulating citations.

Additionally, while papers in higher impact journals may have undergone more scrutiny, there is more pressure on the editors and on the authors of these top journals. This means shortcuts may be taken more often, the last, crucial control experiment may never be done, and the journals end up being less reliable than their reputations imply. This disconnect sometimes generates sniping about how certain journals aren’t as good as they claim to be – which actually keeps everyone on their toes.

While all the controversies surrounding impact factors are real, every researcher knows and thinks about them or other journal ranking systems (SNP – Source Normalised Impact per Paper, SJR – Scientific Journal Rankings, and others) when they are choosing which journal to publish in, which papers to read, and which papers to trust.

Read more: Science isn't broken, but we can do better: here's how

Nothing is perfect

Even if everything is done properly, peer review is not infallible. If authors fake their data very cleverly, for example, then it may be difficult to detect.

Deliberately faking data is, however, relatively rare. Not because scientists are saints but because it is foolish to fake data. If the results are important, others will quickly try to reproduce and build upon them. If a fake result is published in a top journal it is almost certain to be discovered. This does happen from time to time, and it is always a scandal.

Errors and sloppiness are much more common. This may be related to the increasing urgency, pressure to publish and prevalence of large teams where no one may understand all the science. Again, however, only inconsequential mistakes will survive – most important errors will quickly be picked up.

Read more: Not just available, but also useful: we must keep pushing to improve open access to research

Can you trust the edifice that is modern science?

Usually, one can get a feel for how likely it is that a piece of peer reviewed science is solid. This comes through relying on the combination of the pride and the reputation of the authors, and of the journal editors, and of the peer reviewers.

So I do trust the combination of peer review system and the inherent fact that science is built on previous foundations. If those are shaky, the cracks will appear quickly and things will be set straight.

I am also heartened by new opportunities for even better and faster systems that are arising as a result of advances in information technology. These include models for post-publication (rather than pre-publication) peer review. Perhaps this creates a way to formalise discussions that would otherwise happen on Twitter, and that can raise doubts about the validity of published results.

Read more: Bored reading science? Let's change how scientists write

The journal eLife is turning peer review on its head. It’s offering to publish everything it deems to be of sufficient interest, and then letting authors choose to answer or not answer points that are raised in peer review after acceptance of the manuscript. Authors can even choose to refrain from going ahead if they think the peer reivewers’ points expose the work as flawed.

ELife also has a system where reviewers get together and provide a single moderated review, to which their names are appended and which is published. This prevents the problem of anonymity enabling overly harsh treatment.

All in all, we should feel confident that important science is solid (and peripheral science unvalidated) due to peer review, transparency, scrutiny and reproduction of results in science publication. Nevertheless in some fields where reproduction is rare or impossible – long term studies depending on complex statistical data – it is likely that scientific debate will continue.

But even in these fields, the endless scrutiny by other researchers, together with the proudly guarded reputations of authors and journals, means that even if it will never be perfect, the scientific method remains more reliable than all the others.

Authors: Merlin Crossley, Deputy Vice-Chancellor Academic and Professor of Molecular Biology, UNSW

Read more http://theconversation.com/when-to-trust-and-not-to-trust-peer-reviewed-science-99365

Australia Today

Capturing the shadow of Saturn's moon Titan from right here on Earth

David Coward, Associate professor, University of Western Australia

NASA's Cassini spacecraft captures Saturn's largest moon, Titan, passes in front of the planet and its rings.NASA/JPL-Caltech/Space Science InstituteTitan is Saturn’s largest moon, and it is more like a planet than a moon in many respects.It has a thick atmosphere as well as wind, rivers, lake...

Read more

Battle scars reveal the life of 'Mephisto', a WW1 German tank from a century ago

Michael Westaway, Senior Research Fellow, Australian Research Centre for Human Evolution, Griffith University

Mephisto after its capture in France by the Australian 26th Battalion.Queensland Museum, Author providedWhat can we possibly learn from the archaeological study of a World War I battle tank? Quite a lot, it turns out, when the attention is devoted to a rare German-built A7V Sturmpanzerwagen tank kno...

Read more

are phone-obsessed teens at greater risk of ADHD?

Hannah Kirk, Research Fellow in Psychology, Monash University

Hyperactivity and inattentiveness doesn't mean you have ADHD.Ant RozetskyThis week, news outlets across the world reported a study from the Journal of the American Medical Association (JAMA) linking digital media exposure to reduced attention spans and attention deficit/hyperactivity disorder (ADHD)...

Read more

Is the 'Zero Hour' youth climate march a turning point, or more of the same?

Marc Hudson, PhD Candidate, Sustainable Consumption Institute, University of Manchester

This weekend, young climate activists will march through Washington DC’s National Mall. The rally, part of the Zero Hour movement, is another sign of the concern and dismay felt by young people after 30-plus years of prevarication and hesitation by their elders. Just as young Americans are rea...

Read more

Michelle Grattan on the Mayo byelection and crossbenchers in the parliament

Michelle Grattan, Professorial Fellow, University of Canberra

Centre Alliance candidate Rebekha Sharkie is polling strongly ahead of next weekend's byelection in the South Australian seat of Mayo.AAP/Roy VandervegtMichelle Grattan discusses the week in politics with Director of the University of Canberra’s Institute for Governance and Policy Analysis, Ma...

Read more

do neckties reduce blood supply to the brain?

Steve Kassem, Postdoctoral fellow, Neuroscience Research Australia

The fashion advice is generally to tighten ties so they're tight but not too tight.from www.shutterstock.comNews reports about a study from Germany may provide the ultimate excuse for men to dress more casually for work, finding neckties reduce blood supply to the brain.The study showed that wearing...

Read more

Holidays & Travel

Luxury Camping Adventures Around the World

Perhaps in direct response to the dominance of technology in the modern world, many travelers are feeling a strong draw towards nature. While some are able to fully embrace their...

Dredging works underway Plantation Island

Tourists and workers travelling to popular tourist destination Plantation Island, Fiji, are set to benefit from safer access when marine channel dredging is completed at the end of June 2018...

Top 5 Tips for Traveling With Dogs

Travelling with your dog can be thrilling, but sometimes it can turn into a real nuisance. Unless you know how to properly restrain the pooch, whether it’ll be able to...

Top Island Destinations for 2018

When we picture island destinations in our minds, we often imagine a tropical beach of white sand, a sun lounger laid out in the shade of a line of palm...

Why Car Rental Excess Insurance is a Smart Choice

Some car rental agencies include basic damage cover in the standard rental fee, with the option to add on liability reduction, or excess insurance. The addition of excess insurance normally...

More Than a Third of Australians Feel 'Too Busy' With Work to Take a Holiday

New research from InterContinental Hotel & Resorts has unveiled a large number of Australians (32 percent) are simply too busy with work to take some much-needed rest and relaxation. Almost...

Matakauri Lodge and Mahu Whenua Lodge

TRIED, TRUE AND SOMETHING NEW  A luxurious accommodation package at Matakauri Lodge + Mahu Whenua Lodge South Island, New Zealand  Available 1 May, 2018 - 30 September, 2018   Central Otago, NEW ZEALAND –...

Heaven is the Adelaide Hills

Looking for a quick getaway? Enjoy food and wine? How about stunning countryside? Look no further! The Adelaide Hills has something for everyone. We flew to Adelaide for a weekend-getaway...

5 Amazing Australian Holiday Ideas

Australia is one of the most fantastic places in the world to travel, explore, and witness all of the beauty that Mother Nature has to offer. With so much rich...

Fashion & Beauty

Groom To Do List

Sorry Aussie Men, you no longer just pay for the alcohol, flowers and marriage license. Times have certainly changed. Wedding Expert shares what Australian Grooms are now expected to do before...

Simple Secrets To A More Confident Smile

We all know just how important first impressions are. After a handshake, what’s one of the first things people will notice about you? It’s your smile. A friendly smile is...

Improve Your Body Confidence and Feel Great

No matter how good you feel about yourself, sometimes you just can’t help but see a few little flaws that seem gigantic to you and make you feel self-conscious. Whether...

First-Timer Tips For Your First Lip Filler Appointment

Getting injectable lip fillers can be a quick and effective way to plump up your smile with minimal drama. They’re a cost-effective and safe way to boost the natural shape...

The Most Common Skin Conditions and How to Deal with Them

We often underestimate the importance of our skin, but as the largest organ in our body, and our primary layer of defense, it is constantly affected by many environmental factors...

Beach Babe Basics: Everything You Need to Look Stylish at the Beach

Summer is right around the corner, and this year, we all should make the best of it. Right now, it is not just about looking stylish on the streets, it...

HEALTH

Eye Health Dos and Don'ts

Most people take their eyes for granted and spend little to no time thinking about their eye health until they notice something wrong. This leads to plenty of common and...

Non-invasive Cosmetic Treatments

In a world where everyone is chasing perfection, plastic surgeries have become more and more common to the point where it’s odd if you don’t get one during your life...

My Health Yoga

Carrie-Anne Fields of My Health Yoga has taken her Gold Coast yoga teacher training worldwide. She started her health business 20 years ago aged 22 and has grown into a...

Fit Body 101 - Increase Your Fitness Effectiveness

You have to find some space for physical exercise in your weekly schedule in order to live a good life. Embracing this way of life is very beneficial because it...

The World of Dentistry: 5 Most Common Dental Procedures in Australia

Are you dissatisfied with the state of your pearly whites? Do you often experience discomfort and pain while eating and drinking? Well, be sure that you’re not the only one...

How You Can Conquer Your Dental Anxiety

Does the thought of going to the dentist fill you with dread? Well, you will be pleased to know that you are not the only one. A large number of...

LIFE STYLE

Dog Flea Allergy

Severe itchy skin disease in dogs causes Flea bite hypersensitivity. The itch is as a result of an allergic reaction to the saliva of the flea which is injected into...

5 Ways to Turn Your Bedroom into a True Zen Retreat

The bedroom should be our little sanctuary where we come to relax after a long day of work. However, in order to be able to unwind all the accumulated stress...

Things to Consider when Deciding on your Wedding Flowers for the Big Day

Choosing the wedding flowers is a complicated process where you have to take into account many elements, from the wedding dress to the wedding theme. You can discover in this...

Renovating tips for your house

There are very few property owners who move into a house where they don’t want to make any changes. Alterations are almost inevitable every time you move-house, but alterations can...

Tips for Decluttering Your Yard

Decluttering probably sounds like a tedious job, but it’s essential if you want to have a nice, well-decorated backyard. Bad organization habits and hoarding can eventually lead to clutter, and...

Dry Carpet Cleaning Vs Steam Carpet Cleaning: Which one to Choose?

A clean carpet not only improve your interiors, increases your comfort around the office and home but also beneficial for your health. This is why it is necessary to choose...

FOOD & DINING

Morning greens: the breakfast skincare regime

It’s safe to assume that most Aussies fail to eat enough greens. Although this was momentarily resolved with the rise and popularity of the green juice movement, the lack of...

Best Hidden Gem Hong Kong Restaurants

You finally find yourself enjoying the endless beauties of China and this fascinating city. In a place with so much variety, all your senses will be wide awake, and your...

JOHNNY DI FRANCESCO NAMED OFFICIAL AMBASSADOR OF EAT STREET MELBOURNE

Johnny Di Francesco, world champion pizzaiolo and owner of Melbourne’s Gradi Group of restaurants, is excited to announce he has been named official ambassador for Eat Street Melbourne 2018. As Ambassador...

The best Mother's Day gift EVER!

Whether it’s to celebrate a special occasion, or just wind down at the end of a long day, there’s nothing much better than pouring yourself a glass (or two…) of...

Sing for Your Supper at Chargrill Charlie’s

To celebrate the final curtain call of the Barden Bellas, Chargrill Charlie’s is giving you the chance to Sing for your Supper! Show off your best acapella vocals to win...

MAKE IT WINE TIME, ANY TIME

Whether it’s to celebrate a special occasion, or just wind down at the end of a long day, there’s nothing much better than pouring yourself a glass (or two…) of...