Modern Australian Magazine
Men's Weekly

.

Your Guide to Finding the Right GP: What Perth City Doctors Offer Today

Choosing a General Practitioner (GP) is one of the most important health decisions you’ll make. Luckily, Perth’s vibrant CBD now hosts a new ge...

Why Every Mining Operation Needs a Robust Safety Management System

Mining is one of the backbones of the Australian economy, particularly in Western Australia. Back in 2019-20, mining contributed 10.4% of Australia...

Australian Classic Literature Enjoys Resurgence

Welcome back to the good old days of storytelling! As the modern world becomes increasingly more demanding, returning to childhood favourites offers...

Building a Governance Model for Headless Content Management at Scale

Image by pch.vector on Freepik There's never been a better time to implement a headless content management system (CMS) to gain the flexibility and ...

Understanding Trade Insurance: Essential Protection for Businesses

Image by Drazen Zigic on Freepik In the current economic environment, trade insurance is an important element for companies trading both locally an...

Choosing the Right Timber for External Cladding

Timber cladding is one of those finishes that pulls double duty: it makes a building look warm and welcoming, and it quietly shields the frame from ...

Top Services Offered by Diesel Mechanics in Brisbane

Keeping a diesel vehicle running at its best takes more than the occasional oil change. When you invest in regular specialist care, you protect the ...

Top 5 Benefits of Hiring Professional House Removalists

Moving day should feel like the start of a new adventure, not the end of your patience. Yet once the settlement papers are signed and the champagne ...

Navigating the Digital Landscape: Managed IT Solutions and IT Services in Townsville

As technology advances at an unprecedented pace, companies must adapt to embrace the transformation ahead. With an evolving technology landscape, mana...

Types of Catering You Should Consider for Your Next Event

Choosing the right type of catering service can elevate your event from ordinary to unforgettable. Whether it’s an elegant wedding, corporate func...

Understanding the Benefits of Split System Installation for Your Home

Climate control is essential to maintaining comfort in your home, especially during the extreme temperatures that many regions face. Whether you’r...

Best Aluminium Window Sliding Designs for Natural Light and Airflow

Bringing natural light and airflow into a space is one of the most efficient ways to create a healthy and comfortable home. In Sydney and across Aus...

When to trust (and not to trust) peer reviewed science

  • Written by Merlin Crossley, Deputy Vice-Chancellor Academic and Professor of Molecular Biology, UNSW
When to trust (and not to trust) peer reviewed science

The article is part of our occasional long read series Zoom Out, where authors explore key ideas in science and technology in the broader context of society.

The words “published in a peer reviewed journal” are sometimes considered as the gold standard in science. But any professional scientist will tell you that the fact an article has undergone peer review is a long way from an ironclad guarantee of quality.

To know what science you should really trust you need to weigh the subtle indicators that scientists consider.

Read more: Why I disagree with Nobel Laureates when it comes to career advice for scientists

Journal reputation

The standing of the journal in which a paper is published is the first thing.

For every scientific field, broad journals (like Nature, Science and Proceedings of the National Academy of Science) and many more specialist journals (like the Journal of Biological Chemistry) are available. But it is important to recognise that hierarchies exist.

Some journals are considered more prestigious, or frankly, better than others. The “impact factor” (which reflects how many citations papers in the journal attract) is one simple, if controversial measure, of the importance of a journal.

In practice every researcher carries a mental list of the top relevant journals in her or his head. When choosing where to publish, each scientist makes their own judgement on how interesting and how reliable their new results are.

If authors aim too high with their target journal, then the editor will probably reject the paper at once on the basis of “interest” (before even considering scientific quality).

If an author aims too low, then they could be selling themselves short – this could represent a missed opportunity for a trophy paper in a top journal that everyone would recognise as significant (if only because of where it was published).

Read more: Not just available, but also useful: we must keep pushing to improve open access to research

Researchers sometimes talk their paper up in a cover letter to the editor, and aim for a journal one rank above where they expect the manuscript will eventually end up. If their paper is accepted they are happy. If not, they resubmit to a lower ranked, or in the standard euphemism, a “more specialised journal”. This wastes time and effort, but is the reality of life in science.

Neither editors nor authors like to get things wrong. They are weighing up the pressure to break a story with a big headline against the fear of making a mistake. A mistake in this context means publishing a result that becomes quickly embroiled in controversy.

To safeguard against that, three or four peer reviewers (experienced experts in the field) are appointed by the editor to help.

The peer review process

At the time of submitting a paper, the authors may suggest reviewers they believe are appropriately qualified. But the editor will make the final choice, based on their understanding of the field and also on how well and how quickly reviewers respond to the task.

The identity of peer reviewers is usually kept secret so that they can comment freely (but sometimes this means they are quite harsh). The peer reviewers will repeat the job of the editor, and advise on whether the paper is of sufficient interest for the journal. Importantly, they will also evaluate the robustness of the science and whether the conclusions are supported by the evidence.

This is the critical “peer review” step. In practice, though, the level of scrutiny remains connected to the standing of the journal. If the work is being considered for a top journal, the scrutiny will be intense. The top journals seldom accept papers unless they consider them to be not only interesting but also water tight and bullet proof – that is they believe the result is something that will stand the test of time.

If, on the other hand, the work is going into a little-read journal with a low impact factor, then sometimes reviewers will be more forgiving. They will still expect scientific rigour but are likely to accept some data as inconclusive, provided the researchers point out the limitations of their work.

Knowing this is how the process goes, whenever a researcher reads a paper they make a mental note of where the work was published.

Read more: What was missing in Australia's $1.9 billion infrastructure announcement

Journal impact factor

Most journals are reliable. But at the bottom of the list in terms of impact lie two types of journals:

  1. respectable journals that publish peer reviewed results that are solid but of limited interest – since they may represent dead ends or very specialist local topics

  2. so-called “predatory” journals, which are more sinister – in these journals the peer review process is either superficial or non-existent, and editors essentially charge authors for the privilege of publishing.

Professional scientists will distinguish between the two partly based on the publishing house, and even the name of the journal.

The Public Library of Science (PLOS) is a reputable publisher, and offers PLOS ONE for solid science – even if it may only appeal to a limited audience.

Read more: Universities spend millions on accessing results of publicly funded research

Springer Nature has launched a similar journal called Scientific Reports. Other good quality journals with lower impact factors include journals of specialist academic societies in countries with smaller populations – they will never reach a large audience but the work may be rock solid.

Predatory journals on the other hand are often broad in scale, published by online publishers managing many titles, and sometimes have the word “international” in the title. They are seeking to harvest large numbers of papers to maximise profits. So names like “The International Journal of Science” should be treated with caution, whereas the “Journal of the Australian Bee Society” may well be reliable (note, I invented these names just to illustrate the point).

The value of a journal vs a single paper

Impact factors have become controversial because they have been overused as a proxy for the quality of single papers. However, strictly applied they reflect only the interest a journal attracts, and may depend on a few “jackpot” papers that “go viral” in terms of accumulating citations.

Additionally, while papers in higher impact journals may have undergone more scrutiny, there is more pressure on the editors and on the authors of these top journals. This means shortcuts may be taken more often, the last, crucial control experiment may never be done, and the journals end up being less reliable than their reputations imply. This disconnect sometimes generates sniping about how certain journals aren’t as good as they claim to be – which actually keeps everyone on their toes.

While all the controversies surrounding impact factors are real, every researcher knows and thinks about them or other journal ranking systems (SNP – Source Normalised Impact per Paper, SJR – Scientific Journal Rankings, and others) when they are choosing which journal to publish in, which papers to read, and which papers to trust.

Read more: Science isn't broken, but we can do better: here's how

Nothing is perfect

Even if everything is done properly, peer review is not infallible. If authors fake their data very cleverly, for example, then it may be difficult to detect.

Deliberately faking data is, however, relatively rare. Not because scientists are saints but because it is foolish to fake data. If the results are important, others will quickly try to reproduce and build upon them. If a fake result is published in a top journal it is almost certain to be discovered. This does happen from time to time, and it is always a scandal.

Errors and sloppiness are much more common. This may be related to the increasing urgency, pressure to publish and prevalence of large teams where no one may understand all the science. Again, however, only inconsequential mistakes will survive – most important errors will quickly be picked up.

Read more: Not just available, but also useful: we must keep pushing to improve open access to research

Can you trust the edifice that is modern science?

Usually, one can get a feel for how likely it is that a piece of peer reviewed science is solid. This comes through relying on the combination of the pride and the reputation of the authors, and of the journal editors, and of the peer reviewers.

So I do trust the combination of peer review system and the inherent fact that science is built on previous foundations. If those are shaky, the cracks will appear quickly and things will be set straight.

I am also heartened by new opportunities for even better and faster systems that are arising as a result of advances in information technology. These include models for post-publication (rather than pre-publication) peer review. Perhaps this creates a way to formalise discussions that would otherwise happen on Twitter, and that can raise doubts about the validity of published results.

Read more: Bored reading science? Let's change how scientists write

The journal eLife is turning peer review on its head. It’s offering to publish everything it deems to be of sufficient interest, and then letting authors choose to answer or not answer points that are raised in peer review after acceptance of the manuscript. Authors can even choose to refrain from going ahead if they think the peer reivewers’ points expose the work as flawed.

ELife also has a system where reviewers get together and provide a single moderated review, to which their names are appended and which is published. This prevents the problem of anonymity enabling overly harsh treatment.

All in all, we should feel confident that important science is solid (and peripheral science unvalidated) due to peer review, transparency, scrutiny and reproduction of results in science publication. Nevertheless in some fields where reproduction is rare or impossible – long term studies depending on complex statistical data – it is likely that scientific debate will continue.

But even in these fields, the endless scrutiny by other researchers, together with the proudly guarded reputations of authors and journals, means that even if it will never be perfect, the scientific method remains more reliable than all the others.

Authors: Merlin Crossley, Deputy Vice-Chancellor Academic and Professor of Molecular Biology, UNSW

Read more http://theconversation.com/when-to-trust-and-not-to-trust-peer-reviewed-science-99365

Holidays & Travel

Why Personalised Ornaments Make the Perfect Christmas Gift

In a season brimming with sentiment and tradition, gift-giving is one of the most cherished ways to show we care. And while store-bought items can be thoughtful, there’s something especially...

Slowing Down in Nature: My Experience on Orchid Classic Cruise in Lan Ha Bay

Why Orchid Classic is a Top Choice for Lan Ha Bay Lan Ha Bay, a hidden gem adjacent to the more famous Ha Long Bay, offers travelers a quieter and more...

How to Reduce Turbo Lag with an Upgraded Mazda BT-50 Exhaust System

Are you struggling with turbo lag affecting your driving experience?  Did you know upgrading your Mazda BT-50 exhaust system could be the key to unlocking smoother acceleration and enhanced vehicle performance? Exhaust...

Surprise Party Ideas That Will Leave Your Guests Speechless

Throwing a surprise party is one of the most exciting ways to celebrate a special occasion. The thrill of keeping it a secret combined with the joy of watching the...

The Most Scenic Sailing Destinations in Australia and New Zealand

Australia and New Zealand are home to some of the world’s most breathtaking sailing destinations. With vast coastlines, pristine waters, and diverse landscapes, these countries offer incredible opportunities for sailors...

A Guide to Finding Caravans for Sale for Your Australian Road Trip

From the rugged Outback to stunning coastal drives, there's no better way to explore Australia than in a home on wheels. If you're considering joining the caravan community, you're likely...

Must-Have Outdoor Gear for Your Next Trip

 Introduction Getting ready for your next outdoor adventure but not sure what gear makes the cut? Selecting quality equipment optimized for the conditions helps ensure fun and safety during backcountry trips...

5 Innovative Caravan Accessories to Enhance Your Travel Experience

Caravanning offers a unique and liberating way to explore the world around us. Whether you're a seasoned traveler or embarking on your first adventure, having the right accessories can significantly...

Conventional Tent or Roof Top Tent? Which Is Better for Camping in Australia?

For camping in Australia, it depends on factors like terrain and climate. A roof top tent is sturdy in rugged areas and elevates you above ground risks. In flat places like parks...

Fashion & Beauty

The Best Nail Art for Short Nails

Short nails often get overlooked in the world of nail art, but they’re actually the perfect canvas for stylish, sophisticated designs. Whether you prefer bold patterns, minimalist details or soft...

Davines Sustainable Haircare: Revolutionizing Eco-Friendly Beauty Solutions

Davines is a well-known brand in the world of haircare, known for its high-quality products that deliver exceptional results. Its commitment to sustainability and unique haircare philosophy sets Davines apart...

High Visibility Work Shirts: A Must-Have for Workplace Safety

Workplace safety is a necessity. Workers from construction sites to manufacturing units should be visible for their well-being and they have it by wearing high visibility work shirts. Yes, when...

Polo Shirts: Wardrobe Essential for All

Polo shirts have started the journey as sportswear. But now, these are regarded as the best workwear. Also, people love to wear them for personal gatherings and more. These are...

Biz Collection Workwear: Premium Uniforms for Australian Professionals

In this modern era, the presentation of the team matters the most in the growth of companies. The professional appearance attracts clients’ attention and doing the deal becomes easy. The...

Functional and Fashionable: The Rise of Ladies Workwear in Australia

A perfect change in the Australian workforce has been happening in recent days. Women are contributing a lot in the male-dominated workplaces. They are doing excellent there. So, the Australian...