Case study

Cochrane Crowd

Citizen science platform – collaborative volunteer effort

Helping to curate and deliver health evidence


Cochrane’s new citizen science platform, Cochrane Crowd, uses crowdsourcing to help identify research needed to produce quality health evidence. More and more health research is published every day and an army of contributors is required to meet the growing challenge of identifying the research needed to produce high quality and up to date health evidence. This health evidence is a vital part of the Cochrane Central Register of Controlled Trials (CENTRAL). 

CENTRAL is the world’s best source of reports of randomised and quasi-randomised controlled trials – a powerful and respected tool to those wanting to enhance healthcare knowledge and decision-making.  It is a vital resource for systematic reviewers and other researchers trying to find evidence. Cochrane’s work is recognised globally as the gold standard for high-quality, trusted evidence-based information.

Cochrane Crowd has over 4000 people helping to identify and describe health research. Each piece of research is looked at multiple times by different people. Over 1 million classifications have been achieved since its launch in May 2016.

Overall objectives of Cochrane Crowd

To meet the growing challenge of identifying the research needed to produce high-quality and up to date health evidence.

To achieve this by creating a global community of volunteers and train them to identify the research needed to support informed decision-making about healthcare treatments.

To continually identify and process research globally adding classifications to CENTRAL thereby ensuring its integrity as a global evidence-base and comprehensive Register of Controlled Trials.

The community has grown to over 4600 volunteers since its launch in May 2016, across 88 countries and over 1 million classifications have been achieved. The plan is to build on this momentum continually.

The role of Metaxis

To develop a bespoke back-end management system to control imports, deduplication and formatting of data for use in CROWD. Specifically, crowdsourcing ‘agreement’ algorithms were developed to manage record screening outputs by users.

To design an easy to use front-end where users would find it effortless to work through training, and find the record screening intuitive with lots of online help and guidance from anywhere in the system. Being reliant on volunteers, the system had to be appealing and nurture volunteers regardless of their level.

Integrating a ‘weighting’ process using a range of values into the solution automates decision-making by volunteers – determining whether a citation should be included or not is core to the solution.  Each user is assigned a level which weights their decision.  Depending upon that weighting decisions are verified by others before being accepted – this ensures solution integrity.

The solution is user feature-rich and includes:

  1. Interactive training – 20 practice records highlighting what to reject or accept
  2. A quick reference guide as a reminder of what is eligible or not
  3. Helpful highlights based on commonly used words or phrases indicating whether the record might be one to reject or accept.  Speeds up screening.
  4. A ‘Help me decide’ feature that takes the user through a series of questions to help them reach a decision
  5. The ability to screen records of interest which prioritises the records the users screen based on their preferences
  6. Add own highlights – users can highlight certain words to aid their screening process
  7. Feedback – helpful metrics on performance, reduces isolation in this online environment and aids learning, keeping the user engaged

In summary, the CROWD solution is versatile, intuitive and streamlines the arduous task of screening citations. It also includes rich management tools – statistics, graphs, user performance matrix, audit trails and progress reports which ensures continuous enhancement of the solution itself and engagement with the thousands of volunteers signing up to help.

Key benefits:

  • A streamlined way of identifying reports
  • Improved brokering and synthesising of health information globally
  • Capturing, curating and classifying of evidence as it is produced
  • Being able to harness volunteers with no previous experience in evidence production and build a viable, trained community to assist in this important work

About Metaxis Ltd

Metaxis is a software development company based in the UK specialising in content management solutions and workflow control software. Our solutions support digital collaborative working environments where multiple users, regardless of their global location, can access and interact with high quality, relevant data and information in a meaningful and productive way.

Metaxis provides bespoke solutions rooted in best practice Agile methodology. Its software design principles – rapid behaviour-driven development, high performance, close client collaboration and ease of use – underpin the company’s business ethos.  Metaxis makes short work of Big Data.

Metaxis Ltd, Elmbank Court, Main Road, Curbridge, Witney OX29 7NT, UK  Tel: +44 (0)1993 771740