We have performed 100s of literature reviews over the last few years across medical device and pharma, and here’s what we’ve found. We’ve done them by hand, we’ve tested the best systematic literature review software available, and ultimately we’ve built our own.
If you’re reading this, then you or your team have conducted systematic reviews (and will again soon) and more or less hated the experience.
If you’re happy spending hours clicking through excel, careful not to overwrite old cells (and fixing them when you do), our hats off to you. Soldier on. Read this for fun, and then get back to your spreadsheets.
But if you’re like us, you always have an ear to the ground about how to make a time-consuming systematic review process better. You do the mental math of what a 20% efficiency improvement could bring your work life, or your team’s productivity and stars flash in your eyes.
Just think of all the accolades! Medical writing and clinical evidence manager of the year saves team productivity by figuring out a new approach. Audit feedback reduced to nothing, team productivity hours explode, and all of your absurd deadlines set by executives (who have no idea how long these documents take to write) were miraculously met.
And then the jaded and deeply experienced manager part of your brain wakes up, knocks the empty bottle of whiskey over while reaching to smash the alarm clock, and weighs into the conversation.
New processes mean new validation to be had, extra time to ‘train and onboard’ (whatever that entails), and an entire stack of new stuff to do plopped on top of an already way-to-long pile of things to do.
Ugh. Well said, grumpy manager persona.
Here’s the thing, both of you are right in almost all cases. But today we have something to show that even the jaded grumps of your subconscious will sit quietly through.
Something that’s easy to implement (set your team up in 10 minutes), and can benefit your team to the tune of hundreds of hours per year (more depending on how much literature you’re cranking through).
Ready? Let’s get into it.
The Breakdown of a Systematic Literature Review
There are a lot of ways and different types of reviews one can perform on clinical literature. From meta-analysis, full on systematic reviews, to just pulling up some research articles on your drug/device to have some backup for a section in a document you’re writing.
Just so we’re speaking the same language, here’s an overview of our systematic review process. A video run through of this can also be watched here.
- Figuring out what you search (pick your terms, databases etc.)
- Determine the purpose of your review, your inclusion and exclusion criteria, research questions
- Running your searches and compiling all of the results
- Removing duplicates
- Reading through all the abstracts/titles and excluding articles that aren’t useful (with a reason!)
- Going out and collecting all of the full texts to read through
- Pulling out the useful information from each article. We call this data extraction.
- Compiling it all into tables that can be used in a report (Prisma, Summary of Search, Analysis Extraction Tables etc.)
- Producing an audit log of every decision you made, so that an auditor can easily follow your process (key word here is easily).
Phew, still with us? If you skimmed over these bullets and still are then you are probably nodding your head thinking your own process is more or less the same approach.
Now the only question left is… how long does your current systematic review process take you?
Time to Review – The Key Factor
In 8 years of constant phone calls, introductions and conferences, I’ve never met a bored regulatory or quality professional. Almost everyone is overworked (and underappreciated, but that’s for a different article).
As teams continue to stretch, and budgets continue to get slashed, companies have a renewed vigor for finding ways to maximize their existing staff. Long gone are the days of hiring armies of consultants to throw at intimidating projects.
It’s not secret that organizing and reviewing thousands of abstracts to distill into a useable report is a labor intensive task. And we aren’t at the point yet where an AI overlord can simply ‘handle it’ for you.
Time to review adds up. An end to end project might be 40+ hours of review time. That’s ‘focused’ review time, not in addition to 5 meetings a day, surprise audits and various fire fighting with your other product’s submissions.
The Hard To Measure Items
Not everything can be easily measured in minutes with a stop watch. What about risk of mistake? What about the time you spend redoing your summary tables because marketing wants to add a new indication so you had to append your search?
What about the back forth of revisions with your client (or manager), and having to make endless tweaks to your review to get the thing signed and submitted?
It’s tough out there the clinical evidence world, we know it.
For this intangibles, we only can draw from our own experience. The reviews, the heartache, the humanity! OK, it’s not usually that bad,, but you don’t want to be around when someone asks your team to pull up and update an old review that was buried under 2 years of emails, partial drafts and revision requests. Those were some dark days, at least before our tools took shape and started proving their mettle.
While not easy to quantify, these more ‘workflow’ related challenges can add exorbitant amounts of time onto a review cycle. In a future article, we’ll take a stab at quantifying these in a way that’s presentable.
How to Evaluate Software Tools for Systematic Reviews
Evaluating new software can be a massive pain, especially for an already overloaded team. Alas, it must be done if you’re to have any hope of freeing up additional time and boosting productivity. As someone who’s reviewed hundreds of products across all kinds of industries, I can help you out with a buyer’s checklist to make your evaluation process as painless as possible.
Does this tool do Everything I need It To?
Obvious question first, but you’d be surprised how many folks will start their search for new products without being able to articulate how they’re currently solving the problem. If you have a few extra weeks handy to go back and forth on ‘discovery calls’ so that vendors can figure out what you’re doing, great. If you don’t have the time and want to greatly speed up the buying process, have some ready-to-share documentation from the start.
Most of the time if you come with a checklist of features/functions, a single call (or even trip to a good website) is all it will take to figure out if there’s any hope of a product being able to work for you.
Do We Have the Budget for This?
Not always the easiest thing to find out, as a lot of enterprise companies will try and make you sit through 2-3 calls before sharing even the most basic of pricing numbers. For this I have two recommendations:
- Shop around online, and use your referral network to get some ideas of pricing (chances are someone in your network has used the product before).
- Insist on at least a basic idea of pricing on your first demo call. If you can give a rough range of users needed, and your use case the sales rep should be able to at least give you a pricing range of what costs could look like. Don’t waste your time on a second call if your budget is 1/10th what an average contract looks like.
Is the On-boarding Process Simple, or an Intrusive Pain?
Another little detail that’s conveniently forgotten by many a software-salesperson is how long it will take to get you and your team set up and actually using the product. Some systematic review tools (like Endnote) you can download and start using immediately. Others (like DistillerSR from evidence partners), can take weeks of meetings, infrastructure set-up/testing, and workflow cross-training calls.
Both scenarios have their pro’s and cons, weight this factor according to your team set-up.
Will this be a nightmare or a pleasure to maintain documentation for?
There’s nothing like spending months implementing a new tool and training your team only for the majority to leave the company after holiday bonus season. Will it be a snap to get your new hires up to speed and using the shiny new software tools you purchased? Or will you be back to square one with training and knowledge transfer?
Some software tools are designed to be intuitive, with beautifully written suites of documentation and ‘plug and play’ SOPs to re-use within your organization. Many software tools are not at all. If your workflows are based around common knowledge tools like Excel, this is far less of a concern.
Components of Incredible Systematic Review Software
Aside from general advice on buying software products, let’s talk about how to evaluate something you would specifically use to manage systematic reviews, shall we?
The research world is full of different and interesting tools that support a variety of research methods. If your goal is truly to conduct systematic reviews, some tools will be far more useful than others that still fall under the research category.
Here are my top checklist questions for evaluation review software in specific.
Does it support systematic reviews
Systematic reviews can be different from a meta-analysis or even a ‘literature review’ even though the terms are often used interchangeably. [ TK link to different article here ]
Data Extraction Features (Critical Appraisal)
The biggest component of a systematic review is actually extracting data from each relevant study in order to compile a useful and actionable table. This is pulling info on safety, performance, and study/population out of the Full Text of the article chosen.
Without a great interface for data extraction, this process can be incredibly time consuming and error prone.
Reference Management Software Capabilities
After your systematic reviews are complete…what happens to the literature? In most companies, all that effort and analysis gets buried in a sharepoint graveyard never to be fished out again. The next go-around, you run the same searches, review the same articles (again) and even re-buy the full texts because it’s easier than hunting down the previous review data. A great systematic review system has this solved as well, because updating reviews in the regulatory context are just as important as conducting them in the first place.
Intuitive and Streamlined Screening Process
The screening process is another potentially big time suck. I don’t care how smart or experienced you are, everyone’s eyes get blurry after reading hundreds of abstracts crammed into an excel spreadsheet. It’s usually at this point you either make a mistake, throw your laptop through a plate glass window, or start browsing jobs in a more realistic industry.
Collaborative Enabled
Not all existing tools are web based software. Many are still ‘on hardware’ that you install (like most Endnote and Excel versions). Does your team need to see and work on the same reviews together? Would shared access to all citations be useful? Lack of good collaboration features can take a good product and make it unusable for your team.
Protocol Development
One of our favorite time saving features is enabling the user to manage and create their ‘search protocol’ to parallel their reviews. The protocol is critical in regulatory submissions, and also a quick way to make sure that reviewers are understanding of the systematic reviews goals and parameters. Regardless of tool used, your systematic review process needs to be ‘easily replicated’ by an auditor or reviewer. This is where tools (like ours) that generate a protocol based on your review automatically can save time and reduce any confusion/errors down the line come your technical audits.
The Competitors – Systematic Review Tools
In this article, we compare the most prevalent tools for systematic literature review with our own solution, CiteMed.io. Any medical writer worth their salt has extensive experience (and maybe a few horror stories) in the former. Having been around only 6 years, we often get a lot of comparison questions (read skepticism) that boil down to “is it worth the effort and cost to switch to a more dedicated tool versus what I’ve already been doing?”. We hope to let the numbers speak for themselves.
Disclaimer: Not all of these tools we have used personally in our workflows. A lot of the feature based info was pulled off of marketing materials, and the pros/cons through our survey-based data from our substantial network of medical writers.
Citavi
Citavi is a reference manager, in the same realm as Endnote and not a true systematic review tool. But we mention it here as it’s found a way into many a clinical team’s internal processes.
Pros
- Been around a long time, and refined/familiar interface
- Reasonable pricing for individuals and teams
Cons
- not set up for systematic reviews at all
- no real time-saving features built in
DistillerSR from Evidence Partners
Distiller is one of the biggest names in the systematic review industry. Being around a long time, and having a large team they’ve parlayed some great experience into a very robust enterprise solution. Not without their drawbacks (that we’ll explore later), but a comparison article wouldn’t make sense without mentioning their offerings.
Pros
- Industry leading and largest
- Robust platform for data extraction and systematic reviews. AI and machine learning features.
- Very customizable for large teams and implementations
Cons
- Leading in costs and expenses for license fees. Cost prohibitive to medium/small orgs
- Long learning curve, and heavy lift to get set up for your own workflows/reviews
Quosa
Oh what happened to Quosa!? We will give them an honorable mention here as their product was absorbed by Elsevier and ultimately shut down. Rest in peace, reference manager software.
Excel
In corner number one, the reigning champion of the world… we have Excel.
There’s not much you can’t do with excel if given enough time and vlookup statements, so for those experienced it’s a completely viable form of performing a systematic review…mostly.
Pros
- Cost: Doesn’t get much better than ‘free with Office’, right?
- Customization: If you can dream it, you can build it in excel. The result might not be pretty, but in terms of raw data processing ability there’s nothing more powerful to manage from a graphical interface.
Cons
- Mistakes and Audit Trails: It’s all too easy to accidentally copy/paste in the wrong place, or delete a cell without noticing. Come audit time it’s very difficult to prove you have checked everything and not many any mistakes.
- Maintainability: Shared spreadsheets have come a long way, but managing something with thousands of citations that grows year after year… among a small or large team quickly becomes unwieldy.
Medboard Systematic Reviews
If there was one company we’d be happy losing business to, it’s Medboard. They have a nice team, and have built a great regulatory intelligence product, and a nice systematic review tool as well as far as we can tell.
Pros
- Literature solution integrated into a regulatory intelligence platform
- Great founders, and well run company
Cons
- Enterprise Pricing models aren’t
- Not designed to support individuals/freelancers
Endnote – Citation Manager for SLR
An old challenger to excel, is the Endnote Citation Management Platform. Old and (more or less) reliable, you’d be hard pressed to find a medical writer who doesn’t spend a good deal of their manipulating citations in Endnote.
Pros
- Workflow Integration: Most writers are already using Endnote everyday, so no additional tools or training is needed to make it work for you.
- Exports and Cite While You Write: The biggest draw of Endnote is being able to reference citations from your library in word documents easily. Exporting and Importing to excel/RIS is also robust and reliable.
Cons
- Team Collaboration: Despite it’s prevalence as an enterprise tool, sharing research libraries among teams is not simple. Reliability (who’s had a crash and lost all their work?) is also something to be mentioned.
- Lacks Systematic Features: While a systematic process can be wrangled with Endnote, much like excel the platform isn’t set up to track your progress and provide an interface optimized for reviewing articles. Outputs are into excel where you need to perform all of your formatting and can be labor intensive.
CiteMed.io – Full Service Systematic Review Software
The new (6 years!) guy on the block is our beloved CiteMed platform. Designed from the ground up with our team of medical writers (performing CERs and Literature Reviews) over the years to make a painful process as smooth as possible.
Pros
- Design: This is a streamlined solution for lit review that took years to perfect. All of the sticky parts of review with the commercial tools have been re-designed and improved on.
- Sharing and Collaboration: Gone are the days of emailing Endnote libraries and ris files back and forth. Literature Reviews can be collaborated on online, and your citation library is fully backed up and accessible anyway. If you still want to use Endnote, importing and exporting a snap.
Cons
- Inertia and Training: No matter how simple we make things to get going (on-boarding a new projects takes only minutes), it’s still a new tool to integrate into your teams process. We all know how difficult it can be to tackle learning a new platform when you’re already drowning in work.
- Fitting into Your Existing Process: Some companies have workflows so entrenched that even the slightest alteration threatens to send your team through months of cascading ‘process update’ calls and procedures. If your team (or company) needs an absolute 1 to 1 fit with your process, it may prove difficult depending on your workflow.
The Results – A Giant Table of Features Comparison
| CiteMed.io | Citavi | Excel | Endnote | DistillerSR | Medboard | |
|---|---|---|---|---|---|---|
| Automatic Search Protocol Generation | Yes | No | No | No | No | No |
| Auto-Search of Literature Databases | 7 Databases Offered | No | No | No | Yes, PubMed | Yes, PubMed |
| Works 'Out of the Box' for Systematic Reviews | Yes | No | No (assembly required) | No (assembly required) | Setup and Training Required | ? |
| Workflow Customization for SLR | Flexible Design - But not 100% Customizable | No | Blank Slate - BUild Your Own | No | Built by their team to your specifications (or build it yourself) | No |
| De-duplication | Yes, across all databases and reference types | Sometimes | No, manual required | Duplicates Identified if matching IDs, but not recorded | Yes - Not Sure if it's across all article types/databases | Yes - Not Sure if it's across all article types/databases |
| Auto-Download Free Full Texts | Yes | No | No | No | Yes | Yes |
| AI Data Extractions | (Optional) | No | No | No | Yes | Not Advertised |
| Robust Data Extraction Features | Yes | No | (assembly required) | Sometimes (Add Extra Metafields) | Yes | Yes |
| Submission-Ready Word Doc Outputs | Yes | No | No | No | No | No |
| Audit-Trail Ready | Yes | No | No | No | Yes | Yes |
| Built-In Citation Manager | Yes | Yes | No | Yes | No | Yes |
| Free-Trial Available | Free Demo/Sandbox With Your Data For Evaluation | Not Available | Not Available | 30 Day License Trial | Not Available | Not Available |
| Pricing | Flexible, Freelancer and Enterprise Packages | Per User Pricing (Medium) | Single License (Cheap) | Single License (Cheap per User) | Enterprise Pricing (Expensive) | Enterprise Pricing (Unknown) |
So What is the Best Systematic Literature Review Tool?
We’ve shown that our CiteMed.io tool blows the current ‘best practices’ out of the water in terms of time efficiency and accuracy (and reduction of headaches!)… but that doesn’t mean it’ll be the right fit for your team and workflows. Both Excel and Endnote have their place (which is why they are used so globally), so with all things it depends.
In the Enterprise space, there are several players that might fit your needs. DistillerSR continues to be the choice for complex workflows and large teams despite it’s price tag, and Medboard has brought a very nice tool and feature set to the table in nice pairing with it’s regulatory intelligence platform.
No matter what you decide, if you’re performing systematic reviews at all, you should be evaluating a more sophisticated tool set. The time savings (not to mention error-reduction) are always worth the upfront set up and training pains for your systematic review process. Not convinced? You can check out our literature review tools comparison study (We compare our approach with manual tools like Endnote and Excel). Download it here TK
A summary In Bullets – The Top Tools for Systematic Reviews
If you want to maximize your time saved doing literature reviews AND manage multiple projects/citations with your team: Consider Checking out CiteMed.io
If you are only occasionally performing adhoc reviews to support your other medical writing tasks, stick with Endnote for easy ‘cite while you write’ features.
If you have no budget for tools, or time to even think about improving your team’s workflow (then stop reading long articles online!), but Excel may be your best bet to maintain contuinity and not spend any extra money.
If you have budget, and a significant complex organizational workflow, enterprise-ready solutions like CiteMed, DistillerSR, or Medboard are your best bet to start comparing in more depth.
And as always if you’re interested in taking the next step to improve this part of your writing process, please take advantage of our free trial/demo environment offer today. Your systematic review team will thank you.
