Playing devil's advocate - Why the newly popular literature mapping tools may not be useful for the average researcher
I have become increasing bullish on the rise of what I have called innovative literature mapping tools which have been emerging in the last two to three years thanks to the increasing availability of openly Scholarly metadata (in particular title, abstracts and citation data).
I would identify Barney walker's Citation Gecko released in 2018 as the first of it's class of user friendly tools targeted at researchers that aim to help with literature (mostly narrative review) searches.
You typically enter some relevant seed paper and it will attempt to recommend related papers (typically using citation relationships) using it's built-in index of papers drawn from mostly open sources. Some common tools and the indexes/sources of data used are listed below (as of June 2021).
Tool Major index & Sources used Citation Gecko OpenCitations Index of Crossref open DOI-to-DOI Citations(COCI) & OpenCitations Corpus (OCC) Local Citation Network Microsoft Academic Graph, Crossref, OpenCitations Cocites NIH Open Citation Collection (NIH OCC) Connected Papers Semantic Scholar Open Research Corpus Paper Graph Semantic Scholar Open Research Corpus Inciteful Microsoft Academic Graph,Semantic Scholar (abstracts), Crossref,OpenCitations Litmaps Microsoft Academic Graph (primary source) Citation Chaser Lens.org ResearchRabbit Microsoft Academic Graph
31/12/2021 - Update - with the Closure of Microsoft Academic Graph on 31/12/2021, some of the tools above have been forced to move to new sources. Some of the leading ones include Semantic Scholar and Lens.org. The closest successor to MAG appears to be the new OpenAlex but as of writing is too new that most tools have not implemented it yet.
Besides suggesting or recommending papers, you will also usually get a nice visualization or map of both the seed papers and papers recommended. Another commonality I have identified is that most of these tools are implemented as web services so there is no setup cost to install the software (though some open source ones like Citation Gecko do allow you the option to run a local install)
I think after due consideration, a more accurate terminology for them should be Citation based Literature mapping services but for now I will use the generic name "literature mapping tool".
These tools are very new with the latest ones emerging in 2020. Some of them like Connected Papers seems to have struck a chord. For example, this tiktok video on Connected Papers garnered 2.5 million views! My post on r/Phd on these tools (link to this medium post) got 400+ upvotes and dozens of awards and enthusiastic thanks and various social media accounts of these tools are showing very good responses from their fans. But are such reactions only based on first impressions?
As such I think it is often a good idea when advocating for something new to switch hats and consider why the new shiny idea might not be good. That is why in this post I will play devil's advocate and try to argue why such tools really aren't that useful to the average researcher when doing literature review.
In this long post, I will first describe how such tools typically work and then distinguish them from other similar tools such as "Science mapping tools".
Finally, I will then play devil's advocate on why such tools aren't really useful and possible answers to those doubts.
The typical "new generation" literature mapping tools
To set the scene, I'm going to describe how such tools as Citation Gecko, Connected Papers, Litmaps, Inciteful etc generally work and point out similarity and differences (see list of about a dozen such tools here) .
I think after due consideration, a more accurate terminology for them should be Citation based Literature mapping services but for now I will use the generic literature mapping tool.
If you are already familar with such tools do skip to the section where I play devil's advocate.
Step 1 : Input relevant papers into tool
Relevant known papers often termed seed papers are entered into the tool. The most common way to do this is to search for the paper via the search index underlying the tool to add the paper in.
Searching Connected Paper for one seed paper to start
In most cases, the seed paper is available to be added into the tool, but in rare occasions when your seed paper is obscure and not included in the search index underlying the tool you may need to start with another seed paper.
Sometimes a seed paper may be found in the search index but may not be eligible to be added as a seed paper because the tool's algorithms might not have enough information from the seed paper to recommend anything. This typically happens for very new papers with little citations, or items with minimal metadata.
But it is also getting common to input seed papers in other ways, including supporting exports from reference managers (via bibtex), while some even allow connecting to ORCID profiles (Litmaps) to select papers.
Input methods for Citaton Gecko
While some tools like Connected Papers allow only one relevant seed paper to be entered, it is more common to allow more than one paper to be entered e.g Citation Gecko, Inciteful, Local Citation Network
Multiple ways of inputting "seed" papers in Litmaps
Inputting multple seed papers in Inciteful
Step 2 : Tool recommends relevant papers based on seed papers primarily using citation relationships
Once you have entered papers you find relevant, you tell the system, find me more papers like this but what does it use to do recommendations?
Conventional recommenders you are familar with generally do "collaborative filtering", most famously popularized by Amazon's "users who bought this item X also bought this item Y" feature. In the academic context this is rarely employed because you need a lot of usage data. One example of such a system is Exlibris bX Article Recommender which looks at the articles users try to access across link resolver sessions across hundreds of institutions which allows it to eventually do "users who look at X also look at Y". However other than Google Scholar which as a mass of users it seems unlikely collaborative filtering can work for most other players.
The other method recommenders use are "content-based". In other words, recommending papers based on similarity in their content. While one can use similarity in titles, abstract, subject or even full-text to do such recommendations, the tools in this list focus on leveraging citations and references.
Arguably, citations and references made are "votes" by authors on what to read, so it also captures part of the collaborative filtering aspects.
While all the tools in this post essentially they try to use the entered seed papers to recommend related papers using citation relatonships they diverage in the way it is done (there are many strategies).
Most of the tools I focus on use traditional citations. In recent years we have seen development of a few tools that consider citation contexts (e.g. "mentioning vs supporting vis disputing", "cites for background vs method vs resulst) such as Scite (which provides simple visualization) and Semantic Scholar and now even Web of Science has a pilot on this. But these tools are still rare.
Many tools like the early pioneer Citation Gecko etc just try to find related papers using direct cited and citing relationships to seed papers with the idea that papers that are often cited by seed papers or cite seed papers may be relevant.
Citation Gecko Recommended papers are the papers most well cited or referenced by seed papers
Others like Papergraph draw a bigger net and try to find papers within "2 hops".
Papergraph finds papers 2 hops away (maximum 20x20)
In fact, it seems fairly for such tools to do a "two hop" rule for papers (with limits of course) as a first cut for the pool of papers to consider before doing their own special sauce to decide which among these papers are best to recommend.
For example Inciteful runs a "depth-2 localized graph" , with a limit of 150k nodes/papers
"Where the node labeled 0 represents the seed paper, the nodes labeled 1 represent the results of the first search, and the nodes labeled 2 represent the results of the second search. In this process we also capture all of the citations between the nodes in level two as well. From here we run a few different algorithms against this graph to find the most interesting papers."
While Litmap FAQ says
"We do a 2º citation network search from your maps. This means we search through the citation network to find the articles connected to your maps by references and citations. These are the 1º citation search results. We then go one step further and find all the articles connected to those 1º articles."
Of course the sky's the limit on the various ways you can use the citation network to construct some measure of "similarity", tools often use say a combination of Co-citation and Bibliographic Coupling techniques like Connected papers while Inciteful using Adamic/Adar and Salton Index methods.
Many tools particularly the newer ones now allow you to add these suggested/recommended similar tools to the existing network and do iterative runs over the same network (e,g, Litmaps and Inciteful) but a few like Connected papers do not.
This is where I think there is scope for interesting ideas.Inciteful throws up different categories of papers including
Similar papers (uses Adamic/Adar index)
“Most Important Papers in the Graph” (based on PageRank)
Recent Papers by the Top 100 Authors
The Most Important Recent Papers
All of which you can add to your seed papers and you can even modify to some extent the SQL rules used to generate this list of papers.
Another interesting idea is Litmaps' explore function that provides a way to expand your map/graph by combining keyword search , with maps of papers already added , as well as adding data filters and paper exclusions.
Litmaps explore function
Much experimental and innovation lies ahead to see what is really helpful, particular since most of these tools with the possible exception of Cocites has not been formally tested for effectiveness.
Note : As mentioned citation relationships are not the only way to find similar papers, and there are some tools that also or only take into account text semantic similarity in abstracts e.g. Open Knowledge Maps, IRIS.ai, Yewno Discover, Ebsco Discovery concept maps etc , but those tools are often more associated with the semantic knowledge graph area which we will not discuss here. (Also most except for Open Knowledge Maps are commercial as creating semantic tools require huge investments in terms of getting access to the full text & processing costs), while citations are now generally open.
There is also a seperate ecosystem around supporting the conduct of systematic reviews and other formal reviews (see survey here and here) e.g ASReview uses active machine learning for screening of papers found in systematic reviews.
Step 3 : They provide a way to visualize papers found in a network
Everyone like visualizations so these tools typically provide a way to visualize papers entered as well as found.
Again Citation Gecko provides a paradigm example, where you literally see the recommended papers in the visualize and you can choose to add them as seed papers and iterate the search further to find more papers.
Even tools like Citation Chaser which let you enter papers (based on Lens.org index) and get cited and citing papers and Inciteful which has a similar model that does not strictly require visualizations have basic visualizations or are planning some. (Of the 8 options listed above only Cocites does not have visualization option yet)
Inciteful visualization
Citation Chaser Visualization
Visualizations I have seen so far tend to denote indvidual papers as nodes and denote edges as simple citation relationships, arranged in a force directed graph.
Of course tools like Connected papers that recommend papers based on their own similarity measures (still based on citation relationships) will have edges based on those measures and not simple citation relationships, hence the tool specifies that they produce a "similarity graph" not "citation graph" but the principle remains the same I think.
Connected Paper's visualization
The other major way of visualizing the network is to adopt a chronological timeline layout with older papers on the left or the top (e.g. Litmaps , Papergraph, Local Citation Network)
Local Citation Network - arranges papers with newest papers on top
Like use of citation relationships to find similar papers, there is likely to be a lot of room for innovation & improvement in visualizations. For example, ResearchRabbit that is now in closed beta uses a column based , Trello like layout to list collections.
ResearchRabit Casacade - demo
Currently, I find Litmaps' execution of visualizations the most intriguing, thanks the ability to create multiple overlapping brightly colored maps. The idea I think is to create maps to represent each cluster of research and check, explore for connections between them.
Step 4 - Additional perks
So you have found some interesting papers now what? Pretty much all the tools will allow you to export those papers in a variety of formats csv, bibtex, RIS etc.
Some may offer additional features including
Alerts for new recommended papers - Litmaps
Find seminal papers and review papers - Connected papers
Embed graph/map - Litmaps
Sharable link for graph/map - Litmaps
Some of this are so useful, I expect they will become standard.
Different tools for different needs
That said, while the text so far, suggests these tools are pretty similar and hence there is one best tool this is probably not true. I see a hint that these tools as they develop might actually start to target different user needs.
After all, there is not just one type of literature review.
In fact, I had a private chat with the developers of one of these tools and he said that these tools probably reflect the literature review styles of the creators which struck me as a pretty insightful comment
For instance a researcher who is totally unfamilar with the literature and want to explore to see what is out there will probably have different needs then a user who already knows mostly what is out there and wants to see if there are any connections he misses.
Comparing say Connected Papers and Litmaps, one can see that the former is probably designed as a low effort investment tool where you enter one paper, look at the recommended papers and mostly be done with it. While you can use other papers found to create more maps, you can't really alter or modify the maps created.
Compare this with Litmaps, which allows you to curate the nodes/papers you want to add, has features that allow you to annotate the map, get updates of new papers and creation of multiple maps to check overlaps etc. All this suggest a tool where you are encouraged to come back over and over again to slowly improve your map.
I expect more ideas to blossom leading to very different tools targeted for different use cases.
Comparison with other similar classes of products - Science mapping tools & recommender features in search engines
In this section, I discuss and distinguish other related tools with what I have been calling "literature mapping (citation based) tools".
The idea of mapping research using bibliometric scholarly data is hardly a new one and there exists a class of tools often dubbed "Science mapping tools" (see for example a survey here and here) that has been around since the late 90s/early 2000s to the 2010s that are optimized for such tasks (as opposed to general tools for network analysis such as Gephi, Pajek, R)
Today such tools include VOSviewer , Citespace, BibExcel, Sci2 etc. See this recent 2020 review of such tools.
Vosviewer Co-authorship network of my institution using Microsoft Academic Graph
However these older tools differ a lot from the the new generation of literature mapping tools though there is obviously some similarity. What are some diferences?
Firstly these Science mapping tools are often very difficult to use. They typically involve a local install (as opposed to a web service) and beyond that are targeted at bibliometric specialists and as such have a lot of technical jargon exposed in the interface.
Starting Screen of Citespace - look at the complexity
Designed as a research tool for bibliometric specialists, they often expose pretty much every possible function leading to a very flexible tool with a lot of bells and whistles.
Citespace options - exposes all the bells and whistles and parameters
Take VOSviewer or Citespace where you can decide everything from what a node represents (paper, author, instititon) to the citation network you use (direct citations, cocitations, bibliometric mapping), to the clustering and layout parameters and options and this is just scratching the surface (e.g. Text mining capabilities).
VOSviewer functions
These tools also tend not to be connected automatically to citation indexes, so a typical workflow is to run a search in a citation index like Web of Science or Scopus to get >100 papers, export and format them in a certain way and import into the science mapping tool and run a visualization so this adds a little to the inconvenience of using them.
While such tools have their place and you often see research papers titled "A Science mapping of <topic x>, they clearly are targetted at a difference audience.
I personally see Science mapping tools used mostly to give you a bird's eye view of an area, like entering a search for "corporate social responsibility" into Scopus or Dimensions, tossing it into a tool like VOSviewer and see what shakes out (via clustering algos). This is a top down approach. Which is a quite different approach from the new literature mapping tools that focus on putting into few key relevant seed papers and growing it from there - a bottom up approach. Both methods might have value.
I personally find them quite daunting to use, particularly many of these tools have hard to use interfaces coupled with less than sensible default settings (It's doesn't help many of these tools are over a decade old). VOSviewer is the only exception to the difficult to use interface in my opinion and even that has jargon that I think intimdates most researchers not into bibliometrics.
Another major difference is these tools (with the exception of Vosviewer and Citespace which are regularly updated) often support only Web of Science or Scopus rather than the newer and broader indexes such as Microsoft Academic Graph, Semantic Scholar Open Research Corpus, Crossref which the newer tools use.
Still I personally think there is potential for what I have been calling literature mapping tools to move towards offering a few more carefully selected options and settings that improve their range.
For example what about visualization of author networks? With the exception of Local Citation Network and the still in closed beta ResearchRabbit, none of the tools offer author network visualizations
Already Inciteful offers a limited amount of this by allowing advanced users to edit the SQL that generates the lists of papers it's recommends.
Editing the SQL rule used to generate list of important papers generated by Inciteful
For example, the auto cluster labels in Citespace strike me as potentially useful.
Citespace autolabels clusters
Many academic search engines like Google Scholar, Semantic Scholar, Microsoft Academic, Meta also provide features where you can "find similar/related papers" next to each paper or allow you to track/follow papers to be updated when "similar" papers are found.
Google Scholar "Related Articles"
Semantic Scholar, Microsoft Academic offers the ability to create Research Feeds based on papers you add/follow etc. Meta has similar function though you add "concepts", "authors" and "journals" not specific papers.
Adding papers to Research Feed in Semantic Scholar
Research Feeds generate "Recommended Research"
Many of these "find familar" features often do not explain even roughly how they work but it is likely they are using similar citation based and/or perhaps text similarity based techniques that we have already seen.
Still these are search engines first and foremost and keyword search is their main function and "find similar" functions play second fiddle. In comparison, Literature mapping tools invert this and their main if not only option is to start from existing papers.
That said we see an interesting tools like Litmaps that combine search with citation connections - suggesting a move towards the search side from the other end.
More generally stand alone academic recommender systems are hardly a new idea, and example of one recent one is R Discover
However they are typically marketed as current awareness services that will inform you when a new similar paper is found. As such they differ from the mapping tools discussed here as
1, They generally lack visualization options
2. They tend to be designed as current awareness tools and typically recommend papers going forward.
Playing Devil's advocate - Why literature mapping tools may not be useful for the average researcher
Argument 1 : Such Literature mapping tools are no better than a researcher doing a keyword search and going through the top N results
While such literature mapping tools seem cool, there is no evidence that it leads to "better results" (however it is defined) than simply doing a keyword search in Google Scholar and going through the results.
Response : While it is true that most of these tools have not been formally tested, there is actual reason to expect that literature mapping tools can actually be effective under certain circumstances.
One of the few tools that has been formally tested is the life science based tool Cocites. In one study, the tool was used to try to reproduce 250 reviews and it was capable of finding a "median of 75% of the articles that were included in the original reviews".
Take the common use case of a novice researcher who isn't very familar with the area and wants to explore the thick literature jungle around the area.
Any experienced librarian or researcher will tell you figuring out the right keyword to use at the start can be tricky and often you are advised to supplement your keyword search with pearl growing techniques of relevant papers and to particularly look at references and citations to minimise the problem of not knowing the right keywords.
It is often believed that pearl growing techniques can help compensate for poor keyword choice, but the evidence from using such techniques as a complementary techinques to keyword searching in systematic reviews has been unclear (see 2017 review). That said most researchers don't have expert domain knowledge (at least at first) when doing keyword searches and are unlikely to do very exhaustive searches so it is likely pearl growing will be helpful.
But when you think about it , these literature mapping tools basically automate this or make this type of pearl growing much easier by leveraging citation relationships.
Some like Citation Chaser basically do it straight and just list all papers cited or referenced by the seed papers, while others do more complicated calculations like Connected paper's similarity metrics that ultimately are also based on citation relationships.
If you take a common keyword like gender diversity and entering it into Google Scholar and look at say the top 30 articles, you are likely to get a less diverse set of results then entering one or few important seed papers and letting say Connected Paper do it magic based on it's algorithm,
Why? The reason I suspect is the way common search engines like Google Scholar work. Even in the case where the right keyword is used, such tools tend to rank results by citation counts.
"... Google Scholar's ranking algorithm for title searches always weighs citation counts heavily, while the algorithm for full text searches weighs citation counts heavily most of the time" (Beel & Gipp 2008)
As such a search in Google Scholar in an area with a lot of papers particular with a generic search term often gives you papers that are most well cited and oldest and they often tend to be in the same area.
While this is useful, often what you want as a researcher in a new area is a sampling of different papers that represent different sub clusters of topics in the area. My initution and experience tells me that tools based on citation relationships are more likely to suggest a diversity of papers that meet this need than results from the top N keyword search as citation relationships (which reflects domain expert's idea of relateness) can connect ideas in a lateral manner beyond literal keywords matches.
Argument 2 : Such tools have unknown biases and may end up reinforcing the matthew effect among other issues
These tools are extremely untested, and given that they pretty much use citations to suggest papers, they may have biases leading to undesirable results e.g. reinforcing the matthew effect.
Elizabeth Gadd's AI-based citation evaluation tools: good, bad or ugly? post provides a good summary of her worries about using tools that use algorithms to make recommendations.
Also how does reproducibility of searches fit into this when using such tools? While most of these tools use open data, most if not all of them are closed source and we do not really know how the algothrim works for most of them.
Response : Indeed tools based on citation relatonships to calculate similarity will definitely help reinforce the matthew's effect to some extent and other biases but the problem is really here!
I would suggest the problem is already here and in fact worse than what these tools could ever cause. Heavily used search engines like Google Scholar are likely to already reinforce the Matthew's effect as results rankings are heavily weighted by citation counts and God's knows how Google's black box algorithms are affecting what is seen and hence cited!
I believe there is some dispute on the overall "Google Scholar effect", while Google Scholar might heavily weight results by citation count, it's huge index provides visibility to far more content than traditional search indexes in the past, which helps counter the matthew effect by surfacing more obscure items (particularly with keywords that result in fewer hits) and leading to more citing of content outside the elite journals.
Further more many of such tools such as Inciteful rely more on bibliometric coupling (similarities in references) than Cocitations, so this reduces the Matthew's effect, though other known biases might still be there in the algothrim. For example, items which have minimal metadata and/or lack of references included might be less often surfaced.
Just to be clear though, I am generally supportive of the concerns about using algothrim based tools but my argument here is that these tools are unlikely to make them much worse given the current status quo.
Besides the algorithm (which may or may not be tranparent depending on the tool in question), another major issue and bias would result from the citation index used.
You might argue that this is inherent in current practices where every keyword search can only find items indexed in the system.
However, when humans manually does citation tracing of a relevant/seed paper, they can and do track down items not found in the original index where they found the seed paper.
While Google Scholar does minimize this problem to some extent with [citations] only entries, I'm not sure if the indexes such as Microsoft Academic Graph, Lens.org or Semantic Scholar that these tools work on handle this as well (I suspect not). The result being researchers who rely on such tools to do citation tracing missing out on relevant items.
Argument 3 : Many of the visualizations of clusters of papers are just nice to look at but not useful or helpful
A common experience running such tools or the older science mapping tool like VOSviewer is using certain settings you can quickly toss in a bunch of papers , click a few buttons and generate a nice looking but complicated network. But then you go so what?
While such tools can generate nice looking visualizations are they really that useful? Do they really provide additional insight or unexpected connections beyond what you already knew? After all, for a tool to deserve the moniker - "mapping tool", the "maps" created should be helpful and not pure decoration.
And if you already didn't know the area, how much do they really help you to see the connections? Even if the tool has accurately clustered the research (unclear how likely it is) chances are you are still going to have to read a whole bunch of papers to understand what those clusters mean. So do you really save time?
Response : I must admit this is an objection I have the most sympathy for. My experience running such tools is that in areas I do know, tools like this often gives me clusters that don't always make sense (Adding to the problem is some tools don't allow me to remove papers they think are similar e.g Connected Papers) or if they do don't tell me much I didn't already know.
Nice visualization - so what?
A lot of these tools may have thrown in visualization as an afterthought because it seems easy to do as the data already exists, once you have generated potentially similar papers.
Still this is my own personal view others may disagree and find the visualizations add value.
Also as I mentioned before this is a brand new class of product where there are many more new ideas to try. Someone might eventually hit on value adding visualizations that could change my mind and might even become standard maps that are shown at conference presentations or included in articles.
And if visualizations aren't useful for you, just stick to using it to explore the literature to find similar papers, similar to various current awareness tools and find familar functions in other tools.
Argument 4 : Many of these tools are new and of uncertain stability, it is risky to rely on them.
A glance at the tools you list, a lot of these tools seem to be by hobbyists or unknown startups with unclear business models or revenue sources (some don't even have much information on the creator).
Many of them are using indexes that are open and just overlaying a index over it, what if those indexes disappear?
All in all, who knows how long they will still be around in a year's time? Should we really start relying on them?
Response : Again another argument I don't really disagree with. While a few of these tools are open source such as Citation Gecko most are closed source so there is every possibility the tool will disappear if the creator loses interest or if the startup folds or pivots.
To be fair though even support by big companies isn't a guarantee as shown by the recent shock announcement by Microsoft that they will discontinue Microsoft Academic Graph that a lot of such tools rely on directly or indirectly.
Some tools affected directly or indirectly by the closure of Microsoft Academic
Also I suspect many of these tools & services are relatively little weight to maintain as long as usage of it's servers arent too high, so they may stick around longer than you expect.
Lastly, because of the way these tools work, even if they shut down the amount of disruptive is minimal as you already benefited from extracting the papers found.
Of course, if you build your whole ecosystem around it , such as heavily investing in Litmaps maps, this can be quite problematic if Litmaps closes in the middle of your phd.
Conclusion
I tried my best to be skeptical of these tools but I'm unsure if I succeeded. On the balance, I think such tools surely can't hurt.
Watching the reaction of Phds to these tools seems to suggest there is a huge appetite for these tools.
For example this tiktok video on Connected Papers garnered 2.5 million views!
Lifesaving tips if you’re writing a Thesis/Paper 😲 ##fyp ##foryou ##foryoupage ##learnontiktok ##studytips ##phd
Still it's unclear if this reaction is based on first impressions and whether they find it that useful or helpful in the short run is unclear.