Web of Science and Scholarcy adds citation context features with a twist
Web of Science - Pilot- enhanced cited references
In the recent years I have been blogging quite a bit about the idea of citation contexts (or citation intent or citation sentiment or...) in journal articles. While definitions and concepts might vary a little, the idea is not to simply just count citations but also try to figure out what they mean.
To do that you obviously need the sentence with the citation to analyse, typically called the citation sentence.
The definitions for citation sentence or citation context vary slightly but it usually refers to the text sentence containing the reference in the full text. Some definitions would include the whole paragraph or n (where n=1-3) sentences before and after the citation sentence.
scite which probably has the largest database of such data defines
Citation Statements (sometimes known as citances) are the sentences surrounding a citation or reference within a publication.
and Citation Context
which includes not just the Citation Statement but also one sentence before and after that Citation Statement.
In this blog post I will recap the features available in two major systems - Semantic Scholar and Scite that are cross disciplinary + provide citation context features on a large scale.
I will then go on to describe how Web of Science and Scholarcy have recently started to include similar features but with a twist.
What can we do with citation statements or context?
Back in 2018, in "My look at some interesting discovery ideas, trends and features for academic search" under "Doing more with citation relationships" I identified work around citation statements as a potential exciting discovery trend.
Indeed for years there has been quite a lot of academic discussion about how citation statements or context could help with discovery tasks.
The general idea is that the text around the citations is rich in context and provide clues on the citation intent of the author and could be mined by sentiment analysis or trained via various machine learning methods to classify them into different citation types or categories (see also idea of ciTO or Citation Type Ontology).
I personally also like the idea of doing keyword searching restricted to match within citation statements or context, something scite offers under citation statement search as discussed in past posts
However academic discussion is all well and good, but most of them studies were over relatively small sets of data or on particularly subsets that were readily available e.g. (Open Access subsets, as you need full-text to extract citation statements!) How did it work at scale?
I identified some of the first such experimental tools online in 2018 including SciRide Finder and the now defunct R-factor. As noted the first allowed you to search and match keywords within citation statements and the later classified citations and tells you if a paper supports or refutes the paper it is citing.
However both tools were pretty experimental and were never meant for production use.
I believe the first two major systems to make large scale use of Citation Statements were Semantic Scholar (2018) and the already mentioned scite (2019).
My blog has covered scite and Semantic multiple times including
Below shows a rough comparison of various tools and services that categorize citations
Tool Size (as of March 2022) Citation types Others Further background Semantic Scholar 200 Million papers, citation statements - unknown 1. Cites Background
2. Cites Method
3. Cites Results Only identifies Highly Influential Citations Help
Research paper scite 127 Million papers, citation statements 900 million 1. Mentioning cite
2. Supporting cite
3. Contrasting cite Also detects retracted papers and allow citation statement searches Help
Research Paper Elicit.org Unclear data comes from Semantic Scholar 1. "Possible critiques" This uses GPT-3 to look for possible critiques of papers from Semantic Scholar citation intents Web of Science 179 million papers, citation statements - unknown, is beta 1. Background
2. Basis
3. Discussion
4. Support
5. Differ
Unlike the other tools, this does citation context of references (backwards citations) rather than forward citations
Pilot programme - first started in July 22 2021 notes but had the "compare"
type. Feb 18 2022 release split it into "Support" & "Differ" Details Scholarcy No index (as of now), users extract from full-text uploaded 1. Builds on previous work
2. Differs from previous work
3. Confirms previous work
4. Counterpoints previous work Extractions from pdf uploaded. Unlike the other tools, this does citation context of
references (backwards citations) rather than forwards citations
The two pioneers, Semantic Scholar and scite classify citations in quite different dimensions. The former focuses on the purpose or intent of citing - whether a citation is for background, a cite of the method or a cite of the results.
From the horse's mouth - Semantic Scholar does
Background citations provide historical context, justification of importance, and/or additional information directly related to that which exists in a cited paper.
Method citations use the previously established procedures or experiments to determine whether the results are consistent with findings in related studies.
Result citations extend on findings from research that was previously conducted.
Semantic Scholar - cites method, results, background
In addition, Semantic Scholar tries to determine "highly influential cites"
Semantic Scholar labels "influential cites"
scite on the other hand focuses on trying to figure out whether the findings or results of the citing work provides evidence for or against the findings in the cited work. This often interpreted as whether the citing work agrees or disagrees with the cited work.
However, scite claims that it is not determining mere sentiment (agreement and disagreement can be sentiment without supporting evidence) but the deep learning model
is trained to identify three categories of citation statements: those that provide contrasting or supporting evidence for the cited work, and others, which mention the cited study without providing evidence for its validity.
Essentially it seems to me scite's contrasting and supporting cites basically make up Semantic Scholars's "cite results".
scite contrasting cite in the results section
You can't get the exact equivalent of Semantic Scholar "cite methods" in scite but you can get an approximation of this in scite by filtering to citations made in the method sections as those are likely cites for method reasons.
Also in my last blog post, I briefly mentioned how Elicit could find possible critiques of each paper, which is a kind of citation type (possibly sentiment based on GPT-3 interpretion of Semantic Scholar citation statement. This doesn't seem to be a major feature in Elicit however.
Elicit finds possible critiques - a kind of citation context
Backward citation/Reference context - Web of Science and Scholarcy
Thus far, we have looked at citation intent or statement applied on the citations to the target paper. In other words, for each target paper we look at, we see the citation context of papers citing the target paper.
It is possible to do the reverse, for a given target paper show all the references in the target paper and look at the citation types/context for those. ie Why did target paper reference/cite those papers?
From the technical point of view, even though scite and Semantic Scholar display citation context of the cited papers, they process the data by looking at all the references of papers they can get full text on (open access and/or partnerships with publishers), index the results but display results by cited papers. In fact scite does have a references view (example)...
And indeed, Clarivate's Web of Science has also started to do a pilot along these lines
For the July 22, 2021 release, they announced a pilot study on "Enriched cited references" that classifies citations based on "evaluating the author’s exact words in the sentence containing the mention as well as the sentence before and after"
They started off with the following categories
Background—previously published research that orients the current study within a scholarly area.
Basis—references that report the data sets, methods, concepts and ideas that the author is using for her work directly or on which the author bases her work.
Compare—references that the current study’s results can be compared to; the author could be extending on a prior concept or method or confirming past findings or refuting past findings.
Discuss—references mentioned because the current study is going into a more detailed discussion.
but by Feb 18 2022 release split "Compare" into "Support" & "Differ" which seems to mirror scite's supporting cite and constrasting cite.
Web of Science, currently visualizes it's citation classifications in a interesting diagram as seen below
Pilot Enhanced Citation References in Web of Science (click to enlarge)
You get a sense of not just how many of each citation types you have but also where (in terms of position in the paper) each reference in the paper.
Again at the risk of repeating this is not the same as what you get in scite or Semantic Scholar. It is listing all the references (backward citations) of the chosen paper rather than the reverse showing all the citations (forward citations) of the chosen target paper.
And the way it works is when you mouse over the references, it gives you an overview or citation sentence of the reference
Notice that the citation sentence here is extracted from the target paper and not from full text from citing papers.
UPDATE OCT 2022
As of Oct 2022, Web of Science has further evolved this feature. Firstly, it is now easier to find records which has this "enhanced cited references" with a filter.
More interesting some records are starting to classify cites (not references) of the record you are looking at.
Some records will now have a "Citing items by classification" panel and if you click on any of these classifications you will see the cited items!
This is what you see if you click on support citations - 7 citations Web of Science classified as "support"
All of the citing articles are from 2021 and 2022, so perhaps this is a feature that works only for the newest citing articles?
Another similar development has been with Scholarcy which is known as an article summarizer. The main use of Scholarcy is for the user to upload pdfs of articles, which then summarises important information of the article.
Among some of things it extracts includes
Key Concepts (linked to Wikipedia)
Tables (exported as csv)
Figures
Methods
Results
Limitations
Funding information
Ethics section
References (downloadable in bibtex format).
and you can view or export this data in nice flashcard like formats.
Scholarcy has a ton of cool features from being able to export results into latest knowledge management tools like Obsidian, Roam to create Knowledge graphs to turning your thesis into a powerpoint presentation as well as automatically generating a Literature Matrix into excel.
Scholarcy has always played well with other cutting edge tools, so for example when you look at the extracted references you have the option to go from there to services like Google Scholar, scite, or look up full text with Unpaywall or Libkey, allowing you to do citation chaining.
You can even run the reference through Scholarcy again to get its main findings.
Scholarcy extracted references works well with external services like scite badges. You can even run the reference (open Access only) past Scholarcy to get main findings of the referenced work
One might think that with scite badge intergration, Scholarcy wouldnt have its own ambitions to extract citation sentiment or context.
You would be wrong. Scholarcy recently added "Comparative Analysis" with analyses and classifies or the references in the uploaded full text.
Comparative analysis feature in Scholarcy extracts citation types from full-text
Scholarcy has 4 main citation types
Builds on previous work - method related
Differs from previous work - method related
Confirmation of previous work - results related
Counterpoints previous work - results related
Again you can see parallels here with already mentioned work. Builds on previous work is intended to be for borrowing of methods so this is similar with Semantic Scholar "cites for method"
Differs from previous work would be the opposite of Builds on prevous work, I gather this is the case where the authors state specifically they are using different methods from a similar study.
The other two Confirmation of previous work and Counterpoints previous work map nicely to scite's supporting cite and contrasting cite.
Again I must caution this isn't quite the same feature as what Semantic Scholar or scite offers since you are looking at references of each paper uploaded into Scholarcy. You are not looking at citations of the paper you uploaded to Scholarcy.
But this is of course similar to the pilot in Web of Science which we just discussed, where this is listing the references of the target paper (which you uploaded).
In other words, you are looking at all the references (backward citation) of the paper you uploaded to Scholarcy, rather than the citations to the paper you uploaded to Scholarcy.
So when Scholarcy notes a Confirmation of previous work , this is simply saying among all the references in the uploaded paper, one reference in the paper has the citation type "confirmation of previous work".
Is analysing and displaying "reference context" by paper useful?
In general, it feels the way scite and semantic scholar do things are more natural, you want to see who cited your works and how they cited it.
Note : scite can also offer a reference view though it does not show the citation sentence or context.
scite showing references of target paper
The way Web of Science and scite works is the reverse. It takes any paper with full text it has access to and classify the references out from that target paper.
Is this useful? Probably less so if the target uploaded paper is your own work. Presumably you already know why you cited all the papers in the reference section without ML/AI telling you.
Still this can be helpful if you use this on papers you are not familar with. With this new feature in Scholarcy, you can see which of the say 50 references of a paper (or even thesis) might be worth chasing down. e.g If you want to see which of the papers was referenced with the intent to say their findings agree with the referenced/cited work, you can just look only at references with labels Confirmation of previous work .
Another cool feature is that Scholarcy may allow you to check quickly if a reference is not a misquote or wrong attribution of views.
The comparative analysis will allow you to see not just why the reference was made but also the exact words used in the citation sentence.
Say Scholarcy shows in the comparitive analysis, that the paper you are currently looking at has a reference to a paper that basically has the same results and hence Scholarcy classifies it has Confirmation of previous work.
Still the author citation sentence might not be right or tell the full story. How do we know the author didn't make up or misunderstand the work they quoted?
With the Scholacy plugin one can click on the badge and quickly get the main findings of the referenced work and compare with what the citations said. This might allow you to see quickly if the main finding of the referenced papers says what the author claims it does.
Conclusion and sources
The main issue with working with citation context is the lack of data of course. Are there open data available? Either with the sentences themselves or the classification/content?
The Microsoft Academic Graph used to offer citation contexts but has since closed.
It's successor - OpenAlex inherits this data but going forward (2022 onwards) this seems to have been frozen.
Semantic Scholar provides Citation intents in API only not data dump
Beyond these two sources, Scholarly Knowledge Graphs (e.g. Open Research Knowledge Graph (ORKG) ) could potentially have this information if the relationship was captured in properties etc.
For example, Wikidata seems to cover some CiTO intentions but there are only <500 of such statements. See below.
CiTO intentions in Wikidata
All in all it seems citation context features are starting to emerge in academic discovery systems, though as a new area, it is likely there is going to be a lot of experimentation to study what is really useful to users.