TECHNICAL REPORT
Systematic scoping review
on social media monitoring
methods and interventions
relating to vaccine hesitancy
www.ecdc.europa.eu
ECDC TECHNICAL REPORT
Systematic scoping review on social media
monitoring methods and interventions
relating to vaccine hesitancy
ii
This report was commissioned by the European Centre for Disease Prevention and Control (ECDC) and coordinated
by Kate Olsson with the support of Judit Takács.
The scoping review was performed by researchers from the Vaccine Confidence Project, at the London School of
Hygiene & Tropical Medicine (contract number ECD8894). Authors: Emilie Karafillakis, Clarissa Simas, Sam Martin,
Sara Dada, Heidi Larson.
Acknowledgements
ECDC would like to acknowledge contributions to the project from the expert reviewers: Dan Arthus, University
College London; Maged N Kamel Boulos, University of the Highlands and Islands, Sandra Alexiu, GP Association
Bucharest and Franklin Apfel and Sabrina Cecconi, World Health Communication Associates.
ECDC would also like to acknowledge ECDC colleagues who reviewed and contributed to the document: John
Kinsman, Andrea Würz and Marybelle Stryk.
Suggested citation: European Centre for Disease Prevention and Control. Systematic scoping review on social
media monitoring methods and interventions relating to vaccine hesitancy. Stockholm: ECDC; 2020.
Stockholm, February 2020
ISBN 978-92-9498-452-4
doi: 10.2900/260624
Catalogue number TQ-04-20-076-EN-N
© European Centre for Disease Prevention and Control, 2020
Reproduction is authorised, provided the source is acknowledged
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
iii
Contents
Abbreviations ............................................................................................................................................... iv
Glossary ....................................................................................................................................................... iv
Executive summary ........................................................................................................................................ 1
Introduction and background ..................................................................................................................... 1
Aims ........................................................................................................................................................ 1
Methods ................................................................................................................................................... 1
Results .................................................................................................................................................... 1
Discussion ................................................................................................................................................ 2
1 Introduction ............................................................................................................................................... 4
2 Background ................................................................................................................................................ 5
3 Goals and objectives ................................................................................................................................... 7
4. Review methods ........................................................................................................................................ 8
4.1. Search strategy and database search ................................................................................................... 8
4.2. Screening and selection of articles ....................................................................................................... 8
4.3. Data extraction and analysis ................................................................................................................ 9
5. Review results ......................................................................................................................................... 10
5.1 Individuals’ preferences for using social media platforms as a source of information on vaccination and
social medias influence on individuals’ perceptions of vaccination ............................................................... 11
5.2 Social media monitoring ..................................................................................................................... 12
5.3 Using social media monitoring to inform vaccination communication strategies ....................................... 29
5.4 Uses, benefits and limitations of social media as an intervention tool in relation to vaccination ................. 33
6. Discussion ............................................................................................................................................... 36
6.1 Use of social media for vaccination information .................................................................................... 36
6.2 Methodologies to monitor social media in relation to vaccination ........................................................... 36
6.3 Review how social media monitoring methods and information gathered from monitoring can be used to
inform communication strategies .............................................................................................................. 41
6.4 Understanding the uses, benefits and limitations of using social media as an intervention around vaccination .... 42
6.5 Limitations of this systematic scoping literature review ......................................................................... 43
7. Conclusions and the way forward .............................................................................................................. 44
References .................................................................................................................................................. 45
Annexes ...................................................................................................................................................... 52
Figures
Figure 1. Prisma flow diagram ....................................................................................................................... 10
Figure 2. Social media monitoring phases ....................................................................................................... 12
Figure 3. Number of articles published by type of social media and by year until 2018 ....................................... 15
Figure 4. Number of articles published by type of vaccine ................................................................................ 15
Figure 5. Number of articles published by country monitored ........................................................................... 16
Figure 6. Number of studies by type of social media monitoring tools used ....................................................... 17
Figure 7. Keywords most commonly used across all studies (>4 use) ................................................................ 21
Figure 8. Sentiment codes used across all studies ........................................................................................... 25
Figure 9. Number of studies using manual or automated sentiment coding, by social media ............................... 27
Tables
Table 1. Inclusion and exclusion criteria for screening of articles ....................................................................... 8
Table 2. Data extraction categories ................................................................................................................. 9
Table 3. Description of social media platforms identified in the scoping review .................................................. 14
Table 4. Different manual browser searches and limitations mentioned by the studies using the tool ................... 18
Table 5. Social media APIs and their limitations mentioned by the stuides using the tool .................................... 19
Table 6. List of codes, definitions and counts for sentiment analysis used in the identified studies ....................... 26
Table 7. Suggestions for increasing presence on social media identified in scoping review studies ..................................... 31
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
iv
Abbreviations
AI Artificial Intelligence
API Application Programming Interface
CDC United States Centers for Disease Control and Prevention
ECDC European Centre for Disease Prevention and Control
HTML Hyper Text Markup Language
HPV Human Papillomavirus
GDPR General Data Protection Regulation
GPS Global Positioning System
LDA Latent Dirichlet Allocation
MMR Measles Mumps Rubella
PHP Hypertext Preprocessor
UK United Kingdom
URL Uniform Resource Locator
US United States
VCP Vaccine Confidence Project
TM
Glossary
Application Programming Interface Software allowing two applications to talk to each other (e.g.
smartphone software sending text/images to the Twitter
database/platform).
Global Positioning System A system of satellites, computers, and receivers able to determine the
geographical location of an object on Earth.
Latent Dirichlet Allocation A generative statistical model (in natural language processing) used for
topic extraction, representation and analysis from large datasets.
Hypertext Preprocessor Refers to how dynamic web pages (php) are created and accessed with
precompiled and pre-processed code linking to databases, so that
accessing them is faster and easier via different browsers.
Uniform Resource Locator A uniform resource locator is the address of a resource on the Internet.
Sentiment analysis A process that uses natural language processing, text analysis and
computational linguistics to identify positive, negative and neutral
opinions from text and social media.
Reach analysis Defined in social media as the number of people that see content - the
greater the reach, the higher number of people that have seen content.
Vanity metrics In social media vanity metrics are measured by engagement
(comments, shares, likes, clicks, and saves), providing information on
how many people are interacting with content on social media
platforms.
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions around vaccine hesitancy
1
Executive summary
Introduction and background
We are living in an interconnected world, where social media have become part of the everyday life of many
individuals around the globe. People use social media to stay connected to friends and family, to share personal
information, views or beliefs, or to seek information and gather other peoples’ advice about certain topics,
including health. These new communication technologies have also facilitated the recent spread of unsubstantiated
negative information about vaccination online, influencing individuals’ views about vaccination, their levels of
confidence in different vaccines and their willingness to be vaccinated or to vaccinate their children. The online
spread of rumours surrounding vaccination, including adverse events following vaccination, has contributed to the
growth of vaccine hesitancy and in some cases may have contributed to disease outbreaks in unvaccinated
populations. However, social media also constitute an opportunity to spread positive messages about the benefits
of vaccination and to restore trust in vaccination. Listening, monitoring and analysing social media conversations
concerning vaccination could help us to understand low vaccination acceptance and provide valuable information to
counteract the spread of rumours and misinformation.
In this report, social media have been defined as not just a means of communication, but also a space in which
people socialise. Social media are therefore seen as online environments or platforms that see interaction
as a
main purpose. This study focusses on social networking sites and content communities, which can be seen as more
relevant in the context of vaccination.
Aims
The aim of this research project is to map, analyse and summarise knowledge and research on social media and
vaccination. The key objectives were to identify preferences for using different social media platforms as a source
of information on vaccination and the influence that social media have on individuals’ perceptions of vaccination; to
identify different social media monitoring methods or tools in the context of vaccination and their strengths and
weaknesses; to review how social media monitoring methods and information gathered from monitoring can be
used to inform communication strategies, and to identify the uses, benefits and limitations of social media as an
intervention tool around vaccination (i.e. to determine how effective social media are as a tool for increasing
vaccination uptake).
Methods
In order to address these objectives, a systematic scoping review was commissioned by ECDC and conducted by
researchers at the Vaccine Confidence Project
TM
[1]. A comprehensive search strategy was developed, reviewed by
librarians, and adapted to different databases to identify peer-reviewed and grey literature published since 2000.
Two reviewers independently screened all articles by title and abstract and then by full text, based on a set of
inclusion and exclusion criteria. All disagreements were resolved by discussion. The articles included were divided
into three groups: a) preferences for using different social media platforms as a source of information on
vaccination and the influence that social media have on individuals’ perceptions of vaccination, b) social media
monitoring and c) social media interventions. Data extraction was performed by four reviewers and followed by a
descriptive analysis and synthesis.
Results
The systematic scoping review identified 115 articles: 13 on individuals’ preferences for using social media as a
source for vaccination information and any influence on perceptions of vaccination; 85 on social media monitoring,
15 on social media interventions, one on both social media monitoring and social media interventions, and one on
both social media interventions and individuals’ preferences for using social media as a source for vaccination
information and any influence on perceptions of vaccination.
Preferences for using different social media platforms as a source of information on vaccination and
the influence that social media have on individuals’ perceptions of vaccination
The 14 studies included in this category found that social media platforms were commonly used as a source of
information for vaccination but that most of the time consulting social media had a negative influence on vaccine
uptake. Population groups in different countries were found to use social media in a variety of ways, with some
groups experiencing more positive influences from social media.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
2
Social media monitoring
The majority of the articles on social media monitoring were published between 2015 and 2018, and are based on
Twitter, YouTube and Facebook. Most of the studies were based on social media monitoring in relation to
vaccination generally, while some studies monitored particular vaccinations, including human papillomavirus (HPV)
and measles, mumps, rubella (MMR) vaccination. The majority of studies involved conducting a manual search to
identify social media posts on vaccination, using the search tools available within the social media networks. The
second most commonly used method for identifying posts was the Application Programming Interfaces (APIs
software allowing two applications to talk to each other), followed by automatic monitoring systems using
commercial software. Most of the keywords used to search for social media posts related to vaccination or vaccine-
preventable diseases, with some studies also including negative keywords, for example side effects. While many
studies only used a small number of keywords, other studies also used hashtags or longer sentences or questions.
Only a very small number of studies analysed the locations of posts, meaning that most of the studies were not
limited to one country only. In most cases, geo-localisation was performed manually, for instance by screening user
profiles, since Global Positioning System (GPS) information is not often available. Furthermore, most of the studies
looked at social media on a continuous basis, extracting data over a period of 13 hours and for up to 16 years.
Studies that were conducted at one specific point in time were mostly studies where a manual search had been
carried out for the data.
Sentiment analysis
1
was performed in almost all studies included in this review, with most of them conducting
manual coding of data into either positive vs. negative or pro-vaccination vs anti-vaccination sentiments. Those
that used automated systems to code sentiments mostly analysed Twitter using different tools to establish
sentiments. Some studies also included other types of content analysis, such as qualitative thematic analyses.
Finally, around half of the studies also analysed reach to understand how social media information is shared. The
studies visualised data in different ways.
Some studies provided recommendations for health authorities and health professionals on how to use social media
monitoring, in particular to start communicating on social media platforms and to use social media monitoring
findings to inform the development of intervention and communication strategies.
Social media interventions
Three types of interventions were identified: social media as a source of information about vaccination; online
group discussions and interactive websites. Most of these interventions were developed and implemented by
researchers in Canada, Germany, the Netherlands, Taiwan and the United States. Three studies evaluated existing
interventions. The effects of the interventions varied and no strong impact was identified overall. This may be due
to the methodological challenge of linking the specific effects and influence of social media to actual behaviour.
Studies measured the effect of social media interventions on knowledge and attitudes concerning vaccination, risk
perception and concerns, intentions of being vaccinated and vaccine uptake.
Discussion
While social media usage has been associated with a negative impact on public views and behaviour concerning
vaccination, it also presents many opportunities. More evidence is needed on which interventions using social
media to address vaccine hesitancy are effective in different contexts. Furthermore, while many studies have been
conducted on social media monitoring around vaccination, they have used different methodologies (e.g. use of
manual tools to retrieve data compared to APIs or automated software; manual versus automated sentiment
analyses) to obtain and analyse data, and these have not been evaluated. There is a need to evaluate different
methodological approaches to better understand what works best and to eventually provide standardised research
approaches to monitor and analyse social media. Furthermore, while the general data protection regulation (GDPR)
may limit social media monitoring to publicly available data, this also highlights the need for more control over
what happens with data collected online. It would be helpful if a code of conduct for ethical use of social media
information could be developed to ensure that those reporting on social media monitoring results adhere to fair
and responsible values.
Social media monitoring is highly dependent on what platforms have to offer in terms of APIs, geo-location data,
and sentiment analysis. To reduce the number of manual searches and analyses, and thereby improve the quality
of social media monitoring, easier ways of accessing data should be developed, whether through APIs or through
computational software. Health authorities and researchers should also reflect on the consequences for research of
the constant fluidity of online information, particularly since several platforms have decided to remove anti-
vaccination content.
1
The process of computationally identifying and categorising opinions expressed in a piece of text, especially in order to
determine whether the writer's attitude towards a particular topic, product, etc. is positive, negative, or neutral (Lexico
Dictionary, Oxford University Press)
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
3
Finally, the purpose and value of social media monitoring should be clearly defined. While some health authorities
and researchers may try to use social media as a proxy for what the public thinks about vaccination, the reality is
often much more complex. It is unclear whether social media users are representative of the general public. Social
media monitoring should therefore be seen as a way of capturing the essence and the movement of online
discourse around vaccination in order to better understand how it can influence public perceptions and decision-
making around vaccination. Such evidence could then inform the development of targeted interventions to restore
public confidence in vaccination.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
4
1 Introduction
Vaccine hesitancy is increasingly being recognised as a growing problem globally. In 2019, the World Health
Organization (WHO) acknowledged that it constitutes one of the ten biggest threats to global health [2].
Confidence in vaccination is complex and influenced by an array of individual, social, and structural factors; it can
also vary depending on the vaccines and the diseases they prevent. While highly context-dependent on the one
hand, there are growing global networks promoting vaccine hesitancy, connecting across countries and
languagesaided by online translation tools and social media [3]. Vaccine refusers are a loud minority and such
clustering can interfere with the immunisation uptake required for herd protection, risking an increase in the
burden of disease [4]. Recent measles outbreaks across Europe [5-8] demonstrate the consequences of non-
vaccination and confirm recent findings that Europe is the region in the world with the least confidence in the
safety and effectiveness of vaccination [9].
Continuous advancements in communication technologies such as social media have contributed to the unmediated
spread of concerns about safety and adverse events following vaccination. New communication technologies allow
sentiments, rumours and beliefs about vaccination to quickly diffuse among networks across the world, influencing
individuals and groups online as they assess risks and benefits of vaccination [10-12]. A number of studies have
reviewed websites and social media for information on vaccination and found that it is of variable quality, with a
predominance of negative or incorrect content that influences perceptions about the risks and benefits of vaccines
[13-16].
However, social media have great potential to contribute positively to health communication by allowing direct
interactions with individuals; enhanced availability, accessibility, and customisation of information; and individual
and policy advocacy opportunities. The monitoring and measuring of content posted and shared on social media
also provides an opportunity to listen to online discourses and develop targeted, audience-focused
communications. There are some limitations to using social media for health communication, relating to quality,
confidentiality, reliability, transparency, sponsorship and privacy concerns [17,18]. Engaging on social media can
also be resource- and time-intensive for institutions, requiring radical changes in communication strategies to focus
on direct engagement with the public and provide fast and targeted responses. Social media and new
communication technologies are also rapidly evolving, and require constant adaptations to new platforms, tools
and interactions between individuals. Due to these limitations, and the important shift in communication strategies
that social media require, public health communities focussed on vaccination uptake and confidence have been
slow and inconsistent to proactively engage with and invest in social media for monitoring opinion, communicating
evidence-based information and/or countering misinformation. In the absence of a savvy, strong and sustained
public health presence, pseudoscience, confusing information and public rumour have fuelled strong anti-vaccine
sentiment and influenced vaccination decision-making through social media in countries across the world [19,20].
There are growing efforts in the field of public health and academia to better understand what is happening on
social media and how they can be used to increase vaccine confidence and mitigate concerns. ECDC commissioned
the Vaccine Confidence Project
TM
(VCP) [1] to conduct a scoping review on social media monitoring methods and
interventions around vaccination. This research project stems from the necessity to synthesise all quality research
produced to inform how social media monitoring methods and analysis can be used to understand and respond to
public discourse about vaccination on social media and to understand the uses, benefits and limitations of using
social media as an intervention tool around vaccination.
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
5
2 Background
Social media have been defined as a group of internet-based applications that build on the ideological and
technological foundations of Web 2.0, and that allow the creation and exchange of user generated content[21].
Web 2.0 refers to the new way in which software developers and end-users started using the World Wide Web,
where content and applications are continuously modified by all users[21]. However, when defining social media,
any given description is simply one of many and each discipline contributes its own perspective on the nature of
social media. For this scoping review, we understand social media as not just a means of communication, but also
a space where people socialise.
Prior to social media, conversations were either private or public, through broadcasting media. Social media now
allow the dissemination of conversations and opinions within a vast network without mediation, which has
contributed to the positioning of social media as a key tool to support people’s freedom of speech and expression
around the world. However, this unbounded freedom to create and share content with users around the world also
comes with major hazards, as it also facilitates the spread of unverified misinformation. This has been framed as
the ‘postmodern Pandora’s box’ of the internet; whereby data circulate unbounded, shared and re-shared
regardless of quality [22].
Virtual and in-person social interactions are deeply entwined and any definition that tries
to separate both risks inconsistency [23].
As social media are not merely a tool but a social environment in which people operate, much is said about the
various social platforms and how they account for different social spaces. However, social media should not be
seen primarily as the platforms upon which people post, but as the content posted on these platforms. Social
media users directly influence what social media are and what they will become as seen in the recent decisions
by certain social media platforms to censor content and change algorithms to promote or reject certain content.
This also explains why social media will always be a continuously evolving environment. Recent research on social
lives online shows that it is the people using social media who create what social media mean and represent rather
than developers or social media platforms themselves. At the same time, research indicates the inability to
understand any one social media platform in isolation. The different digital platforms must be seen as relative to
each other, as people use the range of available possibilities to select specific platforms or media for particular
genres of social interaction[23].
Social media and vaccination
This new boundless information ecosystem has shaped the nature of conversations about vaccination and related
concerns. Dominant and singular narratives such as vaccines are goodare rejected, and instead vaccine-decisions
are considered vaccine by vaccine, disease by disease, case by case[22]. In this context, facts from authorities
and experts are suspect and non-linear dialogue (dialogue that can flow in multiple ways rather than only
chronologically), is the norm [24-26]. Largescale analyses have highlighted the importance of these social networks
and trust relationships in influencing vaccine decisions [27]. As Leask et al. highlight, a patient’s trust in the source
of information may be more important than what is in the information[28]. Rather than consulting a single,
authoritative source of information, it is more common for participants to want a variety of opinions [29].
At the same time that information is important for risk assessment and decision-making, sentiments about
vaccination can strongly affect individual and group vaccination decisions [30]. New digital media, social media in
particular, have allowed new levels of transmission of sentiments concerning vaccines [31], with negative vaccine
sentiment posts being the most liked and engaged with [32]. The rise of internet-mediated communication has
also had a significant impact on how fast rumours and unsubstantiated concerns can spread, feeding into the
abovementioned negative vaccine sentiments travelling transnationally [30].
The amplification of risk and risk perception through social media, has led some countries and health authorities to
start using social media to counter misinformation, mitigate anxiety and rebuild public trust [33]. Ireland and
Denmark have recently managed to rebuild public trust in human papillomavirus (HPV) vaccination by adopting a
strategy that had social media at its core to engage parents via YouTube and Facebook. Both countries took into
consideration how information about HPV vaccination was consumed online by parents and developed their
strategies accordingly [33,34]. Another central advantage of using social media within the scope of public health
policy is the possibility to listen, in real time, to the concerns of populations and pick up signals at a very early
stage. At the same time, vaccination discourses on social media need to be understood within a digital ecosystem,
as users tend to be influenced by and use a range of social media platforms to express their feelings and beliefs.
This digital ecosystem relates to a virtual environment where a community of interacting platforms is continuously
growing and evolving which speaks to the importance of conducting social media monitoring and the valuable
insights it can bring.
As there are many ways of defining social media, for this scoping review we attempt to understand them within the
environment of public health policy and the impacts that they can have within this field. With regard to vaccination,
we understand that it is pivotal to look at the social interaction processes that may be weighing into decision-
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
6
making and risk assessment. Kaplan et al. classify social media into blogs, collaborative projects (e.g. Wikipedia),
social networking sites (e.g. Facebook), content communities (e.g. YouTube), virtual social worlds (e.g. Second
Life), and virtual game worlds (e.g. World of Warcraft) [21].
Since we define social media in this report as online environments with a strong interaction component as their
main purpose
,
we have made the methodological decision to focus on social networking sites and content
communities. We have chosen to exclude online platforms that did not have social interactions as their main
purpose, even though they had some scope for user interaction (e.g. blogs and websites with comments sections).
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
7
3 Goals and objectives
The aim of this scoping review was to systematically map, analyse and summarise knowledge and research on
social media and vaccination and to identify examples of how information collected can inform communication and
interventions to address vaccine hesitancy. We provide an overview of how social media monitoring and analysis of
vaccination can support those working in public health agencies and immunisation programmes by looking at the
type of social media data to collect, how social media data can be analysed and interpreted, and what types of
intervention can be developed based on data collected to increase vaccine confidence and increase vaccination.
The specific objectives of the systematic scoping review were to:
identify individuals’ preferences for using different social media platforms as a source of information on
vaccination and the influence that social media has on individuals’ perceptions of vaccination;
identify different social media monitoring methods and tools in the context of vaccination and their
strengths and weaknesses;
review how social media monitoring methods and information gathered from the monitoring can be used to
inform communication strategies;
identify the uses, benefits and limitations of social media as an intervention tool in relation to vaccination
(i.e. how effective social media is as an intervention tool for increasing vaccination).
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
8
4. Review methods
Systematic scoping reviews are a relatively new method for mapping existing literature in a given field. Systematic
scoping reviews have been used to clarify working definitions and conceptual boundaries of a topic or fieldand
have been particularly useful as an exploration tool for large, complex and heterogeneous topics usually not
suitable for systematic literature reviews [35,36]. While systematic literature reviews are often focused on
establishing the effectiveness of interventions, systematic scoping reviews take a broader approach and aim to
map international literature or to identify how research has been conducted [35]. For these reasons, it was decided
to conduct a systematic scoping review to address the aims of this study and to summarise methodologies that
have been used to monitor social media in relation to vaccination. The methodology used to conduct this study, as
described below, was based on the work provided by Arksey et.al. and further developed by the Joanna Briggs
Institute [35-37].
4.1. Search strategy and database search
Librarians at ECDC developed the search strategy, balancing feasibility and comprehensiveness and including a mix
of social media and vaccine keywords. The search strategy was developed for use in Embase, and adapted for use
in PubMed, and Scopus by ECDC and in Medline, PsycINFO, PubPsych, Open Grey (grey literature), and Web of
Science (grey literature) by the VCP and is available in Annex 8.1. Librarians at ECDC peer-reviewed the final
search strategies for all databases.
One researcher from the VCP conducted the search in all databases in December 2018 and exported all articles
into Endnote. Duplicates were then removed in accordance with guidelines provided by ECDC.
4.2. Screening and selection of articles
Two reviewers independently screened articles included in the Endnote file by title and abstract, according to a set
of inclusion and exclusion criteria:
Table 1. Inclusion and exclusion criteria for screening of articles
Inclusion criteria
Exclusion
Study settings: no restrictions
Not about vaccines, or not about human vaccines (i.e.
vaccines for animals)
Research topics: articles were included if they studied
the following topics: methods of social media
monitoring around vaccination, use of social media
monitoring to address vaccine hesitancy, use of social
media interventions to address vaccine hesitancy
(knowledge, hesitancy, confidence, awareness or
coverage)
Articles with studies focusing on:
Other types of media (not social media) or online
resources
Articles that only use social media to recruit study
participants
Publication years: From 2000 (incl.), to include all
studies conducted on social media monitoring
Publication types:
Conference abstracts, editorials, commentaries,
letters to the editors
Location: global
Types of studies
Efficacy trials, pre-clinical trial research
Safety research
Serologic investigations, immunogenicity studies
Health economic studies
Languages: The VCP extracted data from and analysed
articles in English, Spanish, and Italian.
Vaccines: Human vaccines
Study design: quantitative and qualitative studies,
observational and interventional studies
In this review, social media included: social networking
sites and content communities (e.g. Facebook, Twitter,
LinkedIn, Instagram, Snapchat, YouTube, Vimeo,
Reddit, Quora, online discussion forums, or Pinterest).
After articles were selected through title and abstract screening, the two reviewers proceeded to full text screening
to confirm the final list of included studies. All disagreements between the reviewers were resolved by discussion.
A summary of the search and selection process are provided with a PRISMA chart (see Figure 1).
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
9
4.3. Data extraction and analysis
During the full text article selection, articles were divided into three categories corresponding to the various objectives
described above: articles describing individuals’ preferences for using different social media platforms as a source of
information on vaccination and the influence that social media have on individuals’ perceptions of vaccination, social
media monitoring articles and articles describing social media interventions to address vaccine hesitancy.
Three researchers from the VCP extracted data into an Excel spreadsheet for these three categories of articles, as
per the information presented in Table 1.
Table 2. Data extraction categories
Social media monitoring articles
User preferences articles and
interventions articles
Author/reference
Year of publication and study
Country of study
Aims/purpose of study
Study population and sample size
Setting
Vaccine
Type of social media
Tool for data collection and details
Keywords selection and exclusion criteria
Sentiment coding and analysis
Geo-location of data
Reach, spread and interaction
Visualisation of data
Other types of analyses
Number of posts and results
Public health implications
How to use social media monitoring, in particularly to
start communicating on social media platforms.
Limitations
Author/reference
Year of publication and study
Country of study
Aims/purpose of study
Study population and sample size
Setting
Vaccine
Type of social media
Methodology
Intervention type and details
Duration of intervention
Outcomes and details
Key findings
Limitations
Four researchers then summarised, charted and analysed the data extracted. The analysis of the included articles
was mainly descriptive (see more details on the analysis conducted for each of the three types of articles below),
as articles were heterogeneous and presented highly diverse purposes, methodologies and study outcomes.
Preferences for use of social media platforms as a source of information on vaccination and the
influence that social media have on individuals’ perceptions of vaccination
Articles about individual preferences regarding social media were analysed by looking at the use of social media to
gain or share information on vaccines and the possible influence of social media on vaccine attitudes and/or
uptake. Results were noted, and the proportion of participants either using social media or being influenced by
social media were listed for each study and then described in the report.
Media monitoring analysis
When analysing the media monitoring articles the key focus was to describe methodologies used in different studies to
monitor social media and their evaluation (if applicable). The researchers therefore provided a descriptive analysis of the
type of data collection tools used to gather data from social media, the keywords and search strategies used (including
duration of search), and the various analytical methods (sentiment or content analysis, analysis of spread, reach and
interaction, and geo-location of data). Data was first extracted to an Excel spreadsheet in accordance with the categories
in Table 1; this allowed reviewers to compare results across all studies, list and identify the frequency of different
methods used for social media monitoring in different studies, and identify common themes. Two reviewers met to
discuss the extraction spreadsheet, the themes identified and the findings of this review.
Suggestions from the studies on how social media monitoring can inform vaccination communication strategies
were also included.
Intervention articles
For the intervention articles, the data were first categorised by type of intervention. The VCP then recorded results
for various study outcomes to provide a clear overview of the effects of the various interventions. For qualitative
studies, a list of key themes was compiled and analysed. Some descriptive information was also provided, such as
the number of studies reporting different types of interventions or conducted with different population groups; the
names of social media platforms used most frequently by different population groups; or the methods used for
monitoring social media platforms around vaccination.
Some analyses were also common to all three categories, such as the types of social media described, the type of
vaccines studied, and the number of articles published over time, to reflect how much attention the topic has
received in recent years.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
10
5. Review results
The search across all databases generated 15 435 articles, from which 7 539 duplicates were excluded (see Figure
1 for PRISMA chart). The remaining 7 896 unique articles were screened by title and abstract using the inclusion
and exclusion criteria listed above. A total of 7 628 articles were excluded, leaving 268 articles for full text review.
From these, 153 articles were excluded (see annexes detailing reasons for exclusion) for the following reasons:
article on media but not social media monitoring (n=96), no data provided in the article (n=19), about websites or
mobile apps but not social media (n=26), conference abstracts or editorials/letters to the editors (n=6), article
containing data already published in another included article (n=1), article not looking at vaccination (n=1).
Additionally, the full text of four articles on social media monitoring was not accessible, even after making enquiries
with multiple libraries.
At the end of the screening process, a total of 115 articles was included for analysis:
13 articles looking at individuals’ preferences for using social media as a source for vaccination information
and the influence that social media have on individuals’ perceptions of vaccination,
85 articles on social media monitoring,
15 articles on social media as an intervention tool around vaccination,
one article that looked at both an intervention and individuals’ preferences for using social media as a
source for vaccination information,
one article that combined social media monitoring and a social media intervention.
Figure 1. PRISMA flow diagram
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
11
5.1 Individuals’ preferences for using social media platforms
as a source of information on vaccination and social media’s
influence on individuals’ perceptions of vaccination
A total of 14 articles explored individuals’ preferences for receiving information via social media, which social media
platforms are used and how information shared on social media influences the perceptions of vaccination. Annex 8.2
provides an overview of the studies relating to the use of social media to gain or share information about vaccines and
the possible influence of social media on vaccine attitudes and/or uptake.
5.1.1 Preferences regarding the use of social media as a source of
information on vaccination
Several studies pointed to social media as a source of health-related information:
Five out of seven parents in one US study US cited social media as a common resource for information [38].
Another study in the US found that 62% of adults questioned used Facebook to find information on the
influenza vaccine, compared to 15% for Twitter [39].
In a survey of undergraduate students in Seoul, South Korea 30% of the respondents cited social media as
a source for information on HPV [40].
In the UK, a study found that in a group of 626 parents who used the Internet to find information about
vaccinations, 13% used Facebook or Twitter and 6% used discussion forums [41].
Another study in the UK focused on pregnant women using social media and found that 21% of the
participants used social media to find information on vaccinations during their pregnancy, with Facebook
and WhatsApp being the most popular platforms [42].
Limited use of social media as a source of information
A Canadian study found that 68% of participating medical students had never used social networking sites
such as Facebook or MySpace to obtain health-related information [43].
Similarly, university student participants in a study in Northern Ireland reported social media to be their least
preferred source of information on awareness of meningitis and vaccines [44].
A dissertation from the US reported that although 66% of parents had seen information about HPV vaccination
on social media, only 4% had actively used social media as their main source of information about HPV
vaccination (a lower percentage than those using information from friends and government health
organisations) [45].
A US study found that 11% of parents who had heard HPV vaccine stories found them on social media and these
accounts were more likely to be negative ‘stories of harm’ than content through other information channels [46].
Regarding overall use of social media, a study conducted among medical students in Canada found that while
66% of participants sometimes or often used YouTube, 24% reported sometimes looking for health-related
information on the platform, and only 2% reported always doing so [43]. Furthermore, 42% reported using
YouTube for health purposes, including educational purposes, but 17% were uncertain about the platform’s
trustworthiness and 36% reported minimal trust in health content provided on YouTube [43].
Willingness to share information on social media
None of the female students in one study on a university campus in the United States shared HPV information
on Facebook, 71% of them were willing to do so in the future [47].
One Spanish study also looked at the willingness of medical students to use/follow/participate in Facebook
pages promoting influenza vaccination. They found that 63% of students would accept an invitation to follow a
Facebook page with formal or technical content on the healthcare worker influenza vaccination campaign,
while 65% would accept an invitation to follow a Facebook page that communicated the same information
informally (such as animations or offbeat news) [48]. In all, 19% of the students would actively participate in
a ‘technical’ Facebook page, compared to 28% of students who would actively participate in an ‘informal’
Facebook page [48].
Preferences regarding the use of social media as a source of information on vaccination:
Between 4 and 62% of various study populations in different countries use social media as a source
of information on vaccination, with results varying by type of social media platform.
Overall Facebook was the most common social media resource for information on vaccination.
Social media users’ perceptions of vaccination:
Most studies suggest a negative relationship between social media use and vaccination uptake and
attitudes, which could sometimes be explained by the important presence of negative content
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
12
5.1.2 Social media as an influence on perceptions of vaccination
Several studies considered not only individuals’ preferences for social media use but also how this usage influenced
their perceptions of vaccination, such as their attitudes and/or uptake of vaccination. Most of these studies
suggested a negative relationship between various social media use and vaccination uptake, while others
suggested the potential for the positive influence of social media on vaccination uptake. Seven of these specifically
referred to Facebook, four referred to Twitter and seven did not specify a social media network or platform.
Negative relationship between social media and vaccination attitude and/or
uptake:
A study in the UK that asked pregnant or recently pregnant women how they searched for information on
vaccinations during pregnancy found that 12% of participants believed the information they found on social
media influenced their vaccination decisions [42]. This influence manifested in a significantly negative
relationship in relation to pertussis vaccination, with women who used social media to gather information
being 58% less likely to receive this vaccination during pregnancy[42].
Another study in the UK reported that parents who used social media, such as discussion forums and
Facebook or Twitter, were more likely to report that they had come across some material that made them
doubt vaccinations (31% of parents who used discussion rooms and 23% of parents who used Facebook or
Twitter versus 8% of all participating parents) [41].
Similarly, three studies in the US found social media had a negative influence on parentsperceptions of
vaccines [49] [45]. In one of these studies, 10% of the participating parents and guardians felt that social
media increased their sentiments of fear around the HPV vaccination [45].
Another study about the HPV vaccine conducted in Seoul reported that undergraduate students felt the
information they obtained about the vaccine via social media increased their perception of barriers to
receiving it [40].
In the UK, a study considered social media as one of the various intervention strategies used to increase
influenza vaccine uptake in healthcare workers over the course of four years. The researchers in this study
reported a significantly reduced vaccination uptake when using promotions on Facebook (22%) and Twitter
(24%) as an intervention, although no reflection on the reason for these results was provided [50].
In India, a study was conducted to assess the influence on trust of a large measles-rubella vaccination
campaign in the southern state of Tamil Nadu. This study found that most parents who rejected the vaccine
for their children also placed higher levels of trust in social media platforms, including WhatsApp[51].
Positive relationship between social media and vaccination attitude and/or uptake
A study on vaccines during pregnancy in the UK found that women who used WhatsApp and LinkedIn were
more likely to receive both the influenza and pertussis vaccines while pregnant [42].
A study in the US found that participants who used Facebook or Twitter as sources of health information
were more likely to be vaccinated [39].
Participants from another study in the United States proposed using social media to circulate positive
messaging about the HPV vaccine [38].
5.2 Social media monitoring
There is a growing body of literature describing social media monitoring methods. For this report the results of the
social media monitoring are organised into three major phases (see Figure 2): 1- preparation, 2- data extraction
and 3- data analysis.
Figure 2. Social media monitoring phases
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
13
Articles on social media monitoring in this review were therefore reviewed in accordance with the following three
phases:
1. Preparation:
Characteristics of the studies in this report - purpose of social media monitoring and platforms monitored
Ethics approval
2. Data extraction:
Data extraction tools
Period of monitoring
Search strategies
Visualisation of data
3. Data analysis:
Geo-localisation
Reach
Trends, content and sentiments
5.2.1 Preparation
Study characteristics
A total of 86 articles monitored and analysed social media in relation to vaccination (see Annex 8.3 for a table
summarising the characteristics and methods used in these articles). While the first study was published in 2006,
only a very small number of studies were published between 2006 and 2014. In 2015, the number of published
articles about social media monitoring increased substantially, with 83% of all articles identified in this review
published since 2015. Nine studies were published in 2015 (11%), 14 in 2016 (16%), 25 in 2017 (29%) and 21 in
2018 (24%). As the search was conducted in December 2018, only two articles, available ahead of print, were
identified for 2019.
Purpose of social media monitoring
It is important to establish the purpose of analysing social media information on vaccination as this will influence
how and which data are collected. For the majority of studies (55/86, 64%) the goal was to increase
understanding of how vaccination is portrayed on social media, through online discourse, sentiment or the way in
which information is produced, engaged with and shared. Other aims included monitoring the reaction after an
outbreak, monitoring the impact of a campaign or intervention or a vaccination programme, monitoring
misinformation or monitoring public concerns and questions.
Key messages
In 2015, the number of articles published on social media monitoring increased substantially, with
83% of all articles identified in this review published since 2015.
The large increase in the number of articles from 2015 was mostly attributed to an increase in studies
conducted on Twitter.
Purpose
The purpose of analysing information about vaccination online will influence how and which data are
collected. Different purposes were identified:
increasing understanding of how vaccination is portrayed on social media through online
discourse, sentiment or how information is produced, shared and engaged with;
monitoring the reaction after an outbreak;
monitoring the impact of a campaign or intervention or a vaccination programme;
monitoring misinformation;
monitoring public concerns and questions in general or over time.
Types of social media
A large majority of studies focused on Twitter, followed by YouTube, Facebook and various online
forums.
Ethics
The questions relating to ethics approval to perform social media monitoring research are growing. In
addition, a number of studies raised the issue of posts not being publically available.
Out of the 86 articles on social media monitoring, only 13 (15%) explicitly mentioned having received
approval from an institutional ethics review board. Some of the other studies considered that they
were exempt from institutional/ethical review as their studies did not directly involve human subjects
or because social media analysis only included publically available data.
Other researchers believe that anonymization is not enough and they urge that other solutions should
be found due to the fact that private data can easily be revealed.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
14
Type of social media monitored
A large majority of studies focused on Twitter (n=42)[30,52-92], followed by YouTube (n=12)[32,93-103],
Facebook (n=11)[45,104-113], and various online forums (n=9)[114-122]. The forums included in the studies
reviewed in this report were babytree (China), Iltalehti and KaksPlus (Finland), Mothering.com (UK), and Mumsnet
(UK). Additionally, five studies either used multiple forums from the same country identified by a Google search,
forums specifically designed for a particular event or study, or failed to name the forum analysed. Other types of
social media monitored included Yahoo! Answers (n=2)[123,124], Pinterest (n=1)[125], Reddit (n=1)[126], and
Weibo (n=1)[127]. An additional seven studies monitored a mix of social media networks, including Digg, Hyves,
Facebook, unspecified forums, LinkedIn, Reddit, Twitter, and YouTube[128-134]. A description of all the different
social media platforms monitored across the 86 studies is provided in Table 3.
Table 3. Description of the social media platforms identified in the scoping review
Digg
Platform allowing users to post, save and share news stories and to vote content up or down.
Facebook
Platform used for social networking, allowing users to create profiles with personal information
about themselves, and to post, interact, comment or share messages, photos, videos and
other media content with other members (friends or followers). The platform also allows
groups and professional pages to be created, with comments, likes and shares of these posts
across both personal profiles and group/pages (depending on privacy settings).
Forums
Type of social media platforms allowing users to write content in message boards or online
discussion sites/threads. The forums included in the studies reviewed in this report were
babytree (China), Iltalehti and KaksPlus (Finland), Mothering.com (UK), and Mumsnet (UK).
Hyves
Platform used for social networking, allowing users to interact with other members (Dutch
equivalent of Facebook, discontinued in 2013).
LinkedIn
Platform used for professional networking and for posting jobs and/or curriculum vitae or
sharing content in the form of short messages, images, videos or links.
Pinterest
Platform for posting, interacting with and sharing images/articles, referred to as pins, as well
as videos and other media content.
Reddit
Platform for posting links, text messages, videos and images. These are then voted up or
down and discussed by other members.
Twitter
Platform for posting, interacting with and sharing short messages (tweets) of maximum 280
characters, video and/or links.
Weibo
Platform for posting, interacting with and sharing short messages (Chinese equivalent of
Twitter).
YouTube
Platform for posting, interacting with and sharing videos and blog posts.
Figure 3 shows the number of articles identified by year and by type of social media (excluding the two 2019
articles). It indicates that the large increase in the number of articles from 2015 was mostly attributed to an
increase in studies conducted on Twitter (n=37, 54% of all articles published between 2015 and 2018), and to a
smaller extent Facebook (n=10, 15%). Articles about less commonly studied types of social media (Pinterest,
Weibo, Reddit, and Yahoo! Answers) were all published after 2015.
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
15
Figure 3. Number of articles published by type of social media and by year until 2018
Note: Forum refers to the different forums included in the articles covered by this report: babytree (China), Iltalehti and KaksPlus
(Finland), Mothering.com (UK), and Mumsnet (UK)
Type of vaccine monitored
Most of the articles identified in this review looked at vaccines in general (40%, n=34) [32,55-57,59-
61,68,72,78,81,83,87,89,90,95,97,98,101,103-107,110-113,118,121,122,125,126,132], HPV vaccination (27%, n=23)
[45,58,63-66,69,70,73-75,82,84,86,92-94,96,99,102,117,123,130] or measles vaccination (14%, n=12)
[52,62,77,80,85,88,91,116,119,129,131,133] (Figure 4). Additionally, five studies monitored social media in relation to
the 2009 A(H1N1) influenza pandemic [30,76,114,120,128] and four studies looked at seasonal influenza
[67,71,124,134]. Other vaccines monitored on social media included polio (n=2) [108,109], diphtheria (n=1) [79],
hepatitis B (n=1) [127], meningococcal B (n=1) [100], pentavalent DTP-HepB-Hib (n=1) [54], and rotavirus (n=1)
[115]. One study also looked both at polio and HPV vaccination [53].
Figure 4. Number of articles published by type of vaccine
Countries monitored
A large proportion of studies did not restrict monitoring to one specific country and therefore contained global results
(n=41) [52-57,59,62-65,67,70-75,77,78,80,81,84,86,90,92-95,97,99-105,110,125,126] (Figure 5). However, some of
these studies have restricted their search to specific languages such as English or Spanish, which may therefore provide
skewed results towards particular regions of the world. Fifteen studies were conducted specifically with data coming from
the United States (US) [30,45,58,61,66,68,83,85,87,88,91,96,107,129,130], seven from Italy [32,60,89,98,106,118,133],
three from the Netherlands [69,131,134] and three from the United Kingdom (UK) [76,116,122]. Other countries
specifically monitored included Canada (n=2) [113,128], China (n=2) [115,127], Israel (n=2) [108,109], Spain (n=2)
[79,114], Australia (n=1) [119], Chile (n=1) [123], Finland (n=1) [120], Japan (n=1) [124], and Romania (n=1) [117].
Four studies looked at data from multiple countries, including Australia, the US, Canada, and the UK [82,111,121,132].
0
5
10
15
20
25
30
2006 2007 2008 2010 2011 2012 2013 2014 2015 2016 2017 2018
NUMBER OF ARTICLES PUBLISHED
YEAR
Forum Youtube Facebook Twitter Mix Pinterest Weibo Reddit Yahoo Answers
1
1
1
1
1
1
2
4
5
12
23
34
Diphtheria
Hepatitis B
Meningococcal B
Pentavalent
Rotavirus
Polio and HPV
Polio
Seasonal influenza
Pandemic influenza (H1N1)
Measles
HPV
Vaccination in general
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
16
Figure 5. Number of articles published by country monitored
Ethics approval - public versus private data
The issue of posts not being publically available was raised in nine studies, either as a pre-defined exclusion
criterion or as a limitation [45,53,69,80,87,108,111,121,133]. This could be linked to certain ethical issues.
Out of the 86 articles on social media monitoring, only 13 (15%) explicitly mentioned having received
approval from an institutional ethics review board in Australia, Canada, Israel, Romania, the UK and the US
[45,58,62-64,76,82,92,105,108,109,113,117].
Additionally, one study did not mention whether they had received ethics approval, but stated guidelines
from the Institutional Review Board have been considered and applied to protect the identity of forum
users [115].
Nine studies also specifically stated that they were exempt from institutional/ethical review as their studies
did not directly involve human subjects or because social media analysis only included publically available
data [56,57,70,74,86,94,95,99,104].
The authors of a study that obtained ethical approval, conducted on Facebook in Israel, further explained
that as they anonymised their data, participant consent was not required as conversations on the Internet
happen in public fora, where subjects would expect to be observed by strangers[108].
However, anonymisation was not enough for the authors of a global study conducted on Twitter, who were keen
for other solutions to be found. They explained that private data could easily be revealed through the integrative
analysis of multiple datasetsand that revealing the identity of social network contributors who may have wished
for it to be kept secret was feasible (the study did not mention seeking ethical approval) [80].
Finally, Tangherlini et al, who analysed comments on forums in the US and Canada (but did not mention
seeking ethical approval) raised the growing challenge of accessing data on social media, as corporations
are constantly reducing access to data [121].
1
1
1
1
1
2
2
2
2
3
3
4
7
15
41
Australia
Chile
Finland
Japan
Romania
Canada
China
Israel
Spain
Netherlands
United Kingdom
Mix of countries
Italy
United States
Global/not specified
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
17
5.2.2 Data extraction
Data extraction tools
In order to collect data from social media on the subject of vaccination, the studies in this review used:
Manual browser searches on web browsers such as Firefox, and Google Chrome. Browser searches are
performed from within social media platformse.g. the basic or advanced search bar usually found at the
top of the page on Twitter, YouTube or Facebook.
Social media APIs (Application Programming Interfaces). The term APIrefers to a software intermediary
that allows two applications to talk to each other. When Twitter is used, the Twitter application connects to
the Internet and sends data (e.g. the text or images posted with a tweet) to a server. The server then
retrieves the data, interprets it, performs the necessary actions and sends it back to the Twitter application
on a user's phone, web browser or a researcher's database, which is then interpreted and shown to the
user in a readable format. APIs work across all social media platforms to pull and interpret data from
servers storing information for Facebook, Twitter, YouTube, Reddit and many more.
Automatic monitoring (commercial software). These can be automated web platforms that are free, open
source (open to development from other developers), or commercial (where access is allowed via a
subscription pricing structure);
Use of both manual searches and APIs.
Use of both automatic software and APIs.
Figure 6 shows the number of software tools used within each category.
Figure 6. Number of studies by type of social media monitoring tools used
Notes to the figure:
API Social media Application Programming Interfaces
Manual - Manual browser searches on web browsers
Automatic or commercial tool - Automated web platforms.
Manual browser searches
A total of 36 studies used web-based manual tools. Studies that used manual browser search functions within
social media platforms tended to collect less data than those accessing the automatic Application Programming
Interfaces (APIs) or software. Some of the limitations of manual browser searches are described in Table 4.
35
20
24
6
1
0 5 10 15 20 25 30 35 40
NUMBER OF ARTICLES PUBLISHED
API and Manual
API and Automatic
API
Automatic or commercial
tool
Manual
Studies that used manual browser search functions within social media platforms tended to collect
less data than those accessing the automatic Application Programming Interfaces (APIs) or other
software.
A large number of studies used the Twitter API to collect data due to the ease of access given by the
Twitter platform to its data stream compared to other platforms.
Studies used a range of commercial software, with the majority accessing paid-for periodical and
historical Twitter data.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
18
Table 4. Different manual browser searches and their limitations mentioned by the studies using the tool
Social media tool
Limitations
YouTube search browser
[32,93-103]
Despite the use of a variety of search terms being seen as a strength of the study, only
the top 50 hits according to relevance were chosen to be analysed (this is in the context
of the three million videos posted on the topic of vaccines currently on YouTube at the
time of the project)[96]
Facebook search tool
[45,104,105,108,109,112,113]
Due to the time intensity that assessing each Facebook site required, it was impractical
to analyse each site in detail. Another limitation mentioned was that the focus of the
assessment was on the most recent posts. The nature of posts on Facebook sites may
change as new information regarding vaccines reaches the general public, such as
during flu season or when school starts, and parents must turn in their child’s
vaccination records. During the short time for the data collection, there was no major
news related to vaccines that had recently been reaching the general public. Finally, the
authors did not gather data regarding individual users and cannot determine whether
activity on the site centred on several engaged users or was spread among site
membership [104]
A small limitation to using a Hypertext PreProcessor (PHP) script supplied by Facebook
as an add-on to the basic Facebook search tool as the script served to only collect each
post’s first 25 comments - this meant that not all comments for every post were
analysed. However, it was not considered a strong limitation since each post or
comment was analysed as an individual unit. From the sampling frame, a sample of
2 289 items were randomly selected using a ‘randomise numbers’ command. This was
considered a representative sample of the initial sampling frame. This study was made
before the data protection ethics and protocols linked to the 2018 General Data
Protection Regulation (GDPR) rules came into place[135], although all the data were
anonymised and available for scientific use, the authors acknowledged that this
methodology may give rise to ethical concerns, given that identifiable comments made
by people on public Facebook pages are scrutinised. Nevertheless, at the time of the
study (2016), according to the Codes of Ethics and Conduct of Internet Research[136],
if an observation of public behaviour takes place in public fora where subjects would
expect to be observed by strangers (such as an open Facebook discussion), explicit
individual consent is not required. If this search was made today, however, the analysis
of potential identifiable user posts on Facebook would be limited[109].
Luisi (2018)[45], found that the Facebook search feature does not allow users to
organise results by date or engagement (e.g. likes/comments). This limits flexibility in
data collection. Technology also limited the ability to archive Facebook posts. When
loading the search results, one would have to scroll down to make the area printable.
Scrolling down too far would cause internet browsers to crash. Moreover, this study only
collected public Facebook content in an effort to only analyse content that would be
available to any Facebook user, because access to private social media feeds is not
possible without specific participant consent.
Suragh et al. (2018)[112], found that a limitation of just using the Facebook browser
search tool was the inability to examine entire social networks, which means that the
fraction represented by the study data of what actually exists is unknown. The study
was also limited to the information included in the online reports, with potential biases
and errors in reporting. Lastly, there was the challenge of conducting searches in
different countries. The findings from the Google and Facebook searches were
dependent upon the geographic location of the reviewer and this reflected on targeting
popularfindings according to the search location and specific algorithms used by these
companies. This limitation could have also been a result of the study methodology,
which only included reports found in the first three pages for Google and top 20 posts
for Facebook. It is possible that if larger search samples (Google produced hundreds of
thousands of URL (Uniform Resource Locator) links per search term) were analysed,
both reviewers would have found exactly the same results. Facebook results were
dependent on the date and time of the search (e.g. the highest placed posts found on
one day were not the same as those found the next day) therefore searches had to be
completed in one sitting and some of the URL links identified in Facebook did not work.
The study used standardised search terms but other reports of cluster immunisation
may probably have been found by including additional search terms and expanding to
different languages, countries and regions.
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
19
Social media tool
Limitations
Forum search tools[116].
In one study by Skea (2006)[116] relating to internet forum discussion on the measles-
mumps-rubella (MMR) vaccine, one limitation found was that the 617 messages
analysed were those posted to only one website, which meant that participants were
probably not demographically representative of the wider population. In addition, a
higher proportion of participants in the fora had refused MMR vaccine than in the
general population. All these factors raised legitimate questions whether it is appropriate
to generalise from the study’s findings to the wider parent population, and whether and
how insights from this study should inform future communications on MMR vaccine or
vaccination more generally.
Pinterest search tool[125]
A study by Guidry et al. (2015) found that a limitation of using the manual browser
search was that Pinterest does not list its pins chronologically and does not list an exact
time stamp for each pin. The authors found that this made using a more conventional
content analysis sampling method, such as a constructed two-week time period,
virtually impossible.
In addition to the various web-based manual tools mentioned in the table above, three other studies used the
Google Search too l[117,118,128], three used a combination of the Twitter and Facebook search tools
[113,130,133], two studies used manual HTML (Hyper Text Markup) extraction [121,122], one study used the
Yahoo! Answers search engine [123], and one study used the basic Twitter search tool [86].
Social media APIs
Thirty-one studies sampled their data directly from their target social media platform’s API. There was a strong
focus on the Twitter API, which pushes focus on Twitter and neglects data on other social media platforms. The
results of the scoping review found that Twitter does not provide a full reflection of all social media discussion, or
indeed general discussion of vaccines overall. There is a need for a broader understanding of discussions on other
platforms, and how they may influence vaccine uptake. Some of the limitations of social media APIs are described
below.
Table 5. Social media APIs and their limitations mentioned by the studies using the tool
Social media APIs
Limitations
Twitter API
[30,54,57,59,61,63-
67,69,71,74,80,82-84,91,92]
One study[69] used a combination of two Twitter APIs. This was because they found
that general Streaming Twitter API did not provide an easy way to retrieve complete
conversations (many of which were partially truncated/cut-off, due to some accounts
being protected/private). Multiple replies to the same tweet can also occur, which the
study’s data retrieval method may not have detected. To retrieve a more complete
sample of conversations on Twitter they used a combination of checking for replies to
tweets within their user group and screen scraping from the Twitter browser page
itself. It was found that data collected through the relevant keywords and accessed via
the basic API stream accounted for only a quarter of the total number of results. This
is, in itself, a highly interesting result: a search using keywords would miss 75% of the
relevant tweets. One possible way to avoid this would be to gain access to the Twitter
Firehose or all tweets via a subscription-based and automated service however, this
may be cost-prohibitive to some studies.
In one study[83], one of the limitations of using the Twitter API was determined by
looking at the scope of open debate on social media analysis, and whether social media
discussions are a valid and accurate proxy for the rationales of the population at large.
This applies both in terms of users' demographics and the potential for fake users (or
automated bots) to spam social networks or post fake and polarising content. There
was also a need to be aware of the (limited, but needed) amount of technical
supervision required when analysing the sentiment and geolocation of tweets - the
system requires computational capacity and server administration, as well as the
creation of machine learning classifiers to annotate large amounts of social media data
(i.e. tweets, images, posts, or comments).
Facebook API[106,107,110]
In one study[111], one of the limitations acknowledged was that the research
conducted was limited by the combined public/private nature of Facebook. While public
Facebook pages do provide a wealth of network information, the authors were unable
to gather information about how information shared from anti-vaccination pages
disseminates through private Facebook pages or personal social media networks.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
20
Automatic monitoring or commercial software
Twenty-six studies used automatic, commercial software to monitor social media with the majority accessing paid-
for periodical and historical Twitter data (see Appendix 8.4 for a list and details of the automated tools and
commercial software used in the studies in this review). Automatic monitoring or commercial software extract large
amounts of data at one time. Costs can be high and there is a need for sufficient human resources to analyse the
data captured.
Period of monitoring
All studies sampled data from different social media platforms over varying periods of time. Below is a summary of
the different types of timed data collection in relation to the monitoring tools used and the amount of data
collected.
Single point in time
Nine studies sampled data over a single point in time. It was found that these specific studies used a manual form
of data collection. Three studies used the YouTube browser search tool [95,102,137]. Three studies used the
search tools within forums that they drew data from [114,115,119]. Two studies used the basic Facebook search
function [104,105]. One study sampled data using the Pinterest browser search tool [125].
Continuous period of time
Seventy-five studies were carried out over a continuous period of time.
Months
Thirty studies were conducted over a period of 120 months [30,52,62-65,68,69,73,74,76,78-80,85,88-90,93,107-
109,113,120,123,127,128,131,132,134]. The studies were performed using a combination of API and automatic
searches.
Years
Thirty-five studies were carried out within a one year and 16-year time-span [32,53-57,59,61,66,67,71,75,77,82-
84,86,87,91,96,98,99,106,110,111,116-118,122,124,126,129,130,133].
The longest period of data collection was by a study that covered 237 112 Italian Facebook posts in Facebook
groups during a 16-year period, with the aim of understanding the linguistic and psychological features of the
language used to talk about vaccinations on social media [106].
Overall, studies that were conducted over a period of years were done with a combination of API and automatic
searches.
Two more studies with large datasets retrieved 6 288,653 vaccine images on Twitter directly from the
Twitter API over one year and eight months [59], and 1 448 010 tweets and data over seven years via the
Twitter API [55].
Two studies collected data covering a period of ten years, using Facebook [45] and YouTube [98]. In the
first study, 6 537 posts were collected using the Facebook browser search tool, with the aim of studying
likes and parent/guardian perceptions, as well as social media representations of the HPV vaccine [45].
In the second study, the YouTube browser search tool was used to sample 560 Italian videos over a period
of ten years, with the aim of analysing any connections with discussions on the reputed vaccine/autism link
or links between vaccines and other serious medical conditions in children [98].
The largest collection of posts from a mixed study, using an automatic tool, was a study on Crimson
Hexagon looking at a mixture of 58 078 Facebook posts and 82 993 tweets - to examine Facebook and
Twitter discussions of vaccination in relation to measles during a period of several widely publicised
outbreaks over a 7.5 year period [129].
The aim and resources available dictate the time period chosen for monitoring, whether in retrospect
at a single point in time or over a continuous period.
Regardless of the data collection period, studies with the highest number of results consistently came
from the use of social media APIs or automatic data sampling.
Yahoo! Answers API provided the largest sample size from a single platform over a sampling period of
five years (16 million messages).
Crimson Hexagon was the automatic platform that provided the largest mixed sample size, with a
mixture of 58 078 Facebook posts and 82 993 tweets.
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
21
Search strategies
List of keywords used for social media monitoring
Seventy-one articles provided a list of keywords used to monitor social media (see Annex 8.5 for the full list). From these
71, only 10 specifically mentioned an extensive search strategy, with Boolean operators to link keywords (e.g. and, or) or
truncations to identify words with different endings (e.g. vaccin*) [52,54,68,69,76,77,91,126,129,132]. Many studies only
used a small number of keywords to monitor social media: 13 studies used only one keyword (e.g. vaccination, measles,
MMR) [32,57,62,73,81,85,103,104,116,120,124,127,131], eight studies used two keywords (e.g. HPV and vaccination,
vaccination and immunisation) [56,69,70,86,94,95,101,115] and five studies used three keywords (e.g. vaccine, vaccines
or vaccination) [54,93,110,118,132]. Studies looking specifically for negative content in relation to vaccination included
keywords related to safety or risk, specific side effects, or certain conspiracy theories or celebrities. Most of the 71 studies
were conducted on Twitter (n=10), YouTube (n=6) and forums (n=4). Three of these studies mentioned the small
number of keywords as a limitation [32,56,96].
Most keywords can be classified as:
words related to vaccines or vaccination (including the generic names and brand names of specific vaccines)
vaccine-preventable diseases
types of side effects
anti-vaccination related keywords
references to certain controversies and names.
The keywords most commonly used across all studies are shown in Figure 7.
Figure 7. Keywords most commonly used across all studies (>4 use)
Use of hashtags
Ten studies also included hashtags in their keywords when searching Twitter [55,56,60,70,74,75,81,86,87,129].
While some of these studies used generic vaccination keywords (#hpv, #vaccine, #vaccination), others searched
for more specific hashtags used in social media discussions (#cdcwhistleblower, #b1less, #hearus, #iovaccino,
#vaccineswork). One study, conducted in the United States, explained that those specific hashtags were chosen
because they were used by journalists in articles covering anti-vaccination beliefs on Twitter [87].
Use of questions
While some studies used simple keywords or hashtags to conduct searches, others used questions. For example, a study
conducted on YouTube concerning HPV used the keywords should I get the HPV vaccineor what can go wrong with the
HPV vaccine[99]. These keywords were selected based on a modified Delphi procedure, where investigators asked for
and reviewed terms provided by non-medical and medical colleagues as well as patient representatives and practitioners.
One of the two studies conducted on Yahoo! Answers typed in questions such as what is papillomavirusand what is the
effectiveness and safety of HPV vaccinesto collect results [123].
31
22
21
16
15
14
13
11
10
8
7
6
5
5
5
5
5
5
5
Social media monitoring studies often used a small number of keywords related to vaccination and/or
vaccine-preventable diseases.
Studies looking specifically for negative content around vaccination included keywords related to
safety or risk, specific side effects, or certain conspiracy theories or celebrities.
Questions, phrases and/or hashtags (e.g. #vaccine) were used in different searches.
The use of three categories of keywords - relevant, semi-relevant and non-relevant - was found to
increase precision in a search.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
22
Use of phrases
A search on Facebook looked for various phrases about mass psychogenic illnesses related to vaccination, including
mass hysteria after vaccineand fainting in school children after vaccine[112]. These terms were developed after
consultations with safety experts at WHO and the US Centers for Disease Control and Prevention (CDC) and then
pilot tested.
Exclude data
Two studies also used keywords to exclude data. A study conducted on Twitter in the US used words such as
malware, caffeine, heroin and needle exchange to exclude irrelevant results [68]. A Dutch study, also conducted
on Twitter, used words such as blood tests or travel vaccinations[69].
Use of languages
Twenty-four studies specified using English keywords or excluding results not published in English
[30,45,52,56,64,65,70,71,74,82,84-86,90,93-96,99,101,104,110,112,128]. Seven of those discussed the decision
as a limitation in their discussion [56,71,86,90,99,112,128]. Fifteen studies used keywords in languages other than
English: six studies used Italian keywords [32,60,89,98,118,133], five used Spanish keywords [79,97,100,102,123],
two used Dutch keywords [69,131], one used Chinese keywords [115], and one used keywords in Hebrew [108].
Another seven studies focused on social media posts in other languages, without necessarily using keywords in
other languages. Four studies, conducted in Italy [106], Israel [109], Canada [113] and Spain [114] did not
provide the keywords but directly extracted data from Facebook groups in Italian, Hebrew and Canadian
French/English, and from YouTube videos in Spanish or Catalan. A Romanian study analysing online forums did not
specify the language of the keywords used, but explained that only Romanian discussion forums were included
[117]. Another study used keywords in English but also included in their analysis tweets identified in other
languages [67]. Finally, one study developed a keyword strategy (pentavalent OR pentavac OR quinvaxem) with
the purpose of retrieving messages from multiple national discussions [54]. However, the authors acknowledged
that their search terms did not include all brand names, which could have limited results from a number of
countries.
Evaluation of keywords
Only two studies evaluated their keywords: one conducted on Twitter in the Netherlands [69], and the other on
Twitter and Reddit in Canada, the UK and the US [132].
The Dutch study used three lists of keywords (in Dutch): relevant (HPV AND vaccination), semi-relevant
(HPV OR vaccination), and non-relevant keywords (words related to other types of injections and other
meanings of the search keywords). These keywords were generated manually by the authors and by using
a quick scan of initial search results. They found that using such a system allows good precision. Semi-
relevant keywords were particularly useful, as 66% of tweets by users from the target group were found to
be relevant. On average, 59% of tweets occurring before and later in the conversations were relevant [69].
The study conducted on Twitter and Reddit used two sets of keywords: one for vaccines (vaccine OR
vaccines or MMR) and the other one to look for discussions around the vaccine-autism link ((vaccine OR
vaccines OR MMR) and autism). The average precision and recall estimates were 95% and 92%,
respectively [132]. However, the authors mentioned that their keywords might not have sufficiently
considered differences in cultures and norms.
Visualisation of data
The most commonly used form of data visualisation was created with Microsoft Office Software (e.g. Excel or
Word), with the table being the most frequently used format; 75 studies used this format to represent their
findings [32,45,52,54,56-70,72-84,86-90,92-107,110-117,119,121-124,126-128,130-134,137].
Various formats were used to visualise data. The most commonly used form of data visualisation was
created with Microsoft Office Software (e.g. Excel or Word), with the table being the most frequently
used format.
Visualisation of social media monitoring results is a developing field. There is a great deal of variation
in interpreting and presenting the results in order to communicate clearly and to have an impact on
policy.
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
23
Other visual representations of data identified were:
bar charts (32 studies) [30,45,53,54,57,59,60,66,67,75-77,79,84,88-90,92,96-
98,103,104,118,119,121,124,126,127,129,132,133]
line charts (27 studies) [30,32,60-65,67,74-76,85,87,89,91,92,106,109,110,124,126-129,131,133]
flow charts (11 studies) [60,62-64,75,77,78,88,89,92,121]
diagrams (seven studies) [60,74,77,78,89,92,121]
scatter charts (six studies) [30,84,88,110,121,126]
calculus diagrams (six studies) [30,60,89,91,121,126]
pie charts (five studies) [60,63,64,90,120]
area charts (two studies) [60,126]
dendrograms (diagrams representing a tree) (two studies)[84,126]
proportional boxes (one study)[54]
treemap (one study) [126]
bubble chart (one study) [68]
whisker graph (one study) [66]
box graph (one study) [66].
Six studies used screenshots to visualise their findings [45,73,78,111,113,125]. Two were screenshots from the
basic Facebook search browser [45,113]. One was from the automated Topsy tool [73], one was from Facebook
data taken from the automated Social-Media Lab [111], while one was from the automated VaccineWatch software
[78]. One study used screenshots from Pinterest [125] and one study used a story gram (creation software
unknown) [121].
Thirteen studies used social network charts [52,62,68,76,81,82,84,85,88,110,111,121,126]these were created
using the Gephi software package [138].
Visualisation of social media monitoring results is a developing field. There is a great deal of variation in the
interpretation and presentation of the results in order to communicate clearly and have an impact on policy.
5.2.3 Data analysis
Once extracted the data were analysed in different ways (see Appendix 8.3 for more information on each of the
studies aims). This included:
calculating the number of posts available over a period of time;
detailed content analysis to identify the frequency of particular concerns or conspiracies relating to
vaccination;
qualitative thematic analysis;
language and discourse analysis;
comparing social media posts to disease incidence or outbreak cases.
The most common type of analysis looked at sentiments expressed towards vaccination (70% of studies, 60/86).
Geo-localisation
Manually extracting location information
Some studies collected meta-data on location information (if available) or by manually screening for
locations within posts [65,82,132].
McNeil et al. filtered their data using location-specific search terms (United Kingdom, Scotland, Wales,
Northern Ireland, etc.) to screen Twitter users’ profile pages [76].
Manual methods do not always provide enough information, as Dunn et al. explain in their study: accurate location
information can be found in only a small proportion of tweets that have coordinates stored in the metadata of the
tweet (geo-tags), which corresponds to only 1% of tweets [66]
.
Automated mechanisms to retrieve location information
All the studies that used automated mechanisms to retrieve location information were conducted on Twitter and
were either global studies [53,54,67,80] or studies focusing on the US [66,87,91]. Four of these studies used
various types of software (Carmen, Geodict, Nominatim, GeoSocial Gauge) to extract location information from
tweets or user profile pages but did not provide detailed information about the software [66,67,80,91].
Only a small number of studies extracted and analysed location data from social media.
Country-specific social media monitoring was difficult due to the very small proportion of tweets,
posts, videos or profile pages with geo-location tags enabled or with public information on location.
Dictionary of terms for geographical entities may be used to automatically identify mentions of
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
24
In their study, Bahk et al. described the Vaccine Sentimeter, a tool that facilitates media monitoring for
vaccination. Their tool allows results to be filtered by location and to display articles geographically. Posts
are automatically tagged for location (if available) using HealthMap technology, followed by human curation
to validate and correct the automated tags [53].
Becker et al. used a dictionary of terms for geographical entities of countries, GeoNames, to automatically
identify mentions of countries (or cities) in tweets. For countries and cities with the same names (e.g. Bali is
both a city in India and an island in Indonesia), the result with the largest population was selected, which may
have led to some misallocations. However, the authors explain that this mechanism does not distinguish
between country mentions that are related to the vaccines being discussed and those that are not [54].
Tomeny et al. used a tool to define geographical entities [87]. As their study was focused on the US, they
used the Office of Management and Budget’s Metropolitan and Micropolitan Statistical areas. The tool
categorises areas into metro (>50 000) and micro areas and a list of counties. Tweets were geo-located
based on these entities by using the Twitter Global Positioning Systems (GPS) coordinates (if available) or
the user’s self-disclosed location in their profile page. While only 1% of tweets identified had GPS
coordinates, 63% provided self-disclosed locations on users’ profiles. The authors then used the Census
Reporter's API to resolve city names to the correct micro or metro areas. An external validity check was also
performed by manually locating 560 random user profiles and found that 531 profiles (95%) of locations
had been correctly identified [87].
Trends, content and sentiment
Categories of sentiments used and definitions
Sixty studies (70%) provided some type of sentiment coding and analysis, two of which did not specify the
categories used to code sentiments [119,120] (Figure 8).
Key messages
The sentiments most commonly used by researchers to categorise content relating to vaccination in
social media were neutral, negative, and positive. Only a small number of studies used more
complex sentiments (e.g. sarcasm, humour or hesitancy).
Manual sentiment analysis:
The majority of studies manually coded sentiments based on a thematic analysis.
Limitations of manual sentiment analysis: time-consuming, requires 2-4 trained coders and
codebooks, highly subjective, difficult to code sarcasm, slang or hyperbole.
Advantages of manual sentiment analysis
:
no computational skills needed, easy to conduct.
Automated sentiment analysis:
The studies using automated systems were mostly conducted on Twitter. They used machine
learning, based on a sample of manually-coded data.
Lightside was identified as an accurate and valid algorithm to code sentiments. Other non-evaluated
algorithms included Latent Dirichlet Allocation, Naïve Bayes, Brightview Classifier and Topsy.
Limitations of automated sentiment analysis: prone to bias as relying on manual coding to train
algorithms requires strong computational skills, and it is difficult to code sarcasm or irony.
Advantages of automated sentiment analysis: more accurate and less time-intensive than manual
coding.
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
25
Figure 8. Sentiment codes used across all studies
With the exception of a few studies that only searched for negative or anti-vaccination sentiments
[61,65,92], studies generally used a set of codes to characterise different types of sentiments in their data.
The most commonly used set of sentiments across all studies was positive, negative or neutral, or similar
(e.g. positive or negative). Thirty-two studies used these exact three sentiments to classify their data, 21
manually coding the data [32,45,54,58,68,69,72,73,79,86,93,96,98-102,112,113,116,127], and 10 using
automated coding [30,53,63,64,66,74,75,83,89,106]. One study also used a combination of manual and
automated coding, using a sentiment score for the automated coding (from 0-100)[70].
The other most commonly used types of sentiments were pro-vaccine, anti-vaccine, neutral, which were
used in 20 studies. Other combinations of these sentiments included pro-vaccination and anti-vaccination,
or simply pro and anti. Thirteen of these studies used manual coding [52,56,57,61,69,90,103-
105,110,114,125,133] and seven automated coding [62,77,87,88,91,92,129].
Some studies used different types of sentiment categorisations. Two studies conducted on YouTube used
the words encouraging, discouraging or neutral[94,95]. Two studies, one on YouTube[97] and one on
Twitter[60] used the terms for/in favour of vaccination, against/not in favour of vaccination, or neutral. A
study conducted on Reddit used the terms affirmative and negative [126]. Huang et al., used the question
does this message indicate that someone received or intended to receive a flu vaccine?(yes/no) to code
sentiments on Twitter using an automated system [67].
Only two studies used a more comprehensive and thorough list of sentiments, including humour/sarcasm,
concern, relief, and minimised risk. One of these was conducted in Dutch on Twitter, Facebook and fora
[131] and the other was conducted on Twitter in Spain [79].
One study, used WHO’s SAGE determinants of vaccine hesitancy framework to design a list of initial codes
and sentiments that was then reviewed after testing it on a few posts [113].
Two studies coded sentiments as binary variables (positive vs. negative), while another two studies
combined positive and neutral sentiments in the analysis [54,127].
One study conducted on Twitter explained that 0.1% of the 1 154 156 tweets identified were coded (1 151),
by selecting the first tweet with a random number generator and then coding every 1000
th
tweet [62].
In summary, the codes most commonly used to characterise sentiments were neutral (n=37), negative (n=33),
positive (n=31), anti-vaccine or anti-vaccination (n=20) and pro-vaccine, or pro-vaccination (n=20).
Twenty-six studies (43%) provided definitions of the different codes used for the sentiment analysis (Table 6).
37
33
31
20
20
6
3
2
2
1
1
1
1
1
1
1
1
1
1
1
1
Neutral, no opinion
Negative
Positive
Pro-vaccination, pro-vaccine
Anti-vaccination, anti-vaccine
Ambiguous, unclear
Hesitant, doubt
Discouraging
Encouraging
Anger
Anxiety
Concern
Humor/sarcasm
Minimised risk
Personal experiences
Question
Relief
Sceptical
Affirmative
Received or intended to receive flu vaccine?
Frustration
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
26
Table 6. List of codes, definitions and counts for sentiment analysis used in the identified studies
Category
Definition
Negative
Negative sentiments, attitudes or arguments against vaccination [32,65,68,74,91,99-
101,113,125,127]
Concerns about safety, efficacy, cost, resistance due to cultural or emotional issues,
programme suspension or other types of concerns [54,63,65,96,99,112,113]
Posts that would lead readers to be less inclined toward vaccination [53]
Source of the post judged unreliable [116].
Anti, anti-vaccine,
anti-vaccination
Posts expressing scepticism or denial of vaccines as a safe and effective way of
preventing diseases, or discussing alleged side effects such as autism [56,87,90,103]
Negative opinion about vaccination[89]
Discouraging people from vaccinating, advocating for the right to choose not to
vaccinate [90]
Discouraging
Uncertainty of effectiveness, possibility of adverse reactions, links/suspected links to
autism, and statistics stating the current absence of childhood diseases therefore
eliminating the need for future immunisations [95]
Positive
Posts communicating public health benefits or safety of vaccination, or encouraging
vaccination [32,53,63,68,74,96,99-101,112,113]
Positive tones, optimistic sentiments, supportive attitudes towards vaccination
[63,91,99,125,127]
Education on a vaccine [99]
Refuting claims the vaccine is dangerous [99]
Describes risk of not vaccinating [113]
Source of the post judged credible [116]
Pro, pro-vaccine,
pro-vaccination
Posts communicating that a vaccine is a safe and effective way of preventing diseases
[56,103]
Positive opinion about vaccination or current policies [89,129]
Expressing opposition to vaccine hesitancy or refuting claims made by anti-vaccination
groups [90,129]
Encouraging people to vaccinate, spreading scientific information about vaccinations
[90]
Encouraging
Positive messaging, such as stating that vaccines are safe and effective, benefit society
as a whole, have not been linked to autism and save thousands of children’s lives each
year [95]
Positive/neutral
No indication of public concern about a vaccine or vaccination programme [54]
Neutral
Posts that do not convey pro- or anti-vaccine messages, approve or disapprove of
vaccination; no sentiment or opinion [32,56,63,74,89,90,95,96,99,100]
Sentiments or attitudes ambivalent (containing both positive and negative sentiments)
or unable to be determined [63,68,98,127]
Reports of research findings, facts [53,74]
Individuals sharing information that they have been vaccinated, without any associated
sentiment [53]
Source of the post neutral [116]
Ambiguous
Contains both disapproving and approving information [32,96,100,101]
Unclear [113]
Hesitant
Universal opposition, opposition to content of selected vaccines or vaccination
schedules, conviction that vaccine-related injuries occur at higher rates than commonly
believed [129]
Indecision or uncertainty on the risks or benefits of vaccination [113]
Frustration
Message contains anger, irritation, contempt, criticism, or source is flabbergasted [131]
Humour/sarcasm
Message is funny or expresses sarcasm [131]
Concern
Message contains fear, concern, anxiety, worry, or grief [131]
Relief
Message contains joy, happiness, relief [131]
Minimised risk
Message minimises risk of vaccine-preventable diseases or complications [131]
Other types of content analyses
In addition to sentiment analyses, some articles provided a more detailed content analysis (for example by looking
at the topics discussed on social media, the prevalence of conspiracy theories, or the type of concerns raised by
social media users).
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
27
Four studies used the Health Belief Model to facilitate content analysis, whether qualitative or quantitative
[45,82,96,125].
Five studies described a qualitative thematic analysis of the content of social media, with an inductive or
deductive identification of codes and themes [71,108,109,119,134].
An additional seven studies focused their qualitative analysis on language and discourse analysis
[76,78,107,115-117,130].
Twenty-one studies coded the content in a quantitative manner, with a pre-defined codebook used to
ascertain the absence or presence of themes in the data [45,52,56,72-74,79,85,86,93-96,99-
101,103,106,113,118,125].
Four studies used supervised machine learning to code content on social media, including: analysis of image
sharing; measuring proportion of vaccine concerns; text-based analysis by comparing percentage of words
within different categories; and developing a method for story aggregation [59,82,105,121].
Four studies compared social media posts to disease incidence or outbreaks [66,124,129,131].
One study analysed content differences between humantweets and those coming from bots [57].
Methods used for coding and analysing sentiments
From the 60 studies that coded sentiments in social media, 40 (67%) used a manual coding system and 19 (32%)
used an automated system. One study used both a manual and automated system [54]. Almost all of the studies
that used an automated system were conducted on Twitter (n=16, 84%). Studies where sentiments were coded
manually most commonly analysed Twitter (n=14, 35%) as well as YouTube (n=12, 30%), Facebook (n=6, 15%)
and fora (n=4, 10%) (Figure 9).
Figure 9. Number of studies using manual or automated sentiment coding, by social media
Manual sentiment coding
Many studies explained that manual coding was conducted based on a thematic or content analysis of sentiments
[32,45,56-58,72,73,79,86,96,113,116,131,133]. Reviewers were provided with training, definitions of codes, and
codebooks to code the data.
While most articles simply coded the social media posts (tweets on Twitter, pages/posts on Facebook, videos on
YouTube, etc.), a small number of articles coded specific parts of the data:
YouTube: in addition to coding videos on YouTube, two studies also coded comments related to those
videos [93,98].
Facebook: while one study mentioned focusing on comments related to a specific post [105], another study
looked for sentiments on pages, groups and places [104], and another one obtained sentiments from main
objectives of the pages or groups, the title, content of the introductory description, and messages posted by
the group creator [133].
Twitter: Three articles also discussed how they coded sentiments on Twitter: one looked at the title,
headline and source or domain of tweets [68], another one used the verbs, adverbs and adjectives within
Tweets [72] and the last one used both Inlink sites and Twitter profile pages [90].
Three studies also coded the sentiments of hashtags used on Twitter, for example coding #vaccineswork as
positive and #killingusslowly, #cdcfraud or #vaxxedthemovie as negative [52,61,86].
One study also explained coding the hashtag #antivaxxers as positive and #provaxxers as negative, to
reflect the fact that social media users against vaccination do not usually talk about themselves as
antivaxxers or vice-versa [52].
Pinterest: the only study conducted on Pinterest coded sentiments from images, captions and links [125].
1
1
0
2
4
6
12
14
0
0
1
1
0
1
0
16
0
2
4
6
8
10
12
14
16
18
Pinterest Weibo Reddit Mix Forums Facebook YouTube Twitter
NUMBER OF ARTICLES PUBLISHED
Manual Automated
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
28
Reviewers to code the data
Fourteen studies used two reviewers to independently code the data
[45,56,62,69,70,73,79,96,97,99,104,110,125,131], with one study specifying that the two reviewers were
blinded to the research question [96].
Ten studies used three reviewers to code the data [32,72,86,101,105,112,113,127] [68,102], with one
study mentioning that the third reviewer was unaware of the study hypothesis [105]. In some of studies
that used two or three reviewers, the second or third reviewer sometimes just analysed a small sample of
the dataset to establish the reliability of the coding frame. Some studies explained disagreements were
resolved by discussion and consensus instead [45,57,69,104,105,113,131].
One study used four reviewers (with the fourth one consulted only for discrepancies) [57].
One PhD dissertation used only one reviewer [116].
Fourteen studies did not specify how many reviewers manually coded the data.
Interrater reliability of the coding
Twenty-three of the 40 articles that manually coded sentiments in social media tested interrater reliability to
identify the extent to which the codes correctly represented the sentiments measured.
Twelve studies used Cohen’s Kappa score to test interrater reliability, and obtained resulted between 0.31-
1.0 [56,62,69,73,79,93,94,102,105,110,127,131].
Two studies used Krippendorff’s alpha and obtained scores of 0.65 and 0.67 [45,96].
One study used Scott’s Pi, and obtained a measure of 0.84 [125].
Eight studies did not specify which methods were used to calculate interrater reliability, but stated
researchers reached 80100% agreement on coding [32,68,70,95,97,99,101,113].
Limitations of manual sentiment coding
Studies that used manual coding of sentiment described limitations such as the analysis resulting from a subjective
coding and the difficulties of manually ascribing sentiments to posts or tweets [69,93,131]. The difficulty of coding
sarcasm, slang and hyperboles was also discussed [52,58,68]. Manual coding was also described as labour
intensive, which could reduce the number of posts analysed [56].
Automated sentiment coding
Fifteen studies, all of them conducted on Twitter, discussed the use of leverage or supervised machine learning to
code sentiments in their datasets [53,60,63-65,67,74,75,77,83,88,89,91,92,106].
Five studies described how they trained the machine to learn how to code different sentiments, by first
manually coding a range of tweets (between 693 and 8 261 tweets) [65,74,75,89,92].
Four other studies used Amazon Mechanical Turk to train the machine in sentiment coding [67,77,83,91]. In
one study, in which Amazon Mechanical Turk was used to code a random sample of 10 000 tweets, the
authors rejected coding from three annotators as their agreement was below 60% [67].
One study also manually coded tweets after the computer-assisted coding to refine the classification results
and randomly selected tweets to validate model classifications after each round of coding [75].
Two studies used topic modelling to code Tweets and Reddit posts [83,126]. The studies used Latent
Dirichlet Allocation (LDA), an unsupervised machine-learning algorithm that automatically determines topics
in a text. One of the two studies explains that LDA assumes words in documents co-locate near other
words (possibly across documents) because they are related, and the algorithm collects and reports groups
of such related words, with the groups representing topics[83].
One study conducted on Twitter evaluated three standard classification algorithms to automatically predict
sentiments based on manually rated tweets: Naive Bayes, Maximum Entropy and a Dynamic Language Model
classifier [30]. In the end, the authors selected Naive Bayes for their study, using the Natural Language Toolkit.
One study conducted on Twitter used Lightside, an open-source platform that performs feature-
extraction[87]. This study first trained the algorithm on 2 000 manually coded tweets, coded independently
by two reviewers, before using Lightside to automatically code the rest of the tweets. The model was
evaluated and judged as accurate and valid by the authors [87].
A study conducted on Facebook and Twitter used BrightView classifier, provided by Crimson Hexagon to
code sentiments [129]. BrightView is a supervised learning algorithm based on stacked regression analysis
of simplified numerical representation of text [129]. To train the algorithm, some of the investigators
manually coded a random sample of tweets and posts before and after the automated coding.
Finally, one study used Topsy to analyse sentiments in tweets. Topsy was software (bought by Apple in 2015) that
used natural language processing to establish a sentiment score ranging from 0 (negative) to 100 (positive) [70].
Limitations of automated sentiment coding
Some studies discussed the limitations of automated machine learning systems for sentiment analysis of social media.
Three studies discussed the difficulties for models to handle sarcasm and irony [60,89,126] and one explained that slang
and abbreviations may make it more difficult for machine learning systems to correctly identify sentiments [70]. This
study also mentioned that when URL are shared in tweets they are not coded, however they include important
contextual information that may help the sentiment analysis [70]. Three studies also mentioned that automated systems
still rely on manual annotation of some part of the data to train the system, which is prone to biases [53,70,126].
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
29
Reach analysis
Forty-nine studies measured potential social media reach by examining interactions of different social network
communities in order to understand how information can spread and be shared on social media.
Twenty-five studies measured the conversational discourse and sentiment in comments to posts on vaccines
[32,45,55,71,72,93,96,98-100,103-105,107-113,117,120,121,125,126].
Five studies included the number of followers that posters had on different social media platforms, while
two of these specifically looked at how the number of followers a user had affected the amount of retweets,
and tone of sentiment in relation to those retweets [30,65,82,86,92].
Nineteen studies mentioned the amount of retweets that key posts received [56,59,65,67,69-
71,73,74,76,79,80,82,84,87,88,129,131,132], with seven of these studies analysing the effect of retweets
on reach and influence in more detail [56,59,65,69,70,74,76,79,80,88,129,131].
The following is a brief overview of some of the studies that assessed the impact of followers and retweets on
vaccine discussion across different types of social network communities, and the effect this had on overall
sentiment towards vaccination.
One study investigated how two types of communities interacted with each other within conversations on
health and its relation to vaccines [88]. From a retweet network, of 660 892 tweets published by 269 623
users the study compared structural communitywith another opinion groupand used community
detection algorithms and auto-tagging to measure the interaction, sentiment and influence that retweets
had in conversations between the two communities.
Another study focussed on shared concerns relating to the HPV vaccine[82] across different countries.
One study looked at communication patterns revealed through retweeting [80]. They assessed the impact
of various sources of information, contrasting diverse types of authoritative content (e.g. health
organisations and official news organisations) and grassroots campaign arguments (with the anti-
vaccination community views serving as a prototypical example).
One study analysed the content and source of the most popular tweets relating to the controversial death of
a child in Spain - an unvaccinated child who contracted and later died from diphtheria [79].
Another study analysed both tweets and retweets together to compare the weekly number of online social
media messages with the weekly number of reported measles cases in a Dutch measles outbreak [131].
One study looked at a combination of 83 551 tweets or retweets from 30 621 users [65]. The study defined
social connections as the sets of users that followed, or were followed by the users that tweeted about HPV
vaccines.
One study specifically looked at the content of two vaccine-related Twitter datasets, with a focus on
retweets and their frequency, and possible influence of retweet frequency on sentiment [56].
5.3 Using social media monitoring to inform vaccination
communication strategies
Studies examined the interactions of different social network communities to try to understand how
information can spread and be shared on social media and how it can impact overall sentiment
towards vaccination.
The majority of studies measured:
the conversational discourse and sentiment in comments
the number of followers that posters had on different social media platforms
Some studies recommended that health authorities, governments and/or healthcare professionals
start monitoring social media to detect increases in online activity, shifts in sentiments, or other
signals that may influence vaccination uptake or confidence in real time.
Some studies also acknowledged that social media monitoring could help health authorities anticipate,
understand and respond to public questions and concerns.
Results from the monitoring also led some studies to discuss the need for health authorities to
increase their presence, and their popularity on social media.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
30
None of the studies formally evaluated how social media monitoring methods and information gathered from
monitoring can be used to inform communication strategies. However, some studies provided recommendations
and suggestions as part of their discussions and data interpretation.
Detect increases in online activity, shifts in sentiments, or other signals
A first set of studies recommended that health authorities, governments and/or healthcare professionals start
monitoring social media to detect increases in online activity, shifts in sentiments, or other signals that may have
an influence on vaccination uptake or confidence in real time [53,64,75,87,88,112,118,128].
A specific recommendation from Kang et al. is that social media monitoring should also incorporate a
semantic network analysis of sentiments to improve understanding of the scope and the variability of public
attitudes and beliefs [68].
Two studies recommended identifying the platforms people use to gather information on vaccines or those
where misinformation or low-quality evidence is shared [56,66].
Seeman et al. also explained that social media monitoring is not only about identifying content and that
health authorities should also measure engagement
2
to detect more popular content and better understand
how it is shared [128].
Two studies explained that social media monitoring can be used to examine the effectiveness of vaccine
promotion strategies and the impact of positive information on vaccination [64,88].
Another study also stressed the importance of flagging anti-vaccine websites
[128].
One study also described social media monitoring as an important source of information for adverse-events
following immunisation surveillance, particularly anxiety-related reactions [112].
Anticipate, understand and respond to public questions and concerns
Some studies also acknowledged that social media monitoring could help health authorities anticipate, understand
and respond to public questions and concerns [32,53,68,82,109,112,113,120,125,131].
Seven studies concluded that real-time information about which topics or issues the public is discussing on
social media would enable health authorities to develop tailored, targeted, cost-effective and responsive
public engagement or communication strategies, for example by informing press releases and vaccination
campaigns, or by directing responses to online content more effectively [53,66,96,117,118,130,133]. More
specifically, health authorities should seek inspiration from highly shared information to guide the
development of improved messaging using visual designs, features and language formatting [128].
Two studies raised the possibility of producing spatio-temporal indicators from social media monitoring
which could allow public health organisations to better target their health information campaigns to reach
those who have concerns about vaccination, for example using an interactive map that would produce
notifications when there is an increase in concern about vaccination in certain areas [84,87].
Two studies also discussed the usefulness of social media monitoring during infectious disease outbreaks as
it may allow health authorities to create adaptive messages during different stages of the crisis, respond to
specific concerns on social media and provide rapid responses carefully planned in advance [85,120].
Finally Seeman et al. referred to the use of counter-marketing strategies, which they defined as a way to
proactively identify and expose misinformation and anecdotal evidenceand engage in publicly viewable
web discussions with authors of anti-vaccine postings[128]. They argue this type of communication shows
constructive and transparent engagement and provokes dialogue rather than shutting down dissent [128].
Increase presence, and popularity on social media
Results from the social media monitoring studies also led some authors to discuss the need for health authorities to
increase their presence, and their popularity on social media [32,66,90,94,106,109,125,127], with one study
discussing the ethical implications of non-engagement [113]. A few suggestions were made for health authorities
wishing to increase their presence on social media and these are summarised in Table 7.
2
Engagement of users in social media can be measured by analysing comments, shares, likes, retweets, mentions, clicks and
saves.
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
31
Table 7. Suggestions for increasing presence on social media identified in scoping review studies
Messages
Address issues related to vaccine hesitancy (safety of vaccines, effectiveness, benefits, etc.) as well as
enhancing public trust in relevant scientific institutions in order to lower institutional distrust [68]
Provide precise information without too much detail [106]
Use lay terms and appropriate language [112]
Use more narrative styles of communication, discuss the protective effects of vaccines, describe the
lives saved through vaccines [74,125]
Share messages of empowerment [130], be respectful [119]
Communicate evidence-based information [112]
Aim to enhance public understanding of science and the scientific methods and emphasise the
importance of scientific information [90,105]
Immunise users against the critical arguments they are likely to encounter in the online platform they
are entering [118]
Share new research by sharing media articles rather than publications to engage the general public [74]
Avoid using the name of brands in messaging which could lead to more negative reactions [75]
Avoid downplaying negative sentiments, for example by saying side effects are minimal [75].
Use of emotions
The use of emotions is controversial. While some authors advise that communication should be
emotional and cognitive [94] and that it should elicit emotions such as anxiety, fear, regret and blame
[117], others recommend not to use anxiety or anger and avoid fear [106]. One study also explains
that fear can be used if balanced properly but should be avoided with certain audiences [130].
Emotion-filled arenas call for new rhetorical strategies to complement logical arguments [120]
Be careful with unclear, hostile, inaccurate messages from pro-vaccination sources [105]
Be non-judgemental and transparent [113].
Mechanisms
Provide online dialogues including hosting real-time Q&A sessions [127]
Create twitter chats [74]
Adapt messages deliver more persuasive messages on weekends instead of the middle of the week
as negative opinions are more prevalent on these days [64]
Ensure sufficient resources are available for regular communication [120].
Reach and engagement
Enable public health institutions to become skilled influencers [32]
Decide whether using individuals or organisations will have a higher impact [120]
Use key opinion leaders on social media to disseminate messages which could reach users who would
not normally follow the social media accounts of health agencies [56,75]
Use hashtags, mentions and links to increase dissemination and reach and to appear in searches and
avoid echo-chamber effects where people only hear from their own circles [74,75]
Non expert sources may be more important than expert sources when reaching parents in online fora [115]
Prioritise parents with children aged two years and under, as they are highly motivated to participate in
discussions, eager and responsive [115]
Capitalise on awareness days to raise engagement and interest [74]
If possible use consistent, long-term activity of communications, rather than only media or event-
driven episodic messaging [124].
Responding to social media posts
Correct misinformation on platforms (advocates and medical professionals) [58] [82,109,112,113]
Develop systems to instantly detect anti-vaccine tweets and directly reply with counter messages [87]
Answer questions using the same social media platform where the question was asked [124]
If faced with bots or trolls, emphasise that the credibility of the source is dubious and that users
exposed to such content may be more likely to encounter malware [57].
In addition to these suggestions and tips, Nicholson et al. provided seven key recommendations as to how
advocates of immunisation, such as vaccination programme managers, public health institutions, or healthcare
professionals may engage in social media, particularly fora with a large group of fence-sitters seeking information
[119] (see Box 1 below).
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
32
Box 1. Recommendations for health professionals and other
advocates of immunisation on how to best participate in online
discussion forums about immunisation [119]
Resolve questions of whether to actively engage. There is often a dilemma about whether to
respond to posts on forums, which are often monopolised by vocal opponents of immunisation. Here
the concern is that participation in the discussion legitimises and even amplifies anti-vaccine
arguments. The absence of vaccine advocates may create a void into which misinformation could reign
uncontested.
Prepare. Engage a group of vaccine advocates; agree on the most important messages; and who will
advance them. Ensure sufficient numbers of designated experts and advocates who are able to type
quickly and are available to respond for the duration of the discussion, or arrange ‘shifts’.
Diversify the support base. Ensure that each participating advocate is able to address various
issues including vaccinology, disease outcomes, primary care practice, and consumers and
professionals who can identify with people experiencing specific outcomes, but who support
immunisation.
Set the agenda. Vaccine advocates should lead the discussion and avoid the traditionally defensive
mode in which debates operate.
Introduce messages known to positively influence behaviour. Promote messages that appeal
to the core parental values of protecting children from diseases and facilitating the telling of stories
around disease impact. Messages known to increase intention to immunise include emphasising
potential regret of not vaccinating in a non-confrontational way; appealing to altruism in terms of
protecting the vulnerable; and the band-wagonphenomenon where learning that others are
vaccinating makes a parent more likely to want to do so. Social media’s advantages include the
capacity to role model positive health behaviour. Those who have immunised could be asked what
factors influenced their decision, which allows wavering parents to see potential advantages of
immunisation that they may not have considered.
Do not oversell the product. It is important to acknowledge that vaccines can produce common
minor side effects and rare but serious reactions. Promotional messages that also acknowledge the
side effects of immunisation paradoxically lead to lower risk perceptions. Giving information on what is
known about vaccine risk defines the boundaries between fact and fiction and signals that the person
conveying the information is knowledgeable and balanced, increasing their trustworthiness.
Do not attack the opposition. Compassion and respect should underscore vaccine advocacy. Direct
attacks of the opposition often result in vitriolic debates played out before ambivalent audiences who
will often make their decisions via an assessment of source credibility.
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
33
5.4 Uses, benefits and limitations of social media as an
intervention tool in relation to vaccination
A total of 17 articles describing social media as an intervention tool in relation to vaccination were identified in this
review. The articles were divided into three categories of intervention, as described below.
Interventions using social media as a source of information
The use of online group discussions to engage the public
The development of websites with an interactive component.
Annex 8.6 provides an overview of the studies on social media as an intervention tool in relation to vaccination,
including a short description of the intervention and outcomes/results.
5.4.1 Interventions using social media as a source of information
Ten articles, published between 2012 and 2019, described how social media could be used to provide information
about vaccination to the public, thereby aiming to address vaccine hesitancy [43,139-147]. Of these, five
interventions used Facebook, one used an online forum, one used YouTube and three used a mix of social media
tools (Twitter, Facebook and Vine; Instagram, Facebook, Twitter and YouTube and Facebook and Instagram). Most
of the studies were conducted in the United States (n=4), with other studies conducted in Germany (n=2), Canada
(n=2), Italy (n=1), and globally (n=1). Three interventions addressed vaccination in general, five focused on HPV
vaccination and two on influenza vaccination.
Exposure to messages and video contests
A qualitative study found that exposure to messages and video contests (participants submitted videos where
they were asked to finish the sentence I received the HPV vaccine so that I have time to...) about HPV
vaccination on Facebook, Twitter, and Vine seem to increase uptake of HPV vaccination among some
participants[147]. None of the quantitative studies assessing the impact of social media information in relation
to vaccination, vaccination uptake or willingness to get vaccinated found a significant effect [144,145].
Impact of advertisements
One study, looking at the impact of advertisements relating to HPV vaccination posted on Facebook
(including cues for action to motivate adolescents to seek vaccination), found that out of 155 110
adolescents reached and 2 106 engaged with the messages (e.g. liked or shared posts), only 152 had
received at least one dose of HPV vaccine (significance not measured) [143].
Information on social media
None of the quantitative studies found that providing information on social media regarding
vaccination significantly increased uptake or willingness to get vaccinated, which may reflect the
methodological challenges of establishing a causal link between vaccine behaviour and social media
exposure.
Information supporting vaccination on Facebook in the US was found to significantly decrease
perceived barriers to HPV vaccination, decrease perceptions of risk, and increase knowledge about
the HPV vaccine.
The content matters: loss-framed messages on Facebook were associated with a significantly higher
intention of getting vaccinated than gain-framed messages.
Narratives containing information about vaccine adverse events corresponded to decreasing intention
of getting vaccinated.
Online group discussions
A Facebook-assisted teaching method increased knowledge and intention of getting vaccinated.
Parents and friends were found to have a strong influence on vaccination decision-making, whether
they shared their views online or in person.
When comparing factual information to personal experiences in fora, no difference was found in the
number of responses to each type of post, but responses to personal experiences were more
emotional.
Interactive websites
Interactive websites with a space for parents to contribute with content and discuss concerns were
found to significantly reduce parental concerns around vaccination but no impact was found on
vaccine-related attitudes or vaccine uptake.
A survey in the US found that 50% of parents who accepted, delayed or refused vaccines would use
interactive websites if available.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
34
Differences between gain-framing (i.e. health benefits from getting the HPV
vaccine) and loss-framing (i.e. negative consequences of not getting the HPV
vaccine) messages[142].
One study found that participants exposed to a loss-framed message on Facebook had a significantly higher
intention of getting vaccinated than those in the gain-framing condition (p<0.05). This same study also
found that participants exposed to Facebook messages perceived lower barriers to getting vaccinated
against HPV than those exposed to a newspaper message (p<0.05), but also perceived the severity of HPV
to be lower (p<0.05). No significant effect was found on the perception of benefits of vaccination [142]
.
Impact of pro-vaccination or anti-vaccination comments on risk perception
One study conducted in Germany on Facebook found that readers exposed to the post and comments
supporting vaccination had a lower perception of risk than readers exposed to comments opposing
vaccination. The study also found that participants who read the post with likes had more positive attitudes
towards influenza vaccination than those who read the version of the post without any likes [145].
Source credibility
Another study conducted in Germany attempted to discern the impact of source credibility on resulting
vaccination perception [140]. These researchers found that, irrespective of the source, narratives containing
information about vaccine adverse events corresponded with decreased intentions of getting vaccinated [140].
Impact of social media on knowledge of the HPV vaccine
A study conducted in the United States found that participants with greater exposure to Facebook post
notifications were more likely to have a higher level of knowledge about HPV and the vaccine, but were no
more likely to be vaccinated [144].
A Canadian study that used social media as one tool in an education campaign on HPV vaccination and
cervical cancer concluded that social media did not have a significant impact on HPV vaccination uptake in
university students [146].
Another study in Canada found that the rhetoric-style of a video (whether evidence-based or anecdotal) did
not have an impact on medical students’ responses to informational questions about seasonal influenza and
vaccines [43].
A qualitative study in the United States found that a large majority of the 18 university students in their
study exposed to messages on Facebook, Twitter and Vine had heard of HPV vaccination following the
campaign and most of them believed the vaccine was successful at preventing cervical cancer [147].
Use of existing national or international social media platforms developed to
respond to vaccine hesitancy
One study looked at Vaccines Today, a website with various associated social media channels (Instagram,
Facebook, Twitter, YouTube) developed to provide factual information about vaccination [139]. The study
found that their most popular content was a post on their Facebook account how measles can change a
life(average reading time: 7 minutes 16 seconds, views: 233 996) and an animation video on YouTube
showing how herd protection works (53 000 views).
The second study described an Italian Facebook campaign by the Italian Alliance of Vaccination Strategies,
designed to share information about vaccination three times a week using short messages (with images)
selected by health professionals and scientific communication experts [141]. The study looked at the
number of likes on each of the posts, and found that Facebook event pages were the most popular type of
communications, followed by press releases and scientific publications. Press releases were the most shared
types of posts, followed by scientific publications and institutional documents. Facebook users were found
to like and share more posts on Fridays.
5.4.2 Use of online group discussions to engage the public
The impact of engaging the public in online group discussions on vaccine uptake and confidence was analysed in
three articles - one study was conducted in Taiwan [148], one in the Netherlands [149], and one in Germany
[150]. The three studies aimed to observe effects of online group discussion on behaviour and attitudes about
vaccination.
The Taiwanese study found that participants receiving a Facebook-assisted teaching method were more
likely to have the intention to be vaccinated than others who received traditional teaching instructions
[148]. Knowledge concerning vaccination was significantly higher in the intervention group than the control
group.
The Dutch study asked parents to share opinions in an online group discussion about vaccinating their
daughters [149]. The results show that parents online had a similar influence to that of friends and offline
peers. Nonetheless, family members appear to have the strongest influence, suggesting that social media
interventions might need to concentrate on family members rather than individual decision-makers.
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
35
The German study compared the use of factual information and personal experience in an Internet forum
post on the subsequent nature and volume of responses [150]. The researchers found that while personal
experience posts did not result in significantly more responses, the responses were more emotional in
nature [150].
5.4.3 Development of websites with an interactive component
Four articles reported on websites with built-in interactive, social-media like components and its effects on vaccine
attitudes and beliefs, detailed below.
Two articles report on the same study, conducted in the United States between September 2013 and 2016
[151,152]. Both studies are based on a single site randomised controlled trial testing the effects of
interactive websites. Despite reporting on the same study, both articles had different outcomes: in one
study, interventions were associated with significant reductions in parental concerns surrounding
vaccination - nonetheless no change was observed in vaccine-related attitudes. In the second article,
infants with parents who had received the intervention were twice as likely to have received MMR
vaccination than infants with parents in the control group, although this was not significant.
A third article reports on an interactive website developed by the Italian health authority to inform the
population in relation to vaccination safety and benefits [153]. The website was intended for the Italian
population in general and for healthcare workers. It used Twitter to advertise its use and drive traffic to the
website. Most visits to the website were from populations in Rome and Milan. The website remained one of
the top results in Google after only one month of intervention, indicating a high number of visits and the
potential of such initiatives.
One article aimed to create a web-based tool to provide evidence-based information where parents can
contribute with content and discuss concerns with other parents and vaccine experts [154]. Based on a
manual medical record review, a set of surveys was sent randomly to parents in the US who accepted
vaccines, parents who delayed vaccination and parents who refused vaccination. Fifty percent of parents in
all three vaccine groups reported they would use the web-based tool more often. The overall results from
this study suggest that the web tool may represent an effective intervention tool to help parents make
informed vaccination decisions for their children.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
36
6. Discussion
This report provides the results from a systematic scoping review, conducted with the following objectives:
to identify preferences for using social media as a source for vaccination information and the influence that
social media have on individuals’ perceptions of vaccination;
to identify different social media monitoring methods and tools in monitoring vaccination and assess their
strengths and weaknesses;
to review how social media monitoring methods and information gathered from the monitoring can be used
to inform communication strategies;
to identify the uses, benefits and limitations of using social media as an intervention tool in relation to
vaccination (i.e. how effective social media are as an intervention tool for increasing vaccination).
An extensive database search led to the inclusion of 115 articles: 13 on the use of social media, 85 on social media
monitoring, 15 on interventions, one with both social media monitoring and a social media intervention, and one
with both an intervention and an analysis of social media users’ opinions about social media. The results from
these three categories are summarised and discussed in detail below.
6.1 Use of social media for vaccination information
In this systematic scoping review, social media were identified as a common source of information on vaccination.
However, the way in which this information is conveyed and consumed was found to vary according to the social
media platform used, countries and specific populations. This finding resonates with a global study on social media
usage, which found that social media dynamics were dependent on the local social dynamics. This study concluded
that, in fact, users shape social media platform dynamics rather than the other way around [23]. Individuals
commonly use social media to look for health information, and more importantly, are often exposed to information
about vaccination online without necessarily looking for it.
Summary of the key findings on the use of social media for
vaccination information and the influence on perceptions
The types of social media platforms and their use vary by country and by population group (e.g. pregnant
women’s consumption of vaccine information on social media was found to be different from the general
population).
There is a need to have an understanding of who may influence decision making, for example one study
included in this review found that family members often have the strongest influence on vaccination-
decision making, highlighting the importance of communication and interventions targeting entire families
and possible influencers rather than solely at the main decision-makers [149].
Social media interventions to address vaccine hesitancy should target specific populations. Public health
authorities should first seek to understand what platforms their local populations use (e.g. parents, versus
adolescents, versus pregnant women), as well as the local context in which individuals use social media,
before developing social media communication strategies.
The negative relationship between the extent of social media usage and views about vaccination should
certainly be evaluated further, especially as social media are an important communication channel in public
health. These findings may reflect the fact that information on social media has traditionally been more
negative concerning vaccination: as users are more often exposed to negative information and views
concerning vaccination online, they themselves become more hesitant to vaccinate [13,22].
6.2 Methodologies to monitor social media in relation to
vaccination
This scoping review found a large number of studies published on social media monitoring and analysis around
vaccination, with significant increase in the number of articles published since 2015. This increase exposes a
growing academic interest in the field of social media in health, particularly within the context of fake news and
post-factual societies[155]. The large number of studies analysing social media relating to vaccination may also
represent an acknowledgement of the contribution of social media to growing vaccine hesitancy and a need to
better understand how users communicate about vaccination on social media and how information about
vaccination spreads between and within online social networks [156].
However, social media monitoring still represents a new methodological approach for health research, with a lack
of standardised methodology for collecting and analysing data. While the large number of articles identified via the
scoping review provided sufficient evidence to summarise methods (e.g. tools for retrieving social media data, or
methods for sentiment and location) that have been used to monitor social media, almost none of the articles
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
37
evaluated the precision and accuracy of their monitoring and analysis methodologies. Lack of evaluation and
standardisation is reflected in the fact that articles published on social media monitoring are extremely varied in
style, methodology, and complexity. Social media monitoring constitutes a new research methodology, with many
challenges for defining methodologies but also opportunities for public health.
A discussion of the findings from the social media monitoring studies is presented below based on the three social
media monitoring phases 1) preparation 2) data extraction and 3) data analysis.
1) Preparation
Purpose of social media monitoring
The purpose of social media monitoring in relation to vaccination dictates the platform/s to be monitored, period of
monitoring, analysis and messaging to be conducted.
Representativeness
Generally, the aim of social media monitoring studies was to provide a better understanding of how the population
thinks and talks about vaccination, thereby requiring representativeness to be able to generalise the analysis
results to whole populations. Studies where the objectives are to obtain an overview of the use of social media and
perceptions of vaccination tend to position social media monitoring as an alternative to surveys or qualitative
interviews in obtaining data on vaccination beliefs and opinions, without acknowledging that the notion of
representativeness radically changes with social media. There are limitations on the lack of representativeness of
social media populations. There is evidence that social media users, especially those discussing vaccination, tend to
represent particular population groups (e.g. younger, often female individuals) [87]. In this way, social media
monitoring cannot be used as a research tool to increase representativeness and access entire populations, but it
can be a research methodology for studying a new type of population.
Social media users could be considered as a new independent population group, and the field of social media
monitoring could be seen as an opportunity to understand what information social media users are exposed to and
how information about vaccination is shared and spread online. However, social media monitoring comes with
challenges in terms of representativeness, as access to data is often limited due to inaccessible private content, the
challenge of studying all social media platforms at once, or limitations imposed by automated software. Redefining
social media monitoring for vaccination as studying a new type of population may prove valuable for public health,
recognising that social media offer opportunities that go beyond updating existing research methodologies. Social
media monitoring opens the door to more dynamic research that continuously evolves and responds to a
constantly evolving world.
Twitter bias
One of the main reasons why there has been such a bias towards the use of Twitter in a majority of studies within
this review may be that Twitter provides the most openly available API, both for free and paid access for
developers and researchers. There is a need to apply caution to results collected solely from Twitter. Studies using
these freely collected tweets only represent a small 1% sample of all tweets posted and are therefore not
representative of all posts on vaccine hesitancy [157]. Accessing the free Twitter API also involves issues related to
periodical collection the API itself is restricted to intermittent collection points, so that Twitter’s servers are not
overloaded with requests. This means that any collections are limited to pockets of time and do not represent
continuous data collection. Therefore even with access to a 1% sample over a period of seven days, there will be
gaps in the stream of tweets collected, as Twitter moderates collection times meaning that even the 1% sample
is truncated [158]. Furthermore, population groups are known to access different types of social media platforms,
and a focus on Twitter may overlook the concerns of younger individuals, such as teenagers, who more commonly
use Instagram or Snapchat [159].
Ethical considerations around social media monitoring
It is worth noting that most studies included in this report did not seek ethics approval, as it is often considered
that information on social media is public (unless otherwise stated, for instance in private profiles on Facebook).
The argument is that social media users who make their data public do not constitute human research subjects,
who would be defined as living individuals about whom an investigator obtains data through interaction with the
individual or identifiable private information [160].
Even though researchers in previous studies may not have been legally compelled to obtain full social media ethics
approval, the lack of guidance on good ethical conduct when using social media information can be considered a
cause for concern. Issues of confidentiality and anonymisation of data still arise, as some studies included in this
review published screenshots of users’ data that included users’ profile names. Another issue relates to data
coming from minors, which should be considered more carefully, even when publicly available [160]. While these
concerns should not unduly hinder the development of social media monitoring as a research field, they should
however highlight that there is a need for guidelines to ensure ethical conduct, respect for social media users, and
therefore the importance of submitting research proposals on social media monitoring to ethics boards for
approval. While some guidelines have been published on ethical conduct regarding studies recruiting participants
online, such as those from the British Psychological Society [136], there is still a large gap regarding social media
monitoring.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
38
Recent controversies with regard to the exploitation of users’ data in the Facebook and Cambridge Analytica
scandal, and the public outcry of users feeling unnerved about being monitored and manipulated, have indeed
opened up discussions and initiatives for legislation concerning the ethics of handling user data from social media
in both corporate marketing and research [161]. Facebook announced a number of API changes designed to better
protect user information between 2017 and 2019. As a result, these restrictions, along with GDPR laws, pose
further limitations on the research that can be conducted on social media platforms [162,163].
General Data Protection Regulation (GDPR)
The overall aim of the recently implemented GDPR is to increase people’s control over their own personal data and
to protect all EU citizens from privacy and data breaches in an increasingly data-driven world[164]. For
companies, organisations and researchers this means obtaining consent to use and retain customers’ personal
data; while granting more rights to the ‘data subject’ to be informed and to control how their personal data are
used. In light of this, digital analytics companies have had to legally adopt the GDPR into data access policies in
order to harvest social media data where there is automated access to social media APIs, such as the Twitter
Firehose, and the Instagram API. Moreover, social media platforms themselves have had to adapt how data are
shared with third parties [135]. Such recent policy changes may in turn change the way that researchers who used
automated software to sample social media, will have to anonymise data in the future. They may also restrict
which sections of social media platforms (e.g. public Facebook pages versus private groups) are available for
research [165]. There have also been steps within the research community to devise ways of anonymising user
data in social media research, by differentiating research use of these data from marketing analytics, and devising
ways to make sure academic use falls both within GDPR and wider ethical guidelines [135].
2) Data extraction
Complexity of monitoring social media on different platforms over long periods of time
This scoping review has found that information on vaccination is available across different social media platforms,
and that different methods and formats of communication are currently being used to post, share and spread
information about vaccination online (e.g. tweets, Facebook and forum posts and comments, images, followers,
videos and Pinterest clippings). This makes it hard to have a standardised way of collecting and analysing data, or
to obtain a continuous sample of what people are discussing or sharing with regard to vaccine hesitancy.
The majority of studies that had the largest datasets, collected over longer time periods, tended to be those that
had access to the API of a social platform such as Twitter, or to automated data collection and media monitoring
tools. Studies with smaller samples experienced more limitations and used less sophisticated keyword searches,
often relying on manual data collection, and were thus constrained by time, resources, and the limitations of the
browser tools being used within the social media platforms.
It was found that the studies using browser search tools within the social media platforms they were analysing had
smaller samples that were less representative of the overall population they were studying, while the periods of
collection were shorter. The use of browser search tools in general can limit the size and time period of data
collected, as these search tools are built for basic searches by the platforms users, who use them on a smaller
scale for general search queries, rather than more detailed comprehensive data retrieval and analysis.
More intensive use of social media APIs requires a strong knowledge of programming, analytics and data harvesting.
The use of automated software for media monitoring through commercial social media analytics platforms offers
access to such data, however these can be cost-prohibitive, or do not give public health organisations and researchers
the kind of data that they require. This is because most automated digital analytics platforms are built with corporate
marketing analysis and branded content in mind and thus not able to directly tackle, annotate or analyse research
questions through a social sciences or public health lens [157]. Once harvested, the data need to be cleaned and
analysed in a way that answers public health questions, rather than those linked to brand marketing (e.g. brand
strength, brand influencers, and brand trends) or product trends [157] [166].
In recent years there have been increasing opportunities for academics to work in partnership with data analytics
companies to forge a better understanding of how to look at social media images and text from a social sciences
and public policy perspective [167]. While manual searches using browser-based tools within social media
platforms have been found to yield smaller sample sizes over shorter periods of time, it might be assumed that
studies that have used the paid version of the Twitter API (the ‘firehose’) via automated monitoring, have a more
representative sample. This is because access to paid data has been found to offer researchers an array of
tweets/posts, users, and more data for analysis, usually with access to all historical and current tweets. However,
there are still issues with the relative openness of the paid access to Twitter, Facebook or YouTube APIs, which
themselves are still proprietary, and do not advertise the mechanisms behind collection or output of data. Thus,
public health organisations and researchers do not know the full details of commercial API sample data, or indeed
what percentage of all data they are given. Thus, the commercial/automated API acts as a blackboxfilter that
may not yield representative data [168,169]. Therefore, not only does the commercial API preclude analysis as to
the representativeness of the sample but it also prevents public health officials and researchers from fully
comparing studies over time, as the API sampling algorithm itself will change.
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
39
Studies of social media may often fail to use standardised methods that permit interpretation beyond individual
studies. In developing methods to analyse social media data, researchers have not drawn on a coherent body of
agreed-upon methodologies. Rather, analysis of the literature shows that methodological choices differ considerably
from one paper to another, and the validity of the chosen methodologies often cannot be adequately demonstrated
[170]. This may be due to the ever-changing landscape of social media platforms, in terms of the structuring,
formatting and access to data. Thus, while there is no standardisation of methods for conducting social media
monitoring, it has been found that studies tend to try a little bit of everything creating an amalgamation of research
methods and analyses. There are currently no standards for the right sample size per social network, no
recommended time period for different types of analysis per platform, and no recommendations for studying outside
or within the extremes of positive or negative views (which are not always representative of the general population).
There is also currently no general standardisation of which specific API tools or analytical classifiers should be used for
good overall analysis of network discourse, interaction or trends, within the academic or commercial data analytics
sphere. However, despite all of these drawbacks, this review has found that the fast-evolving nature of different social
media platforms, the cross-over of shared data, boundaries to privacy, and public policy surrounding public discourse
on vaccine hesitancy and disease outbreaks, may necessitate a more methodologically diverse approach to keep up
with ever-changing developments. It may also be appropriate for methodologies to remain flexible, as the nature and
access to social media discourse on vaccines and public health changes.
Changes in platform content
In terms of data flows, as concerns grow about the presence of anti-vaccine sentiments on social media and the effect
this has on real-world vaccine uptake, social media platforms are beginning to listen to requests from public bodies
and are gradually altering their content. One such platform is Pinterest, which has recently taken anti-vaccine items
off its platform [171]. Facebook and YouTube have also pledged to remove anti-vaccine videos from their platforms
[172]. YouTube has taken revenue-generating ads off anti-vaccine videos, in a pledge to de-monetise anti-vaccine
groups [173]. Facebook has additionally banned anti-vaccine ads targeting specific groups [174], while Twitter has
included a Know the Factsbox for users searching for anti-vaccination tweets [175], and Instagram is using AI to
filter anti-vaccine content via an information box directing users to information on vaccines [176].
These actions from social platforms may change what users see and what researchers study in terms of sentiment
and vaccine sentiment. Anti-vaccine groups may migrate from platforms that no longer monetise or make it easy
for them to share information, to other platforms from which it may be harder to gather data or analyse, as is
possible with Twitter and Facebook APIs. As platforms such as Pinterest, YouTube and Facebook change due to
political pressure, and begin to moderate the sharing of anti-vaccine and far-right content on their platforms,
certain discussions may potentially become even more diversified and complex as a result of the way in which they
are shared, engaged and enacted upon within the ever-changing landscape of the internet and social media [177].
The importance of search queries and the choice of keywords
Overall, studies with platforms such as Pinterest or YouTube used simpler monitoring methods (i.e. using the browser
search tools) due to restrictions in accessing these platforms’ APIs compared to Twitter. For example, Pinterest
necessitates manual sample collection because no API is publicly available to collect pins by either keyword or account
handle [125]. In studies using browser search tools, the keyword search queries were much more simplistic, with just
one to three keywords used, which means that the quality of the data output may be less stringent than data
accessed using more rigorous Boolean search queries when accessing data from the Twitter API. The small number of
studies looking at other social media platforms compared to the large bias towards Twitter may again be because
Twitter has made its API more easily available, and the data output more rigorous. This makes it an attractive, if not
necessarily more representative sample of conversation around vaccine hesitancy across social media.
Poor keyword searches can result in data that are not necessarily representative of the real conversations on the
social network being studied. There has been a recent call to better standardise practices of keyword search terms
in health research on social media, so that the quality of the data query matches the quality of the data that the
platforms give out, and therefore the resulting study [178]. However, there should be enough flexibility to
acknowledge the constantly evolving, context-specific nature of social media. It is important to ensure that
research questions are also grounded within non-digital empirical research and public health frameworks, so that
any social media research around vaccines also demonstrates an awareness of the wider social milieu, and is not
unduly skewed or restricted in scope by the constraints of the social platforms or devices used [179].
Visualisation of data
The lack of methodological standardisation discussed above also constitutes a problem in terms of visualisation of
data. Studies using sophisticated APIs to retrieve data can also benefit from sophisticated tools to analyse data,
such as comprehensive social network analysis charts showing key conversation clusters, influential groups, and
scatter plots of influence over time. The less sophisticated the keywords search or tool of analysis, the less detailed
and informative the data visualisation may be, with simpler diagrams and tables able to show data for smaller data
samples, but these may not necessarily be as representative.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
40
3) Data analysis
The challenge of obtaining location information
Another challenge posed by social media monitoring is the geographical scope of data. Most of the studies
identified in this review were either conducted globally or chose not to restrict data to a specific country. One of
the challenges is for social media APIs and other monitoring tools to provide easier ways of locating tweets,
without infringing on the privacy rights users have chosen. Until this is resolved, social media monitoring with a
national or regional scope will remain extremely difficult to conduct.
It is difficult to obtain geo-location data from social media. Many users may decide to keep their location private,
which means some studies had to resort to consulting the users’ profile pages or other manual techniques to
identify locations (raising ethical considerations). Other studies restricted results by language, generally by using
language-specific keywords. As the specific nature of vaccine hesitancy is country- and context-specific, and should
be addressed as such, the difficulty in obtaining national social media data on vaccination will also pose challenges
when using social media information to inform vaccination interventions and communication strategies.
Analysing complex sentiments
The difficulty of assessing sentiment in the context of public health remains a key problem in almost all articles,
where it was found necessary to code social media data so that they could be analysed beyond the basic negative
or positive sentiment of marketing analysis. There is a difficulty in coding sentiments such as sarcasm and
hesitancy, whether using manual or automated coding [66]. When multiple researchers are used, manual coding
and analysis of sentiments, especially complex ones, is often prone to subjectivity biases, and can be extremely
time-consuming. However, setting-up automated algorithms to analyse data through machine learning requires
advanced computing and programmatic skills.
Coding of images
The coding of images on platforms such as Pinterest and Instagram also poses a challenge with automated coding
of sentiment, as image coding algorithms are still in their infancy, and not always readily available on a large scale
outside of Google’s DeepMind, expensive facial recognition packages or image search. APIs for visual social media
platforms such as Instagram and Snapchat are restrictive or even non-existent, even though those platforms are
increasingly being used, particularly by younger individuals. Studies analysing videos, for example on YouTube are
also more time-intensive and time-costly for researchers, as each video needs to be watched and annotated. This
may change now that as of 2018, YouTube allows the download of transcripts (where available), and these
transcripts can in turn be put into automated software to analyse sentiment [180].
Methodologically, over the last 10 years, it has been challenging to study vast amounts of visual social media data
concerning vaccine hesitancy - studies have focused on examining text [181]. However, while the process of
studying large amounts of video and images has previously been labour-intensive, with the need to use human
coding of images to discern themes and subjects, the progress of image recognition and AI technology over recent
years has meant that conducting rich content analysis studies of big data from images or videos is becoming more
prevalent [182]. There will be new insights into public health discourse coming from new and continuing research
in the field of multimodal sentiment analysis using video, images, and captioned text analysis [183-186].
Summary of the key findings on social media monitoring
There is a growing research interest in social media monitoring concerning vaccination, acknowledging the
role of social media in vaccine hesitancy.
Social media monitoring is a new area of research methodology, which can help national health authorities
understand how information about vaccination is shared and spread online.
Social media monitoring should take into account important ethical considerations in terms of access to
public and private data, the use of human research subjects, and confidentiality and anonymity (particularly
for vulnerable populations).
At present, there are no standardised methods for monitoring and analysing social media in relation to
vaccination, and more tools need to be evaluated.
Social media can easily be monitored manually by researchers or health authorities but this can be time-
consuming, and the results can be of limited accuracy or completeness, limiting analysis to small non-
representative samples. Automated software can provide better results but it requires highly technical
computational skills and can be expensive.
Most automated analyses are limited to Twitter, which creates an unbalanced representation of online
content. Researchers and health authorities should pay attention to the platforms used specifically by their
population of interest.
Vaccine hesitancy is context- and country-specific and social media monitoring should be conducted to
reflect this. However, there is still a lack of methods for obtaining location information on social media.
Sentiment analysis is an important tool for analysing social media content concerning vaccination but is
prone to subjectivity biases, especially when conducted manually. While it should go further than simple
negative/positive categorisation of sentiments to reflect the complexity of sentiments around vaccination, it
can be difficult to code sentiments, such as sarcasm or hope.
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
41
Social media monitoring has public health implications. It can be used to detect increases in online activity,
shifts in sentiments or other signals of decreasing confidence in vaccination; as a tool for health authorities
to increase their presence and popularity on social media; and finally as a real-time listening tool to
anticipate, understand and respond to public questions and concerns.
Social media monitoring should be seen as a constantly evolving area of research methodology, to reflect
the continuous growth of the social media environment, and the constant shift of users and content to
newer platforms.
Suggestions for future research
Evaluation of the different social media monitoring tools (e.g. methods for coding and analysing sentiments could
be compared, to identify those offering more valid and reliable results.) This would allow researchers to develop a
gold standard or best practice guide on how to conduct media monitoring for health research.
Visualisation of social media analytics in public health. A growing body of research, both within this review
and within social sciences in general, is concerned with developing innovative solutions to the data
visualisation problem, looking at both the visualisation of social media discourse and the research field of
visual analytics itself. This may have a more cohesive effect on the visualisation of data over time [187].
Development and improvement of methods for geo-locating tweets.
Improvement of the understanding of complex themes such as hesitancy, promotional content and
discouraging content and development of measurement methods for risk perception, or patterns of change
in sentiment over time [188]. This is particularly important for discussions concerning vaccination, which
can elicit complex sentiments.
Coding of more complex sentiments on social media to be able to understand the nuances in sentiments
with a view to providing better responses to vaccine hesitancy.
6.3 Review how social media monitoring methods and
information gathered from monitoring can be used to inform
communication strategies
The studies from the scoping review did not formally evaluate how social media monitoring methods and
information gathered can be used to inform communication strategies. However, based on various findings, some
studies recommended that health authorities, governments and/or healthcare professionals can monitor social
media in order to detect increases in online activity, shifts in sentiments, or other signals that may have an
influence on vaccination uptake or confidence in real time. The studies also recognised that social media
monitoring could help health authorities anticipate, understand and respond to public questions and concerns. For
example, a study was conducted by the European Medicines Agency in 2015 to monitor online discourses around
HPV vaccination in Europe ahead of the release of new safety data [189]. Findings from the study were used to
identify common public concerns and questions concerning HPV vaccination, which were then addressed during a
press conference. The study demonstrated the utility of media monitoring to support communication preparedness.
In addition, there are benefits that health authorities could gain from increasing their presence, and their
popularity on social media, such as an increase in public trust and recognition.
There is, however, a need for public health risk communicators, researchers and officials to be aware that not all
posts are necessarily made by humans, and that sophisticated automated bots and human trolls (posters of
polarising content) are actively involved in online public health discourse. Research and algorithmic approaches
have found that it is not easy to identify sophisticated bots or trolls, especially as some post both pro- and anti-
vaccination narratives, consistent with a strategy of promoting political discord [57].
Summary of key findings on how social media monitoring methods
and information gathered from the monitoring can be used to inform
communication strategies
Monitoring can be a useful listening tool for public health institutes to gain a broad understanding of
prevailing issues of interest and concerns in certain communities and to detect key themes or questions
concerning vaccination among the population.
Continuous monitoring can be used to investigate how public concerns change over time and how questions
about vaccines have tended to change with periodical consistency, enabling health authorities to develop
tailored, targeted, cost-effective and responsive public engagement.
Consistent, long-term communication activity may be more beneficial than reactive or event-driven public
health communication, a finding that may be useful for planning policy around public health interventions to
increase vaccine confidence.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
42
Other beneficial uses of monitoring to support health authorities in developing communication strategies include:
semantic network analysis of sentiments;
identifying the platforms people use to gather information about vaccines;
measuring engagement to detect more popular content and better understand how it is shared;
examining the effectiveness of vaccine promotion strategies;
flagging anti-vaccine websites;
information on adverse-events following immunisation surveillance;
producing spatio-temporal indicators to determine where and when concerns are growing;
during vaccine-preventable disease outbreaks, creating adaptive messages at different stages of the crisis.
Suggestions for future research
More detailed evaluations of how social media monitoring methods and information gathered from
monitoring have informed different communication strategies
Detailed sharing of practices and country experiences of utilising social media monitoring and analysis in
forming communication messages and strategies.
6.4 Understanding the uses, benefits and limitations of using
social media as an intervention around vaccination
Three types of intervention using social media to address vaccination have been identified through this systematic
scoping review - interventions that use social media as a communication channel to share information with the
public; interventions that engage the public using online group discussions and interventions that use interactive
websites with an integrated social media component to provide information and engage users.
Effect of social media interventions on vaccine uptake
None of the interventions summarised in this scoping review were found to significantly increase vaccine uptake.
However, we should be careful when interpreting this finding. First of all, it is very rare to be able to point to an
intervention and say with precision that an individual was vaccinated specifically because of that intervention (or its
social media component). Furthermore, the results from this review may not specifically reflect the benefits and
limitations of social media as a communication channel since they are dependent on the type of content and
messaging strategies used in these interventions and the fact that social media are only one of many channels for
communicating with the public. This demonstrates the complexity of establishing the benefits and limitations of
social media as a tool for addressing vaccination, and the importance of other factors that may influence vaccine
uptake when communicating (such as targeted messaging, choice of language, style and framing of messages,
engagement of users, or source of information.)
One study showed that loss-framed messages on Facebook were associated with a significantly higher intention to
get vaccinated than gain-framed messages [142]. Another study showed that whenever adverse events are
mentioned in a narrative manner, intentions to vaccinate decrease no matter whether the information comes
from pro- or anti-vaccination sources [140]. These results highlight the fact that social media should not be seen as
a magic bullet for addressing vaccine hesitancy but should be used as part of a broader communication strategy
that acknowledges the influence of these other factors. If used adequately, social media have been seen to
successfully mitigate vaccine hesitancy. For example, Denmark’s and Ireland’s successful strategies of using social
media platforms to engage with vaccine-hesitant parents and restore confidence in HPV vaccination eventually led
to an increase in HPV vaccine coverage [33,34].
Effect of social media interventions on attitudinal change
The impact of social media interventions on attitudinal change seemed easier to verify than on vaccine uptake.
Some interventions conducted with the aim of providing information about vaccination on Facebook in the US were
found to significantly increase knowledge in relation to the HPV vaccine, while decreasing perceived barriers to HPV
vaccination and perceptions of risk [142-144]. Facebook-assisted teaching methods were also found to increase
knowledge and intentions to be vaccinated in Taiwan [148]. A randomised-controlled trial in the US found that
interactive websites with a space for parents to contribute with content and discuss concerns significantly reduced
parental concerns concerning vaccination although no impact was found on vaccine-related attitudes [151].
Many of the metrics that measure user engagement should be analysed carefully, as stated by the concept of
‘vanity metrics’ where positive feedback (such as views and likes) does not necessarily reflect the real performance
of a social media strategy. Much of the data attached to social media are not intrinsically useful for furthering the
aims of specific campaigns. While numbers may look appealing (e.g. a large number of likes and viewers), it is
hard to say how they might be linked to particular behaviour. More detailed evaluations and surveys are needed to
confirm those results.
Outcomes of social media interventions and their content messaging were found to vary by nationality and local
culture and it is therefore difficult to summarise the benefits and limitations of using social media as an
intervention tool in relation to vaccination. Furthermore, local cultures and politics filter through an international
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
43
arena where geography does not explicitly matter. Hence, local concerns and politics about vaccines exist offline
and then encounter each other in various fora which are completely location-agnostic. That may pose challenges
when considering country-specific interventions; knowing how specific campaigns targeted specific geographies or
handled this challenge would be particularly interesting.
Another reason why this scoping review cannot identify the benefits and limitations of social media interventions to
address vaccine hesitancy is the lack of standardised approach, the different methodologies used and the range of
research questions. While this richness in research contributes greatly to the understanding of how vaccine
hesitancy and confidence can be addressed, the downside is that it makes it difficult to compare and establish
common ground for interventions.
Summary of the key findings on social media interventions to address
vaccine hesitancy
Although no social media intervention was found to significantly increase vaccine uptake, it may be difficult
to accurately link an individual’s vaccination status to a specific component of an intervention. This is
particularly true as social media are only the channel for communication and other factors, such as the
content or style of messages (loss-framed vs gain-framed messages, narratives vs. scientific facts) also
influence beliefs and behaviour.
Social media interventions were found to positively influence attitudes related to vaccination, including
knowledge, perceived barriers and risk perceptions.
Results vary by country and population group and it would be very helpful if national health authorities that
have used social media to respond to vaccine confidence evaluated the impact of their interventions and
shared their findings.
Suggestions for future research
More detailed evaluations and surveys are needed to see how metrics, such as number of likes and viewers,
might inform specific strategies and drive particular behaviour.
Further research is needed on different types of communication strategies and how they influence vaccine
uptake, such as targeted messaging, choice of language, style and framing of messages, engagement of
users, or the source of the information.
More studies are needed to confirm the effects of social media interventions in different contexts and there
should be more sharing of experience across different countries.
6.5 Limitations of this systematic scoping literature review
The findings from this systematic scoping review should not be discussed without mentioning some study
limitations. While an extensive search strategy was used to identify articles - screening multiple databases and
using a comprehensive set of keywords - the search was only conducted using English keywords. This could have
influenced the finding that most articles use English keywords to monitor social media. Articles in Spanish and
Italian were included for analysis, and although no articles were found in other languages, a more comprehensive
search using keywords in other languages could have identified a larger number of articles.
In order to strengthen the selection process, two reviewers independently screened all articles by title and
abstract, and then by full text. However, it is important to note that data extraction was conducted by four
researchers, who divided the 115 articles among themselves. Although they used the same data extraction sheet,
this could have led to some inconsistencies in data extraction.
While the search for interventions and social media monitoring strategies was conducted in a systematic manner
that should have identified all articles published on these topics, some articles may have been missed on social
media users’ preferences and vaccination information sources. It is possible that articles which did not specifically
mention social media as a finding in their abstract might have been overlooked and excluded from the study. The
number of articles included in the report for this category should therefore be treated with caution.
Finally, the use of social media to address vaccine hesitancy (whether in the form of social media monitoring or
interventions developed to communicate using social media) is still relatively new. Many experiences and real-life
cases have not been published in publically available peer-reviewed journals or reports, and could therefore not be
included in this review. The review may therefore not represent all the methods and interventions that have been
developed to monitor social media and address vaccine hesitancy.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
44
7. Conclusions and the way forward
As social media are becoming part of everyday life, exposure to online information about vaccination, often
negative, is becoming more common. This has been shown to contribute to a decrease in public confidence in
vaccination. Vaccine hesitancy is now seen by many national and international immunisation specialists as a major
threat to public health.
The increasing use of social media by individuals to find information about vaccination, together with the ever-
growing volume of information against vaccination available on social media should constitute a call for action.
Vaccine hesitancy cannot be addressed unless we are better able to understand the role that social media play in
vaccine decision-making, the type of information that social media users are exposed to and the way in which this
information is spread and shared across the world.
This review has found that while many studies have been conducted with the aim of analysing online content
relating to vaccination, the methodologies used are extremely varied. As a minimum, it is recommended that health
authorities, health professionals or others with an interest in monitoring social media around vaccination collect
data relating to the sentiments and content of social media posts, the reach and influence of these posts, and if
available, geo-location data. Even though automated systems (whether for collecting or analysing data) require
some computational skills and have certain limitations, such as access to certain platforms, they provide the most
robust data and are less time-intensive than manual systems. Different APIs or other software may be used and
will be particularly helpful for ongoing continuous surveillance systems used to detect signals of decreasing
confidence in vaccination. Sentiment analyses, whether conducted using a manual or automated system, should
also aim to move beyond a positive versus negative model to truly reflect the content and emotions of social media
posts. Decisions around sample sizes and periods of data collection monitored will be highly dependent on the
resources available and health authorities should therefore consider the recruitment of staff dedicated to these new
types of surveillance systems. Research into the platforms used by local populations in different countries should
inform the decision to monitor specific social media platforms. For example, monitoring Instagram or Snapchat will
be more beneficial to understand how adolescents share information concerning vaccination.
More evaluation of social media monitoring and analysis techniques, from data collection to content and sentiment
analyses, is still needed in order to inform the development of validated standardised approaches. In the context of
GDPR and discussions around the privacy of online information, an ethical code of conduct relating to media
monitoring should also be developed to ensure the respect and anonymity of social media users.
It is crucial that health authorities and immunisation managers start incorporating social media monitoring as part
of their traditional vaccination surveillance strategies and not only after a confidence crisis occurs. While a gold
standard of evaluated and effective methodologies to monitor social media may be useful for health authorities, it
is also important that automated software, facilitating more accurate and comprehensive social media monitoring
and analysis, is made more accessible and user-friendly, to allow health authorities or those without computational
skills to perform media monitoring.
Finally, the purpose and value of social media monitoring should be clearly defined by both researchers and
immunisation specialists. While some may try to use social media as a proxy for what the public thinks about
vaccination, the reality is often much more complex. Social media monitoring should therefore be seen as a way of
capturing the essence and the movement of online discourse around vaccination in order to better understand how
it can influence public perceptions and decision-making concerning vaccination. Such evidence could then inform
the development of targeted interventions to maintain or restore public confidence in vaccination.
The purpose of this review was to analyse the social media monitoring techniques and interventions. A potential
next step could be to conduct a deeper analysis of the key themes retrieved from the results of the studies
addressing social media monitoring of vaccination and how these results can further support public health institutes
with monitoring and communication related to vaccine hesitancy.
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
45
References
1. London School of Hygiene and Tropical Medicine. Vaccine Confidence Project [internet]. Available from:
https://www.vaccineconfidence.org/
2. World Health Organization. Ten threats to global health in 2019. Available from: https://www.who.int/emergencies/ten-
threats-to-global-health-in-2019.
3. Stahl JP, Cohen R, Denis F, Gaudelus J, Martinot A, Lery T, et al. The impact of the web and social networks on vaccination.
New challenges and opportunities offered to fight against vaccine hesitancy. Medecine et maladies infectieuses. 2016
May;46(3):117-22.
4. Salathe M, Bonhoeffer S. The effect of opinion clustering on disease outbreaks. Journal of the Royal Society, Interface. 2008
Dec 6;5(29):1505-8.
5. Filia A, Bella A, Del Manso M, Baggieri M, Magurano F, Rota MC. Ongoing outbreak with well over 4 000 measles cases in
Italy from January to end August 2017− what is making elimination so difficult? Eurosurveillance. 2017;22(37).
6. McCarthy M. Measles outbreak linked to Disney theme parks reaches five states and Mexico. BMJ (Clinical research ed).
2015 Jan 23;350:h436.
7. Wise J. Largest group of children affected by measles outbreak in Wales is 10-18 year olds. BMJ (Clinical research ed). 2013
Apr 19;346:f2545.
8. World Health Organization. Immunization, Vaccines and Biologicals: measles disease incidence. 2018. Available from:
https://www.who.int/immunization/monitoring_surveillance/data/en/
9. Larson HJ, de Figueiredo A, Xiahong Z, Schulz WS, Verger P, Johnston IG, et al. The State of Vaccine Confidence 2016:
Global Insights Through a 67-Country Survey. EBioMedicine. 2016 2016/10/01/;12:295-301
10. Larson HJ, Bertozzi S, Piot P. Redesigning the AIDS response for long-term impact. Bulletin of the World Health
Organization. 2011 Nov 1;89(11):846-52.
11. Yaqub O, Castle-Clarke S, Sevdalis N, Chataway J. Attitudes to vaccination: a critical review. Social Science & Medicine
(1982). 2014 Jul;112:1-11.
12. Black S, Rappuoli R. A crisis of public confidence in vaccines. Science Translational Medicine. 2010 Dec 8;2(61):61mr1.
13. Betsch C, Brewer NT, Brocard P, Davies P, Gaissmaier W, Haase N, et al. Opportunities and challenges of Web 2.0 for
vaccination decisions. Vaccine. 2012;30(25):3727-33.
14. Kata A. Anti-vaccine activists, Web 2.0, and the postmodern paradigmAn overview of tactics and tropes used online by the
anti-vaccination movement. Vaccine. 2012;30(25):3778-89.
15. Keelan J, Pavri V, Balakrishnan R, Wilson K. An analysis of the Human Papilloma Virus vaccine debate on MySpace blogs.
Vaccine. 2010;28(6):1535-40.
16. Zimmerman RK, Wolfe RM, Fox DE, Fox JR, Nowalk MP, Troy JA, et al. Vaccine criticism on the World Wide Web. Journal of
Medical Internet Research. 2005 Jun 29;7(2):e17.
17. Moorhead SA, Hazlett DE, Harrison L, Carroll JK, Irwin A, Hoving C. A new dimension of health care: systematic review of
the uses, benefits, and limitations of social media for health communication. Journal of Medical Internet Research. 2013 Apr
23;15(4):e85.
18. European Centre for Disease Prevention and Control. Social Media Strategy development: a guide to using social media for public
health communication. Stockholm, Sweden: ECDC, 2016. Available from:
https://www.ecdc.europa.eu/en/publications-data/social-
media-strategy-development-guide-using-social-media-public-health
19. Evrony A, Caplan A. The overlooked dangers of anti-vaccination groups' social media presence. Human Vaccines &
Immunotherapeutics. 2017;13(6):1475.
20. Gu Z, Badger P, Su J, Zhang E, Li X, Zhang L. A vaccine crisis in the era of social media. Natl Sci Rev. 2017;5(1):8-10.
21. Kaplan AM, Haenlein M. Users of the world, unite! The challenges and opportunities of Social Media. Business Horizons.
2010 2010/01/01/;53(1):59-68.
22. Kata A. A postmodern Pandora's box: anti-vaccination misinformation on the Internet. Vaccine. 2010;28(7):1709-16.
23. Miller D, Costa E, Haapio-Kirk L, Haynes N, McDonald T, Nicolescu R, et al. Why We Post. Anthropology News.
2016;57(9):e44-e7.
24. Konijn EA, Veldhuis J, Plaisier XS. YouTube as a research tool: three approaches. Cyberpsychology, behavior and social
networking. 2013 Sep;16(9):695-701.
25. Kelly J, Fealy GM, Watson R. The image of you: constructing nursing identities in YouTube. Journal of Advanced Nursing.
2012;68(8):1804-13.
26. Pink S. Digital Ethnography: Principles and Practices. London, UK: SAGE Publications Ltd; 2016.
27. Brunson EK. The impact of social networks on parents’ vaccination decisions. Pediatrics. 2013;131(5):e1397-e404.
28. Leask J, Kinnersley P, Jackson C, Cheater F, Bedford H, Rowles G. Communicating with parents about vaccination: a
framework for health professionals. BMC Pediatrics. 2012 Sep 21;12:154.
29. Sobo EJ, Huhn A, Sannwald A, Thurman L. Information Curation among Vaccine Cautious Parents: Web 2.0, Pinterest
Thinking, and Pediatric Vaccination Choice. Medical Anthropology. 2016 Nov-Dec;35(6):529-46.
30. Salathé M, Khandelwal S. Assessing vaccination sentiments with online social media: Implications for infectious disease
dynamics and control. PLoS Computational Biology. 2011;7(10).
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
46
31. Bartholomew RE, Wessely S, Rubin GJ. Mass psychogenic illness and the social network: is it changing the pattern of
outbreaks? Journal of the Royal Society of Medicine. 2012;105(12):509-12.
32. Covolo L, Ceretti E, Passeri C, Boletti M, Gelatti U. What arguments on vaccinations run through YouTube videos in Italy? A
content analysis. Human Vaccines and Immunotherapeutics. 2017;13(7):1693-9.
33. Corcoran B, Clarke A, Barrett T. Rapid response to HPV vaccination crisis in Ireland. The Lancet. 2018;391(10135):2103.
34. World Health Organization Regional Office for Europe. Danish health literacy campaign restores confidence in HPV
vaccination 2019 [updated 28 January 2019]. Available from:
www.euro.who.int/en/health-topics/disease-
prevention/vaccines-and-immunization/news/news/2019/01/danish-health-literacy-campaign-restores-confidence-in-hpv-
vaccination?utm_source=WHO%2FEurope+mailing+list&utm_campaign=6f678f7035-
EMAIL_CAMPAIGN_2018_03_27_COPY_01&utm_medium=email&utm_term=0_60241f4736-6f678f7035-110548229
35 Peters MD, Godfrey CM, Khalil H, McInerney P, Parker D, Soares CB. Guidance for conducting systematic scoping reviews.
International Journal of Evidence-based Healthcare. 2015 Sep;13(3):141-6.
36. Joanna Briggs Institute. The Joanna Briggs Institute Reviewers’ Manual 2015: Methodology for JBI Scoping Reviews.
Adelaide, Australia: The Joanna Briggs Institute, 2015.
37. Arksey H, O'Malley L. Scoping Studies: Towards a Methodological Framework. International Journal of Social Research
Methodology. 2005;8(1):19-32.
38. Dilley SE, Peral S, Straughn JM, Jr., Scarinci IC. The challenge of HPV vaccination uptake and opportunities for solutions:
Lessons learned from Alabama. Preventive Medicine. 2018;113:124-31.
39. Ahmed N, Quinn SC, Hancock GR, Freimuth VS, Jamison A. Social media use and influenza vaccine uptake among White
and African American adults. Vaccine. 2018;36(49):7556-61.
40. Kim J. The relationship of health beliefs with information sources and HPV vaccine acceptance among young adults in
Korea. International Journal of Environmental Research and Public Health. 2018;15(4).
41. Campbell H, Edwards A, Letley L, Bedford H, Ramsay M, Yarwood J. Changing attitudes to childhood immunisation in
English parents. Vaccine. 2017;35(22):2979-85.
42. Ford AJ, Alwan NA. Use of social networking sites and women's decision to receive vaccinations during pregnancy: A cross-
sectional study in the UK. Vaccine. 2018;36(35):5294-303.
43. Robichaud P, Hawken S, Beard L, Morra D, Tomlinson G, Wilson K, et al. Vaccine-critical videos on YouTube and their impact
on medical students' attitudes about seasonal influenza immunization: A pre and post study. Vaccine. 2012;30(25):3763-70.
44. Moore PJ, Millar BC, Moore JE. Meningococcal ACWY vaccine uptake and awareness among student freshers enrolled at
Northern Ireland universities. International Journal of Adolescent Medicine & Health. 2017 Jan 18;18:18.
45. Luisi MLR. Who gives a "Like" about the HPV vaccine? Kansan parent/guardian perceptions and social media
representations. Dissertation Abstracts International Section A: Humanities and Social Sciences. 2018;79(4-A(E))
46. Margolis MA, Brewer NT, Shah PD, Calo WA, Gilkey MB. Stories about HPV vaccine in social media, traditional media, and
conversations. Preventive Medicine. 2019;118:251-6.
47. Zhang N, Tsark J, Campo S, Teti M. Facebook for Health Promotion: Female College Students' Perspectives on Sharing HPV
Vaccine Information Through Facebook. Hawai'i Journal of Medicine & Public Health: a journal of Asia Pacific Medicine &
Public Health. 2015;74(4):136-40.
48. Mena G, Llupia A, Garcia-Basteiro AL, Aldea M, Sequera V-G, Trilla A. The willingness of medical students to use Facebook
as a training channel for professional habits: The case of influenza vaccination. Cyberpsychology, Behavior, and Social
Networking. 2012 Jun;15(6):328-31.
49. Hwang J, Shah DV. Health Information Sources, Perceived Vaccination Benefits, and Maintenance of Childhood Vaccination
Schedules. Health Communication. 2018:1-10.
50. Edelstein M, Pebody R. Can we achieve high uptakes of influenza vaccination of healthcare workers in hospitals? A cross-
sectional survey of acute NHS trusts in England. Epidemiology and Infection. 2014;142(2):438-47.
51. Palanisamy B, Gopichandran V, Kosalram K. Social capital, trust in health information, and acceptance of Measles-Rubella
vaccination campaign in Tamil Nadu: A case-control study. Journal of Postgraduate Medicine. 2018;64(4):212-9.
52. Addawood A (editor). Usage of scientific references in MMR vaccination debates on Twitter 2018.
53. Bahk CY, Cumming M, Paushter L, Madoff LC, Thomson A, Brownstein JS. Publicly available online tool facilitates real-time
monitoring of vaccine conversations and sentiments. Health Affairs. 2016;35(2):341-7.
54. Becker BFH, Larson HJ, Bonhoeffer J, van Mulligen EM, Kors JA, Sturkenboom MCJM. Evaluation of a multinational,
multilingual vaccine debate on Twitter. Vaccine. 2016;34(50):6166-71.
55. Bello-Orgaz G, Hernandez-Castro J, Camacho D. Detecting discussion communities on vaccination in twitter. Future
Generation Computer Systems. 2017;66:125-36.
56. Blankenship EB, Goff ME, Yin J, Tse ZTH, Fu KW, Liang H, et al. Sentiment, Contents, and Retweets: A Study of Two
Vaccine-Related Twitter Datasets. Perm. 2018 06 11;22(06):11.
57. Broniatowski DA, Jamison AM, Qi SH, AlKulaib L, Chen T, Benton A, et al. Weaponized Health Communication: Twitter Bots
and Russian Trolls Amplify the Vaccine Debate. American Journal of Public Health. 2018 Oct;108(10):1378-84.
58. Chakraborty P, Colditz JB, Silvestre AJ, Friedman MR, Bogen KW, Primack BA. Observation of public sentiment toward
human papillomavirus vaccination on Twitter. Cogent Medicine. 2017 01 Jan;4 (1) (1390853).
59. Chen T, Dredze M. Vaccine images on twitter: Analysis of what images are shared. Journal of Medical Internet Research.
2018;20(4).
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
47
60. D'Andrea E, Ducange P, Bechini A, Renda A, Marcelloni F. Monitoring the public opinion about the vaccination topic from
tweets analysis. Expert Systems with Applications. 2019;116:209-26.
61. Dredze M, Wood-Doughty Z, Quinn SC, Broniatowski DA. Vaccine opponents’ use of Twitter during the 2016 US presidential
election: Implications for practice and policy. Vaccine. 2017;35(36):4670-2.
62. Du J, Tang L, Xiang Y, Zhi D, Xu J, Song HY, et al. Public perception analysis of tweets during the 2015 measles outbreak:
Comparative study using convolutional neural network models. Journal of Medical Internet Research. 2018;20(7).
63. Du JC, Xu J, Song HY, Liu XY, Tao C. Optimization on machine learning based approaches for sentiment analysis on HPV
vaccines related tweets. Journal of Biomedical Semantics. 2017 Mar;8.
64. Du JC, Xu J, Song HY, Tao C. Leveraging machine learning-based approaches to assess human papillomavirus vaccination
sentiment trends with Twitter data. BMC Medical Informatics and Decision Making. 2017 Jul;17.
65. Dunn AG, Leask J, Zhou XJ, Mandl KD, Coiera E. Associations Between Exposure to and Expression of Negative Opinions
About Human Papillomavirus Vaccines on Social Media: An Observational Study. Journal of Medical Internet Research. 2015
Jun;17(6).
66. Dunn AG, Surian D, Leask J, Dey A, Mandl KD, Coiera E. Mapping information exposure on social media to explain
differences in HPV vaccine coverage in the United States. Vaccine. 2017;35(23):3033-40.
67. Huang X, Smith MC, Paul MJ, Ryzhkov D, Quinn SC, Broniatowski DA, et al. (eds). Examining patterns of influenza
vaccination in social media. The AAAI-17 Joint Workshop on Health Intelligence WS-17-09. 2017.
68. Kang GJ, Ewing-Nelson SR, Mackey L, Schlitt JT, Marathe A, Abbas KM, et al. Semantic network analysis of vaccine
sentiment in online social media. Vaccine. 2017;35(29):3621-38.
69. Kaptein R, Boertjes E, Langley D. Analyzing discussions on twitter: Case study on HPV vaccinations. De Rijke M. et al.
(eds) Advances in Information Retrieval. ECIR 2014. Lecture Notes in Computer Science, vol 8416. 2014. p. 474-80.
70. Keim-Malpass J, Mitchell EM, Sun E, Kennedy C. Using Twitter to Understand Public Perceptions Regarding the #HPV
Vaccine: Opportunities for Public Health Nurses to Engage in Social Marketing. Public Health Nursing. 2017;34(4):316-23.
71. Krittanawong C, Tunhasiriwet A, Chirapongsathorn S, Kitai T. Tweeting influenza vaccine to cardiovascular health
community. Eur J Cardiovasc Nurs. 2017;16(8):704-6.
72. Love B, Himelboim I, Holton A, Stewart K. Twitter as a source of vaccination information: Content drivers and what they are
saying. American Journal of Infection Control. 2013;41(6):568-70.
73. Mahoney LM, Tang T, Ji K, Ulrich-Schad J. The Digital Distribution of Public Health News Surrounding the Human
Papillomavirus Vaccination: A Longitudinal Infodemiology Study. JMIR Public Health and Surveillance. 2015 Jan-Jun;1(1):e2.
74. Massey PM, Budenz A, Leader A, Fisher K, Klassen AC, Yom-Tov E. What drives health professionals to tweet about
#HPVvaccine? Identifying strategies for effective communication. Preventing Chronic Disease. 2018;15(2).
75. Massey PM, Leader A, Yom-Tov E, Budenz A, Fisher K, Klassen AC. Applying multiple data collection tools to quantify human
papillomavirus vaccine communication on twitter. Journal of Medical Internet Research. 2016;18(12).
76. McNeill A, Harris PR, Briggs P. Twitter Influence on UK Vaccination and Antiviral Uptake during the 2009 H1N1 Pandemic.
Frontiers in Public Health. 2016;4:26.
77.
Mitra T, Counts S, Pennebaker JW, (editors). Understanding anti-vaccination attitudes in social media. International
AAAI Conference on Web and Social Media Tenth International AAAI Conference on Web and Social Media 2016
.
78. Numnark S, Ingsriswang S, Wichadakul D. VaccineWatch: A monitoring system of vaccine messages from social media data.
2014:112-7.
79. Porat T, Garaizar P, Ferrero M, Jones H, Ashworth M, Vadillo MA. Content and source analysis of popular tweets following a
recent case of diphtheria in Spain. European Journal of Public Health. 2018 Jul 31.
80. Radzikowski J, Stefanidis A, Jacobsen KH, Croitoru A, Crooks A, Delamater PL. The Measles Vaccination Narrative in Twitter:
A Quantitative Analysis. JMIR Public Health and Surveillance. 2016 Jan-Jun;2(1):e1.
81. Sanawi JB, Samani MC, Taibi M. #Vaccination: Identifying influencers in the vaccination discussion on twitter through social
network visualisation. International Journal of Business and Society. 2017;18(S4):718-26.
82. Shapiro GK, Surian D, Dunn AG, Perry R, Kelaher M. Comparing human papillomavirus vaccine concerns on Twitter: A cross-
sectional study of users in Australia, Canada and the UK. BMJ Open. 2017;7(10).
83. Smith MC, Dredze M, Quinn SC, Broniatowski DA, editors. Monitoring real-time spatial public health discussions in the
context of vaccine hesitancy 2017. Available from: http://ceur-ws.org/Vol-1996/paper2.pdf
84. Surian D, Nguyen DQ, Kennedy G, Johnson M, Coiera E, Dunn AG. Characterizing twitter discussions about HPV vaccines
using topic modeling and community detection. Journal of Medical Internet Research. 2016;18(8).
85. Tang L, Bie BJ, Zhi DG. Tweeting about measles during stages of an outbreak: A semantic network approach to the framing
of an emerging infectious disease. American Journal of Infection Control. 2018 Dec;46(12):1375-80.
86. Teoh D, Shaikh R, Vogel RI, Zoellner T, Carson L, Kulasingam S, et al. A Cross-Sectional Review of Cervical Cancer
Messages on Twitter during Cervical Cancer Awareness Month. Journal of Lower Genital Tract Disease. 2018;22(1):8-12.
87. Tomeny TS, Vargo CJ, El-Toukhy S. Geographic and demographic correlates of autism-related anti-vaccine beliefs on Twitter,
2009-15. Social Science and Medicine. 2017;191:168-75.
88. Yuan X, Crooks AT, editors. Examining online vaccination discussion and communities in Twitter. International Conference
on Social Media & Society, Copenhagen, Denmark (SMSociety). 2018.
89. D'Andrea E, Ducange P, Marcelloni F (editors). Monitoring negative opinion about vaccines from tweets analysis 2017.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
48
90. Ninkov A, Vaughan L. A webometric analysis of the online vaccination debate. Journal of the Association for Information
Science and Technology. 2017;68(5):1285-94.
91. Pananos AD, Bury TM, Wang C, Schonfeld J, Mohanty SP, Nyhan B, et al. Critical dynamics in population vaccinating
behavior. Proceedings of the National Academy of Sciences of the United States of America. 2017;114(52):13762-7.
92. Zhou X, Coiera E, Tsafnat G, Arachi D, Ong MS, Dunn AG. Using social connection information to improve opinion mining:
Identifying negative sentiment about HPV vaccines on Twitter. Stud Health Technol Inform.2015; 216: 761-5.
93. Ache KA, Wallace LS. Human Papillomavirus Vaccination Coverage on YouTube. American Journal of Preventive Medicine.
2008;35(4):389-92.
94. Basch CH, Hillyer GC, Berdnik A, Basch CE. YouTube videos related to human papillomavirus: the need for professional
communication. International Journal of Adolescent Medicine and Health. 2016 Apr 9;30(1).
95. Basch CH, Zybert P, Reeves R, Basch CE. What do popular YouTubeTM videos say about vaccines? Childcare, Health and
Development. 2017 01 Jul;43(4):499-503.
96. Briones R, Nan XL, Madden K, Waks L. When Vaccines Go Viral: An Analysis of HPV Vaccine Coverage on YouTube. Health
Communication. 2012;27(5):478-85.
97. Cambra UC, Díaz VC, Herrero SG. Vaccines and anti vaccines in the social network Youtube. Opcion. 2016;32 (Special Issue
9):447-65.
98. Donzelli G, Palomba G, Federigi I, Aquino F, Cioni L, Verani M, et al. Misinformation on vaccination: A quantitative analysis
of YouTube videos. Human Vaccines and Immunotherapeutics. 2018;14(7):1654-9.
99. Ekram S, Debiec KE, Pumper MA, Moreno MA. Content and commentary: HPV vaccine and YouTube. Journal of Pediatric
and Adolescent Gynecology. 2018, Nov 13.
100. Hernandez-Garcia I, Fernandez Porcel C. Characteristics of YouTubeTM videos in Spanish about the vaccine against
meningococcus B. Vacunas. 2018.
101. Keelan J, Pavri-Garcia V, Tomlinson G, Wilson K. YouTube as a source of information on immunization: a content analysis.
Jama. 2007 Dec 5;298(21):2482-4.
102. Tuells J, Martínez-Martínez PJ, Duro-Torrijos JL, Caballero P, Fraga-Freijeiro P, Navarro-López V. Characteristics of the videos
in Spanish posted on youtube about human papillomavirus vaccines. Revista espanola de salud publica. 2015;89(1):107-15.
103. Venkatraman A, Garg N, Kumar N. Greater freedom of speech on Web 2.0 correlates with dominance of views linking
vaccines to autism. Vaccine. 2015;33(12):1422-5.
104. Buchanan R, Beckett RD. Assessment of vaccination-related information for consumers available on Facebook®. Health
Information and Libraries Journal. 2014;31(3):227-34.
105. Faasse K, Chatman CJ, Martin LR. A comparison of language use in pro- and anti-vaccination comments in response to a
high profile Facebook post. Vaccine. 2016;34(47):5808-14.
106. Furini M, Menegoni, G,. Public Health and Social Media: Language Analysis of Vaccine Conversations. New York: In 2018
International Workshop on Social Sensing.2018:50-5
107. Ma JX, Stahl LN. A multimodal critical discourse analysis of anti-vaccination information on Facebook. Libr Infor Sci Res.
2017 Oct;39(4):303-10.
108. Orr D, Baram-Tsabari A. Science and Politics in the Polio Vaccination Debate on Facebook: A Mixed-Methods Approach to
Public Engagement in a Science-Based Dialogue. Journal of Microbiology & Biology Education. 2018;19(1).
109. Orr D, Baram-Tsabari A, Landsman K. Social media as a platform for health-related public debates and discussions: The
Polio vaccine on Facebook. Israel Journal of Health Policy Research. 2016;5(1).
110. Schmidt AL, Zollo F, Scala A, Betsch C, Quattrociocchi W. Polarization of the vaccination debate on Facebook. Vaccine.
2018;36(25):3606-12.
111. Smith N, Graham T. Mapping the anti-vaccination movement on Facebook. Information Communication and Society. 2017:1-18.
112. Suragh TA, Lamprianou S, MacDonald NE, Loharikar AR, Balakrishnan MR, Benes O, et al. Cluster anxiety-related adverse
events following immunization (AEFI): An assessment of reports detected in social media and those identified using an
online search engine. Vaccine. 2018;36(40):5949-54.
113. Tustin JL, Crowcroft NS, Gesink D, Johnson I, Keelan J, Lachapelle B. User-driven comments on a facebook advertisement
recruiting canadian parents in a study on immunization: Content analysis. Journal of Medical Internet Research.
2018;20(9).
114. Vayreda A, Antaki C. To vaccinate or not? The disqualification of commercial sources of health advice in an online forum.
Communication and Medicine. 2011;8(3):273-82.
115. Goh D, Chi JX. Central or peripheral? Information elaboration cues on childhood vaccination in an online parenting forum.
Comput Hum Behav. 2017 Apr;69:181-8.
116. Ske ZC. Communication about the MMR vaccine: an analysis of newspaper coverage and an internet discussion forum. [U -
Thesis; (Thesis (Ph D )]. In press 2006.
117. Penta MA, Bǎban A. Dangerous agent or saviour? HPV vaccine representations on online discussion forums in Romania.
International Journal of Behavioral Medicine. 2014;21(1):20-8.
118. Fadda M, Allam A, Schulz PJ. Arguments and sources on Italian online forums on childhood vaccinations: Results of a
content analysis. Vaccine. 2015;33(51):7152-9.
119. Nicholson MS, Leask J. Lessons from an online debate about measles-mumps-rubella (MMR) immunization. Vaccine.
2012;30(25):3806-12.
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
49
120. Luoma-aho V, Tirkkonen P, Vos M. Monitoring the issue arenas of the swine-flu discussion. J Commun Manag.
2013;17(3):239-51.
121. Tangherlini TR, Roychowdhury V, Glenn B, Crespi CM, Bandari R, Wadia A, et al. "Mommy Blogs" and the Vaccination
Exemption Narrative: Results From A Machine-Learning Approach for Story Aggregation on Parenting Social Media Sites.
JMIR Public Health and Surveillance. 2016 Nov 22;2(2):e166.
122. Skeppstedt M, Kerren A, Stede M. Vaccine hesitancy in discussion forums: Computer-assisted argument mining with
topic models. Stud Health Technol Inform. 2018; 247:366-70.
123. Lopez CS, Krauskopf E, Villota CE, Burzio LO, Villegas JE. Cervical cancer, human papillomavirus and vaccines: assessment
of the information retrieved from general knowledge websites in Chile. Public Health. 2017;148:19-24.
124. Nawa N, Kogaki S, Takahashi K, Ishida H, Baden H, Katsuragi S, et al. Analysis of public concerns about influenza
vaccinations by mining a massive online question dataset in Japan. Vaccine. 2016;34(27):3207-13.
125. Guidry JPD, Carlyle K, Messner M, Jin Y. On pins and needles: How vaccines are portrayed on Pinterest. Vaccine.
2015;33(39):5051-6.
126. Rivera I, Warren J, Curran J, editors. Quantifying mood, content and dynamics of health forums.
ACSW '16. 2016.
127. Chen B, Zhang JM, Jiang ZG, Shao J, Jiang T, Wang ZT, et al. Media and public reactions toward vaccination during the
'hepatitis B vaccine crisis' in China. Vaccine. 2015 Apr;33(15):1780-5.
128. Seeman N, Ing A, Rizo C. Assessing and responding in real time to online anti-vaccine sentiment during a flu pandemic.
Healthcare Quarterly (Toronto, Ont). 2010;13 Spec No:8-15.
129. Deiner MS, Fathy C, Kim J, Niemeyer K, Ramirez D, Ackley SF, et al. Facebook and Twitter vaccine sentiment in response to
measles outbreaks. Health Informatics Journal. 2017 Nov 1:1460458217740723.
130. Sundstrom B, Aylor E, Cartmell KB, Brandt HM, Bryant DC, Hughes Halbert C, et al. Beyond the birds and the bees: a
qualitative content analysis of online HPV vaccination communication. Journal of Communication in Healthcare.
2018;11(3):205-14.
131. Mollema L, Harmsen IA, Broekhuizen E, Clijnk R, De Melker H, Paulussen T, et al. Disease detection or public opinion
reflection? content analysis of tweets, other social media, and online newspapers during the measles outbreak in the
Netherlands in 2013. Journal of Medical Internet Research. 2015;17(5).
132. Jang SM, McKeever BW, McKeever R, Kim JK. From Social Media to Mainstream News: The Information Flow of the Vaccine-
Autism Controversy in the US, Canada, and the UK. Health Communication. 2019 Jan;34(1):110-7.
133. Aquino F, Donzelli G, De Franco E, Privitera G, Lopalco PL, Carducci A. The web and public confidence in MMR vaccination in
Italy. Vaccine. 2017;35(35):4494-8.
134. Lehmann BA, Ruiter RAC, Kok G, Work, Social P, Fpn WSP, II. A qualitative study of the coverage of influenza vaccination on
Dutch news sites and social media websites. BMC Public Health. 2013;Vol 13.
135. Kotsios A, Magnani M, Rossi L, Shklovski I, Vega D. An Analysis of the Consequences of the General Data Protection
Regulation (GDPR) on Social Network Research. arXiv.org (preprint arXiv:190303196). 5 Oct 2019.
136. British Psychological Society. Ethics Guidelines for Internet-mediated Research. Leicester, UK: BPS, 2013.
137. Guidry J, Messner M. Health misinformation via social media: The case of vaccine safety on pinterest. Social Media and
Crisis Communication2017. p. 267-79.
138. Bastian M, Heymann S, Jacomy M (editors.) Gephi: an open source software for exploring and manipulating networks. Third
international AAAI conference on weblogs and social media; 2009.
139. Finnegan G, Holt D, English PM, Glismann S, Thomson A, Salisbury DM, et al. Lessons from an online vaccine
communication project. Vaccine. 2018;36(44):6509-11.
140. Haase N, Betsch C, Renkewitz F. Source Credibility and the Biasing Effect of Narrative Information on the Perception of
Vaccination Risks. Journal of Health Communication. 2015;20(8):920-9.
141. La Torre G, Miccoli S, Ricciardi W. The Italian alliance for vaccination strategies: Facebook as a learning tool for preventive
medicine and public health. Human Vaccines and Immunotherapeutics. 2014;10(10):2910-4.
142. Lee MJ, Cho J. Promoting HPV Vaccination Online: Message Design and Media Choice. Health Promotion Practice.
2017;18(5):645-53.
143. Mohanty S, Leader AE, Gibeau E, Johnson C. Using Facebook to reach adolescents for human papillomavirus (HPV)
vaccination. Vaccine. 2018;36(40):5955-61.
144. Ortiz RR, Shafer A, Cates J, Coyne-Beasley T. Development and Evaluation of a Social Media Health Intervention to Improve
Adolescents' Knowledge About and Vaccination Against the Human Papillomavirus. Global Pediatric Health.
2018;5:2333794x18777918.
145. Peter C, Rossmann C, Keyling T. Exemplification 2.0: Roles of direct and indirect social information in conveying health
messages through social network sites. Journal of Media Psychology. 2014;26(1):19-28.
146. Piedimonte S, Leung A, Zakhari A, Giordano C, Tellier PP, Lau S. Impact of an HPV Education and Vaccination Campaign
among Canadian University Students. J Obstet Gynaecol Can. 2018;40(4):440-6.
147. Sundstrom B, Brandt HM, Gray L, Pierce JY. It's My Time: applying the health belief model to prevent cervical cancer among
college-age women. J Commun Manag. 2018;22(2):161-78.
148. Lai C-Y, Wu W-W, Ts ai S-Y, Cheng S-F, Lin K-C, Liang S-Y. The effectiveness of a Facebook-assisted teaching method on
knowledge and attitudes about cervical cancer prevention and HPV vaccination intention among female adolescent students
in Taiwan. Health Educ Behav. 2015 Jun;42(3):352-60.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
50
149. Psychologische M, Langley DJ, Wijn R, Epskamp S, van Bork R. Should I get that jab? Exploring influence to encourage
vaccination via online social media. ECIS 2015 Research-in-Progress Papers, Paper 64. 2015.
150. Kimmerle J, Bientzle M, Cress U. Personal experiences and emotionality in health-related knowledge exchange in internet
forums: A randomized controlled field experiment comparing responses to facts vs personal experiences. Journal of Medical
Internet Research. 2014;16(12).
151. Daley MF, Narwaney KJ, Shoup JA, Wagner NM, Glanz JM. Addressing Parents’ Vaccine Concerns: A Randomized Trial of a
Social Media Intervention. American Journal of Preventive Medicine. 2018;55(1):44-54.
152. Glanz JM, Wagner NM, Narwaney KJ, Kraus CR, Shoup JA, Xu S, et al. Web-based social media intervention to increase
vaccine acceptance: A randomized controlled trial. Pediatrics. 2017;140(6).
153. Ferro A, Bonanni P, Castiglia P, Montante A, Colucci M, Miotto S, et al. [Improving vaccination social marketing by
monitoring the web]. Annali di Igiene. 2014 May-Jun;26(3 Suppl 1):54-64.
154. Shoup JA, Wagner NM, Kraus CR, Narwaney KJ, Goddard KS, Glanz JM. Development of an Interactive Social Media Tool for
Parents With Concerns About Vaccines. Health Education and Behavior. 2015;42(3):302-12.
155. Lewandowsky S, Ecker UKH, Cook J. Beyond Misinformation: Understanding and Coping with the “Post-Truth” Era. Journal
of Applied Research in Memory and Cognition. 2017/12/01/;6(4):353-69.
156. Dredze M, Broniatowski DA, Smith MC, Hilyard KM. Understanding Vaccine Refusal: Why We Need Social Media Now.
American Journal of Preventive Medicine. 2016;50(4):550-2.
157. Boyd D, Crawford K. Critical questions for big data: Provocations for a cultural, technological, and scholarly phenomenon.
Information, Communication & Society. 2012;15(5):662-79.
158. González-Bailón S, Borge-Holthoefer J, Rivero A, Moreno Y. The dynamics of protest recruitment through an online network.
Scientific Reports. 2011;1:197.
159. Choi TR, Sung Y. Instagram versus Snapchat: Self-expression and privacy concern on social media. Telematics and
Informatics. 2018;35(8):2289-98.
160. Moreno MA, Goniu N, Moreno PS, Diekema D. Ethics of social media research: common concerns and practical
considerations. Cyberpsychology, Behavior and Social Networking. 2013;16(9):708.
161. González F, Yu Y, Figueroa A, López C, Aragon C. Global Reactions to the Cambridge Analytica Scandal: An Inter-Language
Social Media Study. International World Wide Web Conference, 19 Companion. 2019.
162. Wong JC. Facebook's Zuckerberg announces privacy overhaul: ‘We don’t have the strongest reputation'. The Guardian. 30 April 2019.
163. Rieder B, (editor). Studying Facebook via data extraction: the Netvizz application. Proceedings of the 5th annual ACM web
science conference; 2013: ACM.
164. EU GDPR. GDPR Key Changes: An overview of the main changes under GDPR and how they differ from the previous
directive 2019. Available from: https://eugdpr.org/the-regulation/
165. Carr I, Zimmermann K, van't Veer P, Teare H. Ethical design. At the Interface of Ethics for Big Data and the European
Union’s General Data Protection Regulation. Deliverable D13. 2. EU, 2018. Available from:
https://www.richfields.eu/wp-
content/uploads/2018/11/D13.2_Ethical-Design_RICHFIELDS.pdf
166. Tufekci Z (editor). Big questions for social media big data: Representativeness, validity and other methodological pitfalls.
Eighth International AAAI Conference on Weblogs and Social Media; 2014.
167. Gray V. Learning to look: Pulsar partners with Manchester School of Arts to understand the difference between how
machines and humans look at images: Pulsar; 2018. Available from:
https://www.pulsarplatform.com/blog/2018/learning-
to-look-pulsar-partners-with-manchester-school-of-arts-to-understand-the-difference-between-how-machines-and-humans-
look-at-images/
168. Driscoll K, Walker S. Big data, big questions| working within a black box: Transparency in the collection and production of
big twitter data. International Journal of Communication. 2014;8:20.
169. Morstatter F, Pfeffer J, Liu H, Carley KM, editors. Is the sample good enough? Comparing data from Twitter's streaming API
with Twitter's Firehose. Seventh international AAAI conference on weblogs and social media; 2013.
170. Cihon P, Yasseri T. A biased review of biases in Twitter studies on political collective action. Frontiers in Physics. 2016;4:34.
171. Wilson M. The tech giant fighting anti-vaxxers isn’t Twitter or Facebook. It’s Pinterest Fast Company 2019. Available from:
https://www.fastcompany.com/90310970/the-tech-giant-fighting-anti-vaxxers-isnt-twitter-or-facebook-its-pinterest.
172. Wong JC. How Facebook and YouTube help spread anti-vaxxer propaganda. The Guardian. 1 February 2019.
173. YouTube takes ads off 'anti-vax' video channels. BBC. 2019. Available from:
https://www.bbc.com/news/technology-
47357252
174. Wong JC. Facebook to ban anti-vaxx ads in new push against 'vaccine hoaxes'. The Guardian. Thursday 7 March 2019.
175. Hutchinson A. Twitter adds new warnings on anti-vax related searches. Social Media Today [Internet] 2019. Available from:
www.socialmediatoday.com/news/twitter-add -new-warnings-on-anti-vax-related-searches/554801/
176. Notopoulos K. Buzzfeednews [Internet] 2019. Available from:
https://www.buzzfeednews.com/article/katienotopoulos/instagram-will-use-ai-to-filter-anti-vax-content
.
177. Allen C. Extreme far right: ‘pick’n’mix’ ideologies and direct messaging online make for deadly new combination. The
Conversation. 2019. Available from:
http://theconversation.com/extreme-far-right-picknmix-ideologies-and-direct-
messaging-online-make-for-deadly-new-combination-112331
178. Kim Y, Huang J, Emery S. Garbage in, garbage out: data collection, quality assessment and reporting standards for social
media data use in health research, infodemiology and digital disease detection. Journal of Medical Internet Research.
2016;18(2).
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
51
179. Marres N. Material participation: Technology, the Environment and Everyday Publics. 2012. UK; Palgrave Macmillan.
180. Rieder B, Matamoros-Fernández A, Coromina Ò. From ranking algorithms to ‘ranking cultures’ Investigating the modulation
of visibility in YouTube search results. Convergence. 2018;24(1):50-68.
181. Murthy D, Gross A, McGarry M. Visual Social Media and Big Data. Interpreting Instagram Images Posted on Twitter. Digital
Culture & Society. 2016;2(2):113-34.
182. Manovich L. Digital Traces in Context. 100 Billion Data Rows per Second: Media Analytics in the Early 21st Century.
International Journal of Communication. 2018;12:16.
183. Thelwall M. Social media analytics for YouTube comments: potential and limitations. International Journal of Social
Research Methodology. 2018;21(3):303-16.
184. Majumder N, Hazarika D, Gelbukh A, Cambria E, Poria S. Multimodal sentiment analysis using hierarchical fusion with
context modeling. Knowledge-Based Syst. 2018;161:124-33.
185. Amarasekara I, Grant WJ. Exploring the YouTube science communication gender gap: A sentiment analysis. Public
Understanding of Science. 2019;28(1):68-84.
186. Lama Y, Chen T, Dredze M, Jamison A, Quinn SC, Broniatowski DA. Discordance between human papillomavirus Twitter
images and disparities in human papillomavirus risk and disease in the United States: mixed-methods analysis. Journal of
Medical Internet Research. 2018;20(9):e10244.
187. Stieglitz S, Mirbabaie M, Ross B, Neuberger C. Social media analytics Challenges in topic discovery, data collection, and
data preparation. International Journal of Information Management. 2018 2018/04/01/;39:156-68.
188. Gaspar R, Pedro C, Panagiotopoulos P, Seibt B. Beyond positive or negative: Qualitative sentiment analysis of social media
reactions to unexpected stressful events. Comput Hum Behav. 2016;56:179-91.
189. Bahri P, Fogd J, Morales D, Kurz X, Consortium obotA. Application of real-time global media monitoring and ‘derived
questions’ for enhancing communication by regulatory bodies: the case of human papillomavirus vaccines. BMC Medicine.
2017 May 02;15(1):91.
190. Kakroo U. Topsy is Dead. Here are 4 Alternatives (Trial for free): Semrush.com; 2015 [cited 2019 15.02.2019]. Available
from: https://www.semrush.com/blog/topsy-is-dead-alternative-social-analytics-tools-you-need-to-use-now/
191. González-Bailón S, Wang N, Rivero A, Borge-Holthoefer J, Moreno Y. Assessing the bias in samples of large online networks.
Soc Networks. 2014 2014/07/01/;38:16-27.
192. Crimson Hexagon. Wikipaedia 2019 [cited 15 March 2019]. Available from: https://en.wikipedia.org/wiki/Crimson_Hexagon
193. Gnip Inc. Wikipaedia 2019 [cited 15 March 2019]. Available from: https://en.wikipedia.org/wiki/Gnip
194. Foundation SMR. NodeXL 2019 [cited 15 March 2019]. Available from: https://www.smrfoundation.org/nodexl/.
195. Shulman S. DiscoverText 2019 [cited 15 March 2019]. Available from: https://discovertext.com/.
196. Schlitt JT, Lewis B, Eubank S. ChatterGrabber: A lightweight easy to use social media surveillance toolkit. Online Journal of
Public Health Informatics. 2015;7(1).
197. Salesforce.com. Introducing Social Studio REST API 2019 [cited 15 March 2019]. Available from:
https://developer.salesforce.com/docs/atlas.en-us.api_social.meta/api_social/1-introduction-to-the-social-studio-marketing-cloud.htm
198. Sang ETK. Zoek in tweets (Twiqs.nl) 2019 [cited 2019 15.03.2019]. Available from: http://145.100.58.119/cgi-bin/twitter.
199. HowardsHome.com. HowardsHome 2019 [cited 2019 15.03.2019]. Available from:
https://www.howardshome.com/producten/howardshome-monitoring/
200. Black A, Mascaro C, Gallagher M, Goggins SP (editors). Twitter zombie: Architecture for capturing, socially transforming and
analyzing the Twittersphere. Proceedings of the 17th ACM international conference on supporting group work; 2012: ACM.
201. Data MRO. Microsoft Research Social Media Conversation Corpus 2019 [cited 15 March 2019]. Available from:
https://msropendata.com/datasets/2bda14a7-ee25-4092-8f2f-9272d48ae903
202. BV CN. Clip it online media monitoring 2019 [cited 15 March 2019]. Available from: http://www.clipit.nl/
203. Croitoru A, Crooks A, Radzikowski J, Stefanidis A. Geosocial gauge: a system prototype for knowledge discovery from social
media. International Journal of Geographical Information Science. 2013;27(12):2483-508.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
52
Annex 1.
Search strategy developed for Embase and adapted to
different databases
1
("health 2.0" or "medicine 2.0" or "web 2.0" or "web 2.0s" or "43 things" or "500px" or "about.me" or "academia.edu" or acfun or
advogato or afreecatv or "afreeca tv" or "album2" or android or anobii or aparat or "archive.org" or asianavenue or "asian avenue" or
asmallworld or "a small world" or athlinks or "audimated.com" or "baidu tieba" or bayimg or bbm or bebo or bibsonomy or "biip.no" or
bilibili or bitchute or blackplanet or "black planet" or "blip.tv" or blog* or "bolt.com" or bookmarksync or "bookmark sync" or
"break.com" or busuu or buzznet or cafemom or "care2" or caringbridge or citeulike or "classmates.com" or cloob or
"commons.wikimedia.org" or "community manag*" or connotea or couchsurf* or "couch surf*" or cozycot or crunchyroll or
cucumbertown or "cyber spac*" or cyberspac* or cyworld or dacast or dailybooth or dailymotion or dailystrength or daum or dayviews
or "de.lirio.us" or delirious or "del.icio.us" or delicious or deviantart or diaspora* or digg or diigo or disaboom or "distribution list*" or
"dol2day" or doctissimo or dontstayin or douban or doximity or "draugiem.lv" or dreamwidth or dronestagram or "dxy.cn" or "e health"
or ehealth or "e-health" or elftown or elixio or ello or engagemedia or "engage media" or "english, baby!" or "eons.com" or etoro or "e-
toro" or "experience project" or expotv or "expo TV" or facebook* or faves or fetlife or filmaffinity or "film affinity" or filmow or
fledgewing or "fledge wing" or flickr or flixster or "focus.com" or folksonom* or fora or forums or fotki or fotolog* or "fotopic.net" or
foursquare or friendica or "friends reunit*" or friendster or fuelmyblog or "funnyordie.com" or funshion or furl or fyuse or "gab.ai" or
"gaia online" or gamerdna or "gamer DNA" or "gapyear.com" or "gather.com" or "gays.com" or "gazopa bloom" or "geni.com" or
gentlemint or "geograph britain and ireland" or getglue or gfycat or gifboom or girlsaskguys or "girls ask guys" or gnolia or godtube or
gogoyoko or goodnesstv or goodreads or goodwizz or googl* or govloop or grindr or "grono.net" or habbo or "hd share" or "hi5" or
"hospitality club" or hotlist or "hr.com" or "hub culture" or ibibo or "identi.ca" or imageshack or imessag* or imgur or "imm.io" or
"indaba music" or influencer or instagram* or ipad or ipads or ipernity or iphone* or "irc-galleria" or italki or itsmy or jaiku or jalbum or
jiepang or "kaixin001" or kakaotalk or "king of glory" or kiwibox or "kodak gallery" or laibhaari or "last.fm" or "late night shots" or
"league of legends" or letv or librarything or lifeknot or linkedin* or linkexpats or listography or livejournal or liveleak or livemocha or
lockerz or "ma.gnolia" or makeoutclub or mashup* or "mash up*" or mayomo or meetin or meettheboss or meetup or "meet up" or
mefeedia or megavideo or mendeley or metacafe or mevio or microblog* or millatfacebook or mixi or "mobileme web gallery" or
mocospace or "mouthshut.com" or mubi or mumsnet or "muzu.tv" or "my opera" or myheritage or myspace or "my space" or myvideo
or "nasza-klasa.pl" or naver or netlog or "new media" or newgrounds or newsvine or nexopia or "nico douga" or ning or "odnoklassniki"
or onedrive or oneworldtv or "online communit*" or "on-line communit*" or "open diary" or openfilm or "ora tv" or orkut or
outeverywhere or "ovi share" or panoramio or partyflock or patientslikeme or "patients like me" or pearltrees or phanfare or photoblog*
or photobucket or "photo sharing" or picasa or pinboard or pingsta or pinterest or pixabay or pixorial or plaxo or playfire or
"playlist.com" or plurk or podcast* or poolwo or "qq video" or quechup or quora or qzone or "radar.net" or raptr or ravelry or "rdf site
summary" or "really simple syndication" or reddit or rediff or renren or retweet* or "re-tweet*" or "reverbnation.com" or revver or "rich
site summary" or "rooster teeth" or "rss feed*" or rumble or rutube or ryze or "sapo videos" or schooltube or sciencestage or "second
life" or securetribe or sevenload or sharethemusic or "share the music" or shelfari or shutterfly or simpy or "sina weibo" or sitebar or
skoob or skype or skyrock or smartphone* or smugmug or snapchat* or snapfish or "social media" or "social medias" or "social
medium" or "social mediums" or "social network*" or socialvibe* or "sonico.com" or soundcloud or "sound cloud" or "spot.im" or
"spring.me" or "stage 32" or stickam or streamzoo or streetlife* or "street life*" or "students circle network" or studivz or stumbleupon
or talkbiznow or "tape.tv" or "taringa!" or teachstreet or telegram or "tencent qq" or "tencent qzone" or termwiki or "the sphere" or
thestudentroom or "the student room" or tinder or tinypic or "travbuddy.com" or travellerspoint or "tribe.net" or trilulilu or "trombi.com"
or trooptube or trovebox or tsu or tudou or tuenti or tumblr or "tv uol" or tweet* or twine or twitch or twitter or tylted or unsplash or
untappd or uplike or "user generated content" or "vampirefreaks.com" or "vbox7" or veoh or viadeo or viber or viddler or viddsee or
videolog* or vidme or vidyard or vimeo or vine or vines or virb or "virtual communit*" or vlog* or vox or wattpad or wayn or "we heart
it" or "web 2" or "web page*" or "web site*" or weblog* or webcast* or webmd or webpage* or webshot* or website* or wechat* or
weeworld or weibo or wellwer or "wepolls.com" or weread or "wer-kennt-wen" or whatsapp* or wiki* or wistia or wooxie or wordpress
or "word press" or "world wide web" or "writeaprisoner.com" or xanga or xing or xmarks or "xt3" or yammer or yelp or yfrog or yookos
or youku or youtube* or "you tube*" or zalo or "zing.vn" or "zoo.gr" or zooomr).ti,ab.
2
mobile phone/ or smartphone/ or blogging/ or social media/ or webcast/
3
1 or 2
4
(vaccin* or in*oculat* or immuniz* or immunis* or jab or jabs or shot or shots).ti,ab.
5
vaccination/ or immunization/
6
4 or 5
7
3 and 6
8
limit 7 to yr="2000 -Current"
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
53
Annex 2.
Description of studies that consider the preferences for using social media platforms as a source
of information and influence on perceptions of vaccination
Reference
Study aim(s)
Study details
Using social media to gain/share information
about vaccines
Social media influencing vaccine attitudes and/or uptake
Ahmed 2018[39]
Examine the relationship between social media use
and vaccine uptake and determine if there are
differences by social media platform.
Social media type:
Facebook,
Twitter
Vaccine:
Influenza
Country:
United States
With regard to social media use, the study found
that Facebook was a more popular platform (62% of
participants) than Twitter (15%) to find information
on the influenza vaccine.
This study considered the relationship between social media use
and vaccine uptake. Researchers found that individuals who
used Facebook or Twitter as sources of health information were
more likely to be vaccinated than those who did not use
Facebook or Twitter as a source of health information. However,
the analysed data also revealed that Twitter users were less
likely to be vaccinated than non-Twitter users.
Campbell
2017[41]
Understand parents’ opinion on vaccines and vaccine-
preventable diseases, their vaccination experiences
and what affects their vaccination decisions
Social media type:
Facebook,
Twitter, Discussion forums
Vaccine:
General childhood
vaccines
Country:
United Kingdom
This study found 34% of parents used the Internet
to find information about immunisations. Within this
group (626 parents), 13% used Facebook or Twitter
and 6% used discussion for.
This study found that parents who used chat rooms/discussion
fora to find information about immunisation were more likely to
report that they had seen or read something that made them
doubt vaccinations (31% of parents who used chat
rooms/discussion forums versus 8% of all parents). This was
also true for parents who used Facebook or Twitter (23%).
Dilley 2018[38]
Develop a comprehensive assessment of HPV
vaccination in Alabama, with the goal to make
recommendations for tailored multilevel interventions
Social media type:
N/A
Vaccine:
HPV
Country:
United States
Multiple study participants (5 of 7 parents) cited
social media as a common resource for information.
The study found misinformation on social media to be a
significant barrier to getting vaccinated, but also suggested
social media as a potential facilitator by providing an avenue to
propagate positive messaging about the HPV vaccine.
Edelstein
2014[50]
Ascertain what strategies the National Health Service
(NHS) trusts in England have used to increase
influenza vaccine uptake in their HCWs between
2008/2009 and 2011/2012 and to identify which
specific interventions were associated with an
increased vaccine uptake overall and by staff group,
in order to inform future HCW vaccination strategies.
Social media type:
Facebook,
Twitter
Vaccine:
Influenza
Country:
United Kingdom
The study reported an increase in the use of social
media interventions on Facebook and Twitter to
increase vaccine promotion.
In this study, use of Facebook and Twitter were associated with
a significantly reduced uptake of vaccines (22% and 24%).
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
54
Reference
Study aim(s)
Study details
Using social media to gain/share information
about vaccines
Social media influencing vaccine attitudes and/or uptake
Ford 2018[42]
Determine if the use of social networking sites to gain
information on pregnancy vaccinations is associated
with women’s uptake of the influenza and pertussis
vaccines during pregnancy.
Social media type:
N/A
Vaccine:
Influenza, Pertussis
Country:
United Kingdom
This study focused on pregnant women using social
networking sites to make decisions on vaccinations
during pregnancy. 21% of the participating women
reported using social media to find information on
vaccinations during pregnancy, with Facebook and
WhatsApp being the most used platforms.
12% of the participants reported that the information they
gathered from social media influenced their vaccination
decisions. The study found that women who used social media
to gather information on vaccinations during pregnancy were
58% less likely to receive the pertussis vaccination during their
pregnancy. No significant relationship was found between social
media usage and the influenza vaccination during pregnancy.
However, the study also found some platforms for social media
were not necessarily associated with a decreased vaccination
uptake for example, women who used WhatsApp and LinkedIn
were statistically more likely to receive the pertussis and
influenza vaccines in pregnancy.
Hwang 2018[49]
Investigate the associations between evaluations of
health information sources, parental perceptions of
childhood vaccination benefits, and the maintenance
of vaccination schedules for children (incorporates
social media as a health information source)
Social media type:
N/A
Vaccine:
General childhood
vaccines
Country:
United States
n/a
This study found that using social media as a source for health
information was negatively associated with parents’ perceptions
of the benefits of vaccines.
Kim 2018[40]
Examine the relationships between multidimensional
health beliefs and HPV vaccine acceptance, and what
information sources effectively foster HPV
vaccination-related health beliefs.
Social media type:
Facebook,
Twitter, Instagram, etc.
Vaccine:
HPV
Country:
South Korea
The study surveyed undergraduate students in
Seoul, South Korea found that 30% of the
participants cited social media as a source of HPV
information.
The findings from this study showed that hearing about HPV
and/or the HPV vaccine on social media increased participants’
perception of barriers relating to social norms and talking about
HPV/the vaccine.
Luisi 2018[45]
Explore how Kansan parents/guardians of HPV
vaccine-eligible children perceive the vaccine in the
contexts of the health belief model and the social
amplification of risk framework, parent/guardian
engagement with HPV vaccine-related information,
and Facebook representations by general users and
the Centers fo
r Disease Control and Prevention during
the vaccine’s first decade on the market.
Social media type:
Facebook
Vaccine:
HPV
Country:
United States
In this study 4% of respondents reported that social
media was their main source for learning about the
HPV vaccine. Additionally, 63% of respondents
reported that they had seen some information about
the HPV vaccine on social media. Out of the 50
parents and guardians that participated in the study,
28% reported searching for information on the HPV
vaccine on social media and 28% reported posting or
sharing information about the HPV vaccine on social
media.
Approximately 10% of the participating parents and guardians
(n = 50) felt that social media increased the fear they have of
their children having the HPV vaccine.
Margolis
2019[46]
Understand exposure to HPV vaccine-related stories.
Social media type:
N/A
Vaccine:
HPV
Country:
United States
This study found that 11% of parents who heard
HPV vaccine stories came across them on social
media. Negative accounts or ‘stories of harm’ were
more often found on social media (and traditional
media) than through other channels of information.
n/a
Mena 2012[48]
Analyse willingness of medical students to use
technical and informal Facebook pages promoting
influenza vaccination of HCWs and determine how
many students would actively follow and participate
in these pages.
Social media type:
Facebook
Vaccine:
Influenza
Country:
Spain
This study explored university medical students
usage of the Internet and Facebook to find
information on the influenza vaccination of
healthcare workers. Approximately 63% of students
would accept an invitation to follow a Facebook page
with formal, technical content on the influenza
n/a
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
55
Reference
Study aim(s)
Study details
Using social media to gain/share information
about vaccines
Social media influencing vaccine attitudes and/or uptake
vaccination campaign and 19% of students would
actively participate in such a group. 65% would
accept an invitation to follow a Facebook page that
used informal ways to communicate about the
influenza vaccination campaign (through animations
and offbeat news, etc.) and 28% of students would
actively participate in such a group.
Moore 2017[44]
Examine the uptake of the MenACWY vaccine
amongst a representative population of first-time
university students, attending universities
in Northern
Ireland, and ascertain how effectively the advice to
request the vaccine was conveyed and
the reasons as
to why students did not take up the offer of the
vaccine. Ascertain the level of meningitis awareness
in this student cohort and identify the best practice
mechanisms to promote such awareness in the
future.
Social media type:
N/A
Vaccine:
Meningococcal ACWY
Country:
United Kingdom
This study of university students in Northern Ireland
found that social media was the least preferred
method of communication to promote
meningitis/vaccine awareness.
n/a
Palanisamy
2018[51]
Assess the influence of social capital and trust in
health information on the status of MeaslesRubella
vaccination campaign in Tamil Nadu.
Social media type:
N/A
Vaccine:
Measles-Rubella
Country:
India
n/a
This study found that parents who refused the measles-rubella
vaccine placed greater trust on information gained through
WhatsApp and other social media platforms.
Robichaud
2012[43]
Examine attitudes of first year medical students to
seasonal influenza immunisation and impact of the
most popular vaccine-critical YouT
ube videos on their
attitudes towards the seasonal influenza vaccine.
Social media type:
N/A
Vaccine:
Influenza
Country:
Canada
At the beginning of this study conducted with
medical students, 42% of participants reported using
YouTube for health-related purposes and 12% used
YouTube to search for health information.
Additionally. 68% of participants reported that they
never used social networking sites (Facebook,
MySpace, etc.) to obtain health-related information.
n/a
Zhang 2015[47]
Examine female college students' attitudes, subjective
norms and perceived behavioural control associated
with forwarding information about HPV and chatting
about HPV.
Social media type:
Facebook
Vaccine:
HPV
Country:
United States
In this small study of female college students, none
of the participants shared HPV vaccine information
on Facebook. However, 71% expressed willingness
to share HPV vaccine information on Facebook.
n/a
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
56
Annex 3.
Description of studies on social media monitoring around vaccination
Reference Study aim(s) Study details Methodology Results (taken from the article’s abstract)
Ache2008[93]
Analyse how HPV vaccination is portrayed in
video clips and comments on YouTube.
Social media type
: YouTube
Vaccine
: HPV
Country
: Global/Not specified
Period monitored
: 8 February
2008
Search strategy
: Manual
Sentiment analysis
: Manual
(negative, neutral,
positive)
# results
: 146 videos
“Three quarters (n=109; 74.7%) of the video clips portrayed HPV vaccination in a positive
manner. One third (n=47; 32.2%) of the video clips had generated at least one posted
comment.”
Addawood2018[52]
Analyse scientific information sharing
behaviours on Twitter regarding the
controversy over the supposed linkage
between MMR vaccine and autism.
Social media type
: Twitter
Vaccine
: Measles
Country
: Global/Not specified
Period monitored
: 1 January
- 28 November 2016
Search strategy
:
Automated (Crimson
Hexagon)
Sentiment analysis
: Manual
(anti-vaccine, pro-vaccine)
# results
: 36,428 tweets
“People with anti-vaccine attitudes linked many times to the same URL while people with
pro-vaccine attitudes linked to fewer overall sources but from a wider range of resources,
and they provided fewer total links compared to people with anti-vaccine attitudes.
Moreover, our results showed that vocal journalists have a huge impact on users’ opinions”
Aquino2017[133]
Estimate the correlation between MMR
vaccination coverage in Italy and online
search trends and social network activity on
autism and MMR vaccine.
Social media type
: Mix:
Facebook, Twitter
Vaccine
: Measles
Country
: Italy
Period monitored
: 1 January
2010 - 31 December 2015
Search strategy
: Manual
Sentiment analysis
: Manual
(anti-vaccination, neutral,
pro-vaccination)
# results
: 19 Facebook pages/groups
“A significant inverse correlation was found between MMR vaccination coverage and
Internet search activity, tweets and Facebook posts. New media might have played a role in
spreading misinformation.”
Bahk2016[53]
Describe a publicly available platform for
monitoring vaccination related content,
called the Vaccine Sentimeter.
Social media type
: Twitter
Vaccine
: Polio and HPV
Country
: Global/Not specified
Period monitored
: October
2012 - November 2014
Search strategy
:
Automated (Twitter API+
HealthMap data)
Sentiment analysis
:
Automated (negative,
neutral/unclear, positive)
# results
: Polio: 39,308 relevant polio tweets and 1,534 relevant HPV tweets
“For the first event (polio), Twitter response to the attacks on health care workers
decreased drastically after the first attack, in contrast to mainstream media coverage. For
the second event (HPV), the mainstream and social media response was largely positive
about the HPV vaccine, but anti vaccine conversations persisted longer than the pro vaccine
reaction. Using the Vaccine Sentimeter could enable public health professionals to detect
increased online activity or sudden shifts in sentiment that could affect vaccination uptake”
Basch2016[94]
Identify the most popular videos on
YouTube related to HPV and describe their
content.
Social media type
: YouTube
Vaccine
: HPV
Country
: Global/Not specified
Period monitored
: not
reported
Search strategy
: Manual
Sentiment analysis
: Manual
(discouraging,
encouraging, neutral)
# results
: 70 videos
“The majority of videos (81.4%) provided general information related to HPV, discussed the
association of HPV infection and the development of cancer (81.4%), and addressed HPV
screening (64.3%). Just under one-half (n=34) of the videos addressed vaccination. Fifteen
of these were neutral, while six were encouraging and 13 were discouraging. The videos
included in this study were viewed ~17 million times, which indicates their potential for
influencing public awareness and opinions. Of the videos devoted to HPV vaccination, few
were encouraging.”
Basch2017[95]
Examine YouTube videos dealing with
vaccines.
Social media type
: YouTube
Vaccine
: Any
Country
: Global/Not specified
Period monitored
: 4
September 2007 - 17 October
2015
Search strategy
: Manual
Sentiment analysis
: Manual
(discouraging,
encouraging, neutral)
# results
: 87 videos
“The range of view counts was 25,532 to 6,229,835, with a median of 62,075 views per
video. Most videos (n=74, 85.1%) were devoted exclusively to the topic of vaccination. The
three most common sources of these YouTube videos were consumers (27.6%), TV-based
or Internet-based news (26.4%) and individual health professionals (25.3%). Top topics
covered were autism causality (47.1% of videos), undisclosed or poorly understood risks
(42.5%), adverse reactions (40.2%) and thimerosal or mercury in vaccines (36.8%). The
majority of videos (65.5%) discouraged the use of vaccines.”
Becker2016[54]
Explore the value of monitoring social media
to understand international public discussion
on the paediatric pentavalent vaccine (DTP-
HepB-Hib) programme by analysing Twitter
messages.
Social media type
: Twitter
Vaccine
: Pentavalent (DTP-
HepB-Hib)
Country
: Global/Not specified
Search strategy
:
Automated (Twitter API)
Sentiment analysis
: Manual
(negative, neutral/positive)
# results
: 7,657 tweets
“Only 3.1% of the messages were reactions to other messages, and 86.6% referred to
websites, mostly news sites (70.7%), other social media (9.8%), and health-information
sites (9.5%). Country mentions were identified in 70.4% of the messages, of which India
(35.4%), Indonesia (18.3%), and Vietnam (13.9%) were the most prevalent. In the
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
57
Reference Study aim(s) Study details Methodology Results (taken from the article’s abstract)
Period monitored
: July 2006 -
May 2015
annotated sample, 63% of the messages showed a positive or neutral sentiment about DTP-
HepB-Hib. Peaks in negative and positive messages could be related to country-specific
programme events.”
Bello-Orgaz2017[55]
Detect communities in Twitter, which are
disseminating vaccine opinions in order to
analyse how it could be influencing to the
rest of users in a particular community,
zone, or country.
Social media type
: Twitter
Vaccine
: Any
Country
: Global/Not specified
Period monitored
: 15 April - 8
November 2014
Search strategy
:
Automated (Twitter API)
Sentiment analysis
: n/a
# results
: 1,448,010 tweets
“Firstly, a preliminary analysis using data from Twitter and official vaccination coverage
rates is performed, showing how vaccine opinions of Twitter users can influence vaccination
decision-making. Then algorithms for community detection are applied to discover user
groups opinions about vaccines. The experimental results show that these techniques can
be used to discover social discussion communities providing useful information to improve
immunisation strategies.
Blankenship2018[56]
Investigate if tweets with different
sentiments toward vaccination and different
contents attract different levels of Twitter
users’ engagement (retweets).
Social media type
: Twitter
Vaccine
: Any
Country
: Global/Not specified
Period monitored
: 4 February
2010 - 10 November 2016
(Study A) & 1 January 2014 -
30 April 2015 (Study B)
Search strategy
:
Automated (Twitter API,
Gnip Inc.)
Sentiment analysis
: Manual
(anti-vaccine, neutral, pro-
vaccine)
# results
: 1,545 tweets
“Among #vaccine tweets, provaccine tweets (adjusted prevalence ratio = 1.5836, 95%
confidence interval = 1.2130-2.0713, p < 0.001) and anti-vaccine tweets (adjusted
prevalence ratio = 4.1280, 95% confidence interval = 3.1183-5.4901, p < 0.001) had more
retweets than neutral tweets. No significant differences occurred in retweet frequency for
content categories among anti-vaccine tweets. Among 411 links in provaccine tweets,
Twitter (53; 12.9%), content curator Trap.it (14; 3.4%), and the Centers for Disease
Control and Prevention (8; 1.9%) ranked as the top 3 domains. Among 325 links in anti-
vaccine tweets, social media links were common: Twitter (44; 14.9%), YouTube (25; 8.4%),
and Facebook (10; 3.4%). Among highly retweeted #vaccineswork tweets, the most
common theme was childhood vaccinations (40%; 81/201); 21% mentioned global
vaccination improvement/efforts (42/201); 29% mentioned vaccines can prevent outbreaks
and deaths (58/201).”
Briones2012[96]
Analyse the content of YouTube videos
related to the HPV vaccine.
Social media type
: YouTube
Vaccine
: HPV
Country
: US
Period monitored
: 1
November 2010
Search strategy
: Manual
Sentiment analysis
: Manual
(ambiguous, negative,
neutral, other, positive)
# results
: 172 Videos
“We found that most of these videos were news clips or consumer-generated content. The
majority of the videos were negative in tone, disapproving of the HPV vaccine. In addition,
negative videos were liked more by the viewers than positive or ambiguous ones.
Accusations of conspiracy theory and infringement of civil liberties were manifested in these
videos. The videos also presented mixed information related to the key determinants of
health behaviour, as stipulated in the Health Belief Model.”
Broniatowski2018[57]
Understand how Twitter bots and trolls
promote online health content.
Social media type
: Twitter
Vaccine
: Any
Country
: Global/Not specified
Period monitored
: 14 July
2014 - 26 September 2017
Search strategy
:
Automated (Twitter API)
Sentiment analysis
: Manual
(anti-vaccine, neutral, pro-
vaccine)
# results
: 1,793,690 tweets
“Compared with average users, Russian trolls (c2(1) = 102.0; P < .001), sophisticated bots
(c2(1) = 28.6; P < .001), and “content polluters” (c2(1) = 7.0; P < .001) tweeted about
vaccination at higher rates. Whereas content polluters posted more anti-vaccine content
(c2(1) = 11.18; P < .001), Russian trolls amplified both sides. Unidentifiable accounts were
more polarised (c2(1) = 12.1; P < .001) and anti-vaccine (c2(1) = 35.9; P < .001). Analysis
of the Russian troll hashtag showed that its messages were more political and divisive.”
Buchanan2014[104]
Assess the magnitude, interest, purpose and
validity of information regarding vaccination
available on Facebook and assess whether
this information varies by site viewpoint.
Social media type
: Facebook
Vaccine
: Any
Country
: Global/not specified
Period monitored
: August
2012 (one point in time)
Search strategy
: Manual
Sentiment analysis
: Manual
(anti, neutral, pro)
# results
: 520 Facebook pages, 196 Facebook places and 187 Facebook groups
“Of 30 sites, 43% (n=13) were anti-vaccination, 7% (n=2) neutral and 50% (n=15) pro-
vaccination. Most sites were most popular with American users. Median members were
similar between anti-vaccination (2703 members, range 33733 631 members) and pro-
vaccination sites (2142 members, range 45661 565 members, P = 0.262); however, anti-
vaccination sites accumulated more posts per week by authors (median 15 vs. 3, P=0.031)
and members (median 33 vs. 1, P<0.001). Pro-vaccination sites more commonly had
commercial purpose (53% (n = 8) vs. 8% (n=1), P=0.02). Anti-vaccination sites more
commonly gave medical advice (54% (n=7) vs. 0%, P=0.004). Overall, 48% (n=22) of
author posts were concordant with regulatory recommendations; concordance was more
common on pro-vaccination sites (78% (n=21) vs. 5% (n=1), P=0.0002).”
Cambra2016[97]
Monitor online discussion on vaccination in
Spain on YouTube with the objective of
developing an interpretative theoretical
framework.
Social media type
: YouTube
Vaccine
: Any
Country
: Global/Not specified
Search strategy
: Manual
# results
: 81,100 videos
“The results indicate that there are fewer negative videos, but with a longer duration than
the positive ones. Countries of origin are mainly from Latin America, particularly Mexico.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
58
Reference Study aim(s) Study details Methodology Results (taken from the article’s abstract)
Period monitored
: 15-20 May
(year not provided)
Sentiment analysis
: Manual
(against vaccination, for
vaccination)
Finally, YouTube characteristics were valued to share health education messages and to
design digital programs of Public Health.”
Chakraborty2017[58]
Systematically analyse Twitter messages to
obtain a unique view into public sentiment
around HPV vaccination.
Social media type
: Twitter
Vaccine
: HPV
Country
: US
Period monitored
: 7-13
February 2015
Search strategy
:
Automated (Twitter API,
Python (x,y) and Twython)
Sentiment analysis
: Manual
(negative, neutral,
positive)
# results
: 20,408 tweets
“To maintain feasibility, we used a computerized random generator to obtain a sub-sample
of 2,000 of these tweets for in-depth qualitative coding. The four categories that accounted
for the largest proportion of tweets included news and media coverage of current events
related to the HPV vaccine, discussion of possible associations between receiving the
vaccine and sexual behaviour, safety of the vaccine, and effectiveness of the vaccine.
Multiple inaccurate myths surrounding the vaccine, such as the misconception that it is only
appropriate for females, were noted.”
Chen2015[127]
Examine media and public reactions to the
media coverage of suspected vaccine
adverse events and relevant policy changes
in the interactive media environment,
investigate the relations between online
media coverage, Weibo posts and search
engine searches, explore public sentiments
towards vaccination on Weibo during a
Hepatitis B crisis in China.
Social media type
: Weibo
Vaccine
: Hepatitis B
Country
: China
Period monitored
: 5
December 2013 - 10 January
2014
Search strategy
: Manual
Sentiment analysis
: Manual
(negative, neutral,
positive)
# results
: 277,091 Weibo posts
“A total of 17 infant deaths were reported to be associated with Hepatitis B vaccination.
Three major waves of high media and public attention were detected. The daily indicators
reached their peaks in the second wave after the relevant vaccine was suspended by the
authority (from December 20 to December29, 2013) with 23,200 daily online news reports,
34,018 Sina Weibo posts and 17,832 Baidu search indices. There were significant
correlations between the daily amount of online news, Weibo posts, and Baidu searches (p
< .001). The contents analysis suggested 1343 out of 1608 (83.5%) original Weibo posts
expressed negative sentiment with almost 90% in the second wave.”
Chen2018[59]
Understand how images are used in vaccine-
related tweets and provide guidance with
respect to the characteristics of vaccine-
related images that correlate with the higher
likelihood of being retweeted.
Social media type
: Twitter
Vaccine
: Any
Country
: Global/Not specified
Period monitored
: 11
November 2014 - 8 August
2016
Search strategy
:
Automated (Twitter API)
Sentiment analysis
: n/a
# results
: 1,137,172 tweets
“Most vaccine-related images are duplicates (125,916/237,478; 53.02%) or taken from
other sources, not necessarily created by the author of the tweet. Almost half of the images
contain embedded text, and many include images of people and syringes. The visual
content is highly correlated with a tweet’s textual topics. Vaccine image tweets are twice as
likely to be shared as non-image tweets. The sentiment of an image and the objects shown
in the image were the predictive factors in determining whether an image was retweeted.”
Covolo2017[32]
Examine the content of Italian YouTube
videos related to paediatric vaccinations and
understand the potential of messages in
influencing public awareness and opinions.
Social media type
: YouTube
Vaccine
: Any (childhood
vaccines)
Country
: Italy
Period monitored
: June 2014
- September 2015
Search strategy
: Manual
Sentiment analysis
: Manual
(ambiguous, negative,
neutral, positive)
# results
: 200 videos
“A total of 123 videos were selected. Pro-vaccination videos were 62 (50%), anti-vaccination
28 (23%), neutral or without a clear position in favour or against vaccination 33 (27%).
Focusing on the first 2 groups, pro-vaccination videos had a higher number of views
compared with those unfavourable (1602 ± 6544 vs 1482 ± 2735) (p < 0.001). However,
anti-vaccination videos were liked more by viewers (17.8 ± 31.3) than positive ones (13.2 ±
44.7) (p < 0.001) in addition to being more shared (23 ± 22.6 vs 3.8 ± 5.5, p < 0.001).”
D'Andrea2017[89]
Monitor Italian public opinion from tweets
analysis, with reference to the vaccination
topic.
Social media type
: Twitter
Vaccine
: Any
Country
: Italy
Period monitored
: 1
September - 30 November
2016
Search strategy
:
Automated (Twitter API +
Java Library: Get Old
Tweets)
Sentiment analysis
:
Automated (negative,
positive)
# results
: 17,937 tweets
“An approach to monitor the Italian public opinion from tweets analysis, with reference to
the vaccination topic. By employing the Simple Logistic classifier, we achieved a 75.5%
average accuracy for discriminating negative opinions tweets (i.e., not in favour of
vaccination) from the rest of tweets.”
D'Andrea2018[60]
Propose an intelligent system for real-time
monitoring and analysis of public opinion
about the vaccination topic on the Twitter
stream.
Social media type
: Twitter
Vaccine
: Any
Country
: Italy
Period monitored
: 1
September 2016 - 31 January
2017
Search strategy
:
Automated (Twitter API +
Java Library: Get Old
Tweets)
# results
: 112,397 tweets
“In tuning the system, we tested multiple combinations of different text representations and
classification approaches: the best accuracy was achieved by the scheme that adopts the
bag-of-words, with stemmed n -grams as tokens, for text representation and the support
vector machine model for the classification. By presenting the results of a monitoring
campaign lasting 10 months, we show that the system may be used to track and monitor
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
59
Reference Study aim(s) Study details Methodology Results (taken from the article’s abstract)
Sentiment analysis
:
Automated (in favour of
vaccination, neutral, not in
favour of vaccination)
the public opinion about vaccination decision making, in a low-cost, real-time, and quick
fashion. Finally, we also verified that the proposed scheme for continuous tweet
classification does not seem to suffer particularly from concept drift, considering the time
span of the monitoring campaign.”
Deiner2017[129]
Examine Facebook and Twitter discussion of
vaccination in relation to measles in a period
of several widely publicised outbreaks.
Social media type
: Mix:
Facebook, Twitter
Vaccine
: Measles
Country
: US
Period monitored
: 2009 -
2016
Search strategy
:
Automated (Crimson
Hexagon)
Sentiment analysis
:
Automated, Brightview
classifier (hesitancy, pro-
vaccination)
# results
: 58,078 Facebook posts and 82,993 tweets
“Pro-vaccination posts were correlated with the US weekly reported cases (Facebook:
Spearman correlation 0.22 (95% confidence interval: 0.09 to 0.34), Twitter: 0.21 (95%
confidence interval: 0.06 to 0.34)). Vaccine-hesitant posts, however, were uncorrelated with
measles cases in the United States (Facebook: 0.01 (95% confidence interval: −0.13 to
0.14), Twitter: 0.0011 (95% confidence interval: −0.12 to 0.12)).”
Donzelli2018[98]
Carry out a quantitative analysis of Italian
videos available on YouTube about the link
between vaccines and autism or other
serious side effects in children.
Social media type
: YouTube
Vaccine
: Any (childhood
vaccines)
Country
: Italy
Period monitored
: 27
December 2007 - 31 July
2017
Search strategy
: Manual
Sentiment analysis
: Manual
(negative, neutral,
positive)
# results
: 560 videos
“The analysis revealed that most of the videos were negative in tone and that the annual
number of uploaded videos has increased during the considered period, that goes from 27
December 2007 to 31 July 2017, with a peak of 224 videos in the first seven months of
2017.”
Dredze2017[61]
Examine how different candidates during the
2016 presidential campaign commanded the
attention of Twitter users who
communicated about vaccination on Twitter.
Social media type
: Twitter
Vaccine
: Any
Country
: US
Period monitored
: 1
December 2014 - 1
December 2016
Search strategy
:
Automated (Twitter API)
Sentiment analysis
:
Automated (anti-
vaccination)
# results
: 11,144 tweets
“Before the election, the number of tweets expressing vaccine scepticism and mentioning
Trump was three times that of any other candidate. After the election, the daily median
number of tweets mentioning Mr. Trump increased from 5 to 22, whereas median mentions
of other candidates remained unchanged.
During the eight time periods with the highest total volume of vaccine sceptical tweets, 5
were about Mr. Trump, 1 was about Secretary Clinton, 1 was about Dr. Stein, and 1
included both Mr. Trump and Secretary Clinton. In every case, the reactions were positive
concerning Mr. Trump (e.g., praising his mention of vaccines in a debate, asking him to
assist in supporting the documentary ‘‘Vaxxed”), negative towards Ms. Clinton (when she
voiced support for vaccines), and mixed towards Dr. Stein. From a subset of 100 randomly
selected post-election tweets that mentioned Mr. Trump and an anti-vaccine hashtag, 59%
expressed that Mr. Trump would bring policy change favourable to vaccine refusal. The
most common themes were investigating the CDC, allowing exemptions or removing
mandates, and repealing the National Childhood Vaccine Injury Act, which limits legal and
financial risk for vaccine manufacturers.”
Du2017a[64]
Leverage a hierarchical machine learning
based sentiment analysis system to extract
public opinions towards HPV vaccines from
Twitter.
Social media type
: Twitter
Vaccine
: HPV
Country
: Global/Not specified
Period monitored
: 2
November 2015 - 28 March
2016
Search strategy
:
Automated (Twitter API)
Sentiment analysis
:
Automated (against, not
applicable, pro)
# results
: 1,154,156 tweets
“The evaluation of the unannotated tweets corpus showed that the micro-averaging F
scores have reached 0.786. The learning system deduced the sentiment labels for 184,214
tweets in the collected unannotated tweets corpus. Time series analysis identified a
coincidence between mainstream outcome and Twitter contents. A weak trend was found
for “Negative” tweets that decreased firstly and began to increase later; an opposite trend
was identified for “Positive” tweets. Tweets that contain the worries on efficacy for HPV
vaccines showed a relative significant decreasing trend. Strong associations were found
between some sentiments (“Positive”, “Negative”, “Negative-Safety” and “Negative-Others”)
with different days of the week.”
Du2017b[63]
Propose a machine learning system that is
able to extract comprehensive public
sentiment on HPV vaccines on Twitter with
satisfying performance.
Social media type
: Twitter
Vaccine
: HPV
Country
: Global/Not specified
Period monitored
: 15 July -
17 August 2015
Search strategy
:
Automated (Twitter API)
Sentiment analysis
:
Automated (negative,
neutral, others, positive)
# results
: 184,214 tweets
“A hierarchical classification scheme that contains 10 categories was built to access public
opinions toward HPV vaccines comprehensively. A 6,000 annotated tweets gold corpus with
Kappa annotation agreement at 0.851 was created and made public available. The
hierarchical classification model with optimised feature sets and model parameters has
increased the micro-averaging and macro-averaging F score from 0.6732 and 0.3967 to
0.7442 and 0.5883 respectively, compared with baseline model.”
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
60
Reference Study aim(s) Study details Methodology Results (taken from the article’s abstract)
Du2018[62]
Develop a scheme for a comprehensive
public perception analysis of a measles
outbreak based on Twitter data and
demonstrate the superiority of the
convolutional neural network (CNN) models
on measles outbreak-related tweets
classification tasks.
Social media type
: Twitter
Vaccine
: Measles
Country
: Global/Not specified
Period monitored
: 1
December 2014 - 30 April
2015
Search strategy
:
Automated
(DiscoverText.com)
Sentiment analysis
:
Automated (negative,
neutral, others, positive)
# results
: 6,000 tweets
“Cohen kappa intercoder reliability values for the annotation were: 0.78, 0.72, and 0.80 on
the 3 dimensions, respectively. Class distributions within the gold standard were highly
unbalanced for all dimensions. The CNN models performed better on all classification tasks
than k-nearest neighbours, naïve Bayes, support vector machines, or random forest.
Detailed comparison between support vector machines and the CNN models showed that
the major contributor to the overall superiority of the CNN models is the improvement on
recall, especially for classes with low occurrence. The CNN model with the 2 embedding
combination led to better performance on discussion themes and emotions expressed
(microaveraging F1 scores of 0.7811 and 0.8592, respectively), while the CNN model with
Stanford embedding achieved best performance on attitude toward vaccination
(microaveraging F1 score of 0.8642).”
Dunn2015[65]
Examine the association between exposure
to negative opinions about HPV vaccines and
expression of negative opinions about HPV
vaccines among Twitter users.
Social media type
: Twitter
Vaccine
: HPV
Country
: Global/Not specified
Period monitored
: 1 October
2013 - 1 April 2014
Search strategy
:
Automated (Twitter API)
Sentiment analysis
:
Automated (negative)
# results
: 83,551 tweets
“During the 6-month period, 25.13% (20,994/83,551) of tweets were classified as negative;
among the 30,621 users that tweeted about HPV vaccines, 9046 (29.54%) were exposed to
a majority of negative tweets. The likelihood of a user posting a negative tweet after
exposure to a majority of negative opinions was 37.78% (2780/7361) compared to 10.92%
(1234/11,296) for users who were exposed to a majority of positive and neutral tweets
corresponding to a relative risk of 3.46 (95% CI 3.25-3.67, P<.001).”
Dunn2017[66]
Determine whether state level differences in
exposure to information on Twitter about
HPV vaccine were associated with state level
differences in HPV vaccine coverage in the
US.
Social media type
: Twitter
Vaccine
: HPV
Country
: US
Period monitored
: 1 October
2013 - 30 October 2015
Search strategy
:
Automated (Twitter API)
Sentiment analysis
: n/a
# results
: 258,418 tweets
“Topics corresponding to media controversies were most closely correlated with coverage
(both positively and negatively); education and insurance were highest among
socioeconomic indicators. Measures of information exposure explained 68% of the variance
in one dose 2015 HPV vaccine coverage in females (males: 63%). In comparison, models
based on socioeconomic factors explained 42% of the variance in females (males: 40%).
Ekram2018[99]
Examine the tone of videos for HPV vaccine
and accuracy of information shown of
YouTube videos.
Social media type
: YouTube
Vaccine
: HPV
Country
: Global/Not specified
Period monitored
: 13
November 2006 - 14 April
2014
Search strategy
: Manual
Sentiment analysis
: Manual
(negative, neutral,
positive)
# results
: 35 videos
“Most videos were negative in tone toward the vaccine. The tone of the video was not a
predictor of video popularity. Pro-vaccine videos were 4 times more likely to report
information accurately than anti-vaccine videos. Anti-vaccine videos were more likely to
report information incorrectly and omit information. The most frequent commentary themes
were concerning serious side effects, conspiracy theories, and vaccines generally being
unhealthy.”
Faasse2016[105]
Investigate the types of arguments and
language used by pro- and anti-vaccination
individuals within the same conversational
context in an effort to better understand
underlying thought processes and inform
future attitude- and behaviour-change
attempts.
Social media type
: Facebook
Vaccine
: Any
Country
: Global/not specified
Period monitored
: One week
in January 2016
Search strategy
: Manual
Sentiment analysis
: Manual
(anti-vaccination, pro-
vaccination,
unrelated/unclear)
# results
: 1490 Facebook comments
“Both pro- and anti-vaccination comments used more risk-related and causation words, as
well as fewer positive emotion words compared to control comments. Anti-vaccine
comments were typified by greater analytical thinking, lower authenticity, more body and
health references, and a higher percentage of work-related word use in comparison to pro-
vaccine comments, plus more money references than control comments. In contrast, pro-
vaccination comments were more authentic, somewhat more tentative, and evidenced
higher anxiety words, as well as more references to family and social processes when
compared to anti-vaccination comments”
Fadda2015[118]
Analysing Italian online debates on
paediatric immunisations through a content
analytic approach.
Social media type
: Forums
Vaccine
: Any (childhood
vaccines)
Country
: Italy
Period monitored
: January
2008 - June 2014
Search strategy
: Manual
(Google Search)
Sentiment analysis
: n/a
# results
: 340 forum threads with 6544 posts
“The analysis included 6544 posts mentioning 6223 arguments about paediatric vaccinations
and citing 4067 sources. The analysis of argument posting patterns included users who
published a sufficient number of posts; they generated 85% of all arguments on the forum.
Dominating patterns of three groups were identified: (1) an anti-vaccination group (n =
280) posted arguments against vaccinations, (2) a general pro-vaccination group (n = 222)
posted substantially diverse arguments supporting vaccination and (3) a safety-focused pro-
vaccination group (n = 158) mainly forwarded arguments that questioned the negative side
effects of vaccination. The anti-vaccination group was shown to be more active than the
others. They use multiple sources, own experience and media as their cited sources of
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
61
Reference Study aim(s) Study details Methodology Results (taken from the article’s abstract)
information. Medical professionals were among the cited sources of all three groups,
suggesting that vaccination-adverse professionals are gaining attention.”
Furini2018[106]
Understand specific features of the language
used to talk about vaccinations on social
media platforms.
Social media type
: Facebook
Vaccine
: Any
Country
: Italy
Period monitored
: July 2009 -
October 2017
Search strategy
:
Automated (Facebook API)
Sentiment analysis
:
Automated (anger, anxiety,
negative, positive)
# results
: 237,112 Facebook posts
“The obtained results show that anti-vaccination groups use a language that is difficult to
refute (e.g. not anxious, not focused on specific health issues or on specific diseases),
whereas the analysis of pro vaccination groups reveals much more anxiety and specificity
(e.g. family cases, specific diseases or vaccines).”
Goh2017[115]
Use the elaboration likelihood model (ELM)
to investigate the central and peripheral
message cues generated by Chinese parents
in their online discussions about the
rotavirus vaccine.
Social media type
: Forums
(Babytree.com)
Vaccine
: Rotavirus
Country
: China
Period monitored
: 2007-2015
Search strategy
: Manual
Sentiment analysis
: n/a
# results
: 136 forum discussion threads
“The results indicated that forum users employed both central and peripheral cues as a joint
process when generating information intended to help other parents gain knowledge and
make vaccination decisions. Issue-relevant arguments important to vaccination decision
included the vaccine's necessity, side effects and efficacy. Peripheral cues including site-
generated sorting cues were associated with posts featuring greater elaboration. New
parents had the most doubts, asking the most questions about vaccine issues. Their
elaboration, however, was the weakest.”
Guidry2015[125]
Identify how vaccinations are portrayed on
Pinterest, how users engage with
vaccination content on the platform, how
are HBM constructs represented in
vaccination-focused pins and to what extent
vaccination-related pins mention issues
related to conspiracy theories and civil
liberties.
Social media type
: Pinterest
Vaccine
: Any
Country
: Global/Not specified
Period monitored
: 19-21
March 2014
Search strategy
: Manual
Sentiment analysis
: Manual
(anti-vaccine, neutral, pro-
vaccine)
# results
: 800 selected pins
“The majority of the pins were anti-vaccine, and most were original posts as opposed to
repins. Concerns about vaccine safety and side effects were oft-repeated themes, as was
the concept of conspiracy theory. Pro-vaccine pins elicited consistently more engagement
than anti-vaccine pins.
Hernandez-
Garcia2018[100]
Determine the characteristics of YouTube
videos which provide information in Spanish
about the meningococcal B vaccine.
Social media type
: YouTube
Vaccine
: meningococcal B
Country
: Global/Not specified
Period monitored
: 19-21
February 2018
Search strategy
: Manual
Sentiment analysis
: Manual
(ambiguous, negative,
neutral, positive)
# results
: 62 videos
“A total of 62 videos were analysed, of which 45.2% were produced by television channels,
and 58.1% supported the use of the vaccine. Only 11.3% mentioned at least one of the
vaccination recommendations of the ministry. There were significant differences in the
frequency of the vaccine mode of administration depending on the authorship (OR (95%
CI:18.75 (1.73203.21)), description of its posology (OR (95% CI: 6.36 (1.57–25.75))), and
its price (OR(95% CI: 0.11 (0.010.95)), and in some vaccination recommendations by the
ministry [deficit of properdin, treatment with eculizumab, and asplenia: OR (95% CI: 9.19
(1.3263.87)).”
Huang2017[67]
Measure levels of flu vaccine uptake
aggregated by time, geography, and
demographic group, where geographic and
demographic attributes are inferred from
user profiles.
Social media type
: Twitter
Vaccine
: Influenza
Country
: Global/Not specified
Period monitored
: Three
influenza vaccination seasons
2013-2016
Search strategy
:
Automated (Twitter API)
Sentiment analysis
:
Automated (Does this
message indicate that
someone received or
intended to receive a flu
vaccine? (yes, no))
# results
: 10,000 tweets
“In this study, we build and employ several natural language classifiers to examine and
analyse behavioural patterns regarding influenza vaccination in Twitter across three
dimensions: temporality (by week and month), geography (by US region), and demography
(by gender). Our best results are highly correlated official government data, with a
correlation over 0.90, providing validation of our approach.”
Jang2019[132]
Investigate the flow of information about the
vaccine-autism controversy between social
media and mainstream online news;
compare social media and online news in
terms of the degree to which media pay
attention to the vaccine-autism controversy;
and examine different patterns shown in the
content coming from the US, Canada and
the UK.
Social media type
: Mix:
Reddit, Twitter
Vaccine
: Any
Country
: Mix: Canada, UK,
US
Period monitored
: 1 February
2015 - 30 September 2016
Search strategy
:
Automated (Crimson
Hexagon's ForSight
platform)
Sentiment analysis
: n/a
# results
: 220,458 tweets and 17,661 Reddit posts
“Our time-series analysis shows that Twitter drives news agendas, and Reddit follows news
agendas regarding the vaccine-autism debate. Additionally, the results show that both
Twitter and Reddit are more likely to discuss the vaccine-autism link compared to online
news content.”
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
62
Reference Study aim(s) Study details Methodology Results (taken from the article’s abstract)
Kang2017[68]
Examine current vaccine sentiment on social
media by constructing and analysing
semantic networks of vaccine information
from highly shared websites of Twitter users
in the US and to assist public health
communication of vaccines.
Social media type
: Twitter
Vaccine
: Any
Country
: US
Period monitored
: 16 April -
29 May 2015
Search strategy
:
Automated
(ChatterGrabber)
Sentiment analysis
: Manual
(negative, neutral,
positive)
# results
: 26,389 tweets
“The semantic network of positive vaccine sentiment demonstrated greater cohesiveness in
discourse compared to the larger, less-connected network of negative vaccine sentiment.
The positive sentiment network centred around parents and focused on communicating
health risks and benefits, highlighting medical concepts such as measles, autism, HPV
vaccine, vaccine-autism link, meningococcal disease, and MMR vaccine. In contrast, the
negative network centred around children and focused on organisational bodies such as
CDC, vaccine industry, doctors, mainstream media, pharmaceutical companies, and United
States. The prevalence of negative vaccine sentiment was demonstrated through diverse
messaging, framed around scepticism and distrust of government organisations that
communicate scientific evidence supporting positive vaccine benefits.”
Kaptein2014[69]
Investigate discussions on Twitter around
HPV vaccinations to find out what is an
effective way to retrieve discussions on
Twitter and what are the characteristics of
HPV discussions on Twitter.
Social media type
: Twitter
Vaccine
: HPV
Country
: Netherlands
Period monitored
: March -
April 2013
Search strategy
:
Automated (Twitter API)
Sentiment analysis
: Manual
(anti-vaccination, doubt,
negative, neutral, no
opinion, pro-vaccination,
positive)
# results
: 12639 tweets
“We find that by tracking the conversations on Twitter relevant tweets can be found with
reasonable precision. Although sentiments and opinions change regularly in a discussion, we
find few cases of topic drift.”
Keelan2007[101]
Characterise the available information about
immunisation on YouTube.
Social media type
: YouTube
Vaccine
: Any
Country
: Global/Not specified
Period monitored
: 20
February 2007
Search strategy
: Manual
Sentiment analysis
: Manual
(ambiguous, negative,
positive)
# results
: 153 videos
“Seventy-three (48%) of the videos were positive, 49 (32%) were negative, and 31 (20%)
were ambiguous. Compared with positive videos, negative videos were more likely to
receive a rating, and they had a higher mean star rating and more views. Among the
positive videos, public service announcements received the lowest mean (SD) ratings (2.6
(1.6) stars) and the fewest views (median, 213; interquartile range, 114-409). The most
commonly discussed vaccine topic was general childhood vaccines (38 videos (25% of the
total)). The most commonly discussed specific vaccine was the HPV vaccine (36 videos)
24% of the total)); 20 of these were positive, 4 of which were industry sponsored. Of the
HPV vaccine-related videos, 24 specifically referred to Merck or Gardasil. Of the negative
videos, 22 (45%) conveyed messages that contradicted the reference standard. None of the
positive videos made scientific statements that contradicted the reference standard.”
Keim-Malpass2017[70]
Evaluate the content of messaging regarding
the HPV vaccine on Twitter, and describe
the sentiment of those messages by type of
user.
Social media type
: Twitter
Vaccine
: HPV
Country
: Global/Not specified
Period monitored
: Two weeks
in June 2015
Search strategy
:
Automated (Topsy)
Sentiment analysis
:
Automated (Topsy,
sentiment score) and
manual (negative, neutral,
positive)
# results
: 1794 tweets
“The majority of Twitter posts were written by lay consumers and were sharing commentary
about a media source. However, when actual URLs were shared, the most common form of
share was linking back to a blog post written by lay users. The vast majority of content was
presented as polarising (either as a positive or negative tweet), with 51% of the Tweets
representing a positive viewpoint.
Krittanawong2017[71]
Assess patients’ perception of the influenza
vaccine and the reason for its
underutilisation.
Social media type
: Twitter
Vaccine
: Influenza
Country
: Global/Not specified
Period monitored
: 23 July
2009 - 22 October 2016
Search strategy
:
Automated (Twitter API)
Sentiment analysis
: n/a
# results
: 29243 tweets
“The tweets often pertained to self-reports after receiving the influenza vaccine (14%); the
reason for not receiving the influenza vaccine (12%); emotional language with positive or
negative sentiments (33%); and advertisement, news, or updated research (41%).”
Lehmann2013[134]
Describe the news site and social media
website content about influenza vaccination
on the Internet, as well as the similarities
and differences between these two types of
media content.
Social media type
: Mix:
Facebook, Hyves, LinkedIn,
Twitter
Vaccine
: Influenza
Country
: Netherlands
Period monitored
: February -
April 2012
Search strategy
:
Automated (Clipit)
Sentiment analysis
: n/a
# results
: 3552 posts
“Three overarching themes were found in both media sources: (1) the (upcoming) influenza
epidemic, (2) general information regarding the virus, its prevention and treatment, and (3)
uncertainty and mistrust regarding influenza vaccination. Social media tended to report
earlier on developments such as the occurrence of an influenza epidemic. The greatest
difference was that in social media, influenza was not considered to be a serious disease,
and more opposition to the flu shot was expressed in social media, as compared to news
media.”
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
63
Reference Study aim(s) Study details Methodology Results (taken from the article’s abstract)
Lopez2017[123]
Identify what information is available to the
Chilean population in Spanish on HPV on
Wikipedia, Yahoo Answers and a website
managed by the Chilean Ministry of Health.
Social media type
: Yahoo!
Answers
Vaccine
: HPV
Country
: Chile
Period monitored
: 17 April - 3
May 2015
Search strategy
: Manual
Sentiment analysis
: n/a
# results
: n/a
“The information provided by the Spanish version of Wikipedia was accurate; nevertheless a
few omissions were detected. The quality of the information provided by the Spanish
version of Yahoo Answers was inaccurate and confusing. The Minsal website lacked
important information on several topics about HPV even though it is managed and endorsed
by the government.”
Love2013[72]
Report a content analysis of Twitter posts
about vaccinations, documenting the
sources, the tone, and the medical accuracy
of the conversation.
Social media type
: Twitter
Vaccine
: Any
Country
: Global/Not specified
Period monitored
: 8-14
January 2012
Search strategy
:
Automated (NodeXL, Social
Media Research
Foundation)
Sentiment analysis
: Manual
(negative, neutral,
positive)
# results
:
“A set of 6,827 tweets indicates professional sources were shared most and treated
positively. Two-thirds of shared medical content were substantiated. One-third of messages
were positive, counter to other research and suggesting that users apply critical thinking
when evaluating content.”
Luisi2018[45]
Explore how Kansan parents/guardians of
HPV vaccine-eligible children perceive the
vaccine in the contexts of the health belief
model and the social amplification of risk
framework, parent/guardian engagement
with HPV vaccine-related information, and
Facebook representations by general users
and the Centers for Disease Control and
Prevention during the vaccine’s first decade
on the market.
Social media type
: Facebook
Vaccine
: HPV
Country
: US
Period monitored
: June -
June 2016
Search strategy
: Manual
Sentiment analysis
: Manual
(negative, neutral,
positive)
# results
: 6,537 posts
“Most of the posts had a negative tone (n = 3,501, 53.8%). Concerning the tone towards
the HPV vaccine, slightly fewer than half had a negative tone (n = 2,928, 45.0%). In a large
majority of posts, content that suggested HPV infection susceptibility was absent (n =
6,236, 95.8%), as was content suggesting that HPV infection would be severe (n = 5,487,
84.3%). Barriers to getting the vaccine were mentioned in more posts (n = 3,056, 47.0%)
than benefits (n = 1,281, 19.7%). Concerning cues to action, most posts did not tell people
either way to get or avoid the HPV vaccine; but a few did explicitly tell people to get
vaccinated (n = 208, 3.2), and fewer told people explicitly to avoid the HPV vaccine (n =
119, 1.8%). Very few self-efficacy messages were largely were present in the sample (n =
287, 4.4%). More than half of posts (n = 3,752, 57.7%) neither amplified nor attenuated
the perception of risk to one’s health that the HPV vaccine could cause. However, nearly
forty percent of the posts amplified HPV vaccine risk (n = 2,568, 39.5%) and only a very
few attenuated risks (n = 186, 2.9%). The vast majority of posts did not discuss ripples (n
= 6,110, 93.9%) or impacts (n = 6,312, 97%) from the perceived risks of the HPV vaccine.
The researcher also analysed the CDC Facebook posts. Most of the posts had a positive tone
(n = 20, 64.5%).”
Luoma-aho2013[120]
Find out which issues related to swine flu
interested citizens, how the authorities were
discussed, what attitudes people harboured
towards them and whether the interventions
by Ministry of Social Affairs and Health
affected the content of the debate.
Social media type
: Forums
(Iltalehti , KaksPlus)
Vaccine
: H1N1
Country
: Finland
Period monitored
: March -
May 2010
Search strategy
: Manual
Sentiment analysis
: Manual
# results
: 19 forum discussions with 2264 comments
“The analysis of the media releases revealed that the crisis communication of the authorities
was timely and factual, yet failed both in using understandable concepts and responding to
the emotional needs of people threatened by swine-flu and questioning the safety of the
vaccination. These deficiencies intensified emotion-driven discussion, and when people
opposed to vaccination managed to secure the central ‘issue arenas’ using the words ‘swine
flu’ online, this led to online speculations and exaggeration of threat, excluding the
authorities and logical argument from the discussion.”
Ma2017[107]
Understand what contextual factors in a
public anti-vaccination Facebook group
potentially influence parental assessment of
information sources when seeking and
sharing experiences, information, and
knowledge regarding vaccine safety.
Social media type
: Facebook
Vaccine
: Any
Country
: US
Period monitored
: January
2015 - August 2016
Search strategy
:
Automated (Facebook
Python API)
Sentiment analysis
: n/a
# results
: 122 posts with 1456 comments
“Findings show that parental information seeking and sharing worked to create an isolated,
sentimentalised information context favouring immediacy and emotional impact over
scientific research and statistical evidence. Because participants shared fundamental beliefs
and goals around vaccines, group members held cognitive authority despite the lack of
expertise or evidentiary support in their postings.”
Mahoney2015[73]
Explore how new media influences the type
of public health information users access, as
well as the impact to these platforms after a
major controversy.
Social media type
: Twitter
Vaccine
: HPV
Country
: Global/Not specified
Period monitored
: 1 August -
31 October 2011
Search strategy
:
Automated (Topsy)
Sentiment analysis
: Manual
(negative, neutral,
positive)
# results
: 100 news articles on Google News and 100 articles on Twitter
“Results indicate that 44.0% of the articles (88/200) about the HPV vaccination had a
positive tone, 32.5% (65/200) maintained a neutral tone, while 23.5% (47/200) presented a
negative tone. Protection against diseases 82.0% (164/200), vaccine eligibility for females
75.5% (151/200), and side effects 59.0% (118/200) were the top three topics covered by
these articles. Google News and Twitter articles significantly differed in article tone, source,
topics, concerns covered, types of sources referenced in the article, and uses of interactive
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
64
Reference Study aim(s) Study details Methodology Results (taken from the article’s abstract)
features. Most notably, topic focus changed from public health information towards political
conversation after Bachmann’s comment. Before the comment, the HPV vaccine news talked
more often about vaccine dosing (P<.001), duration (P=.005), vaccine eligibility for females
(P=.03), and protection against diseases (P=.04) than did the later pieces. After the
controversy, the news topic shifted towards politics (P=.01) and talked more about HPV
vaccine eligibility for males (P=.01).
Massey2016[75]
Quantify HPV vaccine communication on
Twitter, and develop a novel methodology to
improve the collection and analysis of
Twitter data.
Social media type
: Twitter
Vaccine
: HPV
Country
: Global/Not specified
Period monitored
: 1 August
2014 - 31 July 2015
Search strategy
:
Automated (Black and
Colleagues and Microsoft
Research)
Sentiment analysis
:
Automated (negative,
neutral, positive)
# results
: 193,379 tweets
“A total of 193,379 English-language tweets were collected, classified, and analysed.
Associated words varied with each keyword, with more positive and preventive words
associated with “HPV vaccine” and more negative words associated with name-brand
vaccines. Positive sentiment was the largest type of sentiment in the sample, with 75,393
positive tweets (38.99% of the sample), followed by negative sentiment with 48,940 tweets
(25.31% of the sample). Positive and neutral tweets constituted the largest percentage of
tweets mentioning prevention or protection (20,425/75,393, 27.09% and 6477/25,110,
25.79%, respectively), compared with only 11.5% of negative tweets (5647/48,940;
P<.001). Nearly one-half (22,726/48,940, 46.44%) of negative tweets mentioned side
effects, compared with only 17.14% (12,921/75,393) of positive tweets and 15.08% of
neutral tweets (3787/25,110; P<.001).”
Massey2018[74]
Characterise and quantify three types of
Twitter messages related to the HPV
vaccine: 1) tweets sent by health
professionals, 2) tweets intended for a
parent audience, and 3) tweets sent by
health professionals and intended for a
parent audience.
Social media type
: Twitter
Vaccine
: HPV
Country
: Global/Not specified
Period monitored
: 1 August
2014 - 31 July 2015
Search strategy
:
Automated (Twitter API)
Sentiment analysis
:
Automated (negative,
neutral, positive)
# results
: 193, 379 tweets
“Of the 193,379 tweets, 20,451 tweets were from health professionals; 16,867 tweets were
intended for parents; and 1,233 tweets overlapped both groups. The content of each spike
varied per group. The largest spike in tweets from health professionals (n = 851) focused
on communicating recently published scientific evidence. Most tweets were positive and
were about resources and boys. The largest spike in tweets intended for parents (n =
1,043) centred on a national awareness day and were about resources, personal
experiences, boys, and girls. The largest spike in tweets from health professionals to
parents (n = 89) was in January and centred on an event hosted on Twitter that focused on
cervical cancer awareness month.”
McNeill2016[76]
Offer a more detailed and nuanced picture
of the opportunities and challenges
associated with pandemic health-
communication on Twitter.
Social media type
: Twitter
Vaccine
: H1N1
Country
: UK
Period monitored
: 1 April
2009 - 1 May 2010
Search strategy
:
Automated (Gnip)
Sentiment analysis
: n/a
# results
: 12,711 tweets.
“Network analysis of retweets showed that information from official sources predominated.
Analysing the spread of significant messages through Twitter showed that most content was
descriptive but there was some criticism of health authorities. A detailed analysis of
responses to press releases revealed some scepticism over the economic beneficiaries of
vaccination, that served to undermine public trust. Finally, the conversational analysis
showed the influence of peers when weighing up the risks and benefits of medication.
Mitra2016[77]
Identify Twitter users who persistently hold
pro and anti-attitudes, and those who newly
adopt anti attitudes towards vaccination and
explore differences in the individual
narratives across the user cohorts.
Social media type
: Twitter
Vaccine
: Measles
Country
: Global/Not specified
Period monitored
: 1 January
2012 - 30 June 2015
Search strategy
:
Automated (Twitter
Firehose)
Sentiment analysis
:
Automated (anti, pro)
# results
: 315,240 tweets
“We find that those with long-term anti-vaccination attitudes manifest conspiratorial
thinking, mistrust in government, and are resolute and in-group focused in language. New
adoptees appear to be predisposed to form anti-vaccination attitudes via similar government
distrust and general paranoia, but are more social and less certain than their long-term
counterparts. We discuss how this apparent predisposition can interact with social media-
fuelled events to bring newcomers into the anti-vaccination movement.”
Mollema2015[131]
Compare the number of social media
messages with the number of online news
articles and with the epidemiological curve
(i.e., the number of reported measles cases)
and assess the usefulness of social media in
tracking factors that might affect vaccination
behaviour.
Social media type
: Mix:
Facebook, Forums, weblogs,
Twitter
Vaccine
: Measles
Country
: Netherlands
Period monitored
: 15 April -
11 November 2013
Search strategy
:
Automated (Twiqs.nl,
HowardsHome)
Sentiment analysis
: Manual
(concern, frustration,
humour/sarcasm, relief)
# results
: 1982 tweets and unspecified mix of 464 social media messages
“There was a stronger correlation between the weekly number of social media messages
and the weekly number of online news articles (P<.001 for both tweets and other social
media messages) than between the weekly number of social media messages and the
weekly number of reported measles cases (P=.003 and P=.048 for tweets and other social
media messages, respectively), especially after the summer break. All data sources showed
3 large peaks, possibly triggered by announcements about the measles outbreak by the
Dutch National Institute for Public Health and the Environment and statements made by
well-known politicians. Most messages informed the public about the measles outbreak (i.e.,
about the number of measles cases) (93/165, 56.4%) followed by messages about
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
65
Reference Study aim(s) Study details Methodology Results (taken from the article’s abstract)
preventive measures taken to control the measles spread (47/132, 35.6%). The leading
opinion expressed was frustration regarding people who do not vaccinate because of
religious reasons (42/88, 48%).”
Nawa2016[124]
Categorise questions by conducting detailed
qualitative analyses from the clinicians’
viewpoint and to investigate how public
concerns regarding influenza vaccinations
change over time, particularly in relation to
seasonal influenza epidemics.
Social media type
: Yahoo!
Answers
Vaccine
: Influenza
Country
: Japan
Period monitored
: 1 April
2004 - 7 April 2009
Search strategy
:
Automated (Yahoo!
Answers API (via Python
scripts))
Sentiment analysis
: n/a
# results
: 16+ million questions
“After filtering data, we obtained 1950 questions regarding influenza vaccinations. The three
most frequently asked questions were regarding the vaccination schedule, safety, and
effectiveness. When we analysed monthly trends in question contents, we noted the
emergence of similar questions in the same period every year. Therefore, we classified the
time periods of each year into three parts: (1) from April to the commencement of seasonal
influenza vaccinations (September), (2) from October until the epidemic period, and (3) the
epidemic period. Two interesting results were obtained: concerns regarding effectiveness
abruptly increased during the epidemic period and pregnant or breastfeeding women
increasingly asked questions regarding feasibility between October and the epidemic
period.”
Nicholson2012[119]
Develop an understanding of how
epidemiological, scientific and anecdotal
evidence interacted with, and shaped, ideas
about the MMR vaccine and provide
recommendations for strategically
responding to future online debates about
safe and effective vaccines like MMR.
Social media type
: Forums
Vaccine
: Measles
Country
: Australia
Period monitored
: Three
hours
Search strategy
: Manual
Sentiment analysis
: Manual
# results
: 466 forum posts
“From 103 distinct branches there were 1193 posts sent over a h period. We selected
the 13 longest branches containing 466 posts from 166 individuals. One third of these
individuals were explicitly critical of MMR immunisation and one third sought information.
The remainder were ambivalent but seeking no information (5%), supportive (14%), or
unstated (15%). Among five author categories, only 4% identified themselves as health
professionals. Topics included alleged adverse effects of immunisation (35%); autism
spectrum disorders treatment and causes (31%); vaccine ingredients (12%); a conspiracy
(9%); immunisation policies (8%); and measles, mumps or rubella (4%). Scientific concepts
of evidence failed to compete with lay concepts and personal anecdotes prevailed.”
Ninkov2017[90]
Identify whether webometrics methods are
effective in analysing the web presence of
vaccine information.
Social media type
: Twitter
Vaccine
: Any
Country
: Global/Not specified
Period monitored
: 120 days
from 16 May - 13 September
2015
Search strategy
:
Automated (Topsy)
Sentiment analysis
: Manual
(anti-vaccination, neutral,
pro-vaccination)
# results
: 100 tweets
“The study found far more anti- than pro-vaccine web domains. The anti and pro sides had
similar web visibility as measured by the number of links coming from general websites and
Tweets. However, the links to the pro domains were of higher quality measured by
PageRank scores. The result from the qualitative content analysis confirmed this finding.
The analysis of site ages revealed that the battle between the two sides had a long history
and is still ongoing. The web scene was polarised with either pro or anti views and little
neutral ground.”
Numnark2014[78]
Propose a monitoring system with
visualisations and analytics of significant
vaccine information from Twitter and RSS
feeds (VaccineWatch).
Social media type
: Twitter
Vaccine
: Any
Country
: Global/Not specified
Period monitored
: 1 April - 20
June 2014
Search strategy
:
Automated (VaccineWatch)
Sentiment analysis
: n/a
# results
: 172,182 social media records (tweets + RSS URLS)
“The VaccineWatch monitoring system aimed to help identify and extract vaccine-related
information from social media data. It provides various visualisations that help users capture
both spatial and temporal information between vaccines, diseases, countries/cities, and
companies, together with the top 50 tagged terms, messages related to vaccine and disease
alerts, and company announcements. The flexible management of data sources and
backend processes provides users the extensible and customizable system.
Orr2016[109]
Map and describe the role played by social
media and mainstream web-based media as
platforms for vaccination-related public
debates and discussions during the Polio
crisis in Israel.
Social media type
: Facebook
Vaccine
: Polio
Country
: Israel
Period monitored
: 28 May -
31 October 2013
Search strategy
: Manual
Sentiment analysis
: n/a
# results
: 10 Facebook posts
“The traditional media mainly echoed formal voices from the Ministry of Health. The
comments on the Facebook vaccination opposition groups could be divided into four groups:
comments with individualistic perceptions, comments that expressed concerns about the
safety of the OPV, comments that expressed distrust in the Ministry of Health, and
comments denying Polio as a disease. In the Facebook group “Parents talk about the Polio
vaccination”, an active group with various participants, 321 commentators submitted 2289
comments, with 64 % of the comments written by women. Most (92%) people involved
were parents. The comments were both personal (referring to specific situations) and
general in nature (referring to symptoms or wide implications). A few (13%) of the
commentators were physicians (n = 44), who were responsible for 909 (40%) of the items
in the sample. Half the doctors and 6 % of the non-doctors wrote over 10 items each. This
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
66
Reference Study aim(s) Study details Methodology Results (taken from the article’s abstract)
Facebook group formed a unique platform where unmediated debates and discussions
between the public and medical experts took place.”
Orr2018[108]
Characterise public engagement with the
2013 polio crisis in Israel in a social media
environment.
Social media type
: Facebook
Vaccine
: Polio
Country
: Israel
Period monitored
: 14 August
- 12 November 2013
Search strategy
: Manual
Sentiment analysis
: n/a
# results
: 40 Facebook conversation threads
“The qualitative analysis suggested that dialogue became more political than scientific
overall, yet the quantitative analysis showed that the discussants did not abandon the
scientific nature of the issue at hand.”
Pananos2017[91]
Detect rates of critical slowing down in
discussion and uptake of vaccines on Twitter
before and after disease outbreaks, and test
possibility of building a mathematical model
or analytical tools to detect populations at
heightened risk of a future episode of
widespread vaccine refusal.
Social media type
: Twitter
Vaccine
: Measles
Country
: US
Period monitored
: 2011-2016
Search strategy
:
Automated (Twitter API)
Sentiment analysis
:
Automated (anti-vaccine,
other, pro-vaccine)
# results
: 27,906 tweets
“We find critical slowing down in the data at the level of California and the United States in
the years before and after the 20142015 Disneyland, California measles outbreak. Critical
slowing down starts growing appreciably several years before the Disneyland outbreak as
vaccine uptake declines and the population approaches the tipping point. However, due to
the adaptive nature of coupled behaviourdisease systems, the population responds to the
outbreak by moving away from the tipping point, causing “critical speeding up” whereby
resilience to perturbations increases. A mathematical model of measles transmission and
vaccine sentiment predicts the same qualitative patterns in the neighbourhood of a tipping
point to greatly reduced vaccine uptake and large epidemics. These results support the
hypothesis that population vaccinating behaviour near the disease elimination threshold is a
critical phenomenon.”
Penta2014[117]
Explore HPV vaccine-related conversations
posted on discussion forums and provide in-
depth insight into people’s perspectives,
factors that restricted uptake and
particularities of communication about the
vaccine.
Social media type
: Forums
Vaccine
: HPV
Country
: Romania
Period monitored
: 2007-2012
Search strategy
: Manual
(Google Search)
Sentiment analysis
: n/a
# results
: 2,240 forum comments
“Positive discourses relying on evidence-based arguments or cancer-related experiences
battled with negative discourses that focused mostly on pseudo-scientific information and
affect-based testimonials. Both camps made use of appeals to authority in order to provide
powerful messages. Critics expressed high levels of mistrust in the health system and
perceived the vaccine as dangerous, as part of a conspiracy, as unnecessary or as a
promoter of promiscuity. By contrast, supporters considered the HPV vaccine to be helpful
and criticised the irrationality of opponents. Ambivalence and uncertainty also emerged,
along with criticism toward the suboptimal organisation of the vaccination programmes.
Findings highlight ways in which views about the vaccine are embedded in broader
perspectives about science, the national medical system, society development and economic
inequality.”
Porat2018[79]
Analyse the most popular tweets in the
context of vaccination, documenting the
source, topic, tone and sentiment, using the
2015 diphtheria episode in Spain.
Social media type
: Twitter
Vaccine
: Diphtheria
Country
: Spain
Period monitored
: 1 May - 15
July 2015
Search strategy
:
Automated (Topsy)
Sentiment analysis
: Manual
(negative, neutral,
positive)
# results
: 722,974 tweets
“A total of 722 974 tweets were collected. Prevalence of terms relating to policy and
misinformation increased at the onset of the case and after the death of the child. Popular
tweets (194) were either pro-vaccination (58%) or neutral, with none classified as anti-
vaccination. Popular topics included criticism towards anti-vaccination groups (35%) and
effectiveness of immunisation (22%). Popular tweets were informative (47%) or opinions
(53%), which mainly expressed frustration (24%) or humour/sarcasm (23%). Popular
Twitter accounts were newspaper and TV channels (15%), as well as individual journalists
and authors of popular science (13.4%).
Radzikowski2016[80]
Study Twitter narrative regarding
vaccination in the aftermath of the 2015
measles outbreak, both in terms of its cyber
and physical characteristics; contribute to
the analysis of the data and present a
quantitative interdisciplinary approach to
analyse open-source data in the context of
health narratives.
Social media type
: Twitter
Vaccine
: Measles
Country
: Global/Not specified
Period monitored
: 1 February
- 9 March 2015
Search strategy
:
Automated (Twitter API)
Sentiment analysis
: n/a
# results
: 669,136 tweets
“The data analysis captures the anatomy of the themes and relations that make up the
discussion about vaccination in Twitter. The results highlight the higher impact of stories
contributed by news organisations compared to direct tweets by health organisations in
communicating health-related information. They also capture the structure of the anti-
vaccination narrative and its terms of reference. Analysis also revealed the relationship
between community engagement in Twitter and state policies regarding child vaccination.
Residents of Vermont and Oregon, the two states with the highest rates of non-medical
exemption from school-entry vaccines nationwide, are leading the social media discussion in
terms of participation.”
Rivera2016[126]
Examine content, mood and general
dynamics of health forum discussions
Social media type
: Reddit
Vaccine
: Any
Search strategy
:
Automated (Reddit API)
# results
: 272,862 Reddit comments
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
67
Reference Study aim(s) Study details Methodology Results (taken from the article’s abstract)
concerning vaccinations, genetically modified
organisms and a gluten-free diet and
explore the ability to extract sentiment from
social media.
Country
: Global/Not specified
Period monitored
: January
2007 - September 2014
Sentiment analysis
:
Automated, LDA
(affirmative, negative)
“Manual annotation resulted in moderate interrater agreement of an average 0.48 Fleiss-
Kappa. Despite that, the disposition models for each topic were able to achieve a balanced
successful prediction rate of between 68% and 74% providing a considerably better than
chance assessment of a commentator's disposition towards each topic. We observed
changes in disposition over time and found areas of disagreement between the supporters
and opponents of each topic. Despite the limitations associated with manual annotations,
we obtained a wider view on the issues concerning the topics of interest than those offered
by previous research.”
Salathé2011[30]
Measure the spatio-temporal sentiment
towards a new vaccine.
Social media type
: Twitter
Vaccine
: H1N1
Country
: US
Period monitored
: August
2009 - January 2010
Search strategy
:
Automated (Twitter API)
Sentiment analysis
:
Automated, NaiveBayes
(negative, neutral,
positive)
# results
: 477,768 tweets
“We validated our approach by identifying a strong correlation between sentiments
expressed online and CDC-estimated vaccination rates by region. Analysis of the network of
opinionated users showed that information flows more often between users who share the
same sentiments - and less often between users who do not share the same sentiments -
than expected by chance alone. We also found that most communities are dominated by
either positive or negative sentiments towards the novel vaccine. Simulations of infectious
disease transmission show that if clusters of negative vaccine sentiments lead to clusters of
unprotected individuals, the likelihood of disease outbreaks is greatly increased.”
Sanawi2017[81]
Explore discussion on issues related to
vaccination on social media platforms,
specifically Twitter and identify the
‘influencers’ in the conversation.
Social media type
: Twitter
Vaccine
: Any
Country
: Global/Not specified
Period monitored
: 11-17 July
2017
Search strategy
:
Automated (NodeXL)
Sentiment analysis
: n/a
# results
: 10,000 tweets
“The findings show that there are six types of influencers that dictate the discourse on
vaccination on Twitter which are: celebrity doctor, media organisations, homeopathy
promoter, government and government agencies, blogger and renowned medical journal. It
also found that some of the influencers have their own circle of audience while some of the
influencers are sharing the same crowd.
Schmidt2018[110]
Assess whether users’ attitudes are polarised
on the topic of vaccination on
Facebook and how this polarisation develops
over time.
Social media type
: Facebook
Vaccine
: Any
Country
: Global/Not specified
Period monitored
: 1 January
2010 - 31 May 2017
Search strategy
:
Automated (Facebook
Graph API)
Sentiment analysis
: Manual
(anti-vaccine, pro-vaccine)
# results
: 298,018 Facebook posts
“Our findings show that the consumption of content about vaccines is dominated by the
echo chamber effect and that polarisation increased over the years. Well-segregated
communities emerge from the users’ consumption habits i.e., the majority of users consume
information in favour or against vaccines, not both.”
Seeman2010[128]
Track whether online postings about the
A(H1N1) influenza vaccine were
undermining ongoing communications
efforts by public health authorities during
the fall of 2009 and whether anti-vaccine
sentiment escalated after Health Canada’s
approval of the vaccine.
Social media type
: Mix: Digg,
Facebook, Twitter, YouTube
Vaccine
: H1N1
Country
: Canada
Period monitored
: 27 October
27 2009 - 6 April 2010
Search strategy
: Manual
(Google Search)
Sentiment analysis
: n/a
# results
: 17,392 search results.
“Websites and blog posts with anti-vaccine sentiment remained popular during the course of
the pandemic.”
Shapiro2017[82]
Conduct an international comparison of the
proportions of tweets about HPV vaccines
that express concerns, the types of concerns
expressed and the social connections among
users posting about HPV vaccines in
Australia, Canada and the UK.
Social media type
: Twitter
Vaccine
: HPV
Country
: Mix: Australia,
Canada, UK
Period monitored
: January
2014 - April 2016
Search strategy
:
Automated (Twitter API)
Sentiment analysis
: n/a
# results
: 43,852 tweets
“Tweets expressing concerns about HPV vaccines made up 14.9% of tweets in Canada,
19.4% in Australia and 22.6% in the UK. The types of concerns expressed were similar
across the three countries, with concerns related to ‘perceived barriers’ being the most
common. Users expressing concerns about HPV vaccines in each of the three countries had
a relatively high proportion of international followers also expressing concerns.”
Skea2006[116]
Examine discussions about MMR among
parents who participated in an online chat
forum.
Social media type
: Forums
(www.mumsnet.com)
Vaccine
: Measles
Country
: UK
Period monitored
: 31 August
2000 - 5 March 2003
Search strategy
: Manual
Sentiment analysis
: Manual
(negative, neutral,
positive)
# results
: 617 messages
“This analysis focuses on discussions about ‘avoiding harm to others,’ which were important
considerations for many of the participating parents. In the context of concerns about MMR
safety, participants expressed a desire to both (a) protect their own child and (b) help
protect others by contributing to herd immunity. Parents made a distinction between
healthy and vulnerable children, which had important implications for their views about who
should bear the burden of vaccination. Some parents were quite critical of those who did
not vaccinate healthy children, and urged them to do so on grounds of social responsibility.
Skeppstedt2018[122]
Code large text collections of Internet
discussions on vaccination and extract
Social media type
: Forums
(www.mumsnet.com)
Search strategy
: Manual
# results
: 943 forum discussion posts
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
68
Reference Study aim(s) Study details Methodology Results (taken from the article’s abstract)
important information that could be
automatically extracted.
Vaccine
: Any
Country
: UK
Period monitored
: 2011-2017
Sentiment analysis
: n/a
“In this study, we applied automatic topic modelling on a collection of 943 discussion posts
in which vaccine was debated, and six distinct discussion topics were detected by the
algorithm. When manually coding the posts ranked as most typical for these six topics, a set
of semantically coherent arguments were identified for each extracted topic. This indicates
that topic modelling is a useful method for automatically identifying vaccine-related
discussion topics and for identifying debate posts where these topics are discussed. This
functionality could facilitate manual coding of salient arguments, and thereby form an
important component in a system for computer-assisted coding of vaccine-related
discussions.”
Smith2017a[83]
Detail a preliminary system of real-time
geographical monitoring and analysis in the
context of the vaccine hesitancy discussion
across the US.
Social media type
: Twitter
Vaccine
: Any
Country
: US
Period monitored
: 8
December 2014 - 2 March
2015
Search strategy
:
Automated (Twitter API)
Sentiment analysis
:
Automated, LDA (negative,
neutral, positive)
# results
: n/a
“We combine various methods in machine learning to geolocate, categorise, and classify
vaccination discussions on Twitter. As a proof of concept, we show analyses with a
prominent anti-vaccine discussion that validate the system with results from traditional
surveys, yet also provide valuable spatial statistical power on top of such surveys on maps
of the United States. We detail limitations and future work, yet still conclude that the system
and the answers it enables are important because they will allow for more targeted and
effective communication and reaction to the discussion as a first step towards monitoring
people’s views.”
Smith2017b[111]
Examine the characteristics of and the
discourses present around childhood
vaccination within six popular anti-
vaccination Facebook pages.
Social media type
: Facebook
Vaccine
: Any (childhood
vaccines)
Country
: Mix: Australia, US
Period monitored
:14 April
2013 - 14 April 2016
Search strategy
:
Automated (Facebook API,
Social-MediaLab)
Sentiment analysis
: n/a
# results
: 14,736 Facebook posts
“We find that present-day discourses centre around moral outrage and structural oppression
by institutional government and the media, suggesting a strong logic of ‘conspiracy-style’
beliefs and thinking. Furthermore, anti-vaccination pages on Facebook reflect a highly
‘feminised’ movement ‒ the vast majority of participants are women. Although anti-
vaccination networks on Facebook are large and global in scope, the comment activity sub-
networks appear to be ‘small world’.”
Sundstrom2018[130]
Investigate online HPV vaccination
communication to provide insight to increase
vaccine uptake through effective messaging.
Social media type
: Mix:
Facebook, Twitter
Vaccine
: HPV
Country
: US
Period monitored
: 1 June
2014 - 31 May 2015
Search strategy
: Manual
Sentiment analysis
: n/a
# results
: 211 tweets and 144 Facebook posts
“Current messaging in South Carolina emphasised the relative advantage of HPV vaccination
as cancer prevention strategy. Two primary misconceptions about the HPV vaccination were
identified: concerns about safety and that the vaccine could increase sexual activity among
adolescents. The content analysis revealed that health care provider support is needed to
normalise HPV vaccination as part of the routine immunisation series. Observing messages
from peers served as a vicarious trial experience of the vaccine for adolescents and young
adults and showed gaps in vaccine uptake among males and lack of series completion
among males and females.
Suragh2018[112]
Assess the possibility of detecting clusters of
anxiety-related adverse events following
immunisation, not otherwise reported in
traditional peer-reviewed systems.
Social media type
: Facebook
Vaccine
: Any
Country
: Global/Not specified
Period monitored
: 2007-2016
Search strategy
: Manual
Sentiment analysis
: Manual
(negative, neutral,
positive)
# results
: 15 Facebook posts
“We found 39 reports referring to 18 unique cluster events. Some reports were only found
based on the geographic location from where the search was performed. The most common
vaccine implicated in reports was human papillomavirus (HPV) vaccine (48.7%). The
majority of reports (97.4%) involved children and vaccination programs in school settings or
as part of national vaccination campaigns. Five vaccination programs were reportedly halted
because of these cluster events. In this study, we identified 18 cluster events that were not
published in traditional scientific peer-reviewed literature.”
Surian2016[84]
Evaluate the use of community structure
and topic modelling methods as a process
for characterising
the clustering of opinions about HPV
vaccines on Twitter.
Social media type
: Twitter
Vaccine
: HPV
Country
: Global/Not specified
Period monitored
: October
2013 - October 2015
Search strategy
:
Automated (Twitter API)
Sentiment analysis
: n/a
# results
: 285,417 tweets
“We analysed 285,417 Twitter posts (tweets) about HPV vaccines from 101,519 users
connected by 4,387,524 social connections. Examining the alignment between the
community structure and the topics of tweets, the results indicated that the Louvain
community detection algorithm together with DMM produced consistently higher alignment
values and that alignments were generally higher when the number of topics was lower.
After applying the Louvain method and DMM with 30 topics and grouping semantically
similar topics in a hierarchy, we characterised 163,148 (57.16%) tweets as evidence and
advocacy, and 6244 (2.19%) tweets describing personal experiences. Among the 4548
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
69
Reference Study aim(s) Study details Methodology Results (taken from the article’s abstract)
users who posted experiential tweets, 3449 users (75.84%) were found in communities
where the majority of tweets were about evidence and advocacy.
Tang2018[85]
Conduct a semantic network analysis of
Twitter content about measles during a
measles outbreak in California.
Social media type
: Twitter
Vaccine
: Measles
Country
: US
Period monitored
: 1
December 2014 - 30 April
2015
Search strategy
:
Automated
(DiscoverText.com)
Sentiment analysis
: n/a
# results
: 1,133,656 tweets
“Four frames were identified based on word frequencies and co-occurrence: news update,
public health, vaccination, and political. The prominence of each individual frame changed
over the course of the pre-crisis, initial, maintenance, and resolution stages of the
outbreak.”
Tangherlini2016[121]
Develop an automated and scalable
machine-learning method for story
aggregation on social media sites dedicated
to discussions of parenting.
Social media type
: Forums
(www.mothering.com and
one unnamed)
Vaccine
: Any (childhood
vaccines)
Country
: Mix: Canada, US
Period monitored
: 2004-2012
Search strategy
: Manual
Sentiment analysis
: n/a
# results
: 1.99 million forum posts
“We discovered that discussions of exemption from vaccination requirements are highly
represented. We found a strong narrative framework related to exemption seeking and a
culture of distrust of government and medical institutions. Various posts reinforced part of
the narrative framework graph in which parents, medical professionals, and religious
institutions emerged as key nodes, and exemption seeking emerged as an important edge.
In the aggregate story, parents used religion or belief to acquire exemptions to protect their
children from vaccines that are required by schools or government institutions, but
(allegedly) cause adverse reactions such as autism, pain, compromised immunity, and even
death. Although parents joined and left the discussion forums over time, discussions and
stories about exemptions were persistent and robust to these membership changes.”
Teoh2018[86]
Quantify personal stories about cervical
cancer on Twitter during cervical cancer
awareness month and determine the
proportion of Twitter messages discussing
prevention (vaccination) and evaluate
positive or negative sentiment of these
messages.
Social media type
: Twitter
Vaccine
: HPV
Country
: Global/Not specified
Period monitored
: January
2016
Search strategy
: Manual
Sentiment analysis
: Manual
(negative, positive)
# results
: 348 top tweets
“During January 2016, 348 top tweets about cervical cancer were identified. Professional
health organisations produced 20.7% of tweets and individuals identifying themselves as
healthcare professionals contributed an additional 4%. In addition to the tweet, 45.1%
attached a photo or video; 54.6% included links to a larger article. Only 11.2% of tweets
included personal stories from cervical cancer patients. Among the top tweets, 70.3% were
focused on prevention through screening and/or HPV vaccination, with 97.4%
recommending such practices. A substantial proportion of the Twitter traffic (24.7%)
referenced the #SmearForSmear campaign by the patient-advocate organisation Jo’s
Cervical Cancer Trust, based in the United Kingdom.”
Tomeny2017[87]
Examine variations in anti-vaccine beliefs
that link vaccines to autism by geographic
distribution and demographics on Twitter.
Social media type
: Twitter
Vaccine
: Any
Country
: US
Period monitored
: 1 January
2009 - 21 August 2015
Search strategy
:
Automated (Social Studio's
Radian6 API)
Sentiment analysis
:
Automated, Lightside (anti-
vaccine, neutral, pro-
vaccine)
# results
: 549,972 tweets
“Fifty percent of our sample of 549,972 tweets collected between 2009 and 2015 contained
anti-vaccine beliefs. Anti-vaccine tweet volume increased after vaccine-related news
coverage. California, Connecticut, Massachusetts, New York, and Pennsylvania had anti-
vaccination tweet volume that deviated from the national average. Demographic
characteristics explained 67% of variance in geographic clustering of anti-vaccine tweets,
which were associated with a larger population and higher concentrations of women who
recently gave birth, households with high income levels, men aged 40 to 44, and men with
minimal college education.
Tuells2015[102]
Identify the characteristics of YouTube
videos in Spanish about HPV vaccination.
Social media type
: YouTube
Vaccine
: HPV
Country
: Global/Not specified
Period monitored
: 26 July
2013
Search strategy
: Manual
Sentiment analysis
: Manual
(negative, positive)
# results
: 170 videos
“A total of 170 videos were classified like: local news (n=39; 37 favourable, 2 unfavourable;
2:06:29; 42972 visits), national news (n=32; 30/2; 1:49:27; 50138 visits), created by
YouTube subscribers (n=21; 21/1; 1:44:39; 10991 visits), advertisements (n=21; 19/2;
0:27:05; 28435 visits), conferences (n=17; 15/2; 3:25:39; 27206 visits), documentaries
(n=16; 12/4; 2:11:31; 30629 visits). From all of the 20 most viewed YouTube videos
predominated those which were favourable to the vaccination (n=12; 0:43:43; 161789
visits) against the unfavourable (n=8; 2:44:14; 86583 visits).”
Tustin2018[113]
Qualitatively analyse and quantify the
content of users’ posts to describe the main
vaccination sentiments and themes of an
online immunisation debate of Facebook
users who commented on posted
advertisements, in order to better
Social media type
: Facebook
Vaccine
: Any
Country
: Canada
Period monitored
: 12
December 2013 - 11 January
2014
Search strategy
: Manual
Sentiment analysis
: Manual
(ambiguous, hesitant,
negative, positive)
# results
: 117 Facebook comments
“Of 117 comments, 85 were posted by unique commentators, with most being female
(65/85, 77%). The largest proportion of the immunisation comments were positive (51/117,
43.6%), followed by negative (41/117, 35.0%), ambiguous (20/117, 17.1%), and hesitant
(5/117, 4.3%). Inaccurate knowledge (27/130, 20.8%) and misperceptions of risk (23/130,
17.7%) were most prevalent in the 130 non-positive comments. Other claims included
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
70
Reference Study aim(s) Study details Methodology Results (taken from the article’s abstract)
understand the vaccination debate and to
identify underlying themes.
distrust of pharmaceutical companies or government agencies (18/130, 13.8%), distrust of
the health care system or providers (15/130, 11.5%), past negative experiences with
vaccination or beliefs (10/130, 7.7%), and attitudes about health and prevention (10/130,
7.7%). Almost 40% (29/74, 39%) of the positive comments communicated the risks of not
vaccinating, followed by judgments on the knowledge level of non-vaccinators (13/74,
18%). A total of 10 positive comments (10/74, 14%) specifically refuted the link between
autism and vaccination.”
Vayreda2011[114]
Identify how the online talk represent the
debate over the need for a campaign of
mass vaccination in Spain.
Social media type
: Forums
Vaccine
: H1N1
Country
: Spain
Period monitored
: n/a
Search strategy
: Manual
Sentiment analysis
: Manual
(sceptical, pro-vaccination)
# results
: 67 forum messages
“We identify the discursive practices that contributors use to valorise certain elements in the
debate (what they cast as science, rationality, and ‘proper’ scepticism) over others
(especially commercial interests, ‘charlatanism, and ‘profiteering’). A forum participant can
be disqualified on the basis of their alleged partiality and interest, if they can be accused of
having a commercial stake in the matter. But no such opprobrium results if they have a
‘scientific’ interest.
Venkatraman2015[103]
Identify how viewpoints vary with the
degree of freedom of speech offered.
Social media type
: YouTube
Vaccine
: Any
Country
: Global/Not specified
Period monitored
: 20-27
November 2013
Search strategy
: Manual
Sentiment analysis
: Manual
(anti-vaccine, pro-vaccine)
# results
: 175 videos
“Support for a link between vaccines and autism is most prominent on YouTube, followed by
Google search results. It is far lower on Wikipedia and PubMed. Anti-vaccine activists use
scientific arguments, certified physicians and official-sounding titles to gain credibility, while
also leaning on celebrity endorsement and personalised stories.”
Yuan2018[88]
Investigate the communicative patterns of
anti-vaccine and pro-vaccine users in Twitter
by studying the retweet network related to
MMR vaccine published by users after the
2015 California Disneyland measles
outbreak.
Social media type
: Twitter
Vaccine
: Measles
Country
: US
Period monitored
: 1 February
- 9 March 2015
Search strategy
:
Automated (Geosocial
gauge)
Sentiment analysis
:
Automated (anti-
vaccination, neutral, pro-
vaccination)
# results
: 660,892 tweets
“Using supervised learning, we classified the users into anti-vaccination, neutral to
vaccination, and pro-vaccination groups. Using a combination of opinion groups and retweet
network structural community detection, we discovered that pro- and anti-vaccine users
retweet predominantly from their own opinion group, while users with neutral opinions are
distributed across communities. For most cross-group communication, it was found that
more pro-vaccination users were retweeting anti-vaccination users than vice-versa.”
Zhou2015[92]
Determine if information about social
connections could be used to improve the
performance of classifiers intended for
ongoing use in public health surveillance.
Social media type
: Twitter
Vaccine
: HPV
Country
: Global/Not specified
Period monitored
: 1 October
2013 - 31 March 2014
Search strategy
:
Automated (Twitter API)
Sentiment analysis
:
Automated (anti-vaccine)
# results
: 42,533 tweets
“From 42,533 tweets posted between October 2013 and March 2014, 2,098 were sampled
at random and two investigators independently identified anti-vaccine opinions. Machine
learning methods were used to train classifiers using the first three months of data,
including content (8,261 text fragments) and social connections (10,758 relationships).
Connection-based classifiers performed similarly to content-based classifiers on the first
three months of training data, and performed more consistently than content-based
classifiers on test data from the subsequent three months. The most accurate classifier
achieved an accuracy of 88.6% on the test data set, and used only social connection
features. Information about how people are connected, rather than what they write, may be
useful for improving public health surveillance methods on Twitter”
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions around vaccine hesitancy
71
Annex 4.
List of commercial software to monitor social media included
in the studies
Commercial software included in
studies
Details
Topsy[70,73,79,90]
Topsy was an automated tool used for social media analytics and deep Twitter searches, which
was purchased by Apple in 2013, and ceased operating in 2015[190]. A deep Twitter search can
be classified as an advanced search via the Twitter API so that one can get more detailed
information beyond a simple keyword search using the Twitter web browser or app search
function. This advanced ”deep” search may be constructed using specified Boolean search formats
that have a mixture of inclusion and exclusion criteria. For example, the following search, would
get all tweets that mention any variation of the term “vaccine”, “vaccination” or “vaccinated”, but
not
tweets related to non-human (e.g. animal) vaccines: ("vaccin*") NOT ("a vet" or veterinary OR
dog* OR cat* OR horse* OR mouse* OR pig* OR cow*)”. Because the Twitter API only gives a
random sample, and limited percentage of tweets depending on the type of access
(free/commercial) making sure one’s search term is specific when utilising the API is key for
getting access to a better quality data sample within the limitations of the API[191]
Crimson Hexagon[52,129,132]
Crimson Hexagon is an AI (Artificial Intelligence)-powered consumer insights company that has a
powerful social media analytics tool. It gives access to the company's online data library, which
consists of over 1 trillion posts, and includes documents from social networks such as Twitter,
Instagram and Facebook as well as blogs, forums, and news sites. The company's ForSight
platform is a Twitter certified product [192]
Gnip[56,76,77]
Gnip, Inc. was a social media API aggregation company, which provided full historical and current
data from many different social media platforms via a single API. Twitter purchased Gnip in April
2014, and is now part of Twitter’s enterprise API platform, which delivers real-time and historical
social data for research and business purposes[193]
NodeXL[72,81]
An open-source network analysis and visualisation software package for Microsoft Excel, that
includes access to social media network data importers, advanced network metrics, and
automation[194]
Discovertext.com[62,85]
A web-based, text analytics toolkit that supports collaborative search, filtering, duplicate detection,
human coding, and machine-learning of social media platforms[195].
HealthMap[53]
A website that delivers real-time intelligence on a broad range of emerging infectious diseases via
disparate data sources, including online news aggregators, eyewitness reports, and expert-curated
discussions and validated official reports, to achieve a unified and comprehensive view of the
current global state of infectious diseases and their effect on human and animal health. Through
an automated process, the system monitors, organises, integrates, filters, visualises and
disseminates information from international online news and health information sources about
emerging diseases in nine languages, facilitating early detection of global public health threats.
Results are available on a platform called the Vaccine Sentimeter. However, the study found that a
limitation of HealthMap and the vaccine sentiment analysed with it is that data sources are not
exhaustive; they only use a small selection of news and public health data and are limited to
publicly available data.
ChatterGrabber[68]
An open source, natural language processing based toolset for public health social media
surveillance[196]
Social Studio's Radian6 API[87]
A web service for retrieving, analysing, and modifying social media data[197];
Twiqs.nl and HowardsHome[131]
Twiqs.nl is a free analytic Dutch tool for tweets[198], and HowardsHome is a Dutch online
monitoring service, which specialises in collecting news and content for content marketing,
content curation and knowledge sharing[199]
Twitter Zombie and Microsoft
Research[75]
Twitter Zombie is a monitoring tool for capturing, socially transforming and analysing
Twitter[200]. The Microsoft Research Open Data project has a collection of Social Media
Conversation Corpus', that contain collections of tweets, which are open to researchers for
analysis[201]
Clipit[134]
A Dutch online media monitoring programme that can be used to search online and social media,
print media, radio & TV and international online media for preselected terms[202]
Geosocial gauge[88]
a social analytics tool that brings together social media and geographical analysis to monitor and
explore people’s views, reactions, and interactions through space and time[203]
VaccineWatch[78]
An online monitoring system with visualisations and analytics of significant vaccine information
from Twitter[78]
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
72
Annex 5
List of identified keywords used in social media monitoring
articles
Facebook
Vaccination in general:
Vaccine[104]
Vaccine, vaccines or vaccination[110]
Vaccine concern, vaccine choice, vaccines and autism, and anti-vaccination[111]
Vaccine specific:
Cervarix, Gardasil, HPV vaccine, and human papillomavirus vaccine[45]
Polio[109]
Vaccine, vaccines, vaccination, vaccinations, to be vaccinated, to vaccinate*[133]
Issue specific:
Mass hysteria after vaccine, mystery illness after vaccine, fainting in school children after vaccine, mass fainting after
vaccine[112]
Forums
Vaccination in general:
Vaccination and forum (
to identify forums
) and Vaccin* (
to identify threads
)[118]
Vaccine specific:
Discussion forum, HPV vaccine and cervical cancer vaccine[117]
MMR[116]
Rotavirus vaccine and autumn diarrhoea vaccine[115]
STMinfluenssainfo[120]
Pinterest
Vaccination in general:
Vaccination, vaccine, vaccines and vaccinate[125]
Reddit
Vaccination in general:
Vaccines, vaccination, vaxxer[126]
Twitter
Vaccination in general:
#vaccination[81]
#vaccine (
study A
) and #vaccineswork (
study B)
[56]
“The vaccine stream”[57]
Vaccine, vaccination, immunization[72]
Vaccine, vaccines, mmr, tdap, flushot, hpv, polio, rotavirus, chickenpox, smallpox, hepatitis, hepa, hepb, dtap,
meningitis, shingles, vaccinate, vaccinated, vaccine, vaccines, vacine, vacines, tetanus, diptheria, pertussis, whooping
cough, dtp, dtwp, chickenpox, measles, mumps, rubella, varicella, diphtheria, haemophilus, papillomavirus,
meningococcal, pneumococcal, rabies, tuberculosis, typhoid, yellow fever, immunizations, immunization, imunization,
immune, imune, cholera, globulin, encephalitis, lyme, zika[59]
Vaccines, vaccinations, trivalent, vaccination cocktail, mercury, vaxxed, big pharma (
Vaccination keywords examples
);
autism, adverse event (
claimed side effects keywords examples
); meningitis, measles, rubella, mumps, varicella
(
vaccine-preventable diseases keywords examples
)*[89]
Vaccines conspiracy; vaccination coverage; vaccine(s); big pharma; vaccine risk(s); vaxxed; trivalent; hexavalent;
vaccinate; quadrivalent; vaccination(s); vaccination freedom; vaccination objection; vaccination age; vaccination
cocktail; vaccine contraindications (
Vaccination keywords
); flaccid paralysis; autism; autoimmune diseases; adverse
event(s) (
claimed side effects keywords
), meningitis, measles; rubella; mumps; whooping cough; polio; varicella; MMR
(
vaccine-preventable diseases keywords
), #novaccino (hashtag for “no vaccine”); #iovaccino (hashtag for “I
vaccinate”); #libertadiscelta (hashtag for “freedom of choice”)*[60]
Outbreak, vaccination, Influenza, H7N9, H5N1, Japanese encephalitis (
examples
)[78]
Vaccine, Vaccinat, Vacine, Vacinate, MMR, Antivac (
conditions keywords
); Autism, Autistic, Conspiracy, Gave my, Gave
me, Oprah, Aspergers, Poison, Jenny mccarthy, Kristin cavallari, Conspiracy, Mercury, Aluminum, Truther, Bravo, Anti,
Manufacturers, Have known, Vaccine choice, Your child, Your right, Cancer, Fertility, Constitution, Risks, Dangerous
(
qualifier keywords
)[68]
Vaccine, vaccinated, immunization, mmr vaccine, mmrvaccine, #b1less, #hearus, heavy metals, leaky gut, mercury,
ethylmercury, methylmercury, thimerosal, preservative, dpt, diphtheria-pertussistetanus, pharmaceutical companies,
big pharma, autism, autistic, Asperger, vacinne, vacine, antivax, anti vax, aspie, asberger, assberger, asd,
#cdcwhistleblower, #cdc whistleblower, #sb277[87]
Vaccine, vaccines, #vaccine or #vaccines[55]
Vaccine specific:
#HPV and #Gardasil[70]
Cervical cancer, #gyncsm[86]
Diphtheria, Olot (the town where the case occurred), anti-vaccination and vaccination/vaccine*[79]
(Flu or influenza) and (shot(s) or vaccine(s) or vaccination(s)[67]
Flu shot and flu vaccine, influenza vaccine, flu vaccine, influenza vaccination, flu vaccination[71]
Gardasil, Cervarix, hpv AND vaccin* and cervical AND vaccin*[66,82]
((H1N1 OR “swine flu” OR swineflu OR pigflu OR “pig flu” OR “pandemic” OR influenza OR flu) AND (vaccin OR
antiviral OR jab OR vacin OR vaccines OR injection OR shot OR Tamiflu)) OR (Tamivir OR Relenza OR Pandemrix OR
Celvapan)[76]
“HPV AND vaccination” (
relevant keywords example
); “HPV OR vaccination” (
semi-relevant keywords example
); words
related to other types of injections, e.g. blood tests, travel vaccinations, and other meanings of the search keywords,
e.g. the Dutch translation of stinging eyes (“prikkende ogen”) contains the word injection(“prik”) (non
relevant
keywords example
)[69]
HPV and vaccine, HPV and vaccination, gardasil, cervical and vaccination, cervical and vaccine, cervarix[84]
HPV, HPV vaccine, HPV shot, Gardasil, and Cervarix (as well as the 5 corresponding hashtags)[74,75]
HPV, human papillomavirus, Gardasil, and Cervarix[63,64]
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
73
HPV, papilloma, pappiloma, papiloma, pappilomavirus, gardasil, gardisil, guardasil, guardisil, cervarix, cervical shot,
cervical shots, cervical vaccine, cervical vaccines, cervical vax, cervical vaxine, cervical vaxines, cervical vaxx, cervical
vaxxine, cervical vaxxines, cervical vaccination, and cervical vaccinations[58]
HPV vaccination[73]
HPV, vaccine, Gardasil, Cervarix, vaccination, cervical, cancer[92]
Human papillomavirus, HPV, vaccine, vaccination, Gardasil, and Cervarix[65]
Measles[62,85]
Pentavalent OR pentavac OR quinvaxem[54]
Vaccination+autism, vaccine+ autism, mmr+vaccination, measles+autism and mmr+vaccine[77]
Vaccination OR vaccine OR vaccinated OR vaccinate OR vaccinating OR immunized OR immunize OR immunization OR
immunizing[30]
Vaccination, vaccine, vaccines, vax, vaxine, and vaxx (
examples
)[80]
("Vaccinations" OR "vaccination" OR "vaccines" OR "vaccine" OR "measles-mumps-rubella" OR "MMR" OR "mmr" OR
"#MMR") AND ("autism" OR "autistic disorder") AND NOT "RT:"[52]
((vaccin* OR immuni*) AND (ingredient* OR risk* OR lies OR disease* OR exemption* OR safe* OR unsafe OR killing*
OR conspiracy OR scandal* OR whistleblower* OR pharmaceutical OR CDC OR documentary OR truth* OR theory OR
health OR infant* OR baby OR babies OR newborn* OR school* OR aluminum OR death* OR dead OR children OR
kid* OR child* OR poison* OR toxic OR mercury OR injur* OR harm* OR brain OR paraly* OR scare* OR fear* OR
autism OR IBS OR autistic or "irritable bowel syndrome")) OR measles OR mmr OR "andrew wakefield"[91]
Vaccine autism, vaccines autism, vaccine measles autism, vaccine measles mumps rubella autism, MMR autism*[133]
Vaccine, vaccines, shot, mmr, tdap, flushot, hpv, polio, rotavirus, chickenpox, smallpox, hepatitis, hep a, hep b, dtap,
meningitis, shingles, vaccinate, vaccinated, vaccine, vaccines, vacine, vacines, tetanus, diptheria, pertussis, whooping
cough, dtp, dtwp, chickenpox, measles, mumps, rubella, varicella, diphtheria, haemophilus, papillomavirus,
meningococcal, pneumococcal, rabies, tuberculosis, typhoid, yellowfever, immunizations, immunization, imunization,
immune, imune, cholera, globulin, encephalitis, lyme[83]
Weibo
Vaccine specific:
Hepatitis B vaccine[127]
Yahoo! Answers
Vaccine specific:
Influenza vaccine[124]
1. What is the papillomavirus? 2. Who can get infected and how is HPV transmitted? 3. What are the health problems
caused by HPV infection? 4. What vaccines against HPV are available to the community? 5. How do HPV vaccines
work? 6. Who should be vaccinated against HPV? 7. What is the effectiveness and safety of HPV vaccines?[123]
YouTube
Vaccination in general:
Autism and vaccine, autism and vaccines, autism and vaccination, autism and vaccinations*[98]
Vaccination and immunization[101]
Vaccinations[32]
“Vaccine safety” and “vaccines and children”[95]
Vaccine, vaccines, anti-vaccine and non-vaccination[97]
Vaccines autism[103]
Vaccine specific:
Gardasil, cervical cancer vaccination, HPV vaccination[93]
HPV and “human papillomavirus[94]
“HPV vaccine”, “cervical cancer vaccine”, “should I get the HPV vaccine”, “what can go wrong with the HPV vaccine”,
“HPV vaccine side effects”, and Gardasil[99]
HPV vaccine, HPV vaccination, HPV immunization, human papilloma virus vaccine, human papilloma virus
immunization, Gardasil, and Cervarix[96]
“Human papilloma virus vaccine”, “HPV vaccine”, “Gardasil vaccine”, “Cervarix vaccine[102]
Meningitis B vaccine, Bexsero®, and Bexsero® vaccine*[100]
Mix
Vaccination in general:
Vaccine OR vaccines OR MMR (
search for media attention to vaccines), (
vaccine OR vaccines OR MMR) and autism
(
search for media attention to vaccine-autism link)[132]
Vaccine specific:
(autism OR mercury OR thimerosal OR wakefield OR mccarthy OR immigrant OR obama OR #vax) AND (vaccine OR
measles OR “MMR vaccine” OR sb40 OR polio OR chickenpox OR hepatitis OR “mmr shot”) AND -$ AND -http AND -
RT[129]
HPV, Human Papillomavirus, HPV vaccine, HPV vaccination, Gardasil, Gardasil9, and Cervarix[130]
Influenza, vaccination, vaccine and epidemic[134]
Measles*[131]
Systematic scoping review on social media monitoring methods and interventions around vaccine hesitancy TECHNICAL REPORT
74
Annex 6.
Social media as an intervention tool in relation to vaccination
Reference
Study aim(s)
Study details
Description of intervention
Outcome and Results about social media
Information sharing
Finnegan
2018[139]
Assess what works in online
communication about vaccines
and offer proposals for
improving the impact of online
vaccine advocacy.
Social media type:
Instagram,
Facebook, Twitter, YouTube
Vaccine:
Any vaccine
Country:
Worldwide
Target population
: online users
Online platform (Vaccines Today) for discussing vaccines and
vaccination to improve vaccine uptake by providing factual
information about vaccination. Project features a website and
several social media channels targeting the general public,
launched in March 2011. In social media interactions, replies are
made with information that would be of value to observers who
are making decisions about vaccination.
The article found that two categories of content were the most popular:
storytelling approaches and answers to questions posed by readers.
The most popular content of the website was published on Facebook:
"How measles can change a life" was written by a parent whose son
developed subacute sclerosing panencephalitis several years after
measles infection in his first year. The article which is relatively long
compared with other content on the site was read for more than
seven minutes, which is long enough to digest the article in full (07:16,
233,996 views). The authors considered that articles widely read were
more successful.
The most-viewed video on YouTube was an animation showing how
herd immunity works (53,000 views).
Haase 2015[140]
Assess the potential moderating
effect of statistic and narrative
source
credibility on the biasing
effect of narrative information
regarding the perception of
vaccination risks.
Social media type:
Online forum
Vaccine:
Any vaccine
Country:
Germany
Target population
: online users
Researchers aimed to understand if the credibility of the source of
a forum post influenced the readers’ perception of vaccination
risk. They did this by presenting identical narratives, but with
different introductory texts (one from a neutral online health
forum and the other from a known anti-vaccination website).
Researchers found that narratives discussing vaccine adverse events
decreased intentions to get vaccinated and increased perceptions of
vaccination risk. This bias occurred irrespective of whether the post was
read on the neutral online health forum or the anti-vaccination website.
La Torre
2014[141]
Pilot a project with the aim of
informing healthcare workers
and the general population
about vaccination through
Facebook and present results of
one year of activity.
Social media type:
Facebook
Vaccine:
Any vaccine
Country:
Italy
Target population
: online users
Facebook messages developed to share information about
vaccination. Information materials chosen by health professionals
and scientific communication experts published three times a
week. Short and regular messages with breaks of one day
approximately between publications. News with images were
chosen more often.
Likes:
Events were the most popular type of news, followed by press
releases, and scientific publications. Institutional Videos and documents
are forms of communication less considered or appreciated by users.
The day of the week in which users were most likely to be attracted by
the contents of the links was Friday.
Shares
: Press releases were the communication form most shared by
Facebook users, followed by scientific publications and institutional
documents. No sharing of video links. Users shared more links on
Fridays.
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
75
Reference
Study aim(s)
Study details
Description of intervention
Outcome and Results about social media
Lee 2017[142]
Investigate whether using
different message framing and
media influences th
e public's
perceived severity, benefits,
barriers and willingness to get
vaccinated.
Social media type:
Facebook
Vaccine:
HPV
Country:
United States
Target population
: 142 college
students
Participants exposed to four scenarios: 1- a gain-framed message
on a fake look-alike Facebook page, 2- a loss-framed message on
a fake look-alike Facebook page; or 3- a gain-framed message on
a fake look-alike New York Times website or a 4- loss-framed
message on a fake look-alike New York Times website.
Gain-framed message: health benefits from getting the HPV
vaccine (e.g. prevention of cervical cancer or genital warts)
Loss-framed message: negative consequences of not getting the
HPV vaccine (e.g. getting cervical cancer or genital warts).
Perceived severity of HPV
: Participants who viewed the Facebook page
perceived a lower severity than those who saw the online newspaper
(p<0.05)
Perceived benefits of getting vaccinated
: No statistically significant
medium effect between newspaper and Facebook (p=.94)
Perceived barriers to getting vaccinated
: Participants who viewed the
Facebook page perceived lower barriers than those who saw the
newspaper (p<.05)
Willingness to get vaccinated
: Participants who viewed the loss-framed
message on Facebook mostly exhibited a higher level of behavioural
intention to get vaccinated than those in the gain-framing condition
(p<.05).
Mohanty
2018[143]
Assess the campaign reach,
engagement, and HPV vaccine
uptake among Philadelphia
adolescents through the 3forME
campaign.
Social media type:
Facebook
Vaccine:
HPV
Country:
United States
Target population
: 155,110
adolescents (13-18) reached
Advertisements from 3forME displayed on the right side of the
Facebook login page with varied themes, images and text. Link
provided to the 3forME Facebook page and website. Messages
addressed perceived susceptibility of HPV disease, severity of HPV
infection, and benefits of getting vaccinated. The pages were
designed as cues to actions to motivate adolescents to seek the
HPV vaccine.
Uptake of HPV vaccination
: On average, each advertising campaign
reached 155,110 adolescents and engaged 2106 adolescents. The
advertising campaigns that focused on HPV disease risk and local
resources were the most successful in engaging adolescents. Overall,
3400 adolescents became fans of the campaign and 176 doses of HPV
vaccine were administered to 152 adolescents, out of which 63 received
the three doses.
Ortiz 2018[144]
Describe the formative research,
execution, and evaluation of a
social media health intervention
to improve adolescents’
knowledge ab
out and
vaccination against HPV.
Social media type:
Facebook
Vaccine:
HPV
Country:
United States
Target population
: 108
adolescents (13-18)
A Facebook page, “About your Health”, with information about
HPV vaccination and notifications received each time a new
message was posted on the page. Maintained by local healthcare
providers. 24 health facts (11 about HPV) were posted throughout
a three-month period (with images and links to credible
websites). Topics included virus susceptibility, virus severity,
vaccine benefits, vaccine barriers and self-efficacy.
Knowledge
: Participants who reported receiving notifications for each
new Facebook post were significantly more likely to have an increase in
their HPV and vaccine knowledge but not in their vaccination rates.
Peter 2014[145]
Investigate the potential of
o
nline discussions on social
network sites to convey health
messages and to affect people’s
judgements regarding health
issues.
Social media type:
Facebook
Vaccine:
Influenza
Country:
Germany
Target population
: 577 adults
Facebook page that featured a fictitious person who posted an
article from an online magazine about influenza vaccination. The
post was followed by five comments about this issue. In one
version (pro vaccination version), four of the five comments
stated a positive attitude toward flu vaccination, and one
comment reported a negative attitude. In the other version
(contra vaccination version), this relationship was reversed (four
negative and one positive comment).
Furthermore, in one version, one of the comments that
represented the opinion of the majority (e.g., a positive comment
in the pro vaccination version) was also liked by five other users.
In the second version, the single minority comment was liked by
five other users. In the third version, no comment was liked.
Finally, the post itself was liked by 24 users.
Perceived flu vaccination rate
: No effect found from being exposed to
user comments in favour of flu vaccination. No significant interaction of
comment likes or post likes.
Risk perception
: Significant effect of the interaction effect between the
exemplars and the evaluation of the stimulus: with a positive stimulus
evaluation, the readers of a version with comments supporting
vaccination expressed a marginally lower perception of risk than
readers of comments opposing vaccination. For participants with a
negative stimulus evaluation, the exact opposite was found. No
significant interaction of comment or post likes.
Personal attitude and behavioural intention
: No main effect of
exemplars on personal attitude, but an almost significant effect on
behavioural intentions. Readers of comments in favour of vaccination
expressed slightly greater intention to be vaccinated than did readers of
comments against vaccination. This interaction effect was more
pronounced for personal attitudes toward flu vaccination. No significant
interaction of comment likes except for the three-way interaction with
regard to personal attitude. A significant main effect for post likes
emerged with regard to personal attitude: individuals who read a
stimulus version with post likes had a more positive attitude toward flu
vaccination than did participants who saw a version without post likes.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
76
Reference
Study aim(s)
Study details
Description of intervention
Outcome and Results about social media
Piedimonte
2018[146]
Determine the level of
knowledge related to HPV and
cervical cancer among university
students and develop a targeted
education and vaccination
campaign to in
crease uptake.
Social media type:
Facebook
and Instagram
Vaccine:
HPV
Country:
Canada
Target population
: 151
This project aimed to use an education campaign on HPV and
cervical cancer to increase HPV vaccine uptake at two university
campuses in Canada. Phase II of the project included the
education campaign through social media, email communication,
information booths and individual solicitation. The precise use of
social media in the education campaign is unclear.
A Facebook event had 535 invitations, 23 attendees, and 6 shares. Four
people posted pictures on Facebook that generated 106 likes. One
picture on Instagram generated 45 likes.
Robichaud
2012[43]
Examine the prior attitudes of
first year medical students to
seasonal influenza immun
isation
(their risk
-benefit calculation,
their sense of vulnerability to
seasonal influenza, their overall
attitudes towards immuni
sation
and their immuni
sation history)
and assess the impact of the
most popular vaccine
-critical
YouTube videos on their
attitudes towards seasonal
influenza vaccine.
Social media type:
YouTube
Vaccine:
Influenza
Country:
Canada
Target population
: 41 medical
students
This study randomly assigned medical students to watch one of
two YouTube videos with different rhetorical styles (evidence-
based versus anecdotal) and measured any change in attitudes
and behaviours before and after watching the video.
The study did not find a significant difference in the responses to the
questions asked before and after watching the videos.
Sundstrom
2018[147]
Describe the development,
implementation and evaluation
of a theory
-
based cervical cancer
prevention communication
campaign for college
-age
women.
Social media type:
Facebook,
Twitter, and Vine
Vaccine:
HPV
Country:
United States
Target population
: 18 university
female students
Messages communicated about perceived threats, benefits and
safety of HPV vaccine. The main campaign message, “It’s my
time”, encouraged the consideration of HPV vaccination and
regular screening, and reminded individuals that it is not too late
to receive the HPV vaccine. Messages were delivered through
mass media and social media (Facebook, Twitter and Vine).
Twitter and Facebook were updated daily with relevant news
articles, pictures and facts and messages included the hashtag
#MyTime. A video contest was also prepared on Vine, where
participants had to submit a video that finished the sentence: “I
received the HPV vaccine so that I have time to…”
Uptake of HPV vaccination
: Despite widespread coverage in the media,
the messaging does not seem to have effectively changed behaviour
given the limited knowledge seen in focus groups.
Among participants, 63% had heard of the campaign. Following the
campaign, 93% had heard of the HPV vaccine and 74% believed the
HPV vaccine was successful at preventing cervical cancer.
Online group discussions
Kimmerle
2014[150]
Examine in what way the
particular type of contribution
(i.e. factual information vs
personal experiences) has an
impact on the subsequent
communication in health
-related
Internet f
orums.
Social media type:
Internet
Forum
Vaccine:
Measles
Country:
Germany (although
unclear whether the forums are
German)
Target population
: The users of
28 various Internet forums
This study compared the differences between using factual
information versus personal experience in a post on health-related
Internet forums. The researchers observed how each condition of
the initial post affected subsequent communication on the forum.
They considered the number of responses each post received and
how emotional the language/content of the posts were.
There was no statistically significant increase in the number of
responses on the personal experience posts as compared to the factual
post. However, responses to the personal experience post were
significantly more emotional than responses to the factual information
post.
Lai 2015[148]
Identify the effectiveness of a
Facebook
-assisted teaching
method for school-based CCPE
Social media type:
Facebook
Vaccine:
HPV
Country:
Taiwan
Cervical cancer prevention education (CCPE) programme based
on the health belief model, developed by Ministry of Health to
assist teens in understanding cervical cancer, Pap Smear
Knowledge, personal attitude and behavioural intention:
Under the
condition of the vaccine being an out-of-pocket expense, students
receiving a Facebook-assisted teaching method were 1.810 (measured
TECHNICAL REPORT Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy
77
Reference
Study aim(s)
Study details
Description of intervention
Outcome and Results about social media
on knowledge and attitudes
about cervical cancer prevention
and on HPV vaccination intention
among female students in a
senior hig
h school in Taiwan.
Target population
: 1,200 female
students (15-17)
screening and HPV vaccination and cervical cancer prevention The
lecturer, a nursing teacher, initiated the discussion with the
question: What is an HPV vaccine? This was followed by a brief
introduction of the HPV vaccine and the CCPE lecture
commenced. The lecture concluded with a brief summary and
discussion, reflecting on the importance of cervical cancer
prevention. Six-hour discussion sessions were offered either with
Facebook-assisted or in-person discussions after class.
two weeks after the Facebook teaching method) and 1.847 (measured
eight weeks after the Facebook teaching method) times more likely to
have the intention to be vaccinated compared with students who
received traditional teaching instruction. Under the condition of
receiving the vaccine free of charge, this figure was 2.531 times higher.
A comparison of change across groups indicates that knowledge
improvement scores in the experimental group were 2.942 points
greater than those in the control group. Attitude improvement scores in
the experimental group were 3.888 points greater than those in the
control group.
Under the condition of the vaccine being an out-of-pocket expense, the
experimental group’s improvement scores were 2.284 times greater
than those in the control group.
Langley
2015[149]
Develop knowledge about
whether and which interventions
can be used online to actively
support offline vaccination
behaviour once negative
information has been spread via
online social media.
Social media type:
Online
discussion forums
Vaccine:
HPV
Country:
Netherlands
Target population
: 184 parents
of daughters who would be
invited to get their HPV
vaccination
Parents took part in discussion forums, where they were exposed
to the following conditions: Participants randomly assigned to the
conditions of a 2 (source peer vs governmental organisation) x4
(influence strategy: source credibility, self-belief, direct challenge,
indirect challenge) between-subjects design. Online discussion
group to discuss raising adolescents and related issues. Parents
were asked to log in on a specified date and time and to be
available to participate in the experiment for 50 minutes.
Some messages were sent by the study team to look like they
came from peers or government officials and were followed by a
pro-vaccination comment with different persuasion strategies.
Personal attitude and behavioural intention:
The manipulation of peer
vs government spokesperson shows no direct relationship to the
participants' intention or valence with respect to the vaccination. The
influence of the different cues to action is negligible.
Other parents online had similar influence to that of friends and offline
peers whom the respondents know well. Opinions relating to the
vaccination within the nuclear family have the strongest relationships,
suggesting that influences via social media may need to concentrate
not just on one decision-maker but on members of the nuclear family.
No effect of the experimental manipulations of the different cues to
action on vaccination decision.
Interactive websites
Daley 2018[151]
Test the impact of a website
with a social media component
on vaccine attitudes and beliefs.
Social media type:
Website with
built-in interactive component
Vaccine:
Non-specific
Country:
United States
Target population
: 1,052
parents (during pregnancy and
early childhood).
Internet-based platform with vaccine information and interactive
social media components
3 arms: 1- vaccine social media (VSM) arm, access to website
with vaccine information and interactive social media
components; 2 Vaccine information (VI) arm (website without
social media), and 3- usual care (UC) arm
To reflect how a Web-based resource would be used in practice,
individuals in the VSM and VI arms were given access to the Web
site but were not required to visit it.
Social media format: blog, discussion forum, chat room. New
blogs posts added by the research team every month covering
timely or controversial issues such as new vaccine safety
research, recent vaccine-preventable disease outbreaks, changes
in policies (either text or audio). Ask a question portal available as
well to direct questions to experts (vaccine safety researcher,
paediatric infectious diseases specialist, general paediatrician, risk
communication specialist) - responses provided within 2 days.
Online chat sessions held each month to engage in conversations
with vaccine experts and between participants. Monthly
newsletters to encourage website use.
Among 542 participants in the VSM study arm, 189 (35%) visited the
study website at least once, with a mean of 1.9 visits (SD = 1.8) and a
range of one to 15 visits.
Personal attitude and behavioural intention:
Interventions were
associated with significant improvements in attitudes regarding
vaccination benefits compared to usual care among vaccine-hesitant
parents.
Interventions were associated with significant reductions in parental
concerns about vaccination risks compared to usual care among
hesitant parents.
Perceived self-efficacy also improved, although a significant change was
only observed when comparing VI. No significant differences were
observed when comparing the VSM versus VI study arms.
Change in attitudes over time among parents who were not vaccine
hesitant at baseline: The VSM and VI interventions were not associated
with any significant changes in vaccine-related attitudes compared to
usual care.
Systematic scoping review on social media monitoring methods and interventions relating to vaccine hesitancy TECHNICAL REPORT
78
Reference
Study aim(s)
Study details
Description of intervention
Outcome and Results about social media
Glanz 2017[152]
Test the impact of a website
with a social media component
on vaccine uptake.
Social media type:
Website with
built-in interactive component
Vaccine:
Hepatitis B, rotavirus,
diphtheria-tetanus-acellular
pertussis, Haemophilus
influenzae type B, pneumococcal
conjugate vaccine, polio, MMR
Country:
United States
Target population
: 1,052
parents (during pregnancy and
early childhood)
Same as above (Daley2018)
Vaccine acceptance:
Mean ranks for days undervaccinated were 438.5,
443.0, and 465.4 for the VSM, VI, and UC arms, respectively. Infants in
the VSM arm had a lower mean rank for days undervaccinated than
infants in the UC arm (p=.02). Mean ranks did not differ significantly
between the VI and UC arms or the VSM and VI arms.
The proportion of infants up-to-date at the end of follow-up were 92.5,
91.3, and 86.6 for the VSM, VI, and UC arms, respectively. Infants in
the VSM arm were more likely to be up-to-date at age 200 days than
infants in the UC arm (OR 1.92; 95% CI, 1.073.47). Up-to-date status
did not differ significantly between the VI and UC arms or the VSM and
VI arms.
For the MMR sub analysis, there were 71% of infants with at least 489
days of continuous follow-up. The proportion of infants who received
MMR by the end of follow-up were 95.6, 95.5, and 91.8 for the VSM,
VI, and UC arms, respectively. Although none of the study arm
comparisons were statistically significant, infants in the VSM and VI
arms were 2 times more likely to have received MMR than infants in the
UC arm.
Ferro 2014[153]
Evaluate the activity of the
Societa Italiana di Igiene’s web
project to address
misinformation online regarding
vaccination.
Social media type:
Website with
built-in interactive component
Vaccine:
Non-specific
Country:
Italy
Target population
: Online users
The Societa Italiana di Igiene (Italian Society of Hygiene) created
a web project to address misinformation online regarding
vaccination, particularly among healthcare professionals. This
comprises of a series of information tools including scientific
articles, educational information, video and multimedia
presentations, a forum, a periodic letter and a Twitter account. A
website (www.vaccinarsi.org) was developed specifically to
counterbalance, with credible and proved information, the diffuse
misinformation about vaccines online. The exhibition and
structuring of the website contains a first level with easy,
accessible information and a second level, exposing information
with more depth. A third level comprises of user's direct
interaction with the website. The website content comprises of
multimedia presentations, informative videos, informational
support and scientific articles.
The website was visited 27.173 times. From those, 25% returned to the
website. The average visit to the website is 10.000 hits per month.
Most visits are from Italy, with 6000 visits from Milan and 5000 visits
from Rome. Other than desktops, smartphones and tables are the most
common devices to access the webpage. Different sections of the
website have different access rates, and the initial pages had 9000 hits,
and page 'against misinformation' had 1460 visits. The website had,
since the first month, a considerable number of visit and this can be
due to Search Engine Optimizer (SEO) and an advertising campaign
online. The fact that the website remains as one of the first search
results on Google means it is a very sought for website. The results
from first month are encouraging and denote the importance of similar
initiatives.
Shoup 2015[154]
Describe a process for designing,
building, and evaluating a
theory
-driven social media
intervention tool to help reduce
parental concerns about
vaccination.
Social media type:
Website with
built-in interactive component
Vaccine:
Non-specific
Country:
United States
Target population
: 443 pregnant
mothers and parents of children
younger than 4 years
The objective was to create a web-based tool that provides
evidence-based information in an interactive environment where
parents can contribute content and discuss concerns with other
parents and vaccine experts. To gauge interest from target
population of parents, authors developed, pilot-tested, and mailed
a survey to assess their hypothetical trust in and use of a social
media web-based tool for vaccine and health information. A
manual medical record review was then conducted on the children
to determine if parents had delayed or refused vaccination for
personal, nonmedical reasons. Surveys were subsequently sent by
mail to a random sample of parents who accepted vaccines (n =
500), all parents who delayed vaccines (n = 227), and all parents
who refused vaccines (n = 127). Parents who delay vaccines, in
particular, are the primary target population for the intervention.
Approximately 50% of parents in all three vaccine behaviour groups
reported that they would use the web-based tool often. More than 60%
of parents who delay or accept vaccines reported that they would trust
the information about vaccines presented on the tool. Regardless of
their vaccine decisions, a high proportion of parents reported that they
would use the tool to ask questions, to receive current vaccine
information, and to review the childhood vaccination schedule.
Approximately 50% of parents who delay vaccines also said they would
discuss their experiences and vaccine concerns using the web-based
tool. In the sub-analysis comparing survey responders to non-
responders, there were no significant differences in age, income, or
home clinic. Overall, these results suggested that websites with built-in
interactive components may represent an effective intervention tool to
help parents make informed vaccination decisions for their children.
European Centre for Disease
Prevention and Control (ECDC)
Gustav III:s Boulevard 40, 16973 Solna, Sweden
Tel. +46 858601000
Fax +46 858601001
www.ecdc.europa.eu
An agency of the European Union
www.europa.eu
Subscribe to our publications
www.ecdc.europa.eu/en/publications
Contact us
publications@ecdc.europa.eu
Follow us on Twitter
@ECDC_EU
Like our Facebook page
www.facebook.com/ECDC.EU
ECDC is committed to ensuring the transparency and independence of its work
In accordance with the Staff Regulations for Officials and Conditions of Employment of Other Servants of the European Union and the ECDC
Independence Policy, ECDC staff members shall not, in the performance of their duties, deal with matters in which they may, directly or
indirectly, have a personal interest that could impair their independence. Declarations of interest must be received from any prospective
contractor before a contract can be awarded.
www.ecdc.europa.eu/en/aboutus/transparency