Government Information Quarterly 39 (2022) 101755

Contents lists available at ScienceDirect

Government Information Quarterly

journal homepage: www.elsevier.com/locate/govinf

Strategic communication? Measurement and evaluation of Twitter use

among municipal governments
Jeffrey A. Stone a, *, Kimberly J. Flanders b, S. Hakan Can c
a
Department of Information Sciences and Technology, Penn State University, 2809 Saucon Valley Rd, Center Valley, PA 18034, USA
b
Department of Corporate Communications, Penn State University, 120 Ridge View Drive, Dunmore, PA 18512, USA
c
School of Public Affairs, Penn State University, 200 University Drive, Schuylkill Haven, PA 17972, USA

A R T I C L E I N F O A B S T R A C T

Keywords: Government entities utilize social media to communicate with primary, secondary, and tertiary constituents. The
Strategic communication microblog Twitter serves as a rapid and inexpensive means of communication, allowing municipal governments
Barcelona Principles to reach a large and diverse audience with limited time or resource costs. Lacking in the existing research are
Government
examinations of whether governments are using social media outlets like Twitter as strategic communication
Social media
Twitter
tools or merely as ad-hoc communication outlets. By applying the Barcelona Principles 3.0 as the theoretical
Municipalities framework, this study explores whether U.S. municipalities measure and evaluate their Twitter activity as part of
Public sector a formal communications process informed by modern public relations standards, an whether differences in
municipal resources and demographics predict the integration of social media measurement and evaluation. The
results suggest that while formal communication plans are reported to exist, there is not a sufficient presence of
goals, policies, procedures, and metrics to consider Twitter a formal, strategic communication tool for partici­
pating municipalities. Total population and the administrative role of the social media account manager were
also found to impact the perception of defined procedures and specific assessment metrics. Though these results
suggest a lack of formal integration of social media measurement and evaluation, more research is needed to
explore the factors which may impact this situation.

1. Introduction explore whether and how U.S. municipalities assess the effectiveness of
their social media efforts (see Agostino, 2013; Bennett & Manoharan,
The advent of social media as a common information source has led 2017; Bonsón, Perea, & Bednárová, 2019). Efforts to promote formal
governments across the world to employ social media platforms for measurement and evaluation of public relations campaigns have surged
public communication. Social media provides an almost ideal platform in the public relations (PR) industry since 2010, but existing evaluation
for connecting governments with citizens, given its wide acceptance, frameworks and methodologies are not often employed in corporate
low cost, ease of use, and preference for short, focused messaging. practice (Buhmann, Macnamara, & Zerfass, 2019), perhaps due to the
Government use of social media outlets facilitates a communication significant knowledge base, time and effort necessary to properly assess
strategy based on multiple forms of communication - i.e., text-based (e. communication efficacy. In U.S. municipal governments, evaluating
g., Twitter, Facebook), image-based (e.g., Instagram), and video-based social media efforts can be a significant challenge given the variance in
(YouTube) - thus allowing governments to reach a large, diverse, and municipal size, resources, and priorities. As a result, questions exist
dispersed audience with minimal time and resource commitments. Be­ about whether U.S. municipalities use social media outlets like Twitter
sides supporting public information efforts, social media also provides as a formal, strategic communication tool or as an ad-hoc, reactive
for direct government-citizen interaction such as the collection of public communications mechanism.
feedback and opportunities for policy co-production. The primary purpose of the study is to assess whether U.S. munici­
While the importance of social media to public communication and palities employ formal policies and procedures to measure the efficacy
public relations is well-established (Arshad & Khurram, 2020; Warren, of Twitter communication. The Barcelona Principles 3.0, a set of public
Sulaiman, & Jaafar, 2014), there have thus far been few studies which relations industry best practices for evaluating communications efforts,

* Corresponding author.
E-mail addresses: stonej@psu.edu (J.A. Stone), kxt303@psu.edu (K.J. Flanders), hakancan@psu.edu (S.H. Can).

https://doi.org/10.1016/j.giq.2022.101755
Received 26 October 2021; Received in revised form 28 July 2022; Accepted 2 August 2022
Available online 18 August 2022
0740-624X/© 2022 Elsevier Inc. All rights reserved.
J.A. Stone et al. Government Information Quarterly 39 (2022) 101755

will be used as a framework for assessing the level of formal integration in their agencies (Wigand, 2010). For example, a study by Graham and
of measurement and evaluation among a sample of municipalities from Avery (2013) indicated that 59% of local governments were utilizing
across the U.S. The results of this study will identify whether municipal Twitter.
Twitter use in the U.S. is more deliberate (i.e., guided by formal policies, Despite the potential for interactivity, existing research has sug­
goals, outcomes, and performance evaluation) or more ad-hoc (i.e., gested that public sector social media – on any platform – is not a
dynamic or reactionary, with limited measurement and/or evaluation), function of an institutional desire to engage the public (Manetti, Bel­
and whether differences in municipal resources and demographics pre­ lucci, & Bagnoli, 2017; Reddick, Chatfield, & Ojo, 2017) and little evi­
dict the integration of social media measurement and evaluation. This dence exists that citizens use social media for interactive dialogue with
study adds to the existing literature on municipal social media their government (Eom, Hwang, & Yoon, 2014; Haro-de-Rosario, Sáez-
communication by exploring how (and if) municipal governments are Martín, & del Carmen Caba-Pérez, 2018). For example, research by
formally evaluating their Twitter use, the factors which may impact the Hemphill, Otterbacher, and Shapiro (2013) concluded that tweets from
integration of evaluative practices, and whether that measurement and members of Congress rarely sought interaction from constituents. Rather
evaluation is informed by modern public relations standards. than seeking interactive dialogue, research suggests that approximately
This article is organized as follows. A review of the existing content 70% of U.S. Twitter users use the service for news gathering (Hughes &
and literature surrounding public sector and municipal social media use, Wojcik, 2019). Perhaps as a result, public sector Twitter feeds are pri­
measurement, and evaluation will be explored, followed by a discussion marily directed at information dissemination. Analysis of tweets by
of the Barcelona Principles 3.0 framework. A series of research questions members of the U.S. Congress indicated that over 50% were informa­
will then be presented along with an accompanying research design to tional in nature (Golbeck, Grimes, & Rogers, 2010) and research by
investigate those questions. The results of the study will then be pre­ Waters and Williams (2011) found that governmental tweets primarily
sented, and a discussion of those results, the study limitations, and av­ fostered one-way communication. The ability to spread information
enues for future study will be provided. rapidly and in real-time via Twitter allows governments to set the
agenda independent of traditional “mass media” such as television or
2. Background newspapers (Aharony, 2012; Park, Kang, Rho, & Lee, 2016), providing
public sector entities with a direct connection to citizens.
2.1. Public sector use of social media
2.2. Measurement and evaluation of public sector social media
Social media, defined as the digital space that facilitates “diffusion of
compelling content, dialogue creation, and communication to a broader While the importance of social media to government communica­
audience” (Kapoor et al., 2018, p. 536), has taken a prominent role in tions is widely accepted, formal measurement and evaluation of those
the strategic communication plans of both public and private organi­ strategic communications is thought to be limited in both research and
zations. Social media provides organizations with access to large practice. Strategic communication involves the use of communication
numbers of geographically dispersed information consumers, providing towards achieving organizational goals and objectives (Macnamara &
accessible and efficient communication channels for information Gregory, 2018). For organizations to be successful in their communi­
dissemination, collaborative problem-solving, and interactive, insightful cation efforts, an ongoing, formal evaluation of their strategic commu­
dialogue (Bertot, Jaeger, & Hansen, 2012). Public sector organizations nication is required to adapt to changing environments (Plowman &
can leverage social media to encourage civic participation as well as help Wilson, 2018). Given the increasingly important role that social media
to build trusting relationships with constituents. Besides the ability to plays in strategic communication/public relations, holistic evaluation
connect with citizens, social media also functions to promote trans­ methods focusing on a wide array of impacts and outcomes are essential.
parency, trust, and confidence in governmental operations (Warren Prior research has suggested that measurement and evaluation of
et al., 2014; Silva, Tavares, Silva, & Lameiras, 2019; Song & Lee, 2016). social media communication – and public relations campaigns in general
All social media platforms are designed to facilitate content creation – has focused on volumetric measures such as outputs, i.e., counts of the
and sharing of content, as well as to connect with others for social number of page views, “clicks”, “likes”, shares (e.g., retweets), or
purposes. Platforms differ, however, on the primary media used for that mentions across multiple media and outlets (e.g., Agostino, 2013;
communication as well as the desired level of interactivity. The micro­ Bonsón et al., 2019; Haro-de-Rosario et al., 2018). The lack of commonly
blog Twitter emphasizes brevity in communication rather than inter­ used scientific, reliable, and valid measurement tools is a known prob­
active dialogue, providing municipal governments with access to a lem in public relations, limiting the ability to perform comparative
potentially large audience for real-time information dissemination. analysis across contexts (Schriner, Swenson, & Gilkerson, 2017); indeed,
Originating in March 2006, Twitter has experienced significant growth it has been said that progress towards generally accepted and valid
over the past 15 years, ranking 4th overall of social networking sites in evaluation methods for public professional communication has been
2020, following Facebook, Instagram, and Snapchat respectively limited for some time (Macnamara, 2018).
(Briggs, 2020). Twitter has over 38 million active users in the United Existing academic research into public sector social media use has
States, and 186 million active daily users across the globe (Iqbal, 2021). also focused on volumetric measures. A survey of research since 2010
Estimates suggest that 23% of U.S. adults use Twitter (Auxier & finds an emphasis on social media “engagement” measures involving
Anderson, 2021) and that 500 million tweets a day are posted to Twitter post views, likes, shares, and mentions. The emphasis on engagement as
(Aslam, 2020). a research metric is perhaps appropriate, given the desire of govern­
As a strategic communication tool, Twitter can improve individuals' mental organizations to efficiently reach and interact with their citizens,
perceptions of trust in governmental organizations (Kim, Park, & Rho, promoting both an informed citizenry and a sense of legitimacy for
2015). Social media has empowered citizens to engage in political government operations (Bonsón, Royo, & Ratkai, 2017). Measures of
conversations with their local government representatives (Warren engagement have included simple quantitative outputs such as likes,
et al., 2014) by potentially reducing the information imbalance between comments, re-tweets, and interactions, either alone or in combination
governments and their citizens. Social media has enabled ease of mes­ with other measures (Bonsón et al., 2017; Chen et al., 2020; Manetti
sage transmission from community members regarding civic engage­ et al., 2017; Stone & Can, 2021), though qualitative assessments using
ment, allowing community members to express their issues of public various forms of content and case analyses (Lappas, Triantafillidou, &
concern and have them addressed (Delli Carpini, 2000). Many govern­ Kani, 2021; Reddick et al., 2017; Stone & Can, 2020; Zeemering, 2021)
mental agencies have found Twitter's cost, accessibility, ease of use, and are also present. Additional research on public sector social media use
speed favorable for creating relationships and have implemented its use includes examinations of building citizens trust in government (Park

2
J.A. Stone et al. Government Information Quarterly 39 (2022) 101755

et al., 2016), issues related to transparency (Bennett & Manoharan, (specifically Twitter) as a strategic communication tool.
2017; Bonsón, Torres, Royo, & Flores, 2012), social media adoption Twitter was chosen as the social media platform for analysis for two
processes (Bennett & Manoharan, 2017; Mergel & Bretschneider, 2013), primary reasons. Twitter has been previously shown to be a commonly
and properties of social media networks, including homophily and used platform for governmental communication and public consump­
reciprocity (McFarlane & Kaza, 2012). Lacking in the existing research tion, providing an easily accessible tool for municipal information
base are examinations of how (and if) governments are measuring and dissemination. Given that all social media platforms have different
evaluating their Twitter activity as part of a formal communications levels of measurement and evaluation methods, the decision was made
process informed by modern public relations standards. by the research team to focus on a commonly used and mature (est.
2006) platform. Twitter represents a proxy for social media use in this
study, though future studies may highlight differences in measurement
2.3. The Barcelona principles framework
and evaluative approaches using a multi-platform, comparative
approach. Moreover, it must be stated that convenience also played a
In June 2010 the Association for the Measurement and Evaluation of
role in this decision, given the authors' prior published research into
Communication (AMEC) along with the Institute for Public Relations
municipal use of Twitter (e.g., Stone & Can, 2020) and familiarity with
held a summit in Spain where the first formal metrics for measuring the
Twitter usage and metrics.
success of public relations campaigns was created (Magee, 2010). Ac­
To accomplish the objectives of this study, a series of research
cording to Magee (2010), industry leaders, public relations pro­
questions aligned with a subset of the Barcelona Principles 3.0 were
fessionals, and delegates gathered to agree on seven key metrics to
explored. The first two research questions focus on whether U.S. mu­
establish the original Barcelona Declaration of Research Principles
nicipalities have specific policies and defined goals for their social media
(Barcelona Principles). The original Barcelona Principles were created to
use. Such artifacts are a necessary pre-requisite to measuring the impact
enable organizations to effectively measure public relations efforts,
of social media communication on organizational goal achievement,
eliminate outdated PR campaign metrics, and replace controversial PR
something that is considered an ongoing challenge (Macnamara, 2018):
metrics including Advertising Value Equivalents (AVEs) and multipliers
(Rockland, n.d.). Additionally, the original Barcelona Principles sought RQ1: Do U.S. municipalities have formally defined social media
to measure outcomes, ensure proper goal setting, incorporate social (Twitter) policies?
media efforts, and uphold standards of transparency (Magee, 2010). In RQ2: Do U.S. municipalities have a defined set of goals and outcomes
short, the goal of the Barcelona Principles was to standardize measure­ for Twitter use?
ment and evaluation procedures using more modern, data-driven, and
A central tenet of the Barcelona Principles is that measuring the
scientific methods (Schriner et al., 2017).
outcomes of communications strategies is preferable to simply measuring
The Barcelona Principles provide a solid foundation for a uniform PR
outputs, but that both quantitative and qualitative measures are neces­
evaluation system across the globe, but as communication channels
sary (Schriner et al., 2017). Outcomes are more difficult to measure, as
continued to evolve, revisions became necessary. Since the original
they involve qualitative, longer-term assessments of change – in
Barcelona Declaration of Research Principles in 2010, Barcelona Prin­
awareness, cognition, opinion, and behaviour – that occur as the result
ciples 2.0 was released in 2015 (Leggetter, 2019), followed by Barcelona
of communications strategies. Outputs, on the other hand, involve only
Principles 3.0 in 2020. According to Weekes (2020), Barcelona Princi­
quantitative measures of public exposure or consumption, such as page
ples 3.0 provides organizations across the globe a holistic approach to
views (Stacks & Bowen, 2013). A long-term outlook on evaluation is
creating and measuring communication tactics. See Fig. 1.
considered essential to effective measurement. The following research
The primary purpose of this study is to assess whether U.S. munici­
questions address the issue of whether municipal use of Twitter involves
palities employ formal policies and procedures to measure the efficacy
sufficient measurement and evaluation methods:
of Twitter communication. The Barcelona Principles 3.0 framework was
used as an analytical lens for exploring the evaluative practices of U.S. RQ3: Do U.S. municipalities measure outcomes, outputs, and po­
municipalities as it applies to their social media communication. tential impacts of their Twitter use?
Examining these communication practices using an accepted public re­ RQ4: Do U.S. municipalities measure outcomes and potential impacts
lations framework extends the existing literature by providing insight of their Twitter use on stakeholders, society, and the municipality?
into whether U.S. municipalities are truly using social media

#1: Setting goals is an absolute prerequisite to communications

planning, measurement, and evaluation.

#2: Measurement and evaluation should identify outputs, outcomes,

and potential impact.

#3: Outcomes and impact should be identified for stakeholders,

society, and the organization.

#4: Communication measurement and evaluation should include both

qualitative and quantitative analysis.

#5: AVEs are not the value of communication.

#6: Holistic communication measurement and evaluation includes all

relevant online and offline channels.

#7: Communication measurement and evaluation are rooted in integrity

and transparency to drive learning and insights.

Fig. 1. The Barcelona Principles 3.0.

3
J.A. Stone et al. Government Information Quarterly 39 (2022) 101755

RQ5: Do U.S. municipalities use both quantitative and qualitative Table 1

methods for measurement and evaluation? Response rates for participating states.

Next, our study wished to explore whether U.S. municipalities use Municipalities Responses % Rate

industry standard methods for measurement and evaluation, and Northeast

whether the use of these methodologies were made available for public Pennsylvania 32 9 28.13
Massachusetts 37 4 10.81
knowledge:
New York 37 5 13.51
RQ6: Do U.S. municipalities use integrity-based and transparent Connecticut 14 1 7.14
TOTALS: 120 19 15.83%
measurement and evaluation methods? West
California 40 7 17.00
Finally, our study wished to explore whether specific social, political,
Arizona 30 8 26.67
and economic factors could predict whether U.S. municipalities use Idaho 20 2 10.00
formal goals, procedures, and metrics for performance evaluation. Be­ Washington 40 5 12.50
sides commonly used factors such as population size, median household TOTALS: 130 22 16.92
income, and educational level of the population, this study also explores South
Tennessee 40 6 15.00
whether the administrative role of the individual responsible for the
North Carolina 40 12 30.00
Twitter account – e.g., IT administrators, social media directors, or Florida 40 8 17.39
communications specialists – and the political leanings of the population Texas 39 7 17.95
were significantly related to the use of formal practices for performance TOTALS: 159 33 20.13
evaluation: Midwest
Ohio 40 7 17.00
RQ7: Which factors (population size, median household income, Michigan 30 9 30.00
Illinois 40 7 17.00
educational level, the administrative role of the individual primarily
Nebraska 13 3 23.08
responsible for Twitter account management, and political leanings) TOTALS: 123 26 21.14
predict the use of formal goals, procedures, and metrics for social
media performance evaluation among U.S. municipalities?
TOTALS: 532 100 18.80

3. Methods Recruitment was performed by sending an e-mail request to munic­

ipal communications or social media officials in the selected munici­
All study methods and procedures were approved by the Penn State palities. Only one (1) survey response was accepted from each of the
University Office of Research Protections.
participating municipalities. The survey was password-protected;
recruited municipalities were provided with the necessary password.
3.1. Data collection Given that respondents self-identified their state and municipality,
supplemental demographic data for the participating municipalities was
Municipal governments in the U.S. were chosen as the population for obtained from the U.S. Census Bureau's American Community Survey
this study. U.S.-based users represent the largest percentage of Twitter estimates for 2019 which, at the time of this writing, represented the
users, with 76.9 million U.S. users as of January 2022 (Statista, 2022) most complete dataset for these municipalities. Data was obtained for
representing 23% of U.S. adults (Auxier & Anderson, 2021). According three variables – total municipal population (Population); median
to the U.S. Census Bureau, there were almost 36,000 general purpose household income (Income), in 2019 dollars; and the percentage of the
local governments in the U.S. as of 2017, including 19,495 municipal­ municipal population 25 years and over with a bachelor's degree or
ities and 16,253 towns and townships (U.S. Census Bureau, 2021). These higher (Education). For one municipality, data for the three demographic
governments – referred to as general “municipalities” for purposes of variables was not available due to the respondent only identifying their
this study – represent a diverse population in terms of population size state; for three others, education data was not available due to the
and citizen demography, among other aspects. This diversity makes municipality's status as defined by the census bureau. Population was
generalization challenging; perhaps as a result, municipalities as a chosen as a potential predictive factor due to prior research suggesting it
research context often focus on larger cities (Bonsón et al., 2017; Gao & influences municipal social media activity (Stone & Can, 2021) or the
Lee, 2017). extent of public sector social media use (Thackeray, Neiger, Smith, &
The data collection method employed by this study attempted to cast Van Wagenen, 2012). Similarly, median income has previously been
a wide net to produce a diverse set of U.S. municipalities. Data was shown to predict municipal social media activity (Stone & Can, 2021).
collected from a set of municipalities from across the continental United Twitter users, on average, tend to be higher educated and have higher
States using a multi-stage process. The 50 states were then separated incomes than average U.S. adults (Hughes & Wojcik, 2019). Table 2 list
into four U.S. Census Bureau regions (Northeast, Midwest, South, and summary statistics for the quantitative demographic variables.
West) and a set of 16 states were selected using stratified random sam­ A fourth variable was used as a proxy for the political sentiment of
pling. The 16 states included a random selection of four (4) states from the participating municipality. Using the self-reported municipality and
each region, selected through a computerized random selection pro­ state information, a Politics variable was created based on the county-
gram. After selecting the 16 states, municipalities from each state were level results in the 2020 Presidential Election. The county level was
selected for survey recruitment. Beginning with the set of each states' chosen based on the availability of data. Values were coded as 0 if the
municipalities in descending order by population, an attempt was made municipality's home county voted for the Republican candidate, and 1 if
to identify the most populous 40 municipalities in each state which had the municipality's home county voted for the Democratic candidate. This
active Twitter accounts (“active” was classified as at least one tweet in resulted in 93 valid cases (50 Democratic, 43 Republican), as six mu­
the last 30 days). A total of eight (8) states had fewer than 40 which met nicipalities were found to reside in multiple counties with split results
this criterion, so in these cases the available qualifying municipalities and one respondent did not identify their municipality.
were selected. The resultant set of 532 municipalities were chosen for A fifth variable was used to represent the administrative role of the
survey recruitment. Table 1 lists the number of municipalities in each individual in charge of the municipal Twitter account. The decision was
state who were selected for recruitment, along with their final response made to re-code the responses to the survey question Who is primarily in
rates. charge of managing your municipality's Twitter account? into a separate

4
J.A. Stone et al. Government Information Quarterly 39 (2022) 101755

Table 2
Descriptive statistics for demographic measures.
Variable N Mean SD Minimum Maximum

Total Population (Population) 99 104,071.15 192,900.30 764.00 1,680,992.00

Total Population (Population)* 99 10.88 1.14 6.64 14.33
Median Income (Income) 99 68,912.38 26,923.85 31,128.00 168,688.00
Median Income (Income)* 99 11.07 0.37 10.35 12.04
Percent College Educated (Education) 96 0.23 0.11 0.07 0.56
*
Values represent the natural logarithm of the original variable.

variable (Role) with two categories – Management and Non-Management. Communication” (Weekes, 2020). This principle was created to ensure
The recoding was based on analysis of responses to both the original organizations are no longer using AVE's as a unit of measurement on the
question and the accompanying open-ended question. effectiveness of advertising and public relations. AVE's, advertising
value equivalents, are a controversial measurement metric still utilized
by some organizations. Research by Lindenmann (1997) contends that
3.2. Survey construction the validity of AVE measurements is questionable, and the Institute for
Public Relations, 2010 determined that AVE's should not be used to
A custom survey was constructed by the authors. After assessing the measure effectiveness of a PR campaign. Since this research sought to
survey for face and content validity, surveys were delivered electroni­ uncover the metrics that municipal governments were utilizing to
cally through the Penn State University Qualtrics server between measure the success of their communication strategies, the fifth guiding
January 2021 and August 2021. The survey netted 100 responses during principle was irrelevant for this study.
this time, for an overall response rate of 18.80% (n = 532). While the The sixth principle states “Holistic measurement and evaluation in­
survey itself collected data on multiple Twitter utilization topics, this cludes all relevant online and offline channels.” This study was created
study reports on a subset of the survey questions which focused on to investigate measurement of a single communication channel, Twitter.
measurement and evaluation. This subset of questions is designed to Since this guiding principle seeks to uncover a holistic measurement, it
address our stated research questions, along with the social and eco­ did not fit into the scope of research for this study; therefore, no ques­
nomic indicators as part of the aforementioned census data. The survey tions were designed for this principle.
instrument is available from the corresponding author upon request. The seventh guiding principle states “Communication measurement
The survey began with a question intended to uncover whether and evaluation are rooted in integrity and transparency to drive learning
municipalities had a formally defined social media communication and insights” (Weekes, 2020.) Two survey questions were used to un­
plan/policy (RQ1). A follow-up question was used to identify those so­ cover whether municipalities use standard evaluation methods and are
cial media platforms which fall under that plan/policy. Once those transparent with their measurement methods and procedures (RQ6).
questions were provided, additional survey questions were constructed
to correlate to five specific sections of the Barcelona Principles. A
3.3. Analysis procedures
mapping of the Barcelona Principles to the specific survey questions
used is provided in Appendix A. By designing questions that mapped
Survey data related to research questions RQ1 through RQ6 were
directly to specific principles, researchers were able to assess whether
analyzed using descriptive statistics. Significant bivariate correlations
the Twitter communication efforts of the respondents were deliberate
between responses were identified using Kendall's Tau-b. ANOVA was
and in accordance with the Barcelona Principles 3.0 framework.
also used to identify significant group differences based on census
The first guiding principle of Barcelona Principles 3.0 states “Setting
region.
measurable goals is a fundamental prerequisite to communications
Analysis for RQ7 involved three multinomial logistic regression an­
planning, measurement, and evaluation” (Weekes, 2020). A single sur­
alyses to determine which combination of five social, political, and
vey question was used to understand whether municipalities had
economic variables – total municipal population (Population), median
developed goals with respect to using Twitter as part of their formal
household income (Income), the percentage of the municipal population
communications plan (RQ2).
25 years and over with a bachelor's degree or higher (Education), the
The second guiding principle states “Measurement and evaluation
political sentiment of the participating municipality (Politics), and the
should identify outputs, outcomes, and potential impact” (Weekes,
administrative role of the individual responsible for the municipal
2020). To determine whether municipal government entities established
Twitter account (Role) – best predicted responses to three specific survey
goals to measure the performance of Twitter along these avenues (RQ3),
questions. For the dependent variables, the decision was made to
six questions were used. These questions were designed to understand if
explore responses to three survey questions focused on the presence of
specific metrics were in place to measure performance with respect to
outputs, outcomes, and potential impacts as well as the frequency in
which Twitter communication was evaluated. Table 3
Subset of survey questions used for RQ7.
The third guiding principle states “Outcomes and impact should be
identified for stakeholders, society, and the organization” (Weekes, Principle #1: Setting measurable goals is a fundamental prerequisite to communications
2020). To assess municipal governments' adherence to this principal for planning, measurement, and evaluation.

their Twitter communication (RQ4), two questions were used to identify

those groups for whom specific metrics are in place as well as the specific • My municipality has a defined set of goals and outcomes for Twitter
outcomes used to measure Twitter performance. communication.
The fourth guiding principle states “Communication measurement Principle #2: Measurement and evaluation should identify outputs, outcomes, and potential
impact.
and evaluation should include both qualitative and quantitative anal­
ysis.” Two questions were used to indicate whether organizations uti­
lized quantitative and qualitative analysis to measure tweets (RQ5). • My municipality has defined procedures for measuring goal achievement of our
Brief examples of each type of analysis were provided to ensure re­ Twitter communication.
spondents understood what classifies as quantitative or qualitative. • My municipality has specific metrics in place to measure the performance of our
Twitter communication efforts.
The fifth guiding principle states that “AVE's are Not the Value of

5
J.A. Stone et al. Government Information Quarterly 39 (2022) 101755

three key indicators of the level of formal integration of measurement 4.3. Measurement of outcomes, outputs, and potential impacts
and evaluation – the presence of defined goals and outcomes: the pres­
ence of defined measurement procedures; and the presence of specific Research question RQ3 focused on whether U.S. municipalities
metrics. See Table 3 for the specific survey questions and associated measure outcomes, outputs, and potential impacts of their Twitter use.
Barcelona Principles. To answer this question, a series of six survey questions were posed. The
The selected survey questions all involved Likert scale responses (1 survey responses suggest that measurement tends to focus on outputs
= Strongly Disagree…5 = Strongly Agree). To reduce the number of rather than outcomes or impacts, though measurement and evaluation
empty cells, each of the dependent variables were recoded from five to occurs infrequently and often without defined procedures and specific
three categories (1 = Disagree, 2 = Neutral/Undecided, and 3 = Agree) metrics.
prior to the regression analyses. Multinomial logistic regression was Survey responses indicated that measurement and evaluation is
utilized for the analysis. Multinomial logistic regression was chosen primarily an internal activity, as only 6.12% (n = 98) agreed or strongly
instead of ordinal logistic regression due to an inability of the data to agreed that external agencies were employed for this purpose, while
meet the proportional odds assumption. 86.73% disagreed/strongly disagreed. However, the presence of internal
Preliminary data analysis of the quantitative factors – population measurement procedures was also found to be lacking; only 26.00% of
size, median household income, and educational level of the population respondents (n = 100) agreed or strongly agreed that defined mea­
– led to the natural log transformation of two factors due to concerns surement procedures were present. Over one-half (51.00%) disagreed or
over normality and linearity: Population and Income. Descriptive statis­ strongly disagreed and 23.00% were neutral. A similar pattern was
tics for both the original and transformed variables can be found in found in terms of specific metrics. Only 35.00% (n = 100) agreed or
Table 2. The transformed variables were used in all subsequent analyses. strongly agreed that metrics for measuring and evaluating performance
Pearson correlation analysis between the continuous factors (Popu­ were present, while 45.00% disagreed or strongly disagreed and 20.00%
lation, Income, and Education) indicated a strong correlation between the were neutral. In terms of the frequency of measurement, municipalities
median household income factor and the percentage of the municipal most often reported measuring performance on a monthly (38.38%, n =
population 25 years and over with a bachelor's degree or higher factor (r 99), yearly (18.18%), weekly (13.13%), or daily (6.06%) basis, while
= 0.74, n = 96, p < 0.01). Given the sensitivity of regression to collin­ 24.24% indicated that they never perform measurement. As can be ex­
earity concerns, the decision was made to remove the Education factor pected, a strong, positive correlation was found between responses to
from further analysis. the statement My municipality has a defined set of goals and outcomes for
Recoding for the Role variable was performed by two of the three Twitter communication and the frequency of performance measurement
authors independently before reconciliation. The distribution of re­ (Kendall's tau-b = 0.40, p < 0.001). Similarly, a strong, positive corre­
sponses was 56.57% Management (n = 99) and 43.43% Non-Management. lation was found and between responses to the statement My munici­
Chi-square analysis found no significant correlation between the pality has defined procedures for measuring goal achievement of our Twitter
dichotomous Politics and Role variables (Х2 = 0.270, df = 1, p = 0.604, n communication and the frequency of performance measurement (Ken­
= 92). dall's tau-b = 0.41, p < 0.001).
Responses suggested that quantitative outputs are commonly used
4. Results for performance measurement. The output with the highest level of
response agreement was the number of times tweets are “liked”
4.1. Existence of formal social media communication plans (70.10%, n = 97) and retweeted (69.07%), followed by the number or
replies (59.79%) and mentions (54.64%). In terms of which outcomes
Research question RQ1 focused on whether U.S. municipalities have and impacts are measured, the most popular option was “None of the
formally defined social media (Twitter) policies. Responses suggest the Above” (55.67%, n = 96). For those that indicated an outcome or impact
widespread presence of defined plans/policies as well as the use of a measurement. Public awareness data was selected most frequently
diverse set of social media outlets. A total of 78.00% (n = 100) agreed or (32.29%), followed by public attitude data (17.71%), public opinion
strongly agreed that their municipality has a social media communica­ data (16.67%), public behavioral indicators (10.42%), and other
tion plan or policy, with 12.00% disagreeing or strongly disagreeing and (5.21%). Only 20.83% of respondents selected more than one metric.
10.00% neutral. When prompted to select which social media platforms When asked to specify the “other” data source, participants referred to
are part of their municipality's communication plan, the results indicate Twitter output measures, external agencies contracted to measure im­
a multi-platform approach is common. All respondents selected at least pacts, and the number of media engagements.
two different social media platforms, with 82.00% (n = 100) selecting
four or more platforms. Twitter was selected by 99.00% of respondents 4.4. Measurement of impact on specific groups
(n = 100), while Facebook was selected by all respondents. Instagram
was selected by 90.00%, YouTube was selected by 82.00%, LinkedIn by Research question RQ4 focused on whether U.S. municipalities
52.00%, and Flickr by only 13.00%. The multi-platform approach varied measure outcomes and potential impacts of their Twitter use on three
significantly by census region, F(3, 96) = 11.344, p < 0.001. Tukey's key groups – stakeholders (e.g., citizens, external organizations), soci­
HSD Test for multiple comparisons found that the mean number of ety, and municipal policies and operations. The results noted in Section
platforms selected was significantly different between Northeast mu­ 4.3 suggest that performance measurement is performed infrequently –
nicipalities and those in the West (p < 0.01, 95% C.I. = [− 2.13, − 0.69]), including but not limited to outcome and impact performance mea­
Midwest (p < 0.01, 95% C.I. = [− 2.03, − 0.64]), and South (p < 0.01, surement – and further investigation found that measurement of impact
95% C.I. = [− 1.83, − 0.51]). on specific groups is limited at best. When asked whether the munici­
pality had specific metrics in place to measure the impact of Twitter
4.2. Existence of defined goals and outcomes communication efforts on stakeholders, society, and the municipality
itself, the survey results indicated that group-specific impacts were not
Research question RQ2 focused on whether U.S. municipalities have often assessed. The responses show that specific metrics were in place to
a defined set of goals and outcomes for Twitter use. Responses suggest measure the impact on stakeholders (e.g., citizens, other governments,
that U.S. municipalities are unlikely to have defined goals. Only 34.00% businesses, non-profits) (17.71%, n = 96), municipal government op­
(n = 100) agreed or strongly agreed that their municipality has defined erations and/or policies (13.54%), and the wider society (13.54%). Only
goals and outcomes while 41.00% disagreed or strongly disagreed and 30.21% (n = 96) or respondents chose at least one specific group; the
25% were neutral. remaining 69.79% chose “none of the above”.

6
J.A. Stone et al. Government Information Quarterly 39 (2022) 101755

Table 4
Multinomial logistic regression results (defined goals and outcomes).
Response B SE Wald df p Odds Ratio

1 = Disagree
Total Population (Population) 0.006 0.228 0.001 1 0.978 1.006
Median Income (Income) − 0.727 0.709 1.053 1 0.305 0.483
Management Status (Role) 0.825 0.531 2.411 1 0.120 2.281
Election Preference (Politics) 0.503 0.523 0.926 1 0.336 1.654
2 = Neutral (Undecided)
Total Population (Population) 0.212 0.255 0.692 1 0.405 1.236
Median Income (Income) − 0.882 0.800 1.215 1 0.270 0.414
Management Status (Role) − 0.471 0.569 0.686 1 0.408 0.624
Election Preference (Politics) − 0.407 0.598 0.463 1 0.496 0.665

***p < 0.01; **p < 0.05; *p < 0.10.

N = 92 (Disagree = 38; Neutral (Undecided) = 24; Agree = 30).
Х2 = 11.554, p = 0.172, Nagelkerke R2 = 0.133, − 2 log likelihood = 187.380.
The reference category is: 3 (Agree).

4.5. Use of quantitative and qualitative methods 4.7. Factors predicting formal practices for measurement and evaluation

Research question RQ5 focused on whether U.S. municipalities uti­ Research question RQ7 focused on the predictive power of specific
lize both quantitative and qualitative methodologies for measurement social, political, and economic factors on whether U.S. municipalities
and evaluation. In keeping with prior results suggesting limited per­ use formal goals, procedures, and metrics for performance evaluation.
formance measurement, 54.55% of respondents agreed or strongly The regression model for predicting the reported presence of defined
agreed that their municipality used quantitative measures to assess municipal goals and outcomes was not statistically significant, Х2 =
performance (n = 99). Only 30.30% disagreed or strongly disagreed 11.554, p = 0.172, Nagelkerke R2 = 0.133. No factors were found to
while 15.15% were neutral. U.S. municipalities appear less likely to significantly predict the reported presence of defined goals and out­
qualitative measures; only 15.15% agreed or strongly agreed that comes. See Table 4.
qualitative measures were used (n = 99) while 67.68% disagreed/ The regression model for predicting the presence of defined pro­
strongly disagreed and 17.17% were undecided. cedures for measuring goal achievement was statistically significant, Х2
= 22.150, p < 0.01, Nagelkerke R2 = 0.245. The model accounts for
4.6. Integrity-based and transparent evaluation methods 24.5% of the variance. Of the four factors, two significantly impacted
responses – total population size and the administrative role of the in­
Finally, research question RQ6 focused on whether U.S. municipal­ dividual in charge of the municipal Twitter account (both p < 0.05). See
ities utilize industry-standard methodologies for Twitter measurement Table 5. These results show that individuals reporting that managers are
and evaluation, and whether use of these methods was made publicly responsible for managing their municipal Twitter accounts are over four
available. Survey results show that participating municipalities have a times more likely to disagree rather than agree that defined measure­
long way to go towards using measurement and evaluation methods that ment and evaluation procedures are present. Similarly, each one unit
are either transparent or integrity-based (standardized). Only 21.65% increase in population size would lower the odds of disagreeing rather
(n = 97) agreed or strongly agreed that their municipality makes Twitter than agreeing that defined evaluation procedures are present by 0.584
measurement methods and procedures publicly available, while 40.21% units (a 44.2% decrease).
disagreed/strongly disagreed and 38.14% were undecided. Similarly, The regression model for predicting the presence of specific metrics
only 32.99% agreed or strongly agreed that their municipality uses in­ for performance measurement was statistically significant, Х2 = 15.877,
dustry standard methods for performance measurement, while 22.68% p < 0.05, Nagelkerke R2 = 0.181. The model accounts for 18.1% of the
disagreed or strongly disagreed and 44.33% were undecided. variance. Once again, both population and the administrative role of the
individual responsible for managing the Twitter account significantly
impacted responses. See Table 6. The results show that individuals

Table 5
Multinomial Logistic Regression Results (Defined Procedures).
Response B SE Wald df p Odds Ratio

1 = Disagree
Total Population (Population)** − 0.584 0.267 4.785 1 0.029 0.558
Median Income (Income) − 0.868 0.776 1.251 1 0.263 0.420
Management Status (Role)** 1.491 0.587 6.448 1 0.011 4.440
Election Preference (Politics) 0.872 0.580 2.264 1 0.132 2.393
2 = Neutral (Undecided)
Total Population (Population) − 0.070 0.278 0.063 1 0.801 0.932
Median Income (Income) − 0.668 0.869 0.591 1 0.442 0.513
Management Status (Role) − 0.261 0.626 0.174 1 0.677 0.770
Election Preference (Politics) 0.529 0.643 0.677 1 0.410 1.698

***p < 0.01; **p < 0.05; *p < 0.10.

N = 92 (Disagree = 47; Neutral (Undecided) = 21; Agree = 24).
Х2 = 22.150, p < 0.01, Nagelkerke R2 = 0.245, − 2 log likelihood = 167.529.
The reference category is: 3 (Agree).

7
J.A. Stone et al. Government Information Quarterly 39 (2022) 101755

Table 6
Multinomial logistic regression results (specific metrics).
Response B SE Wald df p Odds Ratio

1 = Disagree
Total Population (Population)** − 0.494 0.240 4.230 1 0.040 0.610
Median Income (Income) − 0.830 0.720 1.330 1 0.249 0.436
Management Status (Role)** 1.315 0.541 5.916 1 0.015 3.724
Election Preference (Politics) 0.624 0.525 1.413 1 0.235 1.866
2 = Neutral (Undecided)
Total Population (Population) − 0.309 0.276 1.250 1 0.264 0.734
Median Income (Income) − 0.343 0.846 0.164 1 0.685 0.710
Management Status (Role) 0.026 0.611 0.002 1 0.967 1.026
Election Preference (Politics) 0.260 0.627 0.172 1 0.678 1.297

***p < 0.01; **p < 0.05; *p < 0.10.

N = 92 (Disagree = 42; Neutral (Undecided) = 18; Agree = 32).
Х2 = 15.877, p < 0.05, Nagelkerke R2 = 0.181, − 2 log likelihood = 176.308.
The reference category is: 3 (Agree).

reporting that managers are responsible for managing their municipal Although output measures such as retweets and likes have been
Twitter accounts are almost four times more likely to disagree rather defined as an element of engagement by Twitter (2021), one potential
than agree that specific metrics are present. Similarly, each one unit explanation for the limited use of outcome or impact metrics may be that
increase in population size would lower the odds of disagreeing rather quantitative counts such as “likes” and “retweets” are the most acces­
than agreeing that specific metrics are present by 0.830 units (a 39.0% sible measures available. Twitter Analytics (https://analytics.twitter.
decrease). com/) provides access to an account's basic statistics free of charge.
These output measures are perhaps easier to understand and interpret
5. Discussion for many municipal officials, and indeed may provide favorable though
surface-level measures of engagement. Some researchers argue that the
The primary objective of this research was to draw from Barcelona focus on quantitative counts instead of deeper, more qualitative analysis
Principles 3.0 to investigate whether municipalities had formal policies reflects the way social media is often used by governments. As noted
and procedures in place to measure the effectiveness of their Twitter previously, public sector social media tends to focus more on informa­
communication. The extant research base is lacking in examinations of tion dissemination rather than interactive dialogue (e.g., Manetti et al.,
how (and if) municipal governments integrate modern public relations 2017); this strategy is a function of the desire to minimize the risks
standards into their social media communication processes, and whether associated with relationship-building and the privacy concerns found
they are measuring and evaluating their Twitter activity as part of those with social media interaction (Mergel, 2017). In the case of measure­
processes. The presence and use of formal policies, procedures, and ment and evaluation, exposure to follower details may be more likely
metrics can act as an indicator of whether social media use – specifically, during more complex analysis such as network-based data mining or
Twitter use – is considered an important strategic communication tool. sentiment analysis.
Alternatively, the absence of such policies, procedures, and metrics can It may also be the case that municipal employees in charge of
indicate that Twitter acts as a peripheral communication outlet for U.S. maintaining social media communication, including Twitter, may not
municipalities. The study results suggest that formal integration of have enough time or resources to dedicate to systematically measuring
strategic measurement and evaluation of Twitter use among munici­ performance. Many municipalities may lack the resources to conduct
palities is limited at best, and not aligned with the approach championed qualitative analysis (e.g., sentiment analysis, content analysis and focus
by the Barcelona Principles 3.0 framework. groups) and/or higher-level data analysis (e.g., data mining, advanced
This study uncovered that although most participating municipal­ statistics) that require deeper interpretation and evaluation. These
ities reported having formal communication policies, rarely did they techniques, along with decisions on goals and how to measure them,
report having defined goals or outcomes or a defined plan in place to often require significant resources as well as personnel with specialized
measure the effectiveness of their Twitter communication. Furthermore, training. Depending on the municipality, these capacities may be in
formal measurement of Twitter communications is reported to be con­ short supply. Motivation may also be a factor, as prior research has
ducted infrequently if at all. As can be reasonably expected, the results suggested that changes in technology-related practice – including social
indicated that performance measurement did occur more frequently media use – are often driven by external pressures rather than internal
when a defined plan and set of goals and outcomes existed. The study desires (Manetti et al., 2017; Mergel, Edelmann, & Haug, 2019). As a
results show that Twitter performance measurement and evaluation is result, public sector organizations (especially those with limited re­
primarily an in-house, surface level activity. When measuring the suc­ sources) may not see the cost-benefit value of performing deeper anal­
cess of Twitter communication, most participating municipalities ysis. Future research should explore potential relationships between the
measured their performance using basic output metrics and without the use of outcome/impact metrics and municipal resources, as well as the
assistance of an outside firm. Respondents indicated that only output underlying motivations for the performance measures municipalities
measures are frequently utilized, with more than two-thirds of the re­ choose to employ.
spondents agreeing that their municipality uses the number of “likes” Limited research exists exploring municipal transparency (Greco,
and the number of “retweets” as performance measures. The measure­ Sciulli, & D'onza, 2012) and transparency issues exist more frequently at
ment of outputs has been called a traditional element of PR practice, the local government level (Guillamón, Bastida, & Benito, 2011). Results
reflecting the challenge of finding scientifically valid measures (Schriner from this study indicate that transparency of municipal performance
et al., 2017). The common use of output measures also aligns with prior measures for Twitter is very low, as is the perceived use of standard
public sector social media research by Haro-de-Rosario et al. (2018), methods for performance measurement. Furthermore, the large number
Bonsón et al. (2019), and others to measure citizen engagement and of “undecided” survey responses suggest that respondents were un­
impact, representing both breadth and depth measures to understand knowledgeable about transparency and the role it plays in municipal
the effect of social media strategies (Kagarise & Zavattaro, 2017). communication as well as the use of standard evaluation methods. This

8
J.A. Stone et al. Government Information Quarterly 39 (2022) 101755

result is surprising considering the targeted participants of this research identifying factors which may impact the integration of evaluative
play key roles in the external communication activities of their respec­ practices.
tive municipalities, and that enhanced transparency is frequently cited The results of the study suggest that, in general, participating U.S.
as a primary benefit of public sector social media use (Bonsón et al., municipalities do not adhere to industry standard best practices for
2017). The lack of awareness on the importance of transparency and the measurement and evaluation outlined in the Barcelona Principles 3.0
use of standard evaluation methods, coupled with the limited use of framework. Therefore, Twitter communication cannot be considered a
performance measurement procedures, raises questions of whether formal, strategic communication tool for these municipalities, as there is
participating municipalities attach strategic value to measuring and not a sufficient presence of goals, policies, procedures, and metrics to
evaluating Twitter communication. Future research can be directed to­ warrant such a conclusion. Though these results suggest a lack of formal
wards uncovering the strategic intent of municipal Twitter communi­ integration of social media measurement and evaluation, more research
cation and performance evaluation. is needed to explore the factors which may impact this situation. While
An additional element of this research addressed the social, political, municipality demographics (total population) and the administrative
and economic factors which may play a role in the use of formal mea­ role of those responsible for the Twitter account (management or non-
surement and evaluation practices. Regression analyses found that these management) were shown to impact the presence of defined proced­
factors did not significantly predict whether municipalities agreed that ures and metrics, further study is needed to uncover the impact of
they have defined goals and outcomes for their Twitter communication. resource capacity on the choice of performance metrics and the fre­
However, two factors – the total population of the municipality and the quency of evaluation; the contextual rationale for both Twitter
administrative role of the individual in charge of the municipal Twitter communication and its evaluation; and the influence of personnel roles
account – were found to significantly predict whether municipalities on choices related to Twitter measurement and evaluation. Exploring
agreed that they have defined procedures for measuring the achieve­ these factors can shed further light on the use of social media-based
ment of their Twitter goals. Similarly, the total population of the mu­ strategic communication practices within municipal governments.
nicipality and the administrative role of the individual in charge of the This study was not without a few limitations. Five of the seven
municipal Twitter account was found to predict whether municipalities principles of the Barcelona Principles 3.0 framework were addressed in
agreed that they have defined metrics in place. The significant impact of this study. To maintain survey brevity and obtain the desired level of
the total municipal population has similarities to prior research findings participation in this research, efforts were made to limit the number and
linking population size and social media activity (e.g., Stone & Can, scope of the questions posed. This is a recognized limitation of this
2021; Thackeray et al., 2012) and suggests that measurement and study. The authors are currently implementing a qualitative study
evaluation may be more likely in those municipalities with a larger whereby participating municipalities can provide more detailed
population. The demographic context aligns with the predominant reasoning behind their current social media communication plans
context of the Twitter user base: in the U.S., 27% of Twitter users reside (formal and informal) as well as their processes of goal setting, mea­
in urban areas (Auxier & Anderson, 2021). surement, choice of metrics, and more.
For both the presence of defined procedures and specific metrics, the A diverse sampling of municipalities was selected from across the
identification of a non-manager in charge of the municipal Twitter ac­ United States for recruitment, providing geographically disbursed re­
count was more likely to result in affirmative responses. One possible sults. However, the use of municipalities from only the United States is
explanation may be that management personnel in this role may be another recognized limitation. Future studies may strive to gather
unable or unwilling to devote the necessary resources for measuring similar insight into the practices of other nations for means of analysis
Twitter performance due to competing priorities and/or a lack of job and comparison. Future research exploring the use of holistic mea­
specialization. Municipal managers may “wear many hats”, i.e., have surement – the sixth principle, omitted in this study due to the focus on
additional responsibilities within the municipality limiting their social media and Twitter – would provide a more thorough under­
knowledge about the scope of social media communication performance standing of the extent municipalities measure their communication ef­
and evaluation processes; this occurrence many be more common in forts in adherence to Barcelona Principles 3.0. Additionally, this
smaller, more rural contexts where resources are more strained than in research focused on only one social media platform (Twitter). Future
urban contexts (Kim, Aldag, & Warner, 2020). Another possible expla­ investigations involving other social media channels as well as tradi­
nation may be that non-managers are more familiar with these pro­ tional media channels could be conducted to determine whether
cedures and metrics based on job description and prior training. Future communication practices, including measurement and evaluation, were
research may be able to determine the personnel roles and administra­ uniform across all traditional and new media platforms.
tive context surrounding measurement and evaluation of municipal
Twitter communication. Declaration of interests

6. Conclusion The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influence
Social media outlets allow municipal governments to reach a large, the work reported in this paper.
diverse, and dispersed audience with minimal time and resource
commitment. However, evaluating municipal social media efforts can be CRediT authorship contribution statement
challenging due to the variance in municipal size, resources, and pri­
orities, among other factors. To help assess whether U.S. municipalities Jeffrey A. Stone: Conceptualization, Methodology, Formal analysis,
use Twitter as a formal, strategic communication tool, this study Investigation, Data curation, Writing – original draft, Writing – review &
investigated whether municipalities utilize a formal Twitter communi­ editing, Project administration. Kimberly J. Flanders: Conceptualiza­
cation plan and to what extent the efficacy of those communications are tion, Methodology, Investigation, Data curation, Writing – original draft,
measured and evaluated. By utilizing Barcelona Principles 3.0 as the Writing – review & editing, Funding acquisition. S. Hakan Can:
theoretical framework for this research, municipalities across the United Conceptualization, Methodology, Funding acquisition.
States were surveyed. The results of this research contribute to the
existing body of literature by investigating whether U.S. municipalities Acknowledgements
utilize formal policies and procedures to measure the success of their
social media (Twitter) communication, whether that measurement and The authors would like to thank Maria Veniamin and Shannon
evaluation is informed by modern public relations standards, and by Domermuth, undergraduate students at Penn State University, for their

9
J.A. Stone et al. Government Information Quarterly 39 (2022) 101755

assistance with data collection, and the Penn State Scranton Research exploration of these topics.
Development Grant, the Penn State Schuylkill Faculty Student Research Funding Sources: This research was supported by generous grants
Endowment, and the Penn State University College Work Study pro­ from the Penn State Scranton Research Development Grant, the Penn
grams for their financial support of this research. The authors would also State Schuylkill Faculty Student Research Endowment, and the Penn
like to thank Stephen F. Ross, Penn State University, for encouraging our State University College Work Study Program.

Appendix A. Barcelona Principles and survey question mappings

All survey questions involved a five-point Likert scale (1 = Strongly Agree, 2 = Agree, 3 = Neither Agree nor Disagree
[Neutral], 4 = Disagree, 5 = Strongly Disagree) except where noted.
Principle #1: Setting measurable goals is a fundamental prerequisite to communications planning, measurement, and
evaluation.

• My municipality has a defined set of goals and outcomes for Twitter communication.
Principle #2: Measurement and evaluation should identify outputs, outcomes, and potential impact.

• My municipality contracts with an external agency to measure the performance of our Twitter communication.
• My municipality has defined procedures for measuring goal achievement of our Twitter communication.
• My municipality has specific metrics in place to measure the performance of our Twitter communication efforts.
• My municipality measures the performance of our Twitter communication….
o Daily
o Weekly
o Monthly
o Yearly
o Never
• My municipality uses the following information to measure the performance of our Twitter communication efforts
(Check all that apply):
o The number of times our tweets are “liked”
o The number of times users reply to our tweets
o The number of times our tweets are re-tweeted (shared)
o The number of times our account is mentioned within other tweets
• My municipality uses the following information to measure the performance of our Twitter communication efforts
(Check all that apply):
o Public attitude data
o Public opinion data
o Public awareness data
o Public behavioral indicators (e.g. the number of visitors to a park)
o Other, please specify:
o None of the above
Principle #3: Outcomes and impact should be identified for stakeholders, society, and the organization.

• My municipality has specific metrics in place to measure the impact of our Twitter communication efforts on… (Check
all that apply):
o Stakeholders (e.g. citizens, other governments, businesses, non-profits)
o Society
o Municipal Government (e.g. operations, policies)
o Other, please specify:
o None of the above
Principle #4: Communication measurement and evaluation should include both qualitative and quantitative analysis.

• My municipality uses quantitative measures (e.g., counts, statistics, surveys) to assess our Twitter communication
performance.
• My municipality uses qualitative measures (e.g., focus groups, analysis of tweet comments) to assess our Twitter
communication performance.
Principle #5: AVEs are not the value of communication (Not Assessed)
Principle #6: Holistic communication measurement and evaluation includes all relevant online and offline channels. (Not
Assessed)
Principle #7: Communication measurement and evaluation are rooted in integrity and transparency to drive learning and insights.

• My municipality makes all our Twitter measurement methods and procedures publicly available.
• My municipality uses industry standard methods for measuring the performance of our Twitter communication efforts.

References Auxier, B., & Anderson, M. (7 April 2021). Social media use in 2021. Pew Research Center.
https://www.pewresearch.org/internet/2021/04/07/social-media-use-in-2021/.
Bennett, L. V., & Manoharan, A. P. (2017). The use of social media policies by US
Agostino, D. (2013). Using social media to engage citizens: A study of Italian
municipalities. International Journal of Public Administration, 40(4), 317–328.
municipalities. Public Relations Review, 39(3), 232–234.
Bertot, J. C., Jaeger, P. T., & Hansen, D. (2012). The impact of polices on government
Aharony, N. (2012). Twitter use by three political leaders: An exploratory analysis.
social media usage: Issues, challenges, and recommendations. Government
Online Information Review, 36(4), 587–603.
Information Quarterly, 29(1), 30–40.
Arshad, S., & Khurram, S. (2020). Can government’s presence on social media stimulate
Bonsón, E., Perea, D., & Bednárová, M. (2019). Twitter as a tool for citizen engagement:
citizens’ online political participation? Investigating the influence of transparency,
An empirical study of the Andalusian municipalities. Government Information
trust, and responsiveness. Government Information Quarterly, 37(3), Article 101486.
Quarterly, 36(3), 480–489.
Aslam, S. (2020). Twitter by the numbers: Stats, demographics & fun facts. Available at:
https://www.omnicoreagency.com/twitter-statistics/.

10
J.A. Stone et al. Government Information Quarterly 39 (2022) 101755

Bonsón, E., Royo, S., & Ratkai, M. (2017). Facebook practices in Western European Mergel, I. (2017). Building holistic evidence for social media impact. Public
municipalities: An empirical analysis of activity and citizens’ engagement. Administration Review, 77(4), 489–495.
Administration and Society, 49(3), 320–347. Mergel, I., & Bretschneider, S. I. (2013). A three-stage adoption process for social media
Bonsón, E., Torres, L., Royo, S., & Flores, F. (2012). Local e-government 2.0: Social media use in government. Public Administration Review, 73(3), 390–400.
and corporate transparency in municipalities. Government Information Quarterly, 29 Mergel, I., Edelmann, N., & Haug, N. (2019). Defining digital transformation: Results
(2), 123–132. from expert interviews. Government Information Quarterly, 36(4), Article 101385.
Briggs, P. (2020, December 8). Global Twitter users 2020. Insider Intelligence. htt Park, M. J., Kang, D., Rho, J. J., & Lee, D. H. (2016). Policy role of social media in
ps://www.emarketer.com/content/global-twitter-users-2020. developing public trust: Twitter communication with government leaders. Public
Buhmann, A., Macnamara, J., & Zerfass, A. (2019). Reviewing the ‘march to standards’ in Management Review, 18(9), 1265–1288.
public relations: A comparative analysis of four seminal measurement and Plowman, K. D., & Wilson, C. (2018). Strategy and tactics in strategic communication:
evaluation initiatives. Public Relations Review, 45(4), Article 101825. Examining their intersection with social media use. International Journal of Strategic
Chen, Q., Min, C., Zhang, W., Wang, G., Ma, X., & Evans, R. (2020). Unpacking the black Communication, 12(2), 125–144.
box: How to promote citizen engagement through government social media during Reddick, C. G., Chatfield, A. T., & Ojo, A. (2017). A social media text analytics framework
the COVID-19 crisis. Computers in Human Behavior, 110, Article 106380. for double-loop learning for citizen-centric public services: A case study of a local
Delli Carpini, M. X. (2000). Gen. Com: Youth, civic engagement, and the new government Facebook use. Government Information Quarterly, 34(1), 110–125.
information environment. Political Communication, 17(4), 341–349. Rockland, D. (n.d.). Introducing Barcelona Principles 2.0. instituteforpr.org. https://inst
Eom, S., Hwang, H., & Yoon, Y. (2014). Twitter use by local governors: Who uses Twitter ituteforpr.org/wp-content/uploads/Introducing-Barcelona-Principles-2.0-Why-the
and how is it used in public administration. Korean Public Administration Review, 48 -Change-was-Necessary.pdf.
(3), 381–412. Schriner, M., Swenson, R., & Gilkerson, N. (2017). Outputs or outcomes? Assessing
Gao, X., & Lee, J. (2017). E-government services and social media adoption: Experience public relations evaluation practices in award-winning PR campaigns. The Public
of small local governments in Nebraska state. Government Information Quarterly, 34 Relations Journal, 11(1).
(4), 627–634. Silva, P., Tavares, A. F., Silva, T., & Lameiras,, M. (2019). The good, the bad and the ugly:
Golbeck, J., Grimes, J. M., & Rogers, A. (2010). Twitter use by the US congress. Journal of Three faces of social media usage by local governments. Government Information
the American Society for Information Science and Technology, 61(8), 1612–1621. Quarterly, 36(3), 469–479.
Graham, M., & Avery, E. J. (2013). Government public relations and social media: An Song, C., & Lee, J. (2016). Citizens’ use of social media in government, perceived
analysis of the perceptions and trends of social media use at the local government transparency, and trust in government. Public Performance & Management Review, 39
level. Public Relations Journal, 7(4), 1–21. (2), 430–453.
Greco, G., Sciulli, N., & D’onza, G. (2012). From Tuscany to Victoria: Some determinants Stacks, D. W., & Bowen, S. A. (2013). Dictionary of public relations measurement and
of sustainability reporting by local councils. Local Government Studies, 38(5), research. Retrieved from http://www.instituteforpr.org/dictionary-publicrelation
681–705. s-measurement-research-third-edition/.
Guillamón, M. D., Bastida, F., & Benito, B. (2011). The determinants of local Statista. (2022). Leading countries based on number of Twitter users as of January 2022.
government’s financial transparency. Local Government Studies, 37, 391–406. https://www.statista.com/statistics/242606/number-of-active-twitter-users-in-
Haro-de-Rosario, A., Sáez-Martín, A., & del Carmen Caba-Pérez, M. (2018). Using social selected-countries/.
media to enhance citizen engagement with local government: Twitter or Facebook? Stone, J. A., & Can, S. H. (2020). Linguistic analysis of municipal twitter feeds: Factors
New Media & Society, 20(1), 29–49. influencing frequency and engagement. Government Information Quarterly, 37(4),
Hemphill, L., Otterbacher, J., & Shapiro, M. (2013, February). What's congress doing on Article 101468.
Twitter?. In Proceedings of the 2013 conference on Computer supported cooperative work Stone, J. A., & Can, S. H. (2021). Investigating factors of Twitter use among municipal
(pp. 877–886). governments. Journal of Computer Information Systems, 61(3), 267–274.
Hughes, A., & Wojcik, S. (2 August 2019). 10 facts about Americans and Twitter (pp. 1–21). Thackeray, R., Neiger, B. L., Smith, A. K., & Van Wagenen, S. B. (2012). Adoption and use
Pew Research Center. https://www.pewresearch. of social media among public health departments. BMC Public Health, 12(1), 1–6.
org/fact-tank/2019/08/02/10-facts-about-americans-and-twitter/(4). Twitter. (2021). Twitter account activity analytics – Engagement, impressions and more.
Institute for Public Relations. (2010). Barcelona declaration of measurement principles. Twitter. Retrieved October 6, 2021, from https://help.twitter.com/en/managing-
Gainesville, FL: Institute for Public Relations. your-account/using-the-tweet-activity-dashboard.
Iqbal, M. (2021, July 5). Twitter revenue and usage statistics (2021). Business of Apps. htt U.S. Census Bureau. (2021, October 8). 2017 census of governments – Organization.
ps://www.businessofapps.com/data/twitter-statistics/. Retrieved March 17, 2022 from https://www.census.gov/data/tables/2017/eco
Kagarise, W., & Zavattaro, S. M. (2017). Social media: How one city opens the evidence n/gus/2017-governments.html.
black box. Public Administration Review, 77(4), 486–488. Warren, A. M., Sulaiman, A., & Jaafar, N. I. (2014). Social media effects on fostering
Kapoor, K. K., Tamilmani, K., Rana, N. P., Patil, P., Dwivedi, Y. K., & Nerur, S. (2018). online civic engagement and building citizen trust and trust in institutions.
Advances in social media research: Past, present and future. Information Systems Government Information Quarterly, 31(2), 291–301.
Frontiers, 20(3), 531–558. Waters, R. D., & Williams, J. M. (2011). Squawking, tweeting, cooing, and hooting:
Kim, S. K., Park, M. J., & Rho, J. J. (2015). Effect of the Government’s use of social media Analyzing the communication patterns of government agencies on Twitter. Journal
on the reliability of the government: Focus on Twitter. Public Management Review, 17 of Public Affairs, 11(4), 353–363.
(3), 328–355. Weekes, J. (2020, July 22). Barcelona Principles get a facelift. CARMA. https://carma.co
Kim, Y., Aldag, A. M., & Warner, M. E. (2020). A balanced view: Us local government m/the-measurement-standard/commentary/barcelona-principles-get-a-facelift/.
managers’ perceptions of fiscal stress. Public Administration Quarterly, 44(2), Wigand, F. D. L. (2010). Twitter in government: Building relationships one Tweet at a
172–200. time. In Paper presented at the seventh international conference on Information
Lappas, G., Triantafillidou, A., & Kani, A. (2021). Harnessing the power of dialogue: Technology: New Generations (ITNG), Las Vegas, NV, April 12–14.
Examining the impact of Facebook content on citizens’ engagement. Local Zeemering, E. S. (2021). Functional fragmentation in city hall and Twitter
Government Studies, 1–20. communication during the COVID-19 pandemic: Evidence from Atlanta, San
Leggetter, B. (2019, April 15). Barcelona Principles 2.0. PRNEWS. https://www.prnews Francisco, and Washington, DC. Government Information Quarterly, 38(1), Article
online.com/barcelona-principles-2-0/. 101539.
Lindenmann, W. (1997). Guidelines and standards for measuring and evaluating PR
effectiveness. Florida, USA: Institute for Public Relations (1997; 2003). Retrieved
Jeffrey A. Stone is an Associate Professor of Information Sciences and Technology (IST) at
October 10, 2006 from http://www.instituteforpr.org/index.php/IPR/IPR_info/me
Penn State University, Lehigh Valley Campus. His research focuses on public sector use of
asuring_activities/.
information technology, computer and information science pedagogy, and sociological
Macnamara, J. (2018). A review of new evaluation models for strategic communication:
factors related to gaming.
Progress and gaps. International Journal of Strategic Communication, 12(2), 180–195.
Macnamara, J., & Gregory, A. (2018). Expanding evaluation to progress strategic
communication: Beyond message tracking to open listening. International Journal of Kimberly J. Flanders is an Assistant Teaching Professor of Corporate Communication at
Strategic Communication, 12(4), 469–486. Penn State University, Scranton campus. Her research focuses on crisis communication
Magee, K. (2010). First Global Standard of proving value of PR created at European related to blackface, consumers' perceptions of trust and credibility of information through
Summit on measurement. In US Home. Retrieved October 18, 2021 from http various communication channels, and utilization of social media to build personal
s://www.prweek.com/article/1010806/first-global-standard-proving-value-pr relationships.
-created-european-summit-measurement.
Manetti, G., Bellucci, M., & Bagnoli, L. (2017). Stakeholder engagement and public
S. Hakan Can is a Professor of Criminology at Penn State University, Schuylkill and
information through social media: A study of Canadian and American public
Harrisburg campuses. His research areas are law enforcement organizations behaviour,
transportation agencies. The American Review of Public Administration, 47(8),
police stress, and violence. Dr. Can recently extended his research to include opioid use as
991–1009.
maladaptive coping behaviour of stress.
McFarlane, J., & Kaza, S. (2012). State governors on social media. CeDEM Asia, 2012,
31–43.

11