WICBuzz In the Time of Texting

WICBuzz

In the Time of Texting

Jennifer Chancay, M.S., IBCLC

Kevin Koegel, MPH,

Cynthia Pompa, BS., IBCLC

07/31/2022

Acknowledgments

• USDA Food and Nutrition Service

• The Johns Hopkins (HPRIL) Team

• David Paige, MD, MPH

• Laura Caulfield, PhD

• Susan Gross, PhD, MPH, RDN

• Marycatherine Augustyn, PhD

• Elisabet Epps, MPH

• Erin Wick, BS

• Yunhee Kang, PhD

• Arizona Department of Health Services WIC Program

• Marlene Hernandez, State WIC Director

• Anne Whitmire, MPA, IBCLC, RLC & WIC Registered Dietitians

• Emily Moree, MPA, RDN, IBCLC

• Taffery Lowry MS, RDN, PMP, CSP and the Data Management Team

• Pima County Health Department

• Pima County Grants Management & Innovation

• Pima County Communications

• Pima County WICBuzz Participants

Funding Acknowledgement: This work was supported by a sub-award from the Hopkins/USDA

Participant Research Innovation Laboratory for Enhancing WIC Services (HPRIL), a

cooperative agreement between the United States Department of Agriculture Food and Nutrition

Service and the Johns Hopkins Bloomberg School of Public Health, Cooperative Agreement

Number #OPS-WIC-PRIL-JH-2018. The findings and conclusions in this publication are those of

the authors and should not be construed to represent any official USDA or U.S. Government

determination or policy.

Table of Contents Page #

Executive Summary 4

Introduction 9

Methodology 11

Evaluation Design 24

Results 33

Discussion 67

Appendix 72

Executive Summary

The Pima County Health Department (PCHD) administers the Special Supplemental Nutrition

Program for Women, Infants, and Children (WIC) with a caseload of over 9,000 each month.

About 37% of clients who received WIC services from PCHD in 2018 did not take the needed

steps to continue receiving WIC benefits for the next year, a process referred to as recertification.

Local Management Information System (administrative database) data from 2018 demonstrates

that the group least likely to recertify are non-English speaking families with infants receiving

formula benefits, and children over 1 year. Additionally, MIS data corroborates staff reports that

many parents do not recertify after their child is no longer eligible to receive formula benefits.

Staff has also provided anecdotal data that many potentially eligible individuals may not

participate because they do not know that benefits continue until age. In an effort to address gaps

in knowledge about the WIC program and keep participants on the program longer, PCHD

utilized a text messaging campaign known as WICBuzz, a communication strategy that sends, or

"drips," a pre-written set of messages to clients designed to increase nutritional and

programmatic knowledge and timed to encourage program retention.

The goal of this Project was to implement and evaluate this drip-marketing innovation in terms

of feasibility, acceptability, and impact on participation and retention of children in the WIC

program. The SMART objectives are provided below, and a logic model for this Project is

provided in the Appendix. To evaluate the project to see if there is evidence of impact on child

participation and retention, the Project compared WIC MIS data on participation and re-

certification (retention) of infants and children ages one to four from Pima County and Pinal

County WIC both before and after the implementation of WICBuzz. The baseline or pre-

implementation period was January 1, 2019 to December 31, 2019, with the implementation

period being the subsequent year from March 1, 2020 through February 28, 2021.

Description of Innovative Tool

PCHD’s innovative tool was WICBuzz, a drip marketing text message campaign. The campaign

featured targeted nutrition education and WIC brand awareness messages aimed at parents and

guardians of children from birth to age four who were currently enrolled with PCHD WIC. The

messaging was distributed at a minimum monthly, and no more than weekly, based on feedback

from clients. One challenge of this tool was to determine what number and frequency of

messages would be welcomed by WIC Buzz participants, and at what number would the

messages become undesirable and cause participants to opt out. This innovative drip text

massaging campaign was intended to encourage timely recertification, increase participation and

increase redemption of food benefits Additionally, ad hoc messages were utilized to remind

and/or educate caregivers of community events such as free turkey giveaways during the holiday

season. WICBuzz messages were provided in the participants’ primary language (English or

Spanish) and child’s age. Participants had the option to opt out of receiving messages by replying

STOP to a message.

Description of Project Implementation

The PCHD WICBuzz team kicked off the intervention on March 7, 2020 with a ‘Welcome to

WICBuzz’ text that was sent out to 5,851 families that were currently enrolled in the Pima

County WIC program and had indicated that it was permissible for the WIC program to reach to

them by text message. Of those 5,851 welcome messages, 620 (11%) were in Spanish and 5,231

(89%) in English. Over the course of the next 12 months, both English-speaking and Spanish-

speaking families received a monthly general message and up to three age-specific messages per

month, depending on the number and age of children under five they had. In order to tailor the

messages to the families and increase their helpfulness, a mid-implementation survey was

distributed through the WICBuzz texting platform with questions aimed at improving the

WICBuzz experience.

On November 30, 2020, an additional 1,311 participants who enrolled WIC program between

February 7

and November 30

, 2020, were enrolled in WICBuzz. Of the 1,311 new

participants that received the welcome message on November 30, 2020, 1,190 (91%) identified

English as their primary language and were opted into the English language groups, while 121

(9%) were opted into the Spanish-language groups.

At the end of the intervention (February 2021), taking into account attrition due to client opt-outs

and unrouteable messages, 5,271 (74% of those enrolled over the course of the year) continued

to receive WICBuzz messages.

The start of this intervention lined up almost perfectly with the COVID-19 pandemic in Arizona.

The WICBuzz team intended to use promotional materials to encourage clients to opt themselves

into the intervention by texting ‘WICBuzz’ to 85511. The intervention kicked off on March 6

and our clinics moved from in person to remote (telephone and Zoom) appointments by April 1

2020. The WICBuzz team rearranged the order of some messages and focused initially on

messages that emphasized the ability to complete services remotely. Having this new method of

communication with the majority of our WIC families was ideal during the pandemic.

Results

Process Evaluation

There were several deviations from initial planning. Implementation of the intervention was

delayed by three months relative to initial planning. The pandemic also delayed the MIS baseline

data abstraction by the Arizona Department of Health Services relative to the initial schedule

proposed by HPRIL. Baseline data on client characteristics was sought to provide a comparison

period preceding the period in which the intervention was implemented (i.e., the implementation

period). Programmatic changes included addition of a self-opt in option, which was developed to

provide an additional enrollment mechanism in addition to using the MIS database to identify

clients that provided a cell phone and gave permission to receive texts. The pandemic also

changed the way that Pima County WIC promoted the intervention, limiting original plans to use

promotional materials that WIC participants would see and discuss when they came to in-person

visits at the WIC clinic. Finally, local evaluation activities were altered, from planned in-person

interviews and focus groups to virtual focus groups and surveys administered via text message

and online.

Short-Term Impact Evaluation

Of those who responded to the Post-Implementation Client Survey (1,565 valid responses, 31%

response rate), message recipients reported that WICBuzz text messages added value to the WIC

client experience, marked by overwhelmingly positive responses to questions about changes in

knowledge of WIC services and eligibility, ways to maximize benefits, and nutrition education.

The Pima County WICBuzz team interprets these responses, in conjunction with low levels of

clients opting out of receiving the WICBuzz messages, to reflect acceptability of this type of

messaging campaign among WIC clients who had previously indicated that they would like to

receive text messages from WIC (i.e., a majority of WIC clients).

Further, respondents were able to further elucidate responses to closed-ended questions by

providing descriptive answers to the survey’s open-ended questions that reflected examples of

specific knowledge gained and self-reported behavior change. Some responses to open-ended

questions even referenced specific message content, such as certain foods and/or recipes featured

in WICBuzz messages, demonstrating recipient retention of content from the WICBuzz text

message campaign itself.

Impact on Retention and Participation

WICBuzz had a positive impact on recertification, retention, and continuous benefit issuance. At

Pima County, recertification was 6.7% higher (95% CI: 4.0-9.4%), retention in WIC was 7.4%

higher (4.7% to 10.1%), and continuous benefit issuance was 9.7% higher (95% CI: 6.9%-

12.5%) than at Pinal County. When examining the proportion of those timely recertified, the

results were non-significant. In general, the results were stronger for children than for infants in

stratified analyses.

Discussion

It is increasingly difficult to connect with others by phone, as each individual’s communication

preferences and ability to screen for telemarketers and ‘robocalls’ often results in avoidance of

calls from unrecognized numbers. For agencies like WIC, this can present a real obstacle in

engaging with clients and keeping them active in the program. ‘Nontraditional’ forms of

outreach like text messaging present an opportunity for WIC agencies to capitalize on an existing

and widely available medium (the cell phone) and explore other forms of interaction to achieve

these same programmatic goals.

A drip marketing campaign involves pre-programming text messages to send to a group of

people on a given date and time. Once the messages have been developed and programmed

(meaning assigned to whom and when the messages will be sent), the texting platform sends the

texts out automatically. It’s a relatively fast, low-cost way to put WIC information into the palm

of your client’s hand.

Over the course of the 12-month intervention, WICBuzz sent a total of 153,799 messages to

WIC families. Messages that, when surveyed, participants indicated they found helpful in

maximizing their WIC experience (hopefully here we can add language about perhaps

contributing to increased retention and participation.

Introduction

The Pima County Health Department (PCHD) operates four WIC clinics in Tucson, Arizona

with a contracted caseload of 9,300 clients per month during Federal Fiscal Year 2019 and 9,550

for Federal Fiscal Year 2020. By age, the largest category of clients enrolled as of March 2019

were infants, including 969 breastfeeding or partially breastfeeding and 2,146 formula feedings

infants. The second largest category of people served was comprised of 2,697 pregnant and post-

partum women, followed by one-year olds (1,800), two year olds (1,563), three year olds (1,251)

and four year olds (1,234).

In order to receive WIC benefits a client must demonstrate that they meet program participation

requirements by presenting proof of identification, address and income at their enrollment visit.

The enrollment visit is called a certification. After initial enrollment, a client must recertify

annually through the same process. Program enrollment is highest in the infant age group and

diminishes as children age. Keeping children in the program, or retaining them, is an ongoing

challenge for the WIC program. MIS data obtained from the Arizona Department of Health

Services indicates mothers and caregivers involved in WIC programs have a higher risk of not

recertifying when their children are twelve-months or older. About 37% of clients who received

services with Pima County’s WIC program in 2018 did not return, and the proportion of those

not returning varied by age group. MIS data demonstrates that the group least likely to recertify

are non-English speaking families with infants receiving formula benefits and/or children over

one year, while those most likely to recertify are English-speaking families with infants that are

exclusively or partially breastfed. The goal of PCHD WICBuzz is to improve retention rates and

participation for all WIC participants, focusing on those families with children aged 1-4 and

infants receiving formula benefits.

Development of the WICBuzz was informed by the analysis of a combination of State MIS data,

annual customer service surveys conducted by local WIC agencies in Pima County, and

anecdotal evidence from clinic staff. According to MIS data, approximately 44% of individuals

who were eligible to recertify after their child turned twelve months old failed to recertify in

2018. Anecdotally, front-line staff indicate that participant confusion about which foods are

WIC-eligible may lead some participants to choose not to continue in the WIC program. We

believe that increasing participant knowledge on the WIC food benefit and its nutritional value

will empower participants to purchase and use more of their benefits. Providing recipes and

cooking techniques, as well as information on the health benefits of the foods will demonstrate

the full value of benefits and encourage participants to remain in the program for as long as they

are eligible.

MIS data corroborates staff reports that many parents cease recertifying after their child is no

longer eligible to receive formula benefits. Staff has provided anecdotal evidence that many

potentially eligible individuals may not engage because they do not know benefits continue until

their child is five years old. Another recognized barrier is that families assume that they may no

longer meet the program’s income guidelines; believing WIC is only for women and infants

experiencing the highest levels of poverty. WICBuzz addressed these assumptions with drip

marketing designed to increase programmatic knowledge and encourage program retention.

In 2017, PCHD developed scheduling tools, altered clinic policies, and improved staff training in

order to meet the customer services standards set forth as best practices by the State of Arizona

WIC program. These changes resulted in statistically significant improvement in several

customer service elements, including wait times, counselor demeanor and perceived interest in a

participant’s family and activities, and whether WIC benefits are deemed useful to participants.

All PCHD WIC clinics now meet minimum customer service standards, and clinics continue to

be evaluated for differences in customer satisfaction between sites.

PCHD has developed a close relationship with the Arizona Department of Health Services

(ADHS), the government agency that maintains the MIS for WIC programs throughout Arizona.

This relationship enables regular data extraction related to the total number of participants aged

1-4 years who recertified in the program, as well as how many members of the eligible

population received successful issuance of benefits. This data, when divided by age group,

identifies disparate rates of recertification between families with infants receiving formula and

families with toddlers up to age five.

Logic Model

PCHD WICBuzz innovation tool was aimed to increase client participation in the WIC program

and redemption of food benefits, ultimately, increasing year over year client retention from age

one to five and culminating in graduation from the WIC program. Project inputs included

funding, staff from PCHD WIC and ADHS, the PCHD Communications Team, HANDS (the

ADHS MIS data repository), a Data Analyst and/or Evaluator from Pima County, and the WIC

clients themselves. Project activities included: identifying a text messaging service vendor;

developing and testing text messages; designing and configuring the messaging service by client

category; developing WICBuzz branding; developing promotional strategy and materials;

training staff on WICBuzz; launching WICBuzz and beginning promotion; completing training

of WIC staff in customer service; design and implementation of process and outcome

evaluations; and development of WICBuzz implementation protocols. Project short-term

outcomes included: increased knowledge of WIC services and eligibility and ways to complete

eligibility requirements; increased attendance at WIC appointments; and increased knowledge of

food choices and how to maximize benefits. Long-term outcomes included: increased participant

value of WIC; increased monthly participation; increased redemption of food benefits; and

increased year-over-year program retention see Appendix A.

Methodology

In an effort to provide the best customer service and similar customer service across the four

PCHD WIC locations, every staff member completed a customer service online course. The

course utilized was “Using Active Listening in the Workplace Situations” located at WIC Online

Learning platform, see Appendix B. All staff completed the training before the implementation

of the tool.

The HPRIL team also surveyed our agency to determine whether we met the customer service

‘best practices’ encouraged by HPRIL. The information below details PCHD WIC’s customer

service practices. While our services moved to a remote service model due to COVID-19, these

customer service practices remained the same throughout the intervention period, with the

exception of collecting email addresses. As part of our COVID response and remote service

model, we began emailing materials to clients, including DocuSign envelopes to sign the WIC

Rights and Obligations, as well as other required forms. PCHD WIC also began texting clients to

alert of missed appointments and reschedule.

Appointment reminders (phone, text, and/or email)

• All clients that make an appointment at least 24 hours in advance and have agreed to

receiving text reminders, received a text reminder from the Arizona State WIC office

• PCHD WIC did phone reminders for breastfeeding and registered dietician appointments

because they are scheduled up to 3 months in advance

• PCHD WIC did not routinely collect email information.

Missed appointment Follow-up within 24 hours (phone, text, and/or email)

• In 2019 PCHD WIC added calling missed appointments to attempt to reschedule them to

our caseload management policy.

• PCHD WIC did not currently have a system to text missed appointments

• PCHD WIC did not routinely collect email information

Appointment Scheduling (walk-in, same-day, and/or next-day)

Same-day/ next-day scheduling refers to when the participant is issued 3 months of benefits and

then in order to continue receiving benefits, must to schedule their next appointment either the

same day or next day when the previous 3 months of benefits have been utilized or have expired.

Most of our appointment scheduling was the same day or same week. Prior to moving to remote

services, one of our clinics accepted walk-ins every morning at 8am and two of the clinics

accepted walk-in appointments only on Saturdays. The only appointments booked in advance are

High Risk (Registered Dietician) or Breastfeeding. Otherwise, clients were advised to call when

their benefits were about to expire.

Description of the Innovative Tool

The innovative tool utilized by PCHD’s was WICBuzz, a drip marketing text message campaign.

The campaign featured targeted nutrition education and WIC program awareness messages

aimed at parents and guardians of children from birth to age four who at the time were actively

enrolled with PCHD WIC. Based on feedback obtained from clients in the Pre-Launch Client

Survey, the messaging was distributed at a minimum monthly and no more than weekly and was

intended to “add value” to the participants’ WIC experience. In addition to nutrition and

program-related messages, ad hoc messages were utilized to remind and/or educate caregivers of

local community events and resources such as a free turkey giveaway during the holiday season.

WICBuzz messaging was provided in the participants’ primary language (English or Spanish)

and targeted to their child’s age, with messages specifically for caregivers of infants, one-year-

olds and two to five-year olds. This innovative drip text massaging campaign’s intent was to

encourage timely recertification, increase participation and increase redemption of food benefits.

Implementation Protocols

Implementation of the WICBuzz innovative tool required the development and approval of

protocols, a pre-test of the tool’s functions, the development of training materials and the training

of the WIC staff. These steps are detailed below.

Educational Messaging System

Educational Messaging System (EMS) is a texting platform that supports drip-marketing

campaigns. In other words, EMS sends programmed messages to selected groups at selected

dates and times. The WICBuzz team developed the messages, selected the groups and scheduled

the message release times.

WICBuzz automatically sent age and language appropriate (English or Spanish) messages to

Pima County WIC participants. The age groups selected for this intervention were infants (12

months or younger), one-year-olds, and children two to five. The team selected the groups based

on the developmental milestones experienced during these time periods as well as the similarity

in the WIC food packages within the groups.

The Pima County WIC program provided messages in both English and Spanish due to the large

number of Spanish-speaking clients which is reflective of the large Spanish-speaking population

in the region. Providing the intervention in Spanish also allowed the program to assess the

effectiveness of the messaging to Spanish-speaking clients and tailor messaging to this

population. Of 5,851 families initially enrolled in WICBuzz, 5,231 (89%) indicated English-

language preference and 620 (11%) preferred communications in Spanish, per client records in

the agency MIS database.

The EMS system was structured to release four pre-scheduled messages monthly. Originally, the

WICBuzz Coordinator uploaded messages into the EMS monthly and sent them out manually

through EMS at various times of day, but eventually they were programmed to go out at 6:00

pm. After the mid-intervention survey, based on client feedback, time was changed to 10:00 am.

Pima Table 1. Monthly Message Release Schedule

Week

Message Group

Week 1

General Message

Week 2

C1 (one-year-olds)

Week 3

Infant

Week 4

C2-4 (child, two to five years

old)

Week 5

Possible ad hoc message

Selecting Clients for the WICBuzz Intervention

As part of an intake visit, WIC staff ask clients if their primary number is a cell phone and, if so,

are they willing to receive text messages at that number. More than 90% of clients agree to

accept text messages and request to be set up in the automated text appointment reminder

system. The WICBuzz team opted-in all clients that consented to receive texts from WIC into the

WICBuzz intervention. This was permissible as long as in the first message, clients were given

instructions on how to opt themselves out of the intervention. The WICBuzz team added, “Text

Stop to stop receiving messages” to each WICBuzz message.

Creating a Submission for EMS of Active WIC clients in WICBuzz

• Client Data Request to ADHS: The WICBuzz team asked for a report in an Excel format

from the Arizona Department of Health Services (ADHS) of families who opted-in to

receive text messages. The report included the following client variables:

o Family ID #

o Authorized Representative’s Name

o Client ID #

o Client Name

o WIC Category

o Phone Number

o Primary Language

o Date of Birth

o Certification End Date

• The master dataset was then divided into subsections, using the following list of

predetermined criteria:

a. WIC Categories:

i. Infant (IFF, IPN, IPN+, IEN)

ii. 1 – 2 year-old (C1)

iii. 2 – 5 year olds (C2, C3, C4)

iv. Pregnant Women (PG1, PG2)

b. Language Categories:

i. Spanish

ii. All other languages

• In order to subset the master list, the data were filtered using the appropriate age category

identifiers and language to create a single subset list. For example, for 1–2-year-olds

with Primary Language of ‘Spanish,’ clients with Age Category of ‘C1’ and Primary

Language of ‘Spanish’ were filtered and copied into a new spreadsheet. This process was

repeated for each combination of Age/Language categories (eight combinations in total).

• Each worksheet was clearly labeled with the appropriate age/language identifier in order

to be imported into the age- and language-appropriate EMS message campaign.

Preparing Client Data to Upload to Education Messaging Systems (EMS)

• Client identifiable information not needed to associate them with the age- and language-

appropriate message campaign, including WIC ID numbers and last names, was removed

prior to sharing the datasets with EMS. Six variables (labeled ‘Key Words’ by EMS)

were retained: Authorized Representative’s first name, child’s first name, child’s WIC

category, child’s date of birth, certification end date and Authorized Representative’s

phone number.

• Datasets (Excel spreadsheet format) were emailed securely to EMS.

• The Project’s dedicated EMS representative uploaded the prepared datasets into the EMS

platform, ensuring that the clearly labeled age- and language-appropriate datasets were

matched to the corresponding message campaigns.

Rules for Editing Client Data

The WICBuzz team understood that families with multiple children in different age categories

could receive up to four messages per month. It was important to ensure that families would not

receive the same message multiple times. The WICBuzz team together with EMS partners

programmed messages to ensure that participants did not receive duplicate messages.

• If a phone number had more than one child listed in the same WIC category:

o the oldest child in infant and one-year-old categories was kept.

o In the 2-5-year-old category, the youngest was kept.

• If 2 households had the same phone number:

o The WICBuzz coordinator confirmed the phone number in the client’s Health and

Nutrition Delivery System (HANDS) file and updated the phone number when

possible.

o Otherwise, the oldest child in the infant and one-year-old categories was kept.

o In the 2-5-year-old category the youngest was kept.

Text WICBuzz 85511 to Opt-in

The WICBuzz team worked with EMS to set up an opt-in system for WIC clients interested in

participating in WICBuzz that were not enrolled in the intervention at the time of the March

2020 launch. WIC clients were asked by WIC staff if they would like to opt in to WICBuzz at

their scheduled WIC appointments. Also, promotional materials, including posters, fliers and

oven mitts with the WICBuzz logo indicated to clients that they could text WICBuzz to 85511 to

opt themselves in.

The WICBuzz team worked with EMS to create a welcome message for clients that opted into

the program as well as an algorithm that would ensure that the client got into the correct age

group for their child. (see client self-opt-in under the messaging section below).

Required Opt-out Option

By law, individuals must be able to opt themselves out of a text marketing campaign. Therefore,

included in each WICBuzz message was the phrase, “Text Stop to stop receiving messages”. If a

client texted stop, the EMS system is set up to remove them from the list of recipients, and they

no longer received messages. The phone numbers that choose to opt out will be included in an

opt-out report provided by EMS.

WICBuzz Message Development

A group of WIC Registered Dieticians and Internationally Board-Certified Lactation Consultants

held a retreat to brainstorm possible WICBuzz messaging. Their instructions were to create

messages for the caregivers of infants, one-year-olds and 2–5-year-olds that participate in the

WIC program. The messages were about feeding, infant and child growth and development, WIC

foods, and WIC program information, whose knowledge of, would add value to the WIC client

experience. After the retreat, the group sent more than 50 possible messages to use in the

WICBuzz intervention.

Once received, the PCHD WIC staff edited and organized the messages by appropriate age

category and general topic. The Pima County Communications team screened the messages for

appropriate reading level, cultural competency and clarity of message. Pima County WIC staff

translated the messages into Spanish and then sent them to Pima County’s certified Spanish-

language interpreter for edits and final review. Lastly, the USDA Food and Nutrition services

reviewed the messages. The FNS sent questions and edits and requested that links given in

messages be to government sites and asked for further breakdown of messages by short-term and

long-term outcomes. Pima County WIC updated the links. Find the list of approved messages in

Appendix C.

Calendared Messages

The WICBuzz intervention was scheduled for 12 months, from March, 2020 through February,

2021. The WICBuzz team reviewed the approved list of messages and organized the messages

by month that they would be sent, taking into account the season of the year, cultural events and

recognized health observances (Appendix D) over the course of the intervention.

Bitly Links

Bitly is a URL shortening, managing and analyzing service. The WICBuzz team included

relevant links to online resources in several messages. Due to the fact that links can be lengthy

and WICBuzz messages have a limited character count of 255 characters per text message, the

use of Bitly in this intervention was essential. Not only do the links provide additional resources

to participants they also served as a measure of client engagement in the WICBuzz intervention

by tracking the number of WICBuzz participants that click on a given link.

Welcome Messages and Grouping

Initial Mass Opt-In Welcome Message

• English

WICBuzz is starting soon! You'll receive monthly messages from your WIC office on

topics like nutrition, parenting, links to recipes, family-friendly events and how to get the

most from WIC. Text STOP to receive no more messages.

• Spanish

¡WICBuzz está comenzando pronto! Recibirá mensajes mensuales de su oficina de WIC

sobre temas como nutrición, crianza de los hijos, enlaces a recetas, eventos familiares y

cómo aprovechar al máximo de WIC. Para no recibir mensajes text STOP.

Client Self-Opt-in Algorithm

Welcome to WICBuzz! ¡Bienvenidos a WICBuzz! For English messages text 1 Para español text

• English

You’ll soon start to receive monthly messages from your WIC office on topics like

nutrition, parenting, links to recipes, family-friendly events and how to get the most from

WIC. Text STOP2end

Some messages are age specific, would you like to receive a message for an infant age 0-

11 months? Text 1 for yes or 2 for no.

Would you like to receive a message for a child age 12months to 23 months? Text 1 for

yes or 2 for no.

Would you like to receive a message for a child age 2 years to 5 years? Text 1 for yes or

2 for no.

Thank you for your participation!

• Spanish

Recibirá mensajes mensuales de su oficina de WIC sobre temas como nutrición, crianza

de los hijos, enlaces a recetas, eventos familiares y cómo aprovechar al máximo de WIC.

Text STOP2end

Algunos mensajes son específicos de la edad. ¿Le gustaría recibir un mensaje para

un bebe

de 0-11 meses?

¿Le gustaría recibir un mensaje para un niño/a de 12 meses a 23 meses?

¿Le gustaría recibir un mensaje para un niño/a de 2 años a 5 años?

¡Gracias por su participación!

Ad-hoc Messages

In November of 2020 the WICBuzz team piloted sending ad hoc messages to WICBuzz

participants. Ad hoc messages were assessed based on potential benefit relative to potential

costs. The selected ad hoc messages were about food resources for the holidays (Thanksgiving

Turkey Giveaway), rental assistance for those affected by COVID-19, and information about the

newly available WIC self-check-out at various grocery stores.

In-Message Feedback

In January of 2021 the WICBuzz team added a request for feedback into the messages. The

request said to “Rate this message with a thumbs up or thumbs down.”

Use of Educational Messaging System Reports in WICBuzz Management

Bounce Back Report

A ‘bounce back’ refers to a message that was undeliverable to the phone number on file after

several attempts. The WIC Buzz Coordinator reviewed the Bounce Back report the day after a

message was released. If a phone number appeared on the Bounce Back report, the coordinator

verified the client information in the Arizona WIC MIS database, HANDS, and updated the

phone number in EMS, if there was a new number on file.

From Bounce Back pulled for end of day of launch date.

Phone numbers were verified in each file in HANDS:

• If welcome message was sent to the secondary phone number, phone number in our

Master Database was updated and new welcome message was sent to current primary

phone number.

• If the phone number in the client’s file had changed, phone number in our Master

Database was updated and new welcome message was sent to current primary phone

number.

• If no new number was on file, WIC Buzz Coordinator documented in clients’ file in

HANDS to verify phone number at next WIC visit.

• Welcome message sent out to updated phone numbers.

Other reports the WICBuzz Coordinator reviewed at the end of each month included: incoming

messages, outgoing messages, reminders (this included which messages were sent on which day

to how many people), opt-out report, opt-in report and how many people clicked on any Bitly

links during the month.

From Opt-Out Report pulled for end of day of launch date.

Phone numbers were verified in each file in HANDS:

• If welcome message was sent to the secondary phone number, phone number in our

Master Database was updated and new welcome message was sent to current primary

phone number.

• If the phone number in the client’s file had changed, phone number in our Master

Database was updated and new welcome message was sent to current primary phone

number.

• Welcome messages were sent out to updated phone numbers.

Implementation Training Materials

WICBuzz Staff training began with the introduction of the innovation at an all Staff Meeting. A

description of the Project and its purpose as given. Staff discussed what type of text messages

should be sent and the staff held a brainstorming session for the name of the innovation, this is

how the drip marketing campaign was named.

At subsequent Pima County WIC staff meetings, WICBuzz training and discussions continued.

The discussions included brainstorming of WIC topics to cover in the text messages, what sort of

promotional items could be offered to clients to promote awareness of the campaign, logo

improvements and progression of the WICBuzz intervention. PowerPoint presentations included

messages to be used, images of posters and promotional materials and instructions on how to

opt-in to WICBuzz (Appendix E and F).

Innovative Tool Adaptation and Pre-Test

Adaptation of the WICBuzz innovative tool began with the development of English and Spanish-

language messaging focused on increasing the perceived value of the WIC program for parents

and guardians of children in the program. Some of the messages created were general messages

for parents and guardians of all children, while nutrition messages targeted specific age

categories. The categories were birth to age one, ages one to two and two through four.

PCHD WIC staff worked with EMS to develop the WICBuzz messaging platform. PCHD WIC

staff also worked with ADHS to compile a list of all WIC clients with infants or children up to

age five. Once messages were developed and uploaded onto the EMS platform, all PCHD WIC

clients within the category were opted into WICBuzz, with the option to opt out at any time.

The WICBuzz Coordinator tested several messages in a pilot group comprised of PCHD WIC

staff. Staff opted into WICBuzz and received two to three messages. Staff provided feedback on

the potential of these messages to improve participation, redemption, and retention.

WICBuzz Message Testing

The testing of the delivery of WICBuzz messages using the EMS platform took place in several

steps.

In order for the WICBuzz Coordinator to become familiar with uploading client information into

EMS using an Excel spreadsheet, an Excel spreadsheet was created with the names and cell

phone numbers of five PCHD WIC staff. This data was then uploaded into EMS to create a pilot

test group. This group was utilized not only to help understand how to upload clients but also to

test several messages in both English and Spanish, test Bitly links, and test the opt-in and opt-out

features in EMS.

Once uploaded, each member of this pilot group received the pre-programmed welcome message

as expected. The welcome message used in this testing period was the following:

“Welcome to WICBUZZ. Thank you for your participation in helping us develop our

Project. To opt out text STOP”

Several other messages were programed to be delivered over the course of the next few days and

weeks. This enabled the coordinator to become familiar with the message programing features of

the platform and the selection of specific dates and times for message release. The test messages

included messages selected to be utilized in this drip text messaging campaign. These messages

included Bitly Links in order to test their functionality and the analysis function offered through

the Bitly Link application.

The following English messages were tested during the testing and development of WICBuzz:

“Did you know WIC has over 13,500 eligible WIC foods? Which are your favorites? Use

the EzWIC app to check your benefits and scan foods to see if they are WIC approved.”

“Got milk? WIC offers a variety of milk alternatives including goat's milk, soy milk, dry

milk and lactose-free milk. Ask your WIC clinic if you are interested in switching to a

different milk type.”

“Don't have a crusty day! Good food fuels your body and mind! Use your WIC benefits

to buy whole wheat bread to make toast for breakfast or a healthy sandwich for lunch.”

“ABC's are important! Apples to zucchini, learning about different foods is an important

part of childhood. Try to eat a fruit or veggie for each letter of the alphabet. It's a fun way

for your kids to practice their ABC's and to build healthy habits.”

The following Spanish messages were tested during the testing and development of WICBuzz to

test length and the addition of accents:

“¿A tu pequeño no le gusta beber agua? Pruebe agua divertida, agregue frutas como

fresas, arándanos o rebanadas de limón o jugo a agua mineral o agua con gas.”

“¿Sabías que el brócoli construye huesos? Las verduras de hoja verde como el brócoli, las

espinacas y la col rizada contienen calcio.”

“Los bebés lloran por muchas razones, no solo porque tienen hambre. Antes de darles de

comer, juegue al detective para ver si hay alguna otra razón por la que su bebé podría

estar llorando.”

“WIC está aquí para su hijo desde la infancia hasta los 5 años. Si tiene preguntas,

podemos ayudarlo a ser el mejor padre que pueda ser. ¡Háganos saber cómo podemos

ayudar!”

These messages were received correctly, all within one text message and accents used in the

Spanish language were appropriate in their placement and context. These was consensus

amongst the test group and the WICBuzz Team to move forward with program implementation.

To test opt-outs and opt-ins, several of the PCHD WIC Staff were directed to opt themselves out

the text messaging campaign by responding STOP to the text messaging. These members were

successfully opted-out and received the following message:

“Textline Chat: You have successfully unsubscribed. You will not receive any more

messages. For more info visit https://www.preventionpaystext.com/policies/”

This message was programed by EMS and sent in response to receipt of the text “stop”.

After successfully opting themselves out of the campaign, the PCHD WIC staff test group

members were asked to opt themselves back in to test whether clients who had opted out would

be able to rejoin the campaign. These members were successful in opting back into the

campaign.

The tool was further refined using participant feedback gathered through a pre-implementation

survey and a mid-implementation survey. The mid-implementation survey asked participants for

feedback about WICBuzz, the messages received so far, and potential content, timing, and

frequency of future messages. The survey was administered in both English and Spanish.

Updates Due to COVID-19

After the development of text messages targeting various client age groups and their approval by

FNS, implementation began on March 6, 2020, with the first WICBuzz message sent to all

clients who had given permission to receive text messages. Since this date, messages have been

sent according to a message schedule without interruption. While Project implementation did not

pause due to COVID-19, clinic operations shifted from in-person visits to phone visits on March

21, 2020, and the FNS approval of a physical presence waiver for the state of Arizona.

Originally, as part of ongoing evaluation of the acceptability of WICBuzz messages, the

evaluation plan included developing and implementing both English and Spanish-language focus

groups with WIC participants that were receiving WICBuzz messages, and then to revise

messages based on the feedback received. The pandemic, and the resulting need to socially

distance, ultimately altered these plans. In order to address the barrier, the HPRIL team designed

an electronic mid-implementation survey, see Appendix G, to gauge the acceptability and

usefulness of to WICBuzz messaging for participants. The survey was implemented in July of

2020 through the WICBuzz text message platform.

The pandemic also changed the way that WIC promoted the WICBuzz intervention. Originally,

the plan included designing promotional materials that WIC participants would see and discuss

when they came to their visits at the WIC clinic. The materials included posters, see Appendix

H, staff T-shirts with WICBuzz logos and oven mitts with the WICBuzz logo and self-opt-in

information printed on it. Since PCHD WIC clinics moved WIC services from in-person to

remote service delivery, Pima County WIC staff members suggested creating WICBuzz

promotional materials that could be included in all outgoing mail to WIC participants. Moving

WIC services to a remote platform necessitated using the US Postal Service to send out eWIC

cards and other WIC materials. A mini WICBuzz promotional poster was included in most

mailings for the duration of this intervention. The WIC clinics remained open for clients to pick

up breast pumps or WIC electronic benefit cards. When participants came to the clinic, they

received a WICBuzz promotional oven mitt. By the end of the intervention 914 WIC participants

had opted themselves into the intervention.

Evaluation Design

PCHD WICBuzz utilized a quasi-experimental design with a pre- and post-evaluation to

determine the effectiveness of the WICBuzz intervention. The outcome evaluation integrated

quantitative analysis of MIS data and a qualitative summary of stakeholder perspectives to

determine whether the campaign adds value to the WIC experience for clients and improves

target population retention.

Evaluation Questions and Indicators

The Project had four process evaluation questions:

• Was the Project implemented as intended? What were the strengths and

weaknesses of the development and launch of WICBuzz? Indicators included

documents obtained/completed, such as vendor agreements, the protocol for developing

and deploying WICBuzz messages, and promotional materials, as well as

implementation documentation, such as counts of messages sent.

• How did clients engage with WICBuzz? Indicators included number and percentage of

client opt-outs, numbers of WIC participants without text message capability (or who

have opted not to be contacted via text), and the percentage of links clicked, by message.

• How satisfied were clients with WICBuzz? Indicators included the percentage of

clients satisfied with message length, the percentage of clients satisfied with message

frequency, the percentage of clients indicating they would like WICBuzz to continue,

and the percentage of clients indicating that specific messages were helpful to them.

• What is staff perception of WICBuzz? Indicators included staff awareness of

WICBuzz, as expressed via virtual focus group.

The Project had one short-term outcome evaluation question:

• To what extent did WICBuzz add value to the WIC client experience (including

increasing knowledge of WIC services and eligibility, ways to maximize benefits,

and nutrition education)? Indicators included: percentage of clients reporting they

learned something new about their local WIC program through WICBuzz; percentage of

WIC clients indicating that WICBuzz helped them understand the WIC program better;

percentage of clients reporting they learned something new about nutrition/health

lifestyles through WICBuzz; perceptions of changes in client knowledge compared to

before WICBuzz; percentage of clients indicating that WICBuzz helped them redeem

more of their food benefits; percentage of clients stating that WICBuzz helped remind

them to schedule WIC appointments; percentage of clients reporting they purchased a

new food or tried a new recipe mentioned by WICBuzz; and percentage of clients

reporting benefit of having additional outlet for clients to express needs.

(See Appendix L)

Long-Term Outcome Evaluation Methods

Prior to project implementation, HPRIL assisted Pima County WIC in identifying a comparison

group to allow for a contemporaneous comparison evaluation design. Pinal County WIC, a

nearby local WIC agency, served as the comparison group. Pinal County WIC, located directly

north of Pima County, was chosen as a comparison site due to proximity of the agencies in size,

similar policies and practices in relation to customer service and scheduling practices and similar

client demographics, with the exception of a higher percentage of clients that identify as White in

Pinal County (19.2% of children compared to 8.52%) and a higher percentage of Hispanic in

Pima County (32.68% of children compared to 26.78%). In both Pima County Health

Department WIC program and the Pinal County WIC program more than 90% of clients selected

text messaging as their preferred method of communication.

HPRIL obtained Management Information System (MIS) data from the State of Arizona to

conduct statistical analyses to evaluate the impact of WICBuzz on outcomes related to child

retention and participation. Data were obtained for two time periods: a baseline period that was

the 2019 calendar year and an implementation period that was from March 1, 2020 to February

28, 2021. Due to the time needed for messages development and approval, the baseline or pre-

implementation period was changed from December 1, 2018 through November 30, 2019 to the

calendar year of 2019, and the implementation period changed from December 1, 2019-

November 30, 2020 to March 1, 2020 to February 28, 2021. The data request was for all infants

and children who were active in WIC at the beginning of each period. The HPRIL evaluation

sought to compare changes in each outcome over time for the innovation group (i.e., Pima

County WIC) to changes for the comparison group (i.e., Pinal County WIC).

The MIS data set included variables from the USDA minimum data set (MDS) necessary for

describing the characteristics of the participants as well as for calculating each of the outcome

variables. Because the data set included all infants and children active at the start of the period,

we can examine the pattern of participation of a cohort of WIC participants over time. During

any given 12-month period, each participant has an end date for the prior certification period and

can be expected to recertify (or not). Participants can leave the program by not re-certifying, or

they may recertify and then leave the program, and some may move and enroll in another WIC

agency. Thus, at the end of the year, a child may still be active in WIC (that is, retained), inactive

because they left the program, or re-enrolled at another WIC agency (e.g., they moved out of the

area) (HPRIL Table 1).

Each month benefits are issued for each WIC participant, and over a time period different

patterns of issuance can be observed, with less than continuous benefit issuance indicating gaps

in service due to, e.g., missed appointments. Although benefits are issued to a specific WIC

participant, benefit redemption at the individual level is not generally available in MIS data, nor

is partial redemption of benefits (HPRIL Table 1).

The analyses here focused on four core outcomes regarding retention and participation. First,

child recertification was defined as documented recertification of the children during the 12-

month period or during months 13-14 for those with certification end dates during the final 2

months of the period. Second, timely recertification was defined as recertification within 60 days

of the end date of the prior certification period. Third, retention was defined by the child’s status

at the end of each study period (i.e., active or terminated per the MIS). Fourth, child participation

was measured by continuous benefit issuance (11 or 12 months).

HPRIL Table 1. Child Retention and Participation Outcomes

Outcome

Description

Recertification

The proportion of children in the dataset with a recertification date during

the period. Note: includes children who left the agency and/or were not

classified as “active” at the end of the period.

Timely recertification

The proportion of children (out of all children in the cohort) with a

recertification date less than or equal to 60 days after the end of

certification during the period.

Not-timely recertification

The proportion of children (out of all children in the cohort) with a

recertification date greater than 60 days after the end of certification

during the period.

Percent of recertifications

that are timely

The proportion of children (only out of those with a recertification date)

whose recertification date is less than or equal to 60 days after the end of

the certification during the period.

Retention

The number of children active at the end of the data period at the

innovation or comparison agency / (The number of children overall at the

beginning of the period - children at another local agency at the end of the

period)

Continuous benefit

issuance

The proportion of children who were issued 11-12 months of benefits (out

of 12)

Months of benefit

issuance

Median and interquartile range of proportion of children issued benefits

across the year

Percent of cohort issued

benefits

Average proportion of children that were issued benefits each month

The analyses proceeded in stages. Descriptive analyses were conducted to describe the

participant characteristics and outcomes for each group during each time period. We documented

characteristics with a significant percentage of missing values (> 10%), which would limit their

usefulness during analysis. To assess comparability of the innovation and comparison groups

within each time period, HPRIL compared participant characteristics, including participant

category at the beginning and end of the data period; household size; number of WIC

participants in the household; multiple birth status; race and ethnicity; primary language other

than English; need for a translator; participation in other federal assistance programs such as

TANF, SNAP, and Medicaid; and whether the participant was ever breastfed. Pearson chi-square

tests were used to detect any significant differences between innovation and comparison in terms

of participant characteristics and outcomes for each time period. Logistic regression analyses

were also conducted to compare outcomes between groups (innovation versus comparison)

within each time period adjusting for covariates. As noted above, reports of these analyses were

created for each time period.

1 2

To estimate program impact, HPRIL employed a difference in difference (DID) approach. As

noted above, this involves estimation of the changes over time in each outcome in the innovation

versus the comparison group. Analyses were conducted for the overall sample as well as for

infants (IBE, IFF and IBP categories) and children (C1, C2 and C3 categories). Because

participants are not randomly assigned to the innovation or comparison group, analysis of the

impact of WICBuzz is not straightforward. Participants are assigned to a WIC clinic based on

residence which is determined by the participant’s family and based on multiple factors. This

may lead to the problem of selection bias, if these same factors also affect the likelihood of

recertification, retention, or participation.

To address this issue, HPRIL employed propensity score weighting (PSW) to adjust for

differences in participant characteristics between the innovation and comparison groups at each

time period (labelled T1 and T2) as well as differences across the two time periods. Two

common weighting approaches were used. In the first, weights were estimated using multinomial

logistic regression in which observations are weighted as compared to the those in the innovation

group during T1 as per Stuart et al., 2014.

In the second, a kernel approach for repeated cross-

sectional data was used to weight observations relative to the innovation group during T2 as per

Villa 2016.

To illustrate the balance in participant characteristics achieved through weighting,

HPRIL compared the absolute standardized differences (ASD) for the means of each variable

before and after weighting in the overall sample, for infants, and for children. This involved

comparing the balance achieved for the innovation group over time (at T1 and T2), the

innovation group at T1 and comparison group at T1, and the innovation group at T1 with the

comparison at T2. This approach was repeated for analyses involving infants or children.

To fully present the results, the outcomes are shown and compared over time using both

unweighted and weighted data. HPRIL conducted DID analyses for all four outcomes

(recertification, timely recertification, retention, and participation/benefit issuance) overall, for

infants, and for children. Beta coefficients and 95% confidence intervals were calculated using

three models: (1) Crude, unweighted; (2) Adjusted Model 1 (A1): PSW-DID using logit for

propensity score weighting (PSW) and ordinary least squares (OLS) for DID; and (3) Adjusted

model 2 (A2): PSM-DID using Kernel for propensity score matching (PSM) and probit for DID

with repeated cross-sectional option.

Data Collection and Analysis Plans

Pre-launch Client Survey: In order to determine some key texting preferences of our client

population, PCHD WIC developed a pre-launch WICBuzz design client survey. PCHD WIC

obtained a sample of 150 surveys. The survey asked about preferred frequency of messages,

length of messages, general interest in receiving messages and whether the client could open a

link from a text message received on their phone. Survey results were used to inform the final

selection of WICBuzz messages to begin the launch, Appendix I.

Mid-Project Client Survey: The WICBuzz Mid-Project Client Survey was designed to elicit

feedback from Pima County WIC clients who were enrolled in the WICBuzz text messaging

campaign as of mid-August 2020. The mid-Project data collection was initially planned to be

conducted via focus groups, but programmatic adjustments prompted the shift to the online

survey format, Appendix G. By virtue of this, the target population also shifted from program

sub-populations to the entirety of the population participating receiving WICBuzz text messages.

The survey was drafted in English by the WICBuzz Project team and was translated to Spanish

by Pima County Communications. The survey was programmed on the Qualtrics platform to be

accessed by clients via smartphone. The survey went live on the morning of August 17, 2020, as

the WICBuzz Project team sent a text message containing the survey link to clients. The English

message read: “Hello! Please help make WIC text messages more helpful to you by providing

feedback to us: https://pimacounty.sjc1.qualtrics.com/jfe/form/SV_3NNTz67XNyqQQvz” The

message was sent to 4,905 clients, 89% of whom (n=4,353) received the English language

message and survey. Eleven percent of clients (n=552) received the Spanish language message

and survey. A reminder message was sent on September 3, 2020, as part of a planned WICBuzz

informational message. The reminder message read: “Are you team smooth or team crunchy?

Either way, you can purchase peanut butter with your WIC benefits. Remember your ideas can

help make texts from WIC better. Take our survey here: https://bit.ly/3hJshYg Text STOP to

receive no more messages.” The survey was closed on September 21, 2020.

Post Implementation Staff Focus Group: As part of the intervention’s evaluation, the

WICBuzz team held a focus group for Pima County WIC staff on April 7th, 2021. In total seven

WIC staff, from a variety of clinics and with a variety of job duties, provided feedback on their

experience with clients and the WICBuzz intervention. The evaluator asked the staff about their

experience over the last year at WIC, their thoughts about the WICBuzz intervention, their

opinions about the clients’ awareness of the intervention, the use of technology in WIC, and

WIC practices during the pandemic, see Appendix I.

Post-Implementation Client Survey: On June 3, 2021 the WICBuzz team sent out the post‐

implementation survey to 4,513 English and 577 Spanish‐speaking WICBuzz participants,

Appendix K. The team developed the survey in Qualtrics and sent it as a Bitly link in a

WICBuzz message. A reminder message was sent on June 10th and the survey closed June 18,

2021 at 11:59pm Arizona Time. The initial WICBuzz text message sent June 3, 2021 at 10:00am

Arizona Time read: “It’s Zoo time! Please complete our WICBuzz Survey to help us send

messages you want to read. First 100 to complete will receive 2 adult & 2 child passes to the

Reid Park Zoo. https://bit.ly/3vTnwm4 Text STOP for no more messages.” The Pima County

WIC program used incentives to help increase the survey response. The initial invitation to the

survey included the opportunity to win two adult and two child passes to the Reid Park Zoo in

Tucson, Arizona. A second/reminder invite was sent one week later, June 10, 2021 at 10:00pm

Arizona Time: “Zoo pass winners have been notified, but there is still a chance to win 1 of 25

Annual Family Zoo passes. Complete your survey by June 18 to enter the raffle.

https://bit.ly/3vTnwm4 Text STOP for no more messages.” With the second invite, the WICBuzz

team used a second set of incentives: 25 Reid Park Zoo annual family memberships. Each annual

family membership allows two adults and up to four children admittance to the zoo as well as

admittance to many special member events throughout the year. All survey respondents, except

the winners of the two adult/two child passes, were entered into a raffle for a chance to win one

of the 25 memberships.

The survey contained the following questions:

• How helpful are the text messages you get from WIC?

• Do you feel you learned something new about how to understand and use your WIC

food benefits from the WICBuzz text messages?

• Do you feel you learned something new about healthy lifestyles for you or your children

from the WICBuzz text messages?

• Over the past year, did receiving WICBuzz text messages encourage you to do any of

the following? (select all that apply)

• In the future, which of the following topics would you like to receive WICBuzz text

messages about? (select all that apply)

• What kind of recipes are you interested in receiving?

• Do you know up to what age can a child participate on WIC?

• Is there anything else that you would like to share about the text messages you receive

from WIC?

MIS Data Management and Transfer

Per HPRIL, a condition for eligibility for the program was concurrence from the state WIC office

to provide sub-grantees and HPRIL with de-identified MIS data for the evaluation. The baseline

period was the calendar year 2019, and the data were expected to include infants and children

with birthdays between January 1, 2015 and December 31, 2018 who were active in WIC at the

beginning of the data period (January 1, 2019). Within each innovation and comparison group,

these children make up the “cohort” that HPRIL studied.

With input from Pima County WIC and ADHS, HPRIL compiled a unique baseline and

implementation period codebooks. The official MIS codebooks, data requests, and skeleton

databases were sent to Pima County WIC, which relayed the requests for data to ADHS, the state

agency. HPRIL requested that the datasets be provided by ADHS via Pima County WIC as a

wide format CSV file and uploaded to a HIPAA/FERPA-secure Johns Hopkins University

OneDrive folder. While in possession of Pima County WIC, the de-identified client data sets

were securely stored in an encrypted drive on Pima County’s network, accessible only by the

Pima County WIC Program Manager, the WICBuzz Project Coordinator, and the Data

Analyst/Evaluator.

Results

Results of Process Evaluation

Implementation Fidelity

Implementation of the intervention was delayed relative to the initial timeline due to various

issues associated with intervention planning and preparation. Specifically, the innovation

implementation date changed from December 1, 2019, to March 1, 2020 - a “delay” of three

months.

While the strategy selected to maximize participation was to use the MIS database to identify

clients that provided a cell phone and gave permission to receive texts, a self-opt in option was

also developed. When a participant texts WICBuzz to 85511, they could opt themselves into the

program. The benefit of the self-opt in is that it contains triage questions that place the client into

the appropriate messaging group based on their answers to the questions. The self-opt in option

launched March 17, 2020.

The COVID-19 pandemic changed the priorities of the technology support team at the Arizona

Department of Health Services. The team was required to deploy their resources to reprogram

the HANDS MIS system to accommodate for USDA policy waivers enabling remote services.

The MIS baseline data pull was subsequently delayed relative to the initial schedule proposed by

HPRIL.

During the intervention period, the primary local evaluation activity that was planned was to

develop and implement both English and Spanish-language focus groups with WIC participants

that were receiving WICBuzz messages, and then to revise messages based on the feedback. The

pandemic, and the resulting need to socially distance, ultimately altered these plans. In order to

address the barrier, the HPRIL team designed an electronic survey that was distributed to

WICBuzz participants through text message. The survey was distributed in August 2020 and data

were analyzed by Pima County staff. The WICBuzz Team then met and discussed modifications

to be implemented based on the findings. After discussion, it was determined messaging would

continue as scheduled for the remainder of the WICBuzz intervention period. Additionally, based

on the survey findings, a modification was made to the time of day messages are delivered.

Responses indicated clients would prefer to receive a message early in the day (as well as

“anytime”). The time of delivery of WICBuzz messaging was adjusted from 6:00pm to 10:00am

local time.

The pandemic also changed the way that Pima County WIC promoted the WICBuzz

intervention. Originally, the plans included promotional materials that WIC participants would

see and discuss when they came to their visits at the WIC clinic. The materials included posters,

staff T-shirts with WICBuzz logos and oven mitts with the WICBuzz logo and opt-in

information printed on it. By April 1, 2020, PCHD WIC clinics had moved WIC services from

in-person to remote service delivery. In order to address this challenge, Pima County WIC staff

members suggested creating WICBuzz promotional material that could be included in all

outgoing mail to WIC participants. Moving WIC services to a remote platform necessitated using

the US Postal Service to send out eWIC cards and other WIC materials. A mini WICBuzz

promotion poster was included in most mailings since of April 17, 2020. The WIC clinics were

still open for clients to pick up breast pumps or eWIC cards. When participants came to the

clinic, they received a WICBuzz promotional oven mitt.

In November 2020, the WICBuzz team created and delivered two ad hoc text messages. Initially,

the intention of using ad hoc messaging was to allow for the ability to let WIC participants know

about free and low-cost community events, particularly those targeted toward families with

young children and promoting healthy lifestyles. Since WICBuzz launched as the COVID-19

pandemic began in Arizona, most in-person community events were cancelled. The WICBuzz

participant survey conducted in August 2020 indicated that 38% of the respondents were

interested in receiving WICBuzz text messages with information about “community resources to

meet the needs of my family,” so the WICBuzz team decided to share free/low cost food

resources with clients via ad hoc messages. On November 18, 2020, a general message was sent

to participants with information about how to receive a free Thanksgiving turkey:

As we enter the holiday season, there is help here in Tucson. For more information on a

free Thanksgiving turkey click: https://www.facebook.com/events/1014363729062266/

Text STOP to receive no more messages.On November 27, 2020, the WICBuzz

coordinator sent a message with information on rent assistance for those affected by

COVID-19.

Participant Engagement with the Tool

In March 2020, 5,851 families were enrolled in WICBuzz. On November 30, 2020, an additional

1,311 were added. At the end of the intervention (February 2021), taking into account attrition

due to client opt-outs and unrouteable messages, 5,271 (79% of those enrolled over the course of

the year) continued to receive WICBuzz messages.

Of those 5,851 welcome messages, 620 (11%) were in Spanish and 5,231 (89%) in English. All

those who received this initial welcome message. There were 68 “bounce backs” (1% ). As of

March 31, 2020, 78 clients (1% of those who received the welcome message) “opted out” of

receiving messages by replying “STOP” to the messaging service. The majority of these “opt

outs” (55%) occurred on the first day of the campaign. Three additional messages (one targeted

message specific to clients with 1-2 year olds, one specific to clients with infants, and one

specific to clients with children ages 2-5) were sent to clients during March and early April 2020.

Of the 1,311 new participants that received the welcome message on November 30, 2020

(second enrollment of clients into WICBuzz), 1,190 (91%) identified English as their primary

language and were opted into the English language groups, while 121 (9%) were opted into the

Spanish-language groups. Twenty participants (2%) immediately opted out.

Pima Figure 1. Number of opt-outs per month, March 2020 to February 2021

Mid-Project Client Survey

The WICBuzz Mid-Project Client Survey was designed to elicit feedback from Pima County

WIC clients who were enrolled in Pima County WIC’s “WICBuzz” text messaging campaign as

of mid-August 2020. The survey was drafted in English by the WICBuzz Project team and was

translated to Spanish by the Pima County Communications Department. The survey was

programmed on the Qualtrics platform to be accessed by clients via smartphone.

The survey went live on the morning of August 17, 2020, as the WICBuzz Project team sent a

text message containing the survey link to clients. The message was sent to 4,905 clients, 89% of

whom (n=4,353) received the English language message and survey. Eleven percent of clients

(n=552) received the Spanish language message and survey. A reminder message was sent on

September 3, 2020, as part of a planned WICBuzz message.

The survey was closed on September 21, 2020. There was a total of 230 valid responses (i.e.,

recorded surveys containing a valid response to at least one item), resulting in a response rate of

4.7%. Response rates varied by survey language, with 5.0% (n=218) of those who received the

English language survey responding, while 2.2% (n=12) of those who received the Spanish

language survey responded. Sixteen of the 246 survey responses that were received overall were

blank.

How helpful are the text messages you get from WIC?

Pima Figure 2. Message helpfulness by response selected, August 2020

Of those who answered the question (n=211), nearly two-thirds indicated that text messages they

get from WIC are ‘Extremely helpful’ (32.2%, n=68) or ‘Very helpful’ (31.3%, n=66), while

another 25% indicated that the text messages are ‘Helpful (25.1%, n=53). 11.3% of respondents

indicated that text messages from WIC are ’Slightly helpful’ (n=24), and none responded that

text messages are ‘Not at all helpful’.

How do you feel about how often you have been getting text messages from WIC?

Pima Figure 3. Respondent preferences about message frequency, August 2020

Of those who answered the question (n=204), more than three-quarters indicated that they ‘get

about the right amount of messages’ (76.5%, n=156). Another twenty percent (n=41) said that

they ‘would like more messages’ while 3.4% (n=7) said that they ‘get too many messages’.

How do you feel about how long the text messages from WIC are?

Pima Figure 4. Respondent preferences about message length, August 2020

Of those who answered the question (n=200), 80. %, (n=160) indicated that ‘messages are about

the right length’ Another eighteen percent (n=36) said that ‘longer messages would be okay’

while 2.0% (n=4) said that ‘the messages are too long’.

What is the best time of day for you to receive a text message from WIC?

Pima Figure 5. Respondent preferences about message timing, August 2020

Of those who answered the question (n=200), nearly half (49.5%, n=99) selected ‘Anytime’ as

the best time of day to receive a text message from WIC. More than one-quarter (27.0%, n=54)

selected ‘Morning’ as the best time, while 18.5% (n=37) selected ‘Afternoon’. Five percent

(n=10) indicated that ‘Evening’ is the best time.

Of those who responded to the question, “I want to get messages from WIC about… (select all

that apply)”, (n=192), the most popular desired message topics were ‘Reminders to schedule

appointments’ (84.9%, n=163) and ‘When my benefits are going to expire’ (74.0%, n=142).

More than half of respondents selected ‘Recipes I can try with WIC foods’ (59.9%, n=115) and

‘Raising a healthy eater’ (53.1%, n=102), and slightly less than half selected ‘Infant and child

behavior’ (46.9%, n=90) and ‘How to schedule WIC appointments’ (44.2%, n=85). More than

one-third chose ‘How long I can receive WIC benefits’ (38.0%, n=73) and ‘Community

resources to meet the needs of my family’ (37.5%, n=72), while slightly fewer chose ‘How to

use WIC foods’ (30.7%, n=59). Few selected ‘Prenatal nutrition’ (15.6%, n=30) or ‘How to use

my eWIC card’ (8.9%, n=17).

Pima Table 2. Proposed message topics selected by respondents (n=192), August 2020

Topic

Reminders to schedule appointments

163

84.9

When my benefits are going to

expire

142

74.0

Recipes I can try with WIC foods

115

59.9

Raising a healthy eater

102

53.1

Infant and child behavior

46.9

How to schedule WIC appointments

44.3

How long I can receive WIC benefits

38.0

Community resources to meet the

needs of my family

37.5

How to use WIC foods

30.7

Prenatal nutrition

15.6

How to use my eWIC card

8.9

While slightly more than one-third of respondents (35.4%, n=68) selected one to three message

topics, some selected as many as all 11 options, with 27.1% of respondents (n=52) selecting six

or more of the topics.

Of those who responded to the question, “What food challenges have you experienced during the

COVID-19 pandemic? (select all that apply)”, (n=188), 20.2% (n=38) selected ‘No challenges’.

Of those who selected at least one response option other than ‘No challenges’ (n=150), more

than half (51.3%, n=77) selected ‘I cannot order WIC foods online’. More than forty percent

selected ‘WIC does not allow curbside pickup’ (46.0%, n=69), ‘Food costs more now than it did

before the pandemic’ (42.7%, n=64), and/or ‘It is hard to find WIC foods in the grocery store’

(40.7%, n=61). More than a quarter selected ‘I do not want to go to the grocery store because I’m

afraid of exposing myself or my family to the virus’ (38.0%, n = 57), ‘It is hard for me to get

enough food for my family’ (30.0%, n=45), and/or ‘It is hard for me to get to the grocery store’

(26.0%, n=39). Few selected ‘My family does not like WIC foods’ (4.0%, n=6) or ‘I do not

know how to use WIC foods’ (1.3%, n=2).

Pima Table 3. Challenges reported by respondents who selected at least one response option

other than ‘No challenges’ (n=150), August 2020

Challenge

Percentage (%)

I cannot order WIC foods online

51.3

WIC does not allow curbside pick up

46.0

Food costs more now than it did before the pandemic

42.7

It is hard to find WIC foods in the grocery store

40.7

I do not want to go to the grocery store because I’m afraid

of exposing myself or my family to the virus

38.0

It is hard for me to get enough food for my family

30.0

It is hard for me to get to the grocery store

26.0

My family does not like WIC foods

4.0

I do not know how to use WIC foods

1.3

While more than half of those who reported at least one challenge selected one or two challenges

(52.7%, n=79), nearly one-third of these respondents (30.7%, n=46) selected four or more

challenges.

Would you like to be able to send a text message to WIC and get a response from staff?

Pima Figure 6. Respondent preferences about two-way texting functionality, August 2020

Of those who answered the question (n=184), the vast majority (84.2%, n=155) indicated that

they would like to be able to send a text message to WIC and get a response from staff.

Thirty-seven respondents provided comments in response to, “Is there anything else that you

would like to share about the text messages you receive from WIC?”, ten of which were some

form of “No”. Of the remaining comments, ten were complimentary statements like “I like

getting text messages. It makes my life easier” and “I love the occasional reminders. It is

incredibly helpful to be reminded when benefits will expire.” Five commenters noted assistance

needs (“My baby needs formula”) or requested information about upcoming appointments (“I

would like to know when I need to schedule an appointment”). Two respondents noted that they

had received messages that were not aligned with the ages of their children (“Getting texts about

a baby trying solids when my son was already a year old seemed a bit odd to me”). Others made

various suggestions with regards to WIC benefits themselves (“It would be nice if I could use my

WIC card on self-serve registers”; “I would love it if we can order online or for curbside pickup

at a local store”).

Post-Implementation Staff Focus Group: The WICBuzz team held a focus group for Pima

County WIC staff on April 7th, 2021. Seven WIC staff, from a variety of clinics and with a

variety of job duties, participated in the focus group and provided feedback on their experience

with clients and the WICBuzz intervention. The local evaluator (Pima County staff person) asked

the staff about their experience over the last year at WIC, their thoughts about the WICBuzz

intervention, their opinions about the clients’ awareness of the intervention, the use of

technology in WIC, and WIC practices during the pandemic.

It was clear from the focus group that the COVID-19 pandemic played a big role in the WIC

program this past year. The pandemic led to changes in WIC services like remote services via

phone and the Zoom App, changes to the WIC food package, and other technological advances,

including an increase in text messaging to WIC clients. Since clients did not come to the office

during the 12-month WICBuzz intervention period, WIC staff did not play as large a role in

promoting WICBuzz as was originally intended. However, through the focus group we learned

that staff perceived that the WICBuzz messages did influence clients to renew their

certifications, call to schedule their next appointment, learn more about child development and

how to use their WIC foods.

“I’ve probably had one client who is relatively new just say that they had a lot of help

from specifically WICBuzz regarding I think it was their infant child they were breast-

feeding. So it was just general information regarding, I believe, the child’s next stage in

development, or just what to expect - how to breast feed and the changes that mom might

notice. And for her it was a game-changer – it was something that she really appreciated.

But I think just a combination of all of the technology we’ve been using has maybe

opened the eyes for our current clients on just how resourceful WIC is, how good WIC is,

in the sense of just getting more education and not just food.” – Pima County WIC staff

member

The focus group offered great ideas about how to forge better connections between WIC clients

and staff. They felt that if staff were more aware of the messages going out, they could reinforce

those messages in conversations with clients. Staff also showed interest in helping to develop the

messages and wanted to include topics like how to connect to the EzWIC app, the food list, the

Shopper’s Helpline and to provide additional ideas on how to use the WIC Foods.

Post-Implementation Client Survey: On June 3, 2021 the WICBuzz team sent out the post‐

implementation survey to 4,513 English and 577 Spanish‐speaking WICBuzz participants. The

team developed the survey in Qualtrics and sent it as a Bitly link in a WICBuzz message. A

reminder message was sent on June 10th and the survey closed June 18, 2021 at 11:59pm

Arizona Time.

Not only did Pima County WIC receive a larger response than anticipated, but additionally, the

survey responses came in faster than anticipated. Pima County WIC planned to keep the survey

open for one month in order to obtain representative sample of the population that received the

survey, with a target of 360 responses. However, with the rapid and voluminous response

received, the survey remained open for only two weeks. In all, the program received a total of

1,565 valid responses. Of those 1,368 were English (a 30% response rate) and 197 were Spanish

(a 34% response rate).

Pima Figure 7. Respondents by WIC client category, June 2021

Responses were analyzed overall (“all”), by survey language (English or Spanish), and by WIC

client category. Some respondent client category groupings were subsequently merged based on

breastfeeding status: Those who reported participating in one or more of the parent or infant

exclusively or partially nursing client categories (darker orange shaded bars in the graphics

above and below) were grouped as ‘Parent/infant nursing’ (n=204), while those who reported

participating in the parent postpartum and/or infant formula fed client categories (lighter orange

shaded bars) were grouped as ‘Postpartum parent/infant formula fed’ (n=295). Remaining client

categories (blue shaded bars) were not combined.

Pima Figure 8. Respondents by merged client category, June 2021

Pima Figure 9. Respondents reporting they learned something new from WICBuzz text messages,

June 2021

Responses to: “Do you feel you learned something new about how to understand and use

your WIC food benefits from the WICBuzz text messages?”

Of the 1,565 respondents, 1,523 (97%) provided a valid response to the follow‐up question “If

‘Yes’, can you share one or more things you learned?” Among these:

• Seven percent (n=110) specifically referred to ‘fruits’, ‘vegetables’, or ‘veggies’,

including 47 respondents (3%) who remarked on the change in value (to $35) of fruit and

vegetable benefits:

–

“

It helped me learn about fruits and vegetables

”

–

“

How to maximize my coupons for fruits at the store

”

–

“

I was notified that my benefits would include more veggies

”

–

“

The $35 increase in fruits and veggies during the summer

”

• Seven percent (n=103) mentioned ‘recipes’ or learning to ‘cook’, ‘make’, or ‘prepare’

foods:

–

“

Products I can use and recipes to make

”

–

“

How to make fun food for kids

”

–

“

It gives me ideas on what to make that is healthy and yummy.

”

–

“

Other ways to make snacks for my kids. (they love them)

”

• Six percent (n=96) noted learning about ‘updates’ or ‘increases’:

–

“

I like that I received updates and didn

’

t have to search an email or the

website.

”

–

“

Where WIC is allowed, updates on benefits

”

–

“

I learned new benefit updates for my child.

”

–

“

It keeps me updated.

”

• Five percent (n=74) mentioned ‘health’, ‘healthy’, or ‘healthier’:

–

“

’

ve learned what specific foods are healthy for my children.

”

–

“

How to make more meals that are healthy

”

–

“

Healthy recipes and information on the properties that some of the foods we

use have

”

–

“

I love the healthy eating tips.

”

• Five percent (n=71) referenced getting information about the WIC ‘app’ to ‘scan’ or

‘search’:

–

“

’

ve learned how to search food items on the app.

”

–

“

I love the app and the lists of what I can and cannot buy.

”

–

“

How to search for products by using the UPC code

”

–

“

Helps remind me to check the app for my balance

”

Other commonly expressed ideas included relying on the messages for ‘information’ or ‘info’

(n=61), learning about benefit eligibility at ‘farmer’s markets’ (n=57), and being reminded of

‘appointments’ (n=27).

Responses to: “Do you feel you learned something new about healthy lifestyles for you or

your children from the WICBuzz text messages?”

Of the 1,565 respondents, 1,523 (97%) provided a valid response to the follow‐up question “If

‘Yes’, can you share one or more things you learned?” While many offered general statements

about choosing, purchasing, and preparing ‘healthy’ or ‘healthier’ foods, others provided more

specific insights. Among these:

• Some respondents (n=144) specifically referred to ‘fruits’, ‘vegetables’, or ‘veggies’:

–

“

The importance of certain vegetables for our body

”

–

“

Which fruits and veggies are good for what

”

–

“

Different fruits and vegetables I can start giving my baby

”

–

“

How to add more vegetables to my child

’

s diet

”

• Others (n=71) mentioned ‘nutrition’ or referenced specific nutritional terms like

‘calcium’, ‘dairy’, ‘protein’, or ‘vitamins’, among others:

–

“

How products differ and how important nutrition is

”

–

“

’

ve learned new information about the different vitamins and minerals in

certain veggies.

”

–

“

The nutritional properties of foods and how to combine to obtain a balanced

diet”

–

“

To not give so much juice to kids even though it as fruit it contains sugar

”

• In some instances, multiple respondents alluded to topics addressed by specific WICBuzz

messages:

MESSAGE: “Feeling like a treat ‐ Make smoothies at home with WIC approved yogurt

and fruit. https://www.azhealthzone.org/recipes/strawberry‐yogurt‐shake”

–

“

I have learned how to make smoothies.

”

–

“

I received a text about purchasing frozen fruit and it really served more as a

reminder but my kids and I have been

making smoothies and popsicles with the frozen fruit

”

–

“

Yes new ways to sneak in fruits in smoothies

”

MESSAGE: “Did you know broccoli builds bones? Leafy green vegetables like broccoli,

spinach and kale contain lots of calcium.”

–

“

I learned kale was a very good food to eat with its many side benefits.

”

–

“

Ive learned that kale is a good source of calcium, fiber and protein.

”

–

“

I love the reminders to get my family leafy greens!

”

MESSAGE: “Don’t miss out on these superfoods from WIC! Berries, green leafy

veggies, avocados, sweet potatoes, eggs and beans are just a few SUPERFOODS that are

WIC approved.”

–

“

I learned more about superfoods.

”

–

“

The message about super foods made me more knowledgeable.

”

–

“

Yes. Superfoods!

”

Other commonly expressed ideas included ‘facts’ or ‘information’ (n=32), ‘balance’ (n=24), and

‘exercise’ or ‘activities’ (n=15).

Pima Figure 10. Respondent answers on helpfulness of text messages from WIC, June 2021

Pima Figure 11. Respondent knowledge of age through which a child can participate on WIC,

June 2021

Pima Figure 12. Respondents reporting the number of specific actions, of six choices provided,

that WICBuzz text messages encouraged them to take, June 2021

Pima Figure 13. Respondents reporting that receiving WICBuzz messages encouraged them to

take specific actions, June 2021

Pima Figure 14. Actions respondents reported taking in the past year with encouragement from

WICBuzz messages, and preferences for future message content, June 2021

Results Using Vendor Data

Over the course of the one-year Project period, Pima County WIC sent a total of 153,799

WICBuzz messages:

Pima Table 4. Number of messages sent, by quarter

Initially, in March of 2020 5,851 families were enrolled in WICBuzz. On November 30, 2020, an

additional 1,311 were added. At the end of the intervention period, taking into account attrition,

5,271 (79% of those enrolled over the course of the year) remained enrolled.

In terms of engagement with messages, ad hoc messages directed recipients to community

resources, such as the two messages in November 2020 displayed in the graphic below, resulted

in more clicks on the embedded links, than to WIC program or food-related links sent in

previous WICBuzz messages:

Pima Figure 15. Messages sent and message clicks, April – November 2020

QUARTER

NUMBER OF

MESSAGES SENT

FY 2020 Q2

17,735

FY 2020 Q3

32,396

FY 2020 Q4

37,194

FY 2021 Q1

39,622

FY 2021 Q2

26,852

TOTAL

153,799

4,528

4,556

1,576

2,116

4,777

1,631

1,409

1,391

2,215

1,435

1,755

2,311

1,466

5,623

689

720

112

138

180

November 27

October 23

October 1

August 13

June 18

May 28

April 17

Number of recipients

Number of clicks

2020

Results of long-term outcome evaluation:

HPRIL Table 2. Demographic Characteristics of Children 0-3 at Pima County WIC and Pinal County

WIC at baseline (T1) and implementation (T2). Statistically significant differences by group are in bold.

Baseline (T1)

Implementation (T2)

Pima

(n=7,452)

Pinal

(n=6,780)

Pima

(n=5,004)

Pinal

(n=4,599)

Category at start of

IBE

4.8

4.9

5.1

period

IBP

7.8

5.2

7.6

4.8

IFF

26.5

23.2

21.4

21.6

24.0

23.9

31.3

31.2

20.4

22.0

19.8

20.6

16.6

20.0

14.9

16.6

Number of WIC

One

32.1

29.8

34.6

30.5

participants

Two

36.2

34.0

35.7

33.9

Three or more

31.7

36.2

29.8

35.6

Race/ethnicity

American Indian or Alaska Native

4.2

3.9

4.2

Asian

2.4

0.9

2.3

Black or African American

14.1

10.3

10.7

Native Hawaiian or Other

Pacific Islander

1.5

1.0

1.3

White

85.7

90.6

85.8

89.8

Hispanic

67.9

52.0

68.7

Enrolled

TANF

0.7

1.3

0.9

SNAP

28.0

23.8

33.5

29.4

Medicaid

47.9

48.7

60.4

61.5

Primary language

other than English

85.1

92.6

85.7

92.5

Ever breastfed

Yes

78.1

70.5

77.7

21.9

29.5

22.3

Household size

0-4

61.9

52.1

35.6

47.1

Greater than or equal to 5

38.1

47.9

64.3

52.9

Telephone

1.8

2.7

1.2

2.1

Mobile

97.9

98.1

98.5

98.9

Do not text

0.1

0.2

Do not call

0.2

0.3

0.1

0.2

Participants can respond to more than one category so the total percentage may be greater than 100.

Overall, the participants in the comparison and innovation groups were similar for most

demographic characteristics at baseline (T1) and implementation (T2) and over time (HPRIL

Table 1). Although there were statistically significant differences in many characteristics

between groups during both time periods, there did not appear to be many clinically important

differences. On the other hand, there were potentially important differences by race or ethnicity

and primary language spoken in the household.

Pima County had fewer Hispanic participants at both T1 and T2 (52% at T1 and 54% at T2) than

Pinal County (68% at T1 and 69% at T2), fewer participants with a household language other

than English (85% at T1 and 86% at T2) than Pinal County (93% at T1 and 93% at T2), fewer

participants that live in a household with greater than or equal to five people (38% at T1 and

36% at T2) than Pinal County (48% at T1 and 47% at T2), and a greater proportion of

participants who have been ever breastfed (78% at T1 and T2) than Pinal County (70% at T1 and

66% at T2).

There were very few notable differences in participant characteristics in each group over time.

One notable difference was in SNAP participation: At T1, 28% of Pima County participants and

24% of Pinal County participants also participated in SNAP, while at T2, 34% of Pima County

WIC and 29% of Pinal County WIC were SNAP participants.

Recertification

While the crude, unweighted proportion of infants and children recertified in Pima was lower

than in Pinal during T1 (55.8% and 59.6%, respectively), the proportions recertified during T2

were significantly higher in Pima (66.0% and 62.9%, respectively) (HPRIL Figure 1). This was

also true when studying children. However, for infants, the difference between the recertification

rates between Pima and Pinal County was not significant. For sample sizes of these groups,

please see Appendix: HPRIL Table A.1.

HPRIL Figure 1. Proportion recertified (crude, unweighted) at baseline (T1) and

implementation (T2) overall, for infants, and for children at Pima and Pinal Counties. *p< 0.05.

Timeliness of Recertification

The median number of days between the end of the prior certification and recertification date

during the baseline period (T1) was 50 (IQR 22, 111) for Pima County and 38 (IQR 17, 91) for

Pinal County. In Pima County, median days ranged from 37 among C1s to 209 among IBPs; in

Pinal County, median number of days ranged from 40.5 among IBEs to 239 among IBPs (HPRIL

Figure 2). Of the children with a recertification date at Pima County (n=5,020) and Pinal County

(n=4,675), 43.3% and 47.4% were “timely” (i.e., less than or equal to 60 days after the end of

their last certification period), respectively.

59.6

62.9

56.5

63.3

61.1

62.7

55.8

66.0

52.3

61.4

58.0

68.3

100

T1 T2 T1 T2 T1 T2

Overall Infants Children

Pinal Pima

HPRIL Figure 2. Number of days between end of certification and recertification by participant

category at Pima and Pinal Counties during baseline (T1) (truncated at 100 days)

The median number of days between the end of the prior certification and recertification date

during the implementation period (T2) was 50 (IQR 22, 111) for Pima County and 38 (IQR 17,

91) for Pinal County. In Pima County, median number of days ranged from 25 among IFFs to

171 among IBPs; in Pinal County, median number of days ranged from 22 among IBEs to 115

among IBPs (HPRIL Figure 3). 56.4% of recertifications at Pima County during implementation

were “timely” (i.e., less than 60 days after the end of the last certification period), while 64.1%

of recertifications at Pinal County were timely.

HPRIL Figure 3. Number of days between end of certification and recertification by participant

category at Pima and Pinal Counties at implementation (T2) (truncated at 100 days)

The crude, unweighted proportion of participants who were timely recertified was significantly

lower in Pima than Pinal during T1 and T2 overall (48.5% vs. 52.8% at T1 and 59.5% vs. 66.2%

at T2) and for infants (47.8% vs. 52.1% at T1 and 68.4% vs. 74.8% at T2) and children (48.9%

vs. 53.1% at T1 and 55.5% vs. 62.3% at T2) (HPRIL Figure 4).

HPRIL Figure 4. Proportion timely recertified (crude, unweighted) at baseline (T1) and

implementation (T2) overall, for infants, and for children at Pima and Pinal. *p< 0.05.

52.8

66.2

52.1

74.8

53.1

62.3

48.5

59.5

47.8

68.4

48.9

55.5

100

T1 T2 T1 T2 T1 T2

Overall Infants Children

Pinal Pima

Retention

The crude, unweighted proportion of participants who were retained was significantly lower in

Pima than Pinal during T1 overall and for infants and children (HPRIL Figure 4). At T2, the

percent of participants retained was significantly higher in Pima County than Pinal County

overall (66.4% vs. 63.4%, respectively) and for children (69.9% vs. 65.2%, respectively) but not

for infants.

HPRIL Figure 4. Proportion retained (crude, unweighted) at baseline (T1) and implementation

(T2) overall, for infants, and for children. *p< 0.05.

Participation (i.e., benefit issuance)

The median number of months of benefit issuance for all participants at Pima and Pinal during

T1 was 9 (out of 12) (IQR 4, 12) (HPRIL Table 3). During T2, the median number of months of

benefit issuance was 11 for Pima (IQR 7, 12) and 10 for Pinal (IQR 5, 12). The average

percentage of the cohort issued benefits throughout the year was higher in T2 (76.2% in Pima

and 70.7% in Pinal) than T1 (62.9% in Pima and 64.6% in Pinal).

HPRIL Table 3. Benefit Issuance in Pima and Pinal Counties during Baseline and

Implementation Periods

Agency/Group

Baseline (T1)

Implementation (T2)

Pima

Pinal

Pima

Pinal

Months of benefit issuance (median,

IQR)

9 (4, 12)

11 (7, 12)

10 (5, 12)

Percent of cohort issued benefits (%)

62.9

64.6

76.2

70.7

60.7

63.4

55.9

59.3

63.2

65.2

56.9

66.4

51.6

59.5

60.3

69.9

100

T1 T2 T1 T2 T1 T2

Overall Infants Children

Pinal Pima

The crude, unweighted proportion of the samples that had continuous rates of benefit issuance

(11-12 months) was significantly higher in Pinal than Pima during T1 overall and for infants and

children (HPRIL Figure 6). Conversely, during T2 participants in Pima County had significantly

higher rates of benefit issuance than Pinal County overall (55.9% and 49.0%, respectively), for

infants (50.4% and 46.8%, respectively), and for children (58.8% and 50.1%, respectively).

HPRIL Figure 6. Proportion with 11-12 months of benefit issuance (crude, unweighted) at

baseline (T1) and implementation (T2) overall, for infants, and for children. *p< 0.05.

Adjusting for differences between the cohorts

The greatest differences in characteristics between the Pima County cohorts at T1 and T2 were

Medicaid participation, SNAP participation, and being a category C1 (absolute standardized

differences of 0.26, 0.13, and 0.17, respectively) (HPRIL Figure 7). The absolute standardized

difference mean across all characteristics was 0.05. After propensity score weighting, these

absolute standardized differences were all reduced to below 0.05. The absolute standardized

difference mean after weighting was 0.009. For ASDs for infants and children separately, please

see Appendix: HPRIL Table A.2.

40.0

49.0

39.5

46.8

40.2

50.1

36.4

55.9

37.2

50.4

35.9

58.8

100

T1 T2 T1 T2 T1 T2

Overall Infants Children

Pinal Pima

HPRIL Figure 7. Absolute Standardized Differences in Characteristics (unweighted and

weighted) at T1 vs. T2 in Pima County Overall

The greatest differences in characteristics between Pima County and Pinal County at T1 were

household size greater than or equal to five (ASD 0.20), being White (ASD 0.32), and being

Hispanic (ASD 0.23) (HPRIL Figure 8). The absolute standardized difference mean across all

characteristics was 0.09. After propensity score weighting, these absolute standardized

differences were all reduced to below 0.05. The absolute standardized difference mean after

weighting was 0.01. For ASDs for infants and children separately, please see Appendix: HPRIL

Household size greater than 5

Number of WIC participants in

the HH >/=2

Race: American Indian/

Alaskan Native

Race: Asian

Race: Black

Race: Native Hawaiian/Pacific

Islander

Race: White

Hispanic

Multiracial

Need for a translator

Participate in TANF

Participate in SNAP

Participate in Medicaid

Ever breastfed

Has a telephone

Has a mobile phone

Do not text

Do not call

Infant

Category C1

Category C2

Mean Difference

-0.2000 0.0000 0.2000 0.4000 0.6000 0.8000

Absolute standardized difference (ASD)

Unweighted ASD

Weighted ASD

Table A.2.

HPRIL Figure 8. Absolute Standardized Differences in Characteristics (unweighted and

weighted) at T1 at Pima County vs. T1 at Pinal County overall: Infants and Children

The greatest differences in characteristics between Pima County at T1 and Pinal County at T2

were household size greater than or equal to five (ASD 0.18), being White (ASD 0.28), being

Hispanic (ASD 0.23), participating in Medicaid (ASD 0.28), and being ever breastfed (ASD

0.27) (HPRIL Figure 9). The absolute standardized difference mean across all characteristics was

0.11. After propensity score weighting, these absolute standardized differences were all reduced

to below 0.05. The absolute standardized difference mean after weighting was 0.01. For ASDs

for infants and children separately, please see Appendix: HPRIL Table A.2.

Household size greater than 5

Number of WIC participants

in the HH >/=2

Race: American Indian/

Alaskan Native

Race: Asian

Race: Black

Race: Native Hawaiian/Pacific

Islander

Race: White

Hispanic

Multiracial

Need for a translator

Participate in TANF

Participate in SNAP

Participate in Medicaid

Ever breastfed

Has a telephone

Has a mobile phone

Do not text

Do not call

Infant

Category C1

Category C2

Mean Difference

-0.2000 0.0000 0.2000 0.4000 0.6000 0.8000

Absolute standardized difference (ASD)

Unweighted ASD

Weighted ASD

HPRIL Figure 9. Absolute Standardized Differences in Characteristics (unweighted and

weighted) at T1 at Pima County vs. T2 at Pinal County overall: Infants and Children

Outcome analysis after weighting

Using the unweighted data and a crude (unadjusted) analysis, being at Pima County was

associated with a 6.8% increase in recertification overall, a 2.3% increase in infants, and an 8.7%

increase in children (all statistically significant) (HPRIL Figure 10). Using the weighted data

and two adjusted analysis techniques, being at Pima County was associated with a 6.7-8.3%

increase in recertification overall, a 4.7-5.1% increase in recertification among infants, and a 7.6-

7.9% increase in recertification among children (all statistically significant). For the beta

Household size greater than

Number of WIC participants

in the HH >/=2

Race: American Indian/

Alaskan Native

Race: Asian

Race: Black

Race: Native Hawaiian/Pacific

Islander

Race: White

Hispanic

Multiracial

Need for a translator

Participate in TANF

Participate in SNAP

Participate in Medicaid

Ever breastfed

Has a telephone

Has a mobile phone

Do not text

Do not call

Infant

Category C1

Category C2

Mean Difference

-0.2000 0.0000 0.2000 0.4000 0.6000 0.8000

Absolute standardized difference (ASD)

Unweighted ASD

Weighted ASD

coefficients and 95% confidence intervals, see HPRIL Table 4. For the sample sizes of each of

these groups, see Appendix HPRIL Table A.3.

HPRIL Figure 10. Percentage point differences in recertification between Pima and Pinal

Counties overall, for infants, and for children using three models: Crude (unweighted) and two

weighting analysis techniques: A1: PSW-DID using logit for propensity score weighting (PSW)

and ordinary least squares (OLS) for DID; A2: PSM-DID using Kernel for propensity score

matching (PSM) and probit for DID with repeated cross-sectional option. *p< 0.05.

Using the unweighted data and a crude (unadjusted) analysis, being at Pima County was not

associated with any statistically significant differences (HPRIL Figure 11). Using the weighted

data and two adjusted analysis techniques, being at Pima County was only associated with one

statistically significant difference in timely recertification: A 5.8% decrease among infants using

the adjusted model 2 (A2). For the beta coefficients and 95% confidence intervals, see HPRIL

Table 4. For the sample sizes of each of these groups, see Appendix HPRIL Table A.3.

6.8

6.7

8.3

2.3

5.1

4.7

8.7

7.6

7.9

-7.0

-2.0

3.0

8.0

13.0

Crude A1‡ A2‡ Crude A1 A2 Crude A1 A2

Overall Infants Children

Difference in Differences

* *

HPRIL Figure 11. Percentage point differences in timely recertification between Pima and Pinal

Counties overall, for infants, and for children using three models: Crude (unweighted) and two

weighting analysis techniques: A1: PSW-DID using logit for propensity score weighting (PSW)

and ordinary least squares (OLS) for DID; A2: PSM-DID using Kernel for propensity score

matching (PSM) and probit for DID with repeated cross-sectional option. *p< 0.05.

Using the unweighted data and a crude (unadjusted) analysis, being at Pima County was

associated with a 6.8% increase in retention overall, a 4.5% increase in infants, and a 7.6%

increase in children (all statistically significant) (HPRIL Figure 12). Using the weighted data and

two adjusted analysis techniques, being at the innovation clinics was associated with a 7.4-8.6%

increase in retention overall, an 8.1-8.7% increase in retention among infants, and a 6.1-7.2%

increase in retention among children (all statistically significant). For the beta coefficients and

95% confidence intervals, see HPRIL Table 4. For the sample sizes of each of these groups, see

Appendix HPRIL Table A.3.

-2.4

-3.2

-2.9

-2.1

-5.9

-5.8

-2.6

-2.4

-2.8

-7.0

-2.0

3.0

8.0

13.0

Crude A1 A2 Crude A1 A2 Crude A1 A2

Overall Infants Children

Difference in Differences

HPRIL Figure 12. Percentage point differences in retention between Pima and Pinal Counties

overall, for infants, and for children using three models: Crude (unweighted) and two weighting

analysis techniques: A1: PSW-DID using logit for propensity score weighting (PSW) and

ordinary least squares (OLS) for DID; A2: PSM-DID using Kernel for propensity score

matching (PSM) and probit for DID with repeated cross-sectional option. *p< 0.05.

Using the unweighted data and a crude (unadjusted) analysis, being at Pima County was

associated with a 10.5% increase in continuous benefit issuance overall, a 6.0% increase in

continuous benefit issuance in infants, and a 13.0% increase in continuous benefit issuance in

children (all statistically significant) (HPRIL Figure 13). Using the weighted data and two

adjusted analysis techniques, being at Pima County was associated with a 9.7-11.1% increase in

continuous benefit issuance overall, a 6.2-6.9% increase in continuous benefit issuance among

infants, and an 11.3-12.6% increase in continuous benefit issuance among children (all

statistically significant). For the beta coefficients and 95% confidence intervals, see HPRIL

Table 4. For the sample sizes of each of these groups, see Appendix HPRIL Table A.3.

6.8

7.4

8.6

4.5

8.7

8.1

7.6

6.1

7.2

-7.0

-2.0

3.0

8.0

13.0

Crude A1 A2 Crude A1 A2 Crude A1 A2

Overall Infants Children

Difference in Differences

HPRIL Figure 13. Percentage point differences in continuous benefit issuance between Pima and Pinal Counties

overall, for infants, and for children using three models: Crude (unweighted) and two weighting analysis

techniques: A1: PSW-DID using logit for propensity score weighting (PSW) and ordinary least squares (OLS) for

DID; A2: PSM-DID using Kernel for propensity score matching (PSM) and probit for DID with repeated cross-

sectional option. *p< 0.05.

HPRIL Table 4. Difference-in-Difference Weighted Results for Recertification, Timely Recertification,

Retention, and Benefit Issuance Using Crude and Three Adjusted Models Overall and for Infants and

Children

Overall

Infants

Children

Beta

95% CI

beta

95% CI

beta

95% CI

Recertification (crude,

weighted)

0.068

0.043

0.094

0.023

-0.021

0.067

0.087

0.056

0.118

Timely recertification

(crude, unweighted)

-0.024

-0.057

0.0097

-0.021

-0.078

0.0361

-0.026

-0.066

0.0152

Retention (crude,

unweighted)

0.068

0.043

0.093

0.045

0.001

0.089

0.076

0.045

0.107

Benefit issuance (crude,

unweighted)

0.105

0.079

0.130

0.060

0.016

0.104

0.130

0.098

0.161

Recertification

Model A1

0.067

0.040

0.094

0.051

0.003

0.100

0.076

0.046

0.107

Model A2

0.083

0.0575

0.1085

0.047

0.0039

0.0901

0.079

0.0476

0.1104

Timely recertification

Model A1

-0.032

-0.071

0.008

-0.059

-0.122

0.005

-0.024

-0.074

0.027

Model A2

-0.029

-0.062

0.004

-0.058

-0.115

-0.001

-0.028

-0.069

0.013

Retention

Model A1

0.074

0.047

0.101

0.087

0.039

0.135

0.061

0.030

0.093

Model A2

0.086

0.0605

0.1115

0.081

0.0359

0.1261

0.072

0.0406

0.1034

Benefit issuance

Model A1

0.097

0.069

0.125

0.069

0.021

0.117

0.113

0.078

0.147

Model A2

0.111

0.0855

0.1365

0.062

0.0189

0.1051

0.126

0.0946

0.1574

10.5

9.7

11.1

6.0

6.9

6.2

13.0

11.3

12.6

-7.0

-2.0

3.0

8.0

13.0

Crude A1 A2 Crude A1 A2 Crude A1 A2

Overall Infants Children

Difference in Differences

Discussion

Overall, the results suggest that WICBuzz positively affected recertification and retention of

children in WIC as well as continuous benefit issuance. When examining the proportion of those

timely recertified, the results, were non-significant. In general, the results were stronger for

children than for infants in stratified analyses.

The positive impact of WICBuzz was consistently observed in crude DID analyses as well as in

two different adjusted models. Although there were differences in participant characteristics

between groups and over time, and we were able to successfully balance these differences

through weighting these adjustments only minimally affected the estimated effects of the

innovation.

The internal validity of the analysis also depends on whether there were operational differences

between the clinics over time. The implementation period occurred during the COVID-19

pandemic, and WIC operations across the United States transitioned to virtual appointments by

phone. Differences in clinic operations and staffing during the implementation period were likely

because of COVID-19. Conversations with the Pinal County WIC leadership may provide

insights on potential differences, but there are no known differences in operations that would

bias the results presented here.

Survey results help to clarify the impact evaluation results. Of those who responded to the mid-

implementation client survey (n=230, a 5% response rate), the overall sentiment with regards to

current text messaging from WIC, including message helpfulness, length, and frequency, was

positive. The nature of the responses was likely due in part to the survey sample, which

encouraged responses from WICBuzz participants who view the campaign more positively than

those who did not (or could not) receive the survey.

When asked broadly about the text messages they receive from WIC, respondents often alluded

to content that was not a part of the WICBuzz campaign, such as appointment and benefit

expiration reminders. Understanding that respondents view ‘text messages from WIC’ as a

combination of WICBuzz messages and messages from other sources like the Arizona

Department of Health Services (ADHS) highlights a potential opportunity to promote the

branding of WICBuzz in order to differentiate this campaign from other outreach activities.

Identifying the topics that respondents are interested in receiving messages about, on the other

hand, can assist WICBuzz managers to plan future WICBuzz content in a way that is

complementary to, rather than competitive with, other channels, like ADHS and the WIC-related

application (EZWIC app) used in Arizona. For instance, while reminders to schedule

appointments might be accomplished through EZWIC, awareness of the app and its purposes

could potentially be promoted through combined state-level messaging and localized WICBuzz

outreach.

Responses also yield insight into the unique challenges that WIC participants in Pima County

have encountered during the pandemic. 80% of respondents selected at least one of the

challenges listed as response options, with one-third of these respondents selecting four or more

challenges. While many of these pandemic-specific challenges are the result of federal or state

policy and cannot be addressed entirely by a text messaging campaign, these responses provide

WICBuzz managers with a sense of the appropriateness of the message content originally

developed (prior to the pandemic) for the full course of the one-year campaign. Beyond

reconsidering content for planned messages, managers might also use these findings to propose

policy recommendations to state or federal officials in order to address pandemic-specific issues

facing WIC populations locally and across the country.

Finally, survey responses can provide WICBuzz managers with ideas for how the text messaging

platform might be useful beyond preprogrammed messaging. Eighty-four percent of valid

responses (155 of 184) indicated that the respondent would like to be able to send a text and

receive a response from staff. While one Pima County WIC staff member has been responding to

inquiries received via text in response to WICBuzz messages, the overwhelmingly positive

response to this item allows WICBuzz managers to anticipate the increased workload that would

likely accompany any movement to actively promote communication via text message between

WIC participants and staff through the WICBuzz platform. In this case, adequate logistical

preparation for such a potential expansion would require additional staff time and training and

consequential, funding in order to accommodate the anticipated influx of incoming messages

from participants.

Of those who responded to the post-implementation client survey, message recipients reported

that WICBuzz text messages added value to the WIC client experience. This was marked by

overwhelmingly positive responses to questions about changes in knowledge of WIC services

and eligibility, ways to maximize benefits, and nutrition education. The Pima County WICBuzz

team interprets these responses, in conjunction with low levels of clients opting out of receiving

the WICBuzz messages, to reflect acceptability of this type of messaging campaign among WIC

clients who had previously indicated that they would like to receive text messages from WIC

(i.e., a majority of WIC clients).

Further, respondents were able to reinforce responses to closed-ended questions by providing

descriptive answers to the survey’s open-ended questions that reflected examples of specific

knowledge gained and self-reported behavior change. Some responses to open-ended questions

even referenced specific message content, such as certain foods and/or recipes featured in

WICBuzz messages, which can be interpreted as demonstrating recipient retention of content

from the WICBuzz text message campaign itself.

Limitations

Message development needs to be ongoing. While the WICBuzz team started with 200

potential messages, we selected a finite number to send through the various levels of approval

required to use them for this intervention, including approval by the USDA. The process took

several months to complete. The WICBuzz team did not have much flexibility in changing the

message content during the intervention. Continuing with the intervention beyond the 12 months

also proved difficult due to the need for additional messages so that parents/guardians would

continue to receive unique messages.

Changing WICBuzz client category in the EMS Texting Platform. One challenge of using

the texting platform was that children were entered into a particular group, based on their age, at

the beginning of the intervention. If an infant turned age 1 seven months into the intervention

their parent/guardian would continue to receive the infant messages for the remainder of the

intervention. There was not a means in the system to automatically move a child from one age

group to another. In order to make the move, the WICBuzz manager would have to ‘Opt out’ the

child from one group and ‘Opt in’ to the next age group.

Low response rate to mid-Project client survey. The response rate to the mid-Project survey

was lower than the target response rate, meaning that survey findings are not generalizable to the

overall population of Pima County WIC clients who receive WICBuzz text messages. Further,

the low response rate to the Spanish language survey means that responses from the client sub-

population that received this version of the survey cannot be meaningfully compared to the sub-

population that received the English language version.

Potential sampling bias of surveys due to distribution channel. The link to the online-only

mid-Project and post-implementation client surveys were distributed via text message only. As a

result, WICBuzz participants with cell phones without internet access would likely not have been

able to open the survey link. The number and proportion of WICBuzz participants without

smartphone access is unknown, but this sub-population may differ from the client population

with smartphone access with regards to income level and other factors that are relevant to the

goals of the WICBuzz campaign.

Additionally, these samples contained only those who had not opted out of the WICBuzz

campaign. As a result, findings do not include the perspectives of the proportion of the initial

cohort of WICBuzz participants who opted out of the campaign or otherwise stopped receiving

WICBuzz campaign text messages due to inactive phone numbers. Excluding those who opted

out of the campaign from these samples potentially skews the overall tenor of responses in a

direction favorable to the campaign by omitting the voices of those who chose to no longer

participate in it.

Breadth of survey item focus. Mid-Project client survey items were designed to inquire about

WICBuzz participant feedback on any text messages they receive from WIC, including both

Pima County WIC and the Arizona Department of Health Services (ADHS), rather than being

focused exclusively on WICBuzz messages. This design decision was made due to concerns that

some participants might not: 1) associate WICBuzz text messages with the branded WICBuzz

campaign; or 2) be able to differentiate meaningfully between WICBuzz and non-WICBuzz text

messages from WIC in responding to the survey items. As a result, responses cannot be

interpreted as being solely reflective of respondent feedback on WICBuzz alone, but on the

combined set of text messages they receive from WIC.

Lessons Learned

Despite the limitations disused above, the WICBuzz team learned that texting with WIC clients

is a highly effective means of communication and that there is a great demand for two-way

texting. Initially there was concern that clients may become disinterested in the intervention if

texts were too long or too frequent, but client feedback indicated that participant tolerance for

length and frequency of message was higher than anticipated.

Future Implications

Scalability. Developing and managing a drip marketing campaign, like WICBuzz is relatively

low cost and manageable within the context and staffing levels of a medium to large-sized WIC

clinic. An intervention, such as this one, would likely be even more effective when run from a

State WIC office, as the cost of this intervention is scaled to the number of texts sent, with the

price per individual text sent is cheaper for the larger packages. Running this intervention from a

regional office could be effective, but the State level might be ideal so that messages could be

tailored to the population being served, taking into account language needs, regional foods and

culture.

Data collection. Future data collection efforts targeting WIC participants in Pima County should

mitigate sampling challenges by continuing to offer incentives for participation and investing in

outreach to increase participation among sub-populations of interest, such as non-English

speakers and those who choose to opt out of campaigns like WICBuzz. Subsequent data

collection efforts should be directed to answer more nuanced research questions regarding

participant (and non-participant) perceptions of the WICBuzz campaign than these broad surveys

were equipped to explore.

Plans for sustainability

The Pima County Health Department WIC program has continued the WICBuzz intervention. At

this point in time, however, there is a need to develop additional messages that are unique and

relevant to the population being served. The WICBuzz Team will share results of the

intervention with the Arizona WIC State office and discuss the possibility of scaling the

intervention to all WIC participants in Arizona. The WICBuzz Team would also be interested in

continuing to work with the HRPIL team to pilot WICBuzz either in other areas of the State of

Arizona or other WIC agencies throughout the United States.

Recommendations

The PCHD WICBuzz team recommends that the final report be shared with other WIC agencies.

This final report can be used as a how-to guide to start a similar intervention at other WIC

clinics. The WICBuzz team also recommends creating a digital space for a repository of

messages that can be used by WIC programs in their implementation of this program or others

like it. Taking into account the particular language and culture of an area is important, and not

all messages will resonate with all populations, but many messages have general appeal to

parents and guardians.

Appendix

a. Logic Model

b. Active Listening in Workplace Situations Handout

c. Messages by Category and Outcome

d. Messages by Category and Month

e. WICBuzz Staff Training A

f. WICBuzz Staff Training B

g. WICBuzz Mid-Project Client Survey

h. WICBuzz Poster

i. Pre-launch Client Survey

j. Virtual Focus Groups Protocol

k. Post Implementation Survey

l. Evaluation Questions and Indicators

m. HPRIL Table A.1.

n. HPRIL Table A.2.

o. HPRIL Table A.3.

Eppes, E, Kang, Y, Caulfield, L, Gross, S. Hopkins Participant Research Innovation Laboratory for Enhancing

WIC Services (HPRIL) Baseline Period Characteristics Report. 2022.

Eppes, E, Kang, Y, Caulfield, L, Gross, S. Hopkins Participant Research Innovation Laboratory for Enhancing

WIC Services (HPRIL) Implementation Period Characteristics Report. 2022.

Stuart EA, Huskamp HA, Duckworth K, et al. Using propensity scores in difference-in-differences models to

estimate the effects of a policy change. Health Services and Outcomes Research Methodology. 2014;14(4):166-182.

doi:10.1007/S10742-014-0123-Z/TABLES/5.

Villa JM. diff: Simplifying the estimation of difference-in-differences treatment effects. Stata Journal.

2016;16(1):52-71. doi:10.1177/1536867X1601600108.