Introduction
• “Not everything that
matters can be measured
and not everything that
can be measured
matters.”
• Decide what matters most
and whether (and how) it
can be measured.
• Why is it difficult for family literacy programs to demonstrate
effectiveness?
Limited staff capacity and resources for data collection
Limited training and knowledge about data collection
Scarce funding for research
Changing standards for rigorous research and evaluation
Differing measures across programs and funders
• High-quality data are crucial for telling your story and improving
your program to ensure that participants are learning.
Introduction
Goals of Session
• Explore data that can and should be collected to measure
family literacy outcomes and create an environment of
continuous improvement.
• Examine data collection tools.
• Discuss data collection challenges and how to address them.
• Take-aways: tips and tools for high-quality data collection.
• Suggestions for addressing challenges in data collection.
What interested you
about this session?
Why do we collect data?
• External
• To know if we are meeting outcomes
• To obtain information about meeting our goals
• To keep records for accountability
• To disseminate information about achievements to stakeholders
and funders
• Internal
• To identify and solve problems
• To make decisions about program design and implementation
• To improve programs
Data for Continuous Improvement
Why Data Matters: Part 2
• Session at 2017 NCFL in Tucson, AZ asking participants
about:
• Data collected
• Tools used
• Challenges
• Data want to collect that they do not collect currently
• Mapped data collection framework for Family Literacy
Family Literacy Outcomes and Data Collection Framework
CHILDREN
Family Literacy Outcomes and Data Collection Framework
CAREGIVERS
Family Literacy Outcomes and Data Collection Framework
PROGRAM IMPROVEMENT
Family Literacy Outcomes and Data Collection Framework
Outcomes Identified in Part I Session
Quantitative and Qualitative data
• Important to use both – each has rich information
• Some can include both data types (e.g., survey with open-ended
questions)
Quantitative Data Qualitative Data
• Can be counted
• Statistical analysis
• Surveys, experiments
• Standardized assessments
• Interviews, focus groups,
observations, surveys
• Photographs, video
General Tips for Data Collection
• Start thinking about the data before you start collecting it
• If you collect it, use it—if you don’t use it, don’t collect it
• Make sure those collecting the data are trained and
understand what they are doing
• If gathering qualitative data, do not write assumptions or
opinions
• Keep good notes and be organized
• Keep information confidential
Quantitative Data Sources
• Pre-post standardized tests scores
• Pre-post survey data
• Attendance and participation rates
• School records: grade, promotion, disciplinary
actions/referrals, entry to postsecondary
• Employment placement and retention data
• Demographic data
Examples of Quantitative Tools
• PALS Pre-K (standardized)
• Pennsylvania e-Data & End of School Year Report
• Demographic data sample survey questions
Sample Pre-post Survey Question
If your child/ren attended school last school year, did you visit the school more than
just to drop off or pick up the child/ren?
( ) Yes
( ) No
If "yes," what did you do at your child/ren's school? Please check all that apply.
[ ] For a conference or informal talk with child/ren’s teacher, director or principal
[ ] To volunteer in the school office, cafeteria or library
[ ] To volunteer in child/ren's classroom
[ ] To volunteer for school project or trip
[ ] To observe classroom activity
[ ] To attend a child's school event (e. g., play, art show, party)
[ ] To attend a parent advisory meeting or parent-teacher organization (e.g., PTA)
[ ] Other. Please describe:
Sample Pre-post Questions – Technology
How comfortable are you in your abilities to do the following:
Not at all
comfortable
Uneasy
Somewhat
comfortable
Comfortable
Extremely
confident
in abilities
Turn on and turn off a computer
Use a keyboard and/or mouse
Load software
Have skills necessary to utilize
different programs (e.g.,
Windows, Excel, Publisher)
Use the Internet to obtain
information
Surveys
Tips
• Know something about participants’
backgrounds before developing
questions
• Clarify goals and scope of study
before creating questions
• Make surveys accessible for multiple
languages (based on participants’
linguistic backgrounds)
• Make open-ended questions require
a longer response (rather than
yes/no questions)
• Combine quantitative demographic
data with qualitative narrative data
Challenges
• Survey responses might differ
depending on day and time – use
with qualitative methods
• Even with open-ended questions,
participants may not provide in-
depth responses – use interviews
and focus groups for more detailed
responses
• Participants might have different
digital literacy levels – opt for
online and paper-based surveys
• Surveys are based on self-report
reliability
Qualitative Data Collection Sources
• Formative assessments, teacher-made tests
• Interviews and focus groups
• Portfolios
• Logs
• Journals
• Artifacts
• Observations
• Case notes
• Self-appraisals (e.g., Foundation Skills
Framework, Digital Literacy Checklist)
Examples of Qualitative Tools
• Formative assessment
• Digital literacy survey
• Reading apps for children
• Logs and journals
• Family Pathways ILA Log
• TFL Home Activity Learning and Family Mentoring
Log
• Portfolios (children and adults)
• Foundation Skills Framework
• Pennsylvania Indicators of Program Quality
Formative Assessments
Tips
• When creating the test, consider:
• Matching test to your purpose
• Appropriateness for your population
• Students’ and parents’ strengths and
weaknesses
• Create multi-modal assessments
(written, visual, auditory, etc.)
• Make assessments open to linguistic
and cultural diversity
• Use methods such as backward design
to create assessments that examine
participants’ current skills and what
you hope they attain/learn
Challenges
• Local/individually-created assessments
might contradict funders’ assessments –
use these contradictions as a learning
tool for participants
• Linguistically and culturally diverse
assessments might require more
thought, time, and translation
• Assessments may result in skewed data
(based on participants’ language,
comfort with tests, etc.) – use other
methods to gain comprehensive
understanding of learners
• Formative assessments require follow-
up – implement similar assessments
throughout the year
Portfolios
Tips
• Ask for items that relate to
main research questions or
focus of practice
• Include multiple “artifacts”
(e.g., photos, assessments,
lesson plans, assignments)
• Use data from portfolios to
guide follow-up interviews
and focus groups
• Ensure portfolio items
include linguistically and
culturally diverse artifacts
Challenges
• Portfolio artifacts might
vary in size, quality,
quantity difficult to
compare across sites
• Portfolio documents
can exhibit practices
that are not necessarily
taking place (and vice
versa) – use portfolio
documents with other
qualitative methods
Learning/Service Logs
Tips
• Logs draw from participants’ own
documentation (participant
becomes the researcher)
• Include qualitative and quantitative
prompts within the logs
• Like surveys, create open-ended
responses to logs (rather than
yes/no questions)
• Depending on scope of research
and practice, include questions
directed at reflection over time
Challenges
• Participants might not always
complete logs- check in consistently
to ensure log completion
• Accessibility to technology can
impact when and how logs are
completed
• Participants might have differing
digital literacy levels- opt for online
and paper-based learning logs
• Components of logs are based on
self-perception which can decrease
reliability
Journals
Tips
• Useful for assessing not only what is
being said and written, but HOW it is
being said and written
• Provide both prompts and open
responses to incite creativity while
also providing structure
• Use individual and group journaling
(writing alone and together)
• Allow for multiple literacies (language,
visual, etc.)
• Can be completed on paper or using
technology
• Great tool for analyzing and learning
from learners’ narratives
Challenges
• If learners are uncomfortable with
writing, journals can be difficult to
implement- make space for written,
spoken, and drawn journal entries
• Prompts that are too structured or
unstructured can hinder
substantive responses
• Journaling with diverse linguistic
learners may require translation of
written work
Observations
Tips
• Choose the scope of what and who you
will be observing; create an observation
guide
• Within that scope, be aware of what is
said and NOT said, done and NOT done
• Include both small/minor details and
large/major details in observations
• Write notes during the observation or
directly after
• Pay attention to space, time, visuals, body
language, etc. and what people say
• Objectively describe what you see; don’t
place a value judgment
Challenges
• Different people notice different
things – compare observations after
the observation
• Observations only tell one part of a
story – use in tandem with other
data methods
• Observation relies on chance
(demographics might change from
day to day) – do follow-up
observations to get a more
complete view of what is happening
Longitudinal Research
Tips
• Decide what you want to measure
(what long-term change do you want
to see?)
• Outcome must be aligned with
program focus (logic model)
• Consider using existing measures
• Choose sample carefully (e.g., all new
participants vs. only those who finish)
• Use a comparison group if possible
• Decide time frame (6 months, 1 year,
etc.) and # of data points
Challenges
• $$ – partner with university
researchers to seek funding
• Participants move, change
phone – get contact info for 3-
4 people who will always know
how to reach them
• Participants lose interest –
offer $ incentives, build
interest in study
• Helpful to have research
experience – collaborate with a
university
Longitudinal Research
• “The best way to
understand something
is to try to change it”
(Kurt Lewin)
• Iterative process of
posing and solving
problems
Action Research
Tips
• Identify what isn’t working and a
potential—and feasible—solution
• Identify a researchable question
(focused, specific, measurable)
• Collect data that will show whether or
not solution was successful
• How will you know if it worked?
• Use multiple measures (qualitative and/or
quantitative)
• Reflect on results and plan next steps
• Involve participants
Challenges
• Can be time-consuming
• Use data you already collect
• Use AR for multiple purposes
(assessment, evaluation, reports to
funders)
• Investment is worth it – improve
program and outcomes
• Don’t jump to solutions before
identifying the problem
• Some solutions aren’t feasible –
consider time, staffing, cost, etc.
Action Research
Questions?
Email:
Carol Clymer, Ed.D., [email protected]du
Beth McLean, Ph.D., elg6@psu.edu
Esther Prins, Ph.D., esp150@psu.edu
Collecting Data that Matters: Part 2
Family Literacy Outcomes and Data Collection Framework
Outcomes Family Literacy
Typically Collect
Data to Collect to
Determine if Outcome Met
Tools and Data Resources
Children
Increased language and literacy growth
(e.g., phonological awareness, oral
language, vocabulary, alphabetic
awareness)
Pre- and post-tests or formative
assessments from children’s schools that
measure growth in phonological
awareness, vocabulary development,
and other literacy skills
Standardized tests, formative assessments,
portfolios, including Get Ready to Read,
Early Learning Accomplishment
Profile/Learning Accomplishment Profile,
Dynamic Indicators of Basic Early Literacy
Skills, Phonological Awareness and
Literacy Screening (PALS), Concepts
about Print
Educational/literacy outcomes in school
(e.g., reading at grade level, reading
gains, kindergarten readiness,
attendance, reduced discipline referrals,
promotion)
Education outcomes in school measured
with standardized test scores, student
performance reports collected from
schools
PA Family Literacy End of Year Report,
standardized test scores, parent self-report,
report cards, number of school behavior
referrals
Education/literacy outcomes out-of-
school (e.g., increased reading
frequency/enjoyment, library usage, and
literacy/technology practices at home)
Education/literacy outcomes out-of-
school, information on home literacy
activity logs, self-report participation
Home activity logs with questions related to
children, journals of activities
Caregivers
Increased involvement in child’s
language and literacy growth (e.g.,
participation in interactive learning
activities with children)
Involvement in child’s language and
literacy growth-information from
caregiver pre-post survey, home literacy
activity logs; observations in ILA
activities
Home literacy activity/reading logs, pre-
post survey of activities, attendance at ILA
activities, journals
Increased involvement in child’s
education (e.g., engagement with
schools/centers, engaging in school
activities
Participation in parent-teacher
conferences, attending school events,
engagement with school
Information gathered from schools about
attendance and engagement
Outcomes Family Literacy
Typically Collect
Data to Collect to
Determine if Outcome Met
Tools and Data Resources
Increased involvement in everyday
literacy practices (e.g., ability to access
and enjoy reading, library usage,
reading at home)
Information from caregiver pre-post
survey, home literacy activity logs,
journals about reading activities
Home activity/reading logs with questions
about caregiver involvement, pre-post
survey of activities at home, journals
Language and print and digital literacy
skill development (e.g., increased test
scores, literacy engagement, English
proficiency, use of technology)
Language and print and digital literacy
skill development measured with pre-
post standardized assessment, pre-post
surveys, home activity logs, journals
Standardized tests (e.g. TABE, GED,
CASAS) formative assessments, portfolios,
home activity logs, journals, Computer
Skills Survey
Program
Improvement
Staffing
Action research and observations,
surveys of professional development
Goodling Institute Practitioner Action
Research (PAR), surveys
Program structure and design
Action research and observations,
survey of participant outcomes
Goodling Institute PAR, surveys
Retention
Attendance and participation records
Attendance and participation tracking
system such as Excel spreadsheet, PA e-
Data Access database template
Outcomes
Identified in
Part I Session
(last year)
Financial stability
Government assistance, participation in
financial literacy support service,
employment status, wages, financial
hardship
State records; self-report or employer
report form; case notes related to
participation in support services, financial
hardship & security scales
Parent mental health and social support
Self-efficacy, social support, perceived
stress and depression
Self-efficacy, social support, stress, and
depression scales
Workforce readiness
Pre-post data on employability skills
Foundation Skills Framework Self-
Appraisal
Longitudinal data related to educational
achievement (e.g., reading behaviors)
Longitudinal tracking of outcomes
Longitudinal study that includes a survey
or pre-post survey with information on
desired outcomes for caregivers or school
records for children
Act 143 Family Literacy Contract #
Family Code
Child Code
DOB of Child:
PALS Pre-K
ASSESSMENT SCORING SHEET
FOR
Phonological Awareness Literacy Scale (PALS Pre-K)
AND
Peabody Picture Vocabulary Test (PPVT-III)
Instructions: Transfer the child’s scores from each section of the PALS Pre-K Fall/Spring Child Summary
Sheet to the appropriate section in the following table. Do this for both the fall (Time 1 optional) and spring
(Time 2) administrations of the PALS Pre-K. Place the date the assessments were completed in the space
provided.
FALL
Date Completed:
SPRING
Date Completed:
Section I:
Name Writing
(7 possible)
(7 possible)
Section II:
Alphabet Knowledge
Part A: Upper Case
(26 possible)
Part B: Lower Case
(26 possible)
Part C: Letter Sounds
(26 possible)
Part A: Upper Case
(26 possible)
Part B: Lower Case
(26 possible)
Part C: Letter Sounds
(26 possible)
Section III:
Beginning Sound Awareness
(10 possible)
(10 possible)
Section IV:
Print and Word Awareness
(10 possible)
(10 possible)
Section V:
Rhyme Awareness
(10 possible)
(10 possible)
Section VI:
Nursery Rhyme Awareness
(10 possible)
(10 possible)
---------------------------------------------------------------------------------------------------------------------------------------
PPVT-III or PPVT-IV
Instructions: Place the date the assessments were completed in the space provided. Transfer the child’s raw
score from the PPVT-III/PPVT-IV Performance Record to the space below.
Time 1 date completed:
Time 2 date completed:
Raw score: (0 to 204 possible)
Raw score:___________________(0 to 228 possible)
Raw score: (0 to 204 possible)
Raw score:___________________(0 to 228 possible)
Pennsylvania Department of Education, Division of Adult Education 2018-2019 6/14/2018
!
1!
Sample!Demographic!Survey!Questions!
!"#$%&'()*(+,-.(/0120.3(
(!)!Male!
(!)!Female!
!
!!#($%&'()*(+,-.(4).'%2&'03(56678879999#(
_________________________________________________!
!
!:#($%&'()*(+,-.(.&;0(,.(0'%1);)'+3(5Interviewer:*Read*responses*to*adult*and*check*all*that*
the*adult*responds*affirmatively*to.#(
[!]!American!Indian!or!Alaska!Native!
[!]!Asian!
[!]!Native!Hawaiian!or!Other!Pacific!Islander!
[!]!Black!or!African-American!
[!]!Hispanic!or!Latino(a)!
[!]!White!or!Caucasian!
[!]!Other!(Specify):!_________________________________________________!
!
!<#($%&'()*(+,-.(=&.)'&>(*'&'-*3?(
(!)!Not!Married!(includes!divorced/separated)!
(!)!Married!
(!)!Other!(describe):!_________________________________________________!
(!)!Prefer!not!to!respond!
!
!@#($%&'()*(+,-.(;,-1'.+(,A(,.)/)13(
_________________________________________________!
!
!B#(C,D(=&1+(+0&.*(%&E0(+,-(>)E02()1('%0(F1)'02(G'&'0*3(
(!)!Less!than!1!year!
(!)!1!to!3!years!
(!)!4!to!5!years!
(!)!6!to!10!years!
(!)!11!to!15!years!
(!)!16!to!20!years!
(!)!21!or!more!years!
(!)!Entire!life!
(!)!Prefer!not!to!respond!
!
!H#(I*(J1/>)*%(+,-.(1&')E0(>&1/-&/03(
(!)!Yes!
(!)!No!
!
!
5-1')'>02#(
!
2!
!
!K#($%&'()*(+,-.(1&')E0(>&1/-&/03(
_________________________________________________!
!
!
5-1')'>02#(
!
!L#(I1(+,-.(%,=0(2,(+,-(*M0&NO(PPPPPPP(5please*fill*in*the*blank*using*the*interviewee's*
native*language#(
(!)!more!than!English!
(!)!less!than!English!
(!)!equally!as!much!as!English!
!
!
Q-&.2)&1(J2-;&'),1&>(R&;N/.,-12(
!
:S#($%&'()*('%0(%)/%0*'(/.&20(,.(+0&.()1(*;%,,>(+,-(;,=M>0'023(
(!)!No!schooling!
(!)!Kindergarten-5th!grade!
(!)!6th!grade!
(!)!7th-8th!grade!
(!)!9th!grade!
(!)!10th-11th!grade!
(!)!Some!of!12th!but!didn't!receive!a!diploma!
(!)!High!school!diploma!or!GED!
(!)!Attended!some!technical!school!or!college,!but!did!not!complete!
(!)!Technical!school,!completed!
(!)!2-year!degree,!completed!
(!)!4-year!degree,!completed!
(!)!Graduate!level!degree,!completed!(Master's,!Professional!or!Doctorate)!
!
:"#($%0.0(2)2(+,-(&''012(*;%,,>(,.(;,>>0/03?(
(!)!ONLY!in!the!U.S.!
(!)!ONLY!outside!the!U.S.!
(!)!BOTH!in!and!outside!the!U.S.!
!
!
Q-&.2)&1(J=M>,+=01'(&12(C,-*0%,>2(I1;,=0(
!
:!#(T>0&*0('0>>(=0(D%);%(,A('%0(A,>>,D)1/()*('.-0(&4,-'(+,-.(0=M>,+=01'(*)'-&'),1U(
(!)!Not!currently!employed!
(!)!Currently!employed!
(!)!Prefer!not!to!respond!
!
!
3!
::#(C,D(=&1+(%,-.*(&(D00N(2,(+,-(D,.N3(
(!)!1!to!9!hours!
(!)!10!to!19!hours!
(!)!20!to!29!hours!
(!)!30!to!35!hours!
(!)!36!to!40!hours!
(!)!41!or!more!hours!
!
:<#($%&'()*('%0(','&>(+0&.>+()1;,=0(A,.(+,-.(%,-*0%,>2(A.,=(&>>(*,-.;0*3(
(!)!Less!than!$5,000!
(!)!$5,000!to!$9,999!
(!)!$10,000!to!$14,999!
(!)!$15,000!to!$19,999!
(!)!$20,000!to!$24,999!
(!)!$25,000!to!$29,999!
(!)!$30,000!to!$34,999!
(!)!More!than!$35,000!
(!)!Prefer!not!to!respond!
!
!
DIGITAL LITERACY SURVEY
1. Do you own a smart phone, tablet, or other device such as a game system? If so, please
list.
2. How many hours of screen time do you get each day? (please circle one)
0-1 1-2 3-4 4 or more
3. Do you use apps? What are they?
4. Does your child own a smart phone, tablet, or other device such as a game system? If so,
please list.
5. What do they use it for?
6. About how many hours of screen time does your child get each day? (please circle one)
0-1 1-2 3-4 4 or more
7. Do you set rules and expectations for your child’s usage?
8. What are they?
9. Do you use media (radio, television, the internet) with your child?
10. Do you find ways to connect media to real life?
11. Do you watch television with your child? Do you talk about what you see?
12. Do you talk about commercials and other advertising?
13. Do you model the media behavior you want to see in your kids?
!
eData V2 Child Maintenance
June 14, 2018 PDE Division of Adult Education 1
*Child’s First Name _______________________ Middle Initial ______ *Last Name______________________________
*Date of Birth ___/__/_____ *Gender F M
*Date of First Enrollment ___/__/_____
Ethnicity
*Are you Hispanic/Latino? Yes No
*What is your race? (If applicable choose one or more)
American Indian or Alaskan Native
Asian
Black or African American
Native Hawaiian or other Pacific Islander
White
Family Last Name_______________________________ Family Code _______________
Child Code ________
Type (select one) Preschool (birth to 5) School Age (K-6) Keyword ________________________________
Current Year Enrollment Date ___/__/_____ Participating Y N
PreK Year Before K Y N
Preschool Program Year Data Entry and Exit (Select all that apply at entry and again at exit)
Entry
Early Intervention Y N
Early Head Start Y N
PCHP Y N
Nurse Family Partnership Y N
Head Start Y N
Title I Preschool Y N
Private Preschool Y N
K at 3 years Y N
K at 4 years Y N
Pre-K Counts Program Y N
Keystone Stars Program Y N
Family Literacy Early Literacy Y N
Daycare w/Educational Comp. Y N
Daycare Y N
Other Y N
Specify _____________________
None Y N
Exit
Early Intervention Y N
Early Head Start Y N
PCHP Y N
Nurse Family Partnership Y N
Head Start Y N
Title I Preschool Y N
Private Preschool Y N
K at 3 years Y N
K at 4 years Y N
Pre-K Counts Program Y N
Keystone Stars Program Y N
Family Literacy Early Literacy Y N
Daycare w/Educational Comp. Y N
Daycare Y N
Other Y N
Specify _____________________
None Y N
*EC Assessment __________________________________ Time1 ___/__/_____ Time2___/__/_____
eData V2 Child Maintenance
June 14, 2018 PDE Division of Adult Education 2
School Age Program Year Data Entry and Exit (Select all that apply at entry and again at exit)
Grade ________
Entry
Free or Reduced Lunch Y N
Home School Y N
Special Education Y N
ESL Y N
Before/After School Program Y N
Daycare with Ed Component Y N
Daycare Y N
Other Y N
Specify ___________________________
None Y N
Exit
Free or Reduced Lunch Y N
Home School Y N
Special Education Y N
ESL Y N
Before/After School Program Y N
Daycare with Ed Component Y N
Daycare Y N
Other Y N
Specify ___________________________
None Y N
Information from End of School Report Y N
School-age Promotional/Retention (Select One)
Promoted
Grade ______________
With IEP Y N
Other Specify ___________________________________
Retained
Grade ______________
Referred for testing Y N
Other Specify ___________________________________
Other
Specify _________________________________________
Child’s Reading Level (select one)
Below Basic
Basic
Proficient
Advanced
Unexcused Absences (days) _________
Contact parent had with teacher (Select all that apply)
None
Open House
Conference
Phone
Note/Email
Other Specify ____________________________
eData V2 Adult Intake Family Literacy Details
June 14, 2018 PDE Division of Adult Education 1
Family Literacy Details
*Family Surname _______________________________________________ *Family Code _____________
*Date Adult Entered Family Literacy ______/_______/______________
*Relationship to Enrolled Child (select one)
Parent
Grandparent
Other relative
Other
*Household Status (select one)
Head of single parent household
Head or spouse/partner in two-parent household
Head or spouse/partner no dependents
Dependent member of household
Dependent and single parent
Living in group quarters
Living alone
*Adult Code ________
eData V2 Family Maintenance Intake/Exit
1
June 14, 2018 PDE Division of Adult Education
*Family Code ____________ *Family Last Name (Surname) ________________________________________
*Zip Code (+4) _______________ *County ______________________ *Date of First Entry ___/__/_____
Site ID __________ Keyword ______________________________
*Current Year Entry Date ___/__/_____
*Household Income
0-$5000
$5001-$12,140
$12,140-$16,460
$16,460-$20,780
$20,780-$25,100
$25,100-$29,420
$29,420-$33,740
$33,740-$38,060
$38,060-$42,380
Over $42,380
Unable to Obtain
*Number of Family Members __________
*Contracts _______-________-_____________ _______-_______-___________
_______-________-_____________ _______-_______-___________
Exit Date ___/__/_____
Act 143 Family Literacy Contract #_
Family Literacy 2018-2019
End of the Year School Progress Report
Child's Name: Date of Birth:
Child Code: Family Code:
Date Entered Family Literacy Program: Date Form Completed:
School Name: School District:
SCHOOL ATTENDANCE
Please enter the number of days the child had unexcused absences from school during the 2018-19
school year.
Number of Unexcused Absences
SCHOOL ACHIEVEMENT
Please enter end of the school year information for the child identified above.
1. For each subject listed below, 1) check () the type of assessment/s used OR that you were unable to
assess the child in that subject, and 2) using the categories defined above, check () the most appropriate
category to describe this child's overall performance at the end of the school year in each subject areas.
Subject
Assessment
Check all that apply
Below Basic
Two years below
grade level
Basic
One year below
grade level
Proficient
On grade level
Advanced
Above grade level
Reading
Standardized Test
Portfolio
Observation
Other (specify)
Unable to assess
Writing
Standardized Test
Portfolio
Observation
Other (specify)
Unable to assess
Mathematics
Standardized Test
Portfolio
Observation
Other (specify)
Unable to assess
2. Please put a check () in the appropriate box to complete this statement: This child will be:
1
Promoted to 1
st
grade 2
nd
grade 3
rd
grade 4
th
grade 5
th
grade 6
th
grade 7
th
grade
with IEP other Specify
2
Retained in K 1
st
grade 2
nd
grade 3
rd
grade 4
th
grade 5
th
grade 6
th
grade
referred for testing other Specify
3
Other (specify)
3. Did the parent contact you in any of the following ways (check all that apply)
None Open House Conference Phone
Note/Email Other Specify
Pennsylvania Department of Education, Division of Adult Education. 2018-2019 06/14/2018
Family'Pathways'Activities'Log'
'
Family'Name:'________________________________''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''Month:________________________'
!
DATE'
TIME'SPENT'
ACTIVITY/EXPERIENCE'
COMMENTS/REFLECTION'
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
Action Research
in Family
Learning
Dr. Esther Prins ([email protected])
Goodling Institute for Research in Family Literacy
Pennsylvania State University
What is action research
(AR)?
• “A reflective process of progressive problem-solving that
helps you to answer questions you have about issues in your
program or practice. In a community of learners,
practitioners decide their own questions & work
collaboratively to find ways to improve their practice &
programs.” (Weirauch, p. 1)
• In AR, “we systematically analyze a problem, review the
literature & relevant experience, set a baseline for purposes
of comparative analysis, systematically gather evidence on
observed change(s), & collectively reflect on the
outcomes” (Quigley, 2006, p. 171)
• Research is not just for professors
! “Poking & prying with a purpose”
Why use AR?
• Ethical
! Educators participants have right & capacity to construct
knowledge about own programs
! “Practitioners & their learners can begin to voice their own
realities through action research. They can begin to create & own
their own knowledge for practical change & they can begin to
build momentum for public & policy change with supportive
data.” (Quigley, 1999, p. 260)
• Pragmatic
! Benefits for educators
! Hone skills in problem-posing & problem-solving
! Deepen knowledge about program & teaching practices
! Improve program effectiveness
! Gather evidence to document program outcomes
! Stimulate change on multiple scales: AR “has the potential to
influence—if not transform—the researcher, the research
participants, their collective work or activities, the setting they
are part of, & the culture they share.” (Quigley, 1999, p. 257)
How does it work?
• Group of “critical friends”
• Iterative cycle of problem-posing and problem-solving
(Quigley, 2006, p. 174)
“The best way to
understand something
is to try to change it.”
(Kurt Lewin)
Examples of AR in practice
• SEQUAL (Statewide Educational Quality for Family
Literacy) – FL programs in PA (2007-2011)
! Choose among pre-determined topics or identify their own
! Question: “To what extent can we increase the quality of parent-
child interaction as measured by the ACIRI tool through teaching
basic literacy skills, increased focus on quality parenting
education, implementation of the core messages, as well as the
incorporation of the behaviors on the ACIRI tool?”
Planned'
Outcome'
Intervention(s)' Data'source(s)' Results'
!" #$%&'()'&)(
&*+,'&(-+,(
.$,'/)&(0(
*1%23,'/(+/()1'(
4567(8'&)(
9.1+/+2+:%*$2(
$;$,'/'&&(
2%)',$*<(
&*,''/%/:=(
((
7)$>'1+23',?(
@$A%2%'&(
((
((
((
7'.)'AB',(CDDE(
• F+/)%/G'(-+*G&%/:(+/(
B$&%*(2%)',$*<(&>%22&(-+,(
$3G2)&(0(*1%23,'/((
• H$I'(.$,'/)&(0(
*1%23,'/(.,$*)%*'(5FJ#J(
&>%22&($&(.$,)(+-(J65(
K%/)',$*)%I'(2%)',$*<(
$*)%I%)%'&L(
((
7'.)'AB',(CD!D(
JA.2'A'/)(Parenting)for)
Academic)Success:)A)
Curriculum)for)Families)
Learning)English(
(M$<(CD!D(0(M$<(
CD!!(
4567(&*+,'&(-+,(
$3G2)&(0(*1%23,'/(
CDDENCD!D(F1%23(7*+,'&(
5I',$:'(*1%23(4567(&*+,'(
%/*,'$&'3(9&''(1$/3+G)=(
(
CD!DNCD!!(F1%23(7*+,'&(
5I',$:'(*1%23(4567(&*+,'(
%/*,'$&'3(9&''(1$/3+G)=(
(
CDDENCD!DOCD!DNCD!!(
53G2)(7*+,'&(
5I',$:'($3G2)(4567(
&*+,'(%/*,'$&'3(
Planned'
Outcome'
Intervention(s)' Data'source(s)' Results'
C" JA.,+I'()1'(
PG$2%)<(+-()%A'(
.$,'/)&(&.'/3(
;%)1()1'%,(
*1%23,'/(B+)1(
$)(1+A'(0($)(
&*1++2(
((
7)$>'1+23',?(
@$A%2%'&(
((
((
((
((
((
((
7'.)'AB',(CDDE(
• Q&'(5FJ#J(
*G,,%*G2GA(%/(
*2$&&,++A&(
• H$I'(.$,'/)&(0(
*1%23,'/(.,$*)%*'(
5FJ#J(&>%22&($&(.$,)(
+-(J65(
((
M$,*1(CD!D(
Q&'(2'$,/%/:(:$A'&(
$&(.$,)(+-(J65(0(
.$,'/)%/:('3G*$)%+/(
((
5FJ#J(8'&)(
((
(
(
(
(
(
(
(
4$,'/)(R+G,/$2&(S(
,'-2'*)(+/()%A'(
&.'/)($)(1+A'(0(
&*1++2(;%)1(*1%23(
+/(5FJ#J(2'&&+/(
CDDENCD!D(
• 53G2)(0(*1%2 3($I',$:'(5FJ#J(
&*+,'&(%/*,'$&'3(9&''(
1$/3+G)=(
CD!DNCD!!(
• 53G2)(0(*1%2 3($I',$:'(5FJ#J(
&*+,'&(%/*,'$&'3(9&''(
1$/3+G)=(
(
• 4$,'/)(R+G,/$2('/),%'&(
,'-2'*)'3()1'()%A'()1'<(;','(
&.'/3%/:(;%)1()1'%,(*1%23,'/"(
81'<($22+;'3()1'()'$*1',&()+(
&''(1+;()1'(.$,'/)&(;','(
G&%/:(>/+;2'3:'()1$)()1'<(
;','(2'$,/%/:(B+)1(3G,%/:(J65(
0($)(1+A'"(
(
Planned'
Outcome'
Intervention(s)' Data'source(s)' Results'
T" @+*G&'3(
.$,'/)%/:(
'3G*$)%+/(;%22(
'/1$/*'()1'(
'--'*)%I'/'&&(
+-(J65&(
((
7)$>'1+23',?(
@$A%2%'&(
((
((
((
((
((
((
7'.)'AB',(CDDE(
Q&'(5FJ#J(*G,,%*G2GA(
%/(*2$&&,++A&(
((
M$,*1(CD!D(
Q&'(2'$,/%/:(:$A'&(
$&(.$,)(+-(J65(0(
.$,'/)%/:('3G*$)%+/(
((
7'.)'AB',(CD!D(
JA.2'A'/)(Parenting)
for)Academic)Success(
((
Q&'(U$,2<(6'$,/%/:(
7)$/3$,3&($&($(
.$,'/)('3G*$)%+/(
)++2(
5FJ#J(8'&)(
((
4$,'/)(&G,I'<&(0(
$*)%I%)%'&(-,+A(
Parenting)for)
Academic)Success)
)
((
J65(4+,)-+2%+(
((
(
(
(
J65(VB&',I$)%+/(
F1'*>2%&)(
5FJ#J(&*+,'&(%/*,'$&'3((
((
VI',$22W(.$,'/)&(,'.+,)'3(+/(
&G,I'<&()1$)()1'<('/R+<'3(
Parenting)for)Academic)
Success(2'&&+/&(0(-+G/3()1'A(
1'2.-G2(
((
J65(4+,)-+2%+(3+*GA'/)&(
.$,'/)N*1%23(%/)',$*)%+/&(0(
$*)%I%)%'&(3+/'(;%)1()1'(
-$A%2%'&(
((
8'$*1',&(3+*GA'/)'3()1'(
-$A%2%'&X(%/)',$*)%+/&(3G,%/:(
J65()1,+G:1()1'(<'$,(G&%/:()1'(
J65(+B&',I$)%+/(*1'*>2%&)((
Phases
• #1: Problem-posing
! What is the problem, issue, or concern? What isn’t working?
! Identify intervention
! What will you do to improve the problem?
! Which actions will be most likely to produce the desired outcome?
! Considerations
! Feasibility
! Justification
Phases
• #2: Planning
! Timeline
! Informed consent
! Institutional approval
! Resources
! Data collection techniques
! Use several types
! Qualitative and/or quantitative
! Survey questions
! Standardized measures
! Interviews
! Focus groups
! Observational notes
! Video analysis
! Photographic analysis
! Document analysis
Phases
! Criteria for success (identify benchmarks)
! How will you know if you have succeeded?
! What were the initial patterns of behavior, knowledge, attitudes, etc., and
how have they changed?
! “What’s different, and compared to what?” (Quigley, 2006, p. 181)
! Surveys: pre-post, not retrospective
Type When? Example
retrospective end of intervention “I read to my child more often than
I did before starting the class.”
• yes/no; “strongly agree” to
“strongly disagree”
pre-post T 1: before it begins
T 2: after it ends
“How often do you read to your child
each week?”
• “never” to “every day”
Phases
! Craft a researchable question
! Focused, specific, & measurable
! Include problem area, goal, & intervention
Instead of… Try…
How can we increase
enrollment?
Will distance learning increase
enrollment by 20%?
How can we improve adult
education gains?
To what extent will the use of
authentic materials increase
the frequency of parents’
reading and writing outside the
classroom?
Phases
• #3: Observing
! Systematic data collection & analysis
! Detailed record-keeping
! Careful selection of sample
• #4: Reflecting
! What did you learn from the data?
! How do the results compare to your benchmarks?
! If intervention was effective: Should you do another AR cycle?
! Same results: builds trustworthiness & dependability of findings
! If not effective: Revise intervention, try a new one, or pursue a
different problem?
! Which aspects of the intervention will you keep?
! How can you share your results?
Activity
• Problem area
• Planned outcome: What do you want to happen?
! Participants will ________.
• What intervention could you try?
• What evidence will you use? (2+ data sources)
References & Resources
• DeBruin-Parceki, A., & Edwards, P. (2007). Lets read together: Improving
literacy outcomes with the Adult-Child Interactive Reading Inventory
(ACIRI). Baltimore: Brookes.
• Goodling Institute for Research in Family Literacy http://www.ed.psu.edu/
educ/goodling-institute
• Greenwood, D. J., & Levin, M. (2007). Introduction to action research: Social
research for social change (2
nd
ed.). Thousand Oaks, CA: Sage.
• On sampling strategies, see Michael Patton. (1990). Qualitative evaluation
and research methods (pp. 169-186). Beverly Hills, CA: Sage. http://
legacy.oise.utoronto.ca/research/field-centres/ross/ctl1014/Patton1990.pdf
• Quigley, B. A. (1999). Naming our world, claiming our knowledge: Research-
in-practice in adult literacy programs. The Alberta Journal of Educational
Research, 45(3), 253-262.
• Quigley, B. A. (2006). Building professional pride in literacy: A dialogical
guide to professional development for practitioners of adult literacy and basic
education. Melbourne, FL: Krieger.
• Weirauch, D. (n.d.) SEQUAL Practitioner Action Research, 2009-2010.
University Park, PA: Goodling Institute for Research in Family Literacy.
http://www.ed.psu.edu/educ/goodling-institute/family-literacy-resources/
sequal
