School Counselors and Program Evaluation

School Counselors and Program Evaluation. Washington School Counselors Association John Carey National Center for School Counseling Outcome Research UMass Amherst www.cscor.org. Data-Driven School Counseling Programs.

Share Presentation

Embed Code

Link

• process data
• participate
• results data process data
• what data
• washington school counselors association
• project explorers example

johana + Follow

School Counselors and Program Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

Presentation Transcript

1. School Counselors and Program Evaluation Washington School Counselors Association John Carey National Center for School Counseling Outcome Research UMass Amherst www.cscor.org
2. Data-Driven School Counseling Programs • Implement comprehensive programs based on national design and local need • Use data to determine directions (data driven decision making, needs assessment) • Measure results (program evaluation) • Share successes
3. What Data Do We Use? ASCA National Model asserts there are three broad categories of data sources: • Student Achievement Data • Achievement-Related Data • Standards and Competency Data
4. Student Achievement Data 1. Norm-Referenced Standardized Tests • Scores referenced to national average • PSAT, SAT, ACT, Iowa, Metropolitan • Predictive Validity 2. Criterion-Referenced Standardized Tests • Scores referenced to performance standards • State achievement tests (AIMS) • Content related to state curriculum frameworks • Content Validity
5. Student Achievement Data 3. Performance tests or changes in achievement levels(advancement in Math or English, for example) 4. Portfolios 5. Course grades and GPA 6. Completion of college prep requirements 7. Drop-out rate
6. Achievement-Related Data • Attendance rates • Behavioral problems • Student attitudes • Discipline referrals • Suspension rates • Drug, Tobacco, and Alcohol use patterns • Parent involvement • Extracurricular activities
7. Standards and Competency Related Data 1. College Placements 2. Financial Aid Offers 3. Vocational Placements 4. Percentage of students who: Have 4- or 6-year plans Participate in job shadowing Have completed career interest inventories ASCA National Standards
8. Data Sources vs. Data Types The ASCA National Model identifies three data types for use in program evaluation: • Process Data – What was done for whom? • Perception Data – Attitudes, opinions, beliefs - generally self-report data • Results Data – Objective and measurable student outcomes such as academic achievement, attendance, and disciplinary interventions
9. Program Evaluation: Process Data • Process Data: What was done for whom? • Who received services? • Ninth graders? Students at risk of failing math? • What did they receive? • Curriculum intervention? Small-group intervention? • When did they receive it? • All year? Twice? For 30 minutes? • Where and How was it provided? • In the classroom? After school?
10. Program Evaluation: Process Data • Process data alone does not tell us whether or not the student is different (in behavior, attitude or knowledge) as a result of this activity. • Coupled with results data, process data can help identify what factors may have led to success in an intervention.
11. Program Evaluation: Perception Data • Perception data measures how students are different as a result of an intervention • Did students gain competencies? • Every 10th grade student completed a career interest inventory. • 85% of 10th graders identified the 4 steps in the career decision making process. • Did they gain knowledge? • 87% of 9th graders demonstrated knowledge of graduation requirements. • Were there changes in their attitudes or beliefs? • 86% believe that pursuing a non-traditional by gender career is acceptable.
12. Program Evaluation: Perception Data Differences in student knowledge, competency and attitudes are measured through: • Pre-post tests • What do students know/believe before and after the intervention? • Completion of an activity • Completion of a 4-year plan • Surveys • What do students say they believe or know?
13. Program Evaluation: Results Data • Results data is the proof that the intervention has or has not influenced behavior. • An intervention may occur (process data), students may know the information (perception data), but the final question is whether or not the students are able to utilize the knowledge, attitudes and skills to affect behavior (results data).
14. Program Evaluation: Results Data • Results data can be complex because many factors impact behavior change. • An increase in enrollment at a vocational high school may be due to an intervention implemented for MS students. Conversely, finding no changes in results data does not mean that an intervention has necessarily been unsuccessful.
15. Clarifying Terms • Research • Action Research • Program Evaluation
16. Program Evaluation Process • Program Evaluation allows practitioners to evaluate programs and interventions in their specific contexts. • Practices can change immediately and in an ongoing manner as data are collected and analyzed.
17. Program Evaluation Process 1. Identify the construct(s) 2. Review what is known 3. Develop specific hypotheses or questions you would like to answer and plan the program evaluation accordingly 4. Gather the data 5. Analyze the data 6. Interpret results and disseminate and use findings 7. Evaluate the process
18. Conducting Program Evaluation: Identify the Construct(s) 1. Review the mission and goals of your program to identify the constructs you would like to look at. Ask yourself: • How are people different as a result of the school counseling program? • What is a question I want to answer? • What do I wish was different?
19. Conducting Program Evaluation: Identify the Construct(s) Constructs often have “sub-constructs” Define the construct/question in clear, specific language. The more specific you are, the easier it will be to measure later. You can use the ASCA National Model to help you define your construct
20. Conducting Program Evaluation: Identify the Construct(s) The ASCA National Model domains • Personal/Social Development Domain • Academic Domain • Career Domain
21. Conducting Program Evaluation: What is Already Known? 2. Review what is known about your questions. • Has anyone in your organization asked this question before? • Who might have information? • What is the relevant research in professional journals? • What does an Internet search find on this topic?
22. Conducting Program Evaluation: Develop Hypotheses 3. Develop hypotheses or questions you would like to answer and plan the research process accordingly. • Ask yourself what you think the answer(s) to your question(s) will be. • Identifying the hypothesis helps you identify your biases, which may impact your process. • What is the opposite of your hypothesis (the “null hypothesis”)? What would the data look like if you were wrong?
23. Considerations • What are your biases/Mental Models? How are they impacting the questions you’re asking and the places you’re looking? • Evaluating findings from research literature and internet searches • What is the source? How reliable is it? • What are the strengths/weaknesses of the research design, sampling, effect size, measures used, treatment fidelity, researcher bias, instrument reliability and validity?
24. Considerations • Data: • How accurate is the data you’ve chosen to use? • What’s missing? • Instruments: • Reliability and validity • Just because it exists doesn’t mean it’s well done
25. Considerations • Sampling and Research Design: • Size of sample • Comparability of sample and control • Matching vs. random assignment • Assuring fidelity of treatment • Doing same thing across different groups?
26. Considerations • Ethical Considerations: • Consent • Human Subjects Review • Denying access to interventions - remediation? • Data Analysis • Do you have the capacity to analyze the data?
27. Conducting Program Evaluation: Develop Hypotheses EXAMPLE: Project Explorers – after school program at vocational high school for MS students Hypotheses/Questions: Does participation in in Project Explorers lead to: • Increased enrollment? • Increased awareness of the offerings of the VHS? • Career exploration? • Improved career decision making abilities?
28. Activity #1 With your SC program in mind, think about these questions to help you identify constructs: • How are people different after participating in my program? • What is a question I want to answer? • What do I wish was different? • Use these questions to develop a specific, measurable question you would like to answer, and place the answer on the planning tool
29. Conducting Program Evaluation: Gather Data 4. Gather the data. • What information do you need in order to answer your question? • Use multiple sources of data, or multiple outcome measures, wherever possible (triangulation). • Decide whether you need to consider student achievement data, psychosocial data, career data, school data, process data, perception data, and/or results data.
30. Conducting Program Evaluation: Triangulation Results Data Process Data THE QUESTION Perception Data
31. Conducting Program Evaluation: Triangulation INCREASED ENROLLMENT Process Data THE QUESTION Perception Data
32. Conducting Program Evaluation: Triangulation INCREASED ENROLLMENT PARTICIPATION – # OF SESSIONS # OF STUDENTS THE QUESTION Perception Data
33. Conducting Program Evaluation: Triangulation INCREASED ENROLLMENT PARTICIPATION – # OF SESSIONS # OF STUDENTS THE QUESTION SURVEY DATA
34. Project Explorers Example • Results Data • Did the total number of students enrolling in vocational programs increase when compared to the last three years? • Perception Data • Survey using combination of validated items from pre-existing survey (MCGES), and “hand written” items
35. Project Explorers Example(Perception Data) Circle the correct response for questions 1 - 3. 1) Which school (s) prepares its graduates for skilled employment, such as a mechanic? VHS My local high school Both schools 2) Which school(s) prepares its graduates for 2-year colleges, such as Middlesex Community College? VHS My local high school Both schools 3) Which school(s) prepares its graduates for 4-year colleges, such as UMASS? VHS My local high school Both schools 4) List as many of the shops at VHS that you are aware of:
36. Project Explorers Example(Perception Data) 5) Check all that apply. I have used the following resources to learn about careers: ___Internet ___ Expert or person working in the field ___Counselor/Teacher ___Other (please specify)_____________ ___Family member ___ I have not researched any career 6) Circle the answer that indicates the level of your confidence for each item. (copyright laws prohibit duplication of these items – Missouri Comprehensive Guidance Evaluation Survey (MCGES), Lapan) 7) Please check the 3 areas of employment that you are most interested in at this time. ____ Agriculture ____ Education ____ Arts & Communication ____ Business and Computer Services ____ Construction ____ Health and Hospitality Services ____ Manufacturing ____ Transportation
37. Project Explorers Example(Process Data) • Process Data documents “what was done for who” • Number of students participating (overall) • Attendance at each shop
38. Activity #2 • Identify the PROCESS, PERCEPTION, and RESULTS data needed to answer your question / hypotheses • Use the planning tool!
39. Conducting Program Evaluation: Gather Data • Where is the data? • Does it already exist? • School records, intake forms, test results • Will you generate your own data? • Surveys • Interviews • Observations • Multiple data sources help you more accurately get at the complexity of a situation, whereas one measure or data source will give you a snapshot view.
40. Conducting Program Evaluation: Gather Data • Select and/or develop the instruments you will use to gather the data. Possibilities include: (more later) • Surveys • Tests (of achievement, aptitude, attitude, etc.) • Behavioral checklists or observations • School records • Performance assessments • Interviews
41. Conducting Program Evaluation:Gather Data • Identify and follow ethical and legal standards: • No participant should be exposed to physical or psychological harm. • Permission to use confidential data must be obtained. • Participation in a study is always voluntary. • Participants may withdraw from the study at any time. • Participants’ privacy rights must be respected.
42. Conducting Program Evaluation:Gather Data • Identify the group/sample to be studied: • Ideally either the entire sample is involved in the study (the class, grade, or school) or the group studied is a random sample of the population. • Stratified sampling uses a smaller sample which has the same proportions as the larger sample. • Systematic random sampling is when every x number of students is chosen from the whole population (every 4th student on the attendance list, for example).
43. Conducting Program Evaluation:Gather Data • Once data sources and measures are identified, ethical standards are considered, and the sample is identified, data can be gathered! • Ways to collect data include: • Questionnaires or surveys • School records • Interviews • Observation
44. Collecting Data: Demographics • Regardless of what other types of data you collect and analyze, demographic data is a critical first step • Collect the types of demographic data that are important for making sense of your results, and for describing who contributed to the data you collected
45. Collecting Data: Demographics • Questions to consider: • What information will I need to adequately describe my sample? • Think of the information you will need to provide to let unfamiliar people know who your program served • What information will I need to analyze the data the way I want to? • Think of the various ways you can describe your results . For example, the impact on different ethnic groups, special education vs. regular education, etc
46. Demographic Variables • Ethnicity • Gender • Class (Parent Educational Level) • Language Level (Limited English Proficient) • Low Income (Free or Reduced School Lunch) • Acculturation, Migration (Mobility) • Special Needs • School Performance (GPA, Achievement Quartile)
47. Demographic Variables (school) • Student Participation in School Programs • Student Participation in Extracurricular Activities • Grade Level • Age • Number of Years in Current School • Parent Participation Level
48. Identifying Effective Surveys • Key questions to consider: • How does the survey relate specifically to my program’s goals? • Is the survey appropriate for my age group of students? • How reliable and valid is the survey?
49. Creating Your Own Surveys 1. Define Constructs 2. Select/Write Items 3. Write Instructions 4. Test Survey 5. Edit Items and Instructions
50. 1. Define Constructs 2. Select/Write Items 3. Write Instructions 4. Test Survey 5. Edit Items and Instructions Developing Your Own SurveysWriting Items Sometimes pre-existing surveys and measures are not able to adequately capture the attainment of the goals of your program, or the question you would like to answer Open-ended vs. closed questions: Open-ended questions can provide rich data, but are hard to summarizeWhat do you think about Project Explore? Why is career planning important? Closed questions are most common in surveys because the results are easy to summarize Load More .

• Related
• More by User

High School Counselors’ Workshop

High School Counselors’ Workshop. Looking toward 2017-2018. General Information. Remind you – always seek good partners in the processes related to financial aid. Schools and seasoned professionals are available and should be willing to assist.

470 views • 27 slides

Evaluate School Counselors to Sustain Your Program

Evaluate School Counselors to Sustain Your Program. Gail M. Smith Supervisor of School Counseling Julie Hartline Professional School Counselor Campbell High School Cobb County School District. Today’s Agenda. Over-Arching Question Laying the Groundwork The Framework for the Evaluation

338 views • 20 slides

Middle School Counselors

Middle School Counselors. Workshop Goals. Understand the importance of talking about financing education after high school early Provide tools and resources to increase early financial planning for postsecondary education. Parent Aspirations.

376 views • 29 slides

School Library Program Evaluation

School Library Program Evaluation. Background & History NYSED SLMPE rubric. 2010 SLMS/NYLA Leadership Retreat L4L: Implementing the AASL Standards and Guidelines in your Program. School Library Program Evaluation. Background & History NYSED SLMPE rubric.

384 views • 20 slides

School Counselors : What and Why?

School Counselors : What and Why? . Nauveen Dubash MS/M P hil Clinical & Counselling Psychology Institute of Psychology Beaconhouse National University. WHAT: THE ROLE OF STUDENT COUNSELORS

308 views • 18 slides

Welcome School Counselors!

Welcome School Counselors!. Who We Are…. Founded in 1861 and joined SUNY in 1948 Today… $800 Million Capital Construction Campus Center- Convocation Center and Ice Arena The Village Townhouses Lakeside Complex- Johnson, Riggs & Lakeside Dining Halls School of Business

259 views • 13 slides

Loveland High School Counselors

Loveland High School Counselors. A-E. F- LA. LE-Q. R-Z ELL. GT & IB. This is the transcript you see on portal. 1. Log onto portal 2. Click on reports on on left side. 3. Click on unofficial transcript. LHS vs. 4-Year College Entrance Requirements.

423 views • 30 slides

Role of School Counselors:

Role of School Counselors:. National Center for Transforming School Counseling- (Adapted from Ed Trust 4/19/2012) CED607 Unit II (Unit 2.a) School Counselors- Leading from the “Heart ” Harvey Hoyo, Ed.D. Course Lead, CED607 Nu- Costa Mesa. Leading from the Heart.

539 views • 32 slides

School Counselors

School Counselors. Experts in leadership, accountability and advocacy for students. Courtney Garza - CNDV 5330 - CNA. What are the responsibilities of a School Counselor?.

787 views • 12 slides

Hazelwood School District Program Evaluation

Hazelwood School District Program Evaluation. Mary Piper. Designing Evaluation . How will information be gathered? What is measured or assessed? How will information be used ?. Levels of Program Evaluation . Levels of Program Evaluation . Levels of Program Evaluation .

337 views • 18 slides

School Counselors’ Evaluation System Orientation SY11-12

School Counselors’ Evaluation System Orientation SY11-12. Guidance Services Office, SSS Gail Smith, Supervisor Evaluation Systems Office, HR Jan Holley, PHR, Director. Standards-Based Alignment ASCA National Model CCSD Strategic Plan & Focus. Cobb Keys School Counselor Evaluation System.

474 views • 33 slides

School Counselors’ Evaluation System Orientation SY13-14

School Counselors’ Evaluation System Orientation SY13-14. School Counseling Office, SSS Gail Smith, Supervisor Evaluation Systems Office, HR Dr. Michael Shanahan, CHRO. Standards-Based Alignment ASCA National Model CCSD Strategic Plan & Focus. School Counselor Evaluation System.

508 views • 33 slides

HCES School Counselors

HCES School Counselors. Kristy Barton Grades K, 2, 4 Lauren Hooten Grades 1,3,5. What Counselors D o. Guidance lessons- developed using the TEA guidelines for elementary guidance curriculum Small Group Guidance/Counseling Individual Guidance/Counseling 504 Red Ribbon Week Activities

283 views • 10 slides

Welcome School Counselors!

Welcome School Counselors!. Who We Are…. Founded in 1861 Joined SUNY in 1948 Today… $800 Million Capital Construction Campus Center- Convocation Center and Ice Arena (2007) The Village Townhouses (2010) Lakeside Complex- Johnson, (2003), Riggs & Lakeside Dining Halls (2007)

237 views • 13 slides

School Counselors

School Counselors. Leaders on Campus. Courtney Garza - CNDV 5330 - CNA. What is Leadership?. “the process of influencing the activities of an individual or a group in efforts toward goal achievement in a given situation”

440 views • 16 slides

School Counselors’ Time

School Counselors’ Time. Jeannine O’Brian. Delivery of Services to Students. The American School Counselor Association (ASCA) suggests that a minimum of 80% of school counselors’ time is allocated for the delivery of services to students. Services.

269 views • 14 slides

School Counselors Evaluation Validation Study Training

School Counselors Evaluation Validation Study Training. Linda Brannan K-12 Student Support Services Consultant Curriculum & Instruction Cynthia Martin Professional Development Leads Education Recruitment & Development. School Counseling Wikispace.

670 views • 52 slides

Recognized American School Counselors Association Model Program (RAMP)

Recognized American School Counselors Association Model Program (RAMP) Is your school counseling program ready to RAMP?. Session Goals. What is RAMP? Is the school counseling program ready to RAMP? What are the first steps? What tools are available to help a school counseling program RAMP?.

598 views • 40 slides

School Counselors

School Counselors. Education Specialist – Kim Niehues Back to School 2014. School Guidance and Counseling. Texas Education Agency http:// www.tea.state.tx.us/index2.aspx?id=2828 http :// www.statutes.legis.state.tx.us/Docs/ED/htm/ED.33.htm#33.002 Texas Counseling Association

302 views • 15 slides

Elementary School Counselors.

Elementary School Counselors help students focus on academic, personal/social and career development, so they not only achieve success in school, but are prepared to lead fulfilling lives as responsible members of society. Elementary School Counselors.

494 views • 32 slides

Counselors as School Diplomats:

Counselors as School Diplomats:. “Delivering Bad News in a P.C. (Parentally Correct) World”. Rebecca A. Cox, School Counselor, M.Ed., NCC, NCSC Chris D. Cox, Assistant Principal, M. Ed, Ed. Leadership and Supervision. Lighting the fuse to an Explosive Parent. Academics Behavior

260 views • 14 slides

School Counselors

School Counselors. …are certified/licensed educators trained in school counseling with unique qualifications and skills to address the needs of all students in three major domains:. academic personal/social career. School Counselors….

337 views • 15 slides