Focusing on Student Learning

1
Focusingon Student Learning

• Developing Assessment Systems
• for
• Student and Program Development
• May 24, 2006
• Forsyth Tech Community College

2
Agenda

• Why are we doing this?
• Translating instructors intentions into ?
  student learning outcomes
• Developing
• Instructional objectives
• Assessment tasks
• Evaluation tools

3
Task 1 Pre-Assessment

• Please complete the survey, marking the left hand
  column (how much you know prior to the workshop)

4
Why did Lucy get a C?

• Write down at least one question that comes to
  mind when you read the cartoon.
• What is Lucy questioning?
• How could the teacher avoid questions regarding
  his/her basis for grading?

5
Task 2 Why did Lucy get a C?
6
Why did Lucy get a C?

• Write down at least one question that comes to
  mind when you read the cartoon.
• What is Lucy questioning?
• How could the teacher avoid questions regarding
  his/her basis for grading?

7
What are some questions?

• How do we clearly communicate performance
  expectations to students?
• What is fair or equitable? Do we give some
  students a break because their parents get
  faulty clothes hangers?
• Do we reward effort? If so, how do we define it?
• Do we advantage or disadvantage certain students
  by how we measure performance/achievement?
• If students dont perform well on assessments,
  does that mean we did not teach well? How do we
  know whether or not we did? Do faculty
  evaluations provide a better picture of how well
  we teach?
• Is it important to have students create coat
  hanger sculptures? Or to perform well on any of
  our assessments?
• Important to whom? Why?
• What kinds of student outcomes do we value
  personally as teachers and collectively as a
  faculty?

8
Focusing on Student Learning

• Institutional assessment should not be concerned
  about
• valuing what can be measured,
• but, instead, about measuring that which is
  valued.
• -- Banta, T.W.
  et. al.

9
Goals for today

• Participants will be able to
• Explain how student assessment can enhance
  student engagement/ learning and promote program
  development
• Begin or continue the process to
• a) Articulate goals for student learning
• b) Translate goals into instructional
  objectives stated terms of student
• learning outcomes
• c ) Identify or develop assessment tasks
• d) Select or develop evaluation tools.
• Describe the changing context of assessment,
  accreditation, and accountability in higher
  education.

10
Starting with what we do well

• Being accountable
• Institutional Effectiveness

11

• Community colleges are prominent among the
  leaders in higher education in establishing
  indicators of institutional effectiveness,
  gathering benchmark data, and using findings to
  improve the satisfaction of students and other
  community constituents. Trudy Banta, Editors
  Notes, 1995

12
Assessment, planning, and budget are integrated.
Objectives established by departments during
program review become the basis for budget
allocations.
13
North Carolina Community College Performance
Measures

• Progress of Basic Skills Students
• Passing Rates on Licensure/Certification Exams
• Goal Completion for Completers
• Employment Rate of Graduates
• Performance of College Transfer Students
• Passing Rates in Developmental Courses

• Success rate of developmental students in
  subsequent college level courses
• Student satisfaction
• Retention, graduation rates
• Employer satisfaction
• Business/Industry satisfaction with services
  provided
• Program enrollment

14
Forsyth Techs Record

• Basic skills students meet state benchmarks (82
  compared to system average of 79)
• Aggregate passing rates on licensure/certification
  exams is equal to NC 86 pass rate
• Employment rate is reported at 99.05
• 90 (state 80) students pass developmental
  courses
• Satisfaction of completers 93 (state 97)
• Business/industry satisfaction with services 100
  (100)

15
Back to Trudy Banta

• Peterson (1999)
• 2,524 non-proprietary postsecondary institutions
  AS/BS
• 1,393 (55) responded

Compared to all institutions, associate of arts
institutions are less likely to collect cognitive
and affective data, less likely to use
student-centered methods in collecting data, and
less likely to conduct studies of student
performance
16
How would this change what we do?

• Outcomes-based (MBO) assessment
• Outcomes (objectives on the tactical plans),
• developed by all instructional departments and
  administrative and educational support service
  departments,
• are statements describing what each departments
  staff/faculty members desire to be the results of
  their efforts.
• Annual FTCC Plan, 2004-2005

• Assessing Student Learning Outcomes
• Directly examining
• the knowledge, skills, and abilities
• that a student has attained
• at key points in his or her progress
• through a set of higher education experiences and
  
• in the first years of practice.

17
Creating Assessment Systems--Shared Commitments

• Faculty share a commitment to
• A set of student learning outcomes
• Common assessments within programs
• across course sections
• Collecting, compiling, analyzing, reporting, and
  using the results of assessments of student
  learning
• To improve candidate performance
• To improve programs
• To improve policies and procedures.

18
What am I going to get out of this?

• Shared expectations of student performance
• Clear communication of expectations
• Enhanced student performance
• Ability of students, faculty, programs/departments
  to self-advocate
• A system of program evaluation which includes
  evaluation of student learning outcomes.

19
Why?

• What are some of the potential benefits of
  establishing student assessment systems?

20
Why assess student learning?

• Enhances Student Engagement
• Continuous improvement of Curriculum,
  Instruction, and Student Performance
• Promotes Professional Community (inquiry,
  reflection, scholarship of practice)
• Enables students, faculty, programs, and
  institutions to Self-Advocate
• -- able to participate in data-based
  decision-making
• Better reflects the complexity, extent, and
  impact of Faculty Work
• Helps us achieve our Institutional Mission
• Develops Public Trust

21
Student Engagement

• Embedded assessment student has to be actively
  engaged (cannot be passive learner)
• Clear expectations, models of performance
• Self-evaluation
• Assessing knowledge, skills, dispositions
  required for practice (meaningful)
• Can use products of assessments in job search,
  etc.

22
Improve Curriculum, Instruction, Student
Performance

• JMU major dividend of ongoing assessment has
  been greater faculty involvement. This process
  ensures that curriculum decisions remain in the
  hands of those who deliver the curriculum.
• curriculum assessment curriculumreal-time,
  on-going examination and improvement of teaching
  and curriculum
• Provides a focus for instruction.
• Clarifies expectations of students.
• Ensures we provide appropriate opportunities for
  reaching expectations.
• Clearly communicates target, average, below
  average, unsatisfactory performance
• Avoids overlap or oversight ensures
  comprehensiveness of curriculum
• Grounds curriculum in practice

23
Inquiry, Growth, and Professional Community

• Dialog values, commitments, what matters most,
  common expectations, level of performance,
  opportunities for learning
• Become a learning organization
• Cross-disciplinary connections
• Leadership opportunities among faculty
• Establish connections with practitioners
• Highlight successes of students, programs,
  faculty

24
Self-Advocacy

• Students have concrete evidence of what they know
  and can do
• Faculty are able to document their impact on
  students
• Contributes to the scholarship of practice
• Programs, divisions, institutions have array of
  information

25
Provides a Fuller Picture of Faculty Work

• Outcomes such as graduation/retention rates dont
  always capture the impact of faculty efforts
• Helps us tell our story
• efforts in providing learning opportunities
• the complex and multidimensional nature of
  learning
• Helps make students more accountable for their
  part in the learning process

26
Enhance Public Trust

• General public, potential students and families,
  politicians
• Federal, state, public push for accountability
• The role of anecdotes
• Quality assurance accreditation

27
Accomplish Institutional Mission

• Mission  
• Graduates of Forsyth Tech are
• technically skilled,
• regionally and globally oriented,
• prepared for lifelong learning and full civic
  engagement and employment.

28
Assessment and Accountability

• Professional standards boards have moved from
  input measures
• ? to management by objectives
• to documentation of student
• outcomes
• and have moved from
• examining institutions
• ? to examining programs.

29
Assessment and Accountability

• Council for Higher Education Accreditation CHEA
• American Association of Community Colleges
• Commission on the Accreditation of Allied Health
  Education Programs
• Joint Review Committee on Educational Programs in
  Diagnostic Medical Sonography
• Joint Review Committee on Educational Programs in
  Nuclear Medicine Technology
• Joint Review Committee on Education in Radiologic
  Technology
• North Carolina Board of Nursing
• Technology Accreditation Commission of the
  Accreditation Board of Engineering and Technology
  
• Commission on Accreditation of Allied Health
  Education Programs

30
Council for Higher Education Accreditation

• Accrediting organizations are responsible for
  establishing clear expectations that institutions
  and programs will routinely
• define,
• collect,
• interpret, and
• use evidence of student learning outcomes.

31
Council for Higher Education Accreditation

• More specifically
• regularly gather and report concrete evidence
  about what students know and can do as a result
  of their respective courses of study,
• framed in terms of established learning outcomes
  and
• supplied at an appropriate level of aggregation.
• Supplement this with information about other
  dimensions of effective institutional or program
  performance
• Prominently feature relevant evidence of student
  learning outcomes.
• Statement of Mutual Responsibilities for Student
  Learning Outcomes Accreditation, Institutions,
  and Programs (September, 2003)

32
SACS

• 3.4.1 The institution demonstrates that each
  educational program for which academic credit is
  awarded (a) is approved by the faculty and the
  administration, and (b) establishes and evaluates
  program and learning outcomes.
• 3.5.1 The institution identifies college-level
  competencies within the general education core
  and provides evidence that graduates have
  attained those competencies.

33
Commission on Accreditation of Allied Health
Education Programs

• Evaluations of students must be conducted on a
  recurrent basis and with sufficient frequency to
  provide both the students and program faculty
  with valid and timely indications of the
  students progress toward and achievement of the
  competencies and learning domains stated in the
  curriculum.

34
ABET Technology Accreditation Commission

• Each engineering technology program must have in
  place published educational objectives consistent
  with mission and with ABET criteria
• Must utilize multiple assessment measures in a
  process that provides documented results to
  demonstrate that the program objectives and
  outcomes are being met..
• examples include student portfolios, student
  performance in project work and activity-based
  learning national exams employer and graduate
  surveys..

35
Joint Review Committee on Education in Radiologic
Technology

• A programs goals are a more specific expression
  of the programs intended student learning
  outcomes. The goals should be written using
  behavioral terms and should address the
  cognitive, affective, and psychomotor domains.
  They must be measurable, preferably through the
  use of more than one measurement tool.
• -- JRCERT Guide for Program Analysis (05/05)

36
Summarizing

• Identify, communicate, and assess clear,
  measurable student learning outcomes
• Behavioral statements cognitive, affective,
  psychomotor learning domains
• Based on institutional, departmental/program,
  state, and national standards
• Establish a system for directly assessing student
  achievement of objectives
• Multiple assessments across time
• including graduate and employer surveys
• Collect, compile, report, and use results to
  improve student performance, programs, and
  organization

37
Identifying Student Outcomes

• Translating program goals and instructors
  intentions into instructional objectives
• stated in terms of student learning outcomes.

38
Communicating goals and objectives

• Beginning with some examples and
• the importance of verbs

39
Clear, Observable Behavior (cant measure what
you cant see)
40
Behavioralcognitive, affective,
psychomotorUsing a Framework to guide us

• Benjamin Blooms
• Taxonomy of Learning Domains
• Cognitive mental skills (Knowledge)
• Affective growth in feelings or emotional area
  (Attitudes)
• Psychomotor manual or physical skills (Skills)

41
Writing Instructional Objectives
There are a number of approaches to writing
instructional objectives

• Mager -- Behavioral objectives

• Eisner -- Expressive objectives

Gronlund -- General/specific objectives
42
Writing Instructional Objectives
Mager proposes writing specific statements about
observable outcomes that can be built up to
become a curriculum (an inductive approach).

• An example of a behavioral objective

Given 3 minutes of class time, the student will
solve 9 out of 10 multiplication problems of the
type 5 X 4 _____.
43
Writing Behavioral Objectives
Three Parts of a Behavioral Objective
In an oral presentation,
the student will paraphrase Dr. Martin Luther
Kings's I Have a Dream address,
mentioning at least 3 of the 5 major points
discussed in class.
44
Writing Instructional Objectives
Eisner proposes that not all instructional
objectives should focus on outcome some should
focus on the learning process itself (expressive
objectives).

• Examples of expressive objectives

• Students will attend a live symphony
• performance.

b. Students will use multiplication in
everyday activities.
45
Writing Instructional Objectives
Gronlund proposes starting with a general
statement and providing specific examples of
topics to be covered or behaviors to be observed
(a deductive approach).
46
Stating Instructional ObjectivesCurricular
Questions

• Create a basic document in a spreadsheet.
• Enter text and values into an application.
• Write formulas to calculate simple and
  multi-segment problems
• Format cells to display data appropriately
• Embed charts into the spreadsheet
• Display the sheet in worksheet and formula views
• Print document in both views and in landscape or
  portrait format.

47
Writing Instructional Objectives
Examples of general/specific objectives

• Students will detect the use of stereotypes.
• identify situations in which stereotypes might
  emerge
• recall or identify indicators or clues of
  stereotyping
• use of overgeneralization, exaggeration
• linking features together that are not
  logically linked (blonds are dumb)
• use of vague words (shifty)
• use of extremes or absolutes (never, none, all)
• absence of individual attributes or variations
• locate other information and examples which
  counter stated characteristics
• determine if communication includes indicators
  of stereotyping

48
Writing Instructional Objectives
Examples of general/specific objectives
(affective)

• Displays a scientific attitude
• demonstrates curiosity in identifying problems
• seeks natural causes of events
• demonstrates open-mindedness when seeking
  answers
• suspends judgment until obtains all possible
  evidence
• respects evidence from credible sources
• shows objectivity in analyzing evidence and
  drawing conclusions
• seeks ways to verify results
• shows willingness to revise conclusions as new
  evidence becomes available

49
Writing Instructional Objectives

• Examples of General Objectives
• Write an essay.
• Apply systematic strategies to monitor and
  improve personal health.
• Set up and operate graphics design equipment.
• Apply principles of radiation safety and
  protection.
• Process appointments in a timely and accurate
  manner.
• Develop a basic database using a database
  application.
• Other examples?

50
Task 3 Trying our hand.

• Translating goals and intentions into
  instructional objectives stated in terms of
  student learning outcomes

51
Stating Instructional Objectives

• Principles of electricity.
• Comprehends principles of electricity.
• Topic vs. student learning outcome.

52
Stating Instructional Objectives

• Comprehends assigned reading material.
• To increase students reading ability.
• Describe students learning behavior rather than
  teachers teaching behavior.

53
Stating Instructional Objectives

• Gains knowledge of basic principles of radiation
  safety and protection.
• Applies basic principles of radiation safety and
  protection in new situations.
• 1 the learning process rather than the
  learning outcome (knows, develops skills in,
  acquires, understands, learns). Another pitfall
  includes describing the learning activity
  create a diorama read seven journal articles,
  etc.

54
Stating Instructional Objectives

• Explains the scientific method and applies it
  effectively.
• Explains and applies---avoid more than one
  verb. Students may be able to do one but not the
  other, so is the objective met?

55
Guided Practice Breaking verbs down into
specific learning objectives

• Students will evaluate an Articles of
  Incorporation, a patient care plan, nuclear
  medicine image, essay.
• Determine the purpose for analyzing
• Identify the criteria to use
• Identify idealized standards for each criterion
• Examine the information and identify evidence
  related to criteria
• Judge the degree of match of the evidence with
  idealized standards
• State results of analysis by summarizing patterns
  and giving examples of meeting/not meeting
  standards within the criteria.

56
Stating Specific Instructional Objectives Guided
Practice

• Generic objectives can often guide the
  development of content-specific objectives.
• Students will demonstrate their knowledge of
  legal principles regarding the formation and
  maintenance of corporations or partnerships.
• Comprehend basic principles
• States principle in own words
• Identifies examples of the principles
• Distinguishes between correct and incorrect
  applications of the principle
• Predicts an outcome based on the principle.
• 
  -- Gronlund, 2004, p. 19

57
Stating Instructional ObjectivesCurricular
Questions

• The nuclear medicine technologist provides
  patient care.
• Acquires adequate knowledge of the patients
  medical history
• Provides for proper comfort and care before,
  during, and after procedures
• Recognizes surgical and disease factors that may
  create artifacts or variants on images
• Identifies when data acquisition or data
  processing protocol must be modified
• Provides safe and sanitary conditions
• Establishes and maintains good communication with
  each patient.
• IF THESE WERE OVERALL PROGRAM GOALS, how can we
  describe expectations for students when they are
  beginning their program and starting to learn
  about patient care?
• KNOWS BASIC TERMS ? COMPREHENDS CONCEPTS AND
  PRINCIPLES? APPLIES PRINCIPLES?INTERPRETS,
  EVALUATES

58
Task 3 Stating General and Specific
Instructional Objectives

1. In groups of 2-3, select or state a general
   instructional objective.
2. Identify specific learning objectives for the
   general learning objective.

59
Evaluation Tools

• Checklists
• __ 1. Sands and prepares surface properly.
  (check, /-)
• Numerical rating scales
• 4 3 2 1 a) Uses tools
  correctly for each task.
• Numerical rating scales with descriptors
• Selects appropriate equipment.
• __Needs to be ___________________ Needs some help
  _____________selects proper
• told what to use
  in selecting
  equipment independently
• 1
  2 3
  4 5
• Rating scales
• 3 always, 2 sometimes, 1 never
• a) Pays attention when problems are explained.

60
Rubrics Indicators of Quality
Criteria Performance Level 1 Performance Level 2 Performance Level 3 Performance Level 4
Thesis Unclear or unidentifiable topic Possibly vague and/or overly simplistic topic needs considerable revision. Clearly stated, potentially strong topic with some revisions. Clearly stated, strong topic original insightful.
Structure Unclear, often because these is weak or non-existent. Transitions confusing or unclear. Few topic sentences. Generally unclear wanders or jumps from point to point generally lacks transitions some paragraphs without clear topics. Generally clear and logical may lack a few clear transitions or some paragraphs may not be clear and solid. Ideas flow logically. Excellent transitions from point to point. Paragraphs support sold topic sentences.
Use of Evidence Very few or very weak examples or explanations. General failure to support statements or evidence not relevant quotes poorly integrated. Examples and explanations support some points points often lack evidence quotes may be poorly integrated. Examples and explanations support most points some evidence does not support point or may appear not relevant. Relevant materials support several points reasons supported with good explanations and examples excellent integration of quoted material.
Analysis Lack of evidence or very few or very weak examples weak attempts to relate to argument. Evidence is often merely stated and not explained or connected to the argument. Evidence often related to argument or reasons though not always clearly or completely explained. Clearly relates evidence to reasons poses new ways to think of thesis.
Logic and argumentation Simplistic view of topic no effort to grasp alternative views might contain logical fallacies. Logic might fail at time argument might be unclear might not address opposing views or states them but does not address them. Acknowledges opposing views and addresses several aspects of them, but not always in complete manner. Anticipates and defuses counter-arguments. All ideas flow logically. Makes novel connections which illuminate thesis.
61
Student Assessment PlansIdentify program level
student learning outcomes.Identify decision
points as students progress and the assessments
used to make decisions.
General Program-Level Student Learning Outcomes Phase I Phase 2 Follow Up
Content knowledge Assessment such as comprehensive exams in courses. Assessment such as licensure exam. Employer survey Alumni survey
Professional skills Assessment such as class demonstrations, simulations, or projects. Assessment such as more involved clinical experiences. Employer survey Alumni survey
62
Student Assessment Plans

• Individual Student Inventory
• Identify program-level student learning outcomes
• Instructors document which outcomes students
  demonstrate within courses/clinical experiences.

63
Summary

• Objectives student learning outcomes
• Objectives
• focus instruction,
• guide learning,
• provide criteria/standards for assessment,
• convey instructional intent to others,
• enable us to evaluate instruction.
• Frameworks (Blooms Taxonomy) comprehensiveness
• Verbs are the operative words!
• Pitfalls topic, teacher behavior, learning
  process/activity
• Evaluation tools convey expectations for student
  performance (various levels of specificity)
• Programs develop plans for documenting student
  performance

64
NEXT STEPS Task 4 Teaching Goals Inventory

• Help college teachers become more aware of what
  they want to accomplish in individual courses and
  across programs.
• http//www.uiowa.edu/centeach/tgi/background.htm

65
NEXT STEPS

• Review, update, or develop course-based
  assessment
• Identify general instructional objectives
• Identify specific instructional objectives
• Identify assessment tasks
• Develop evaluation tools
• Review, update, or develop program-level student
  assessment plan
• Identify program- or departmental student
  learning outcomes
• Identify a systematic process for assessing
  outcomes and for collecting, compiling,
  analyzing, reporting, and using the assessment
  data.