Transfer Student Presentation - PowerPoint PPT Presentation

1 / 31
About This Presentation

Transfer Student Presentation


Measure educational outcomes to identify and enhance effective educational practices ... Student Learning Outcomes ... VSA Core Outcomes Evaluation was Premised ... – PowerPoint PPT presentation

Number of Views:31
Avg rating:3.0/5.0
Slides: 32
Provided by: LoriVa7


Transcript and Presenter's Notes

Title: Transfer Student Presentation

The VSA Responding to a New Era of Accountability
Accountability Today Tomorrow AACU, AASCU,
APLU Williamsburg, VA October 6-8, 2009
Voluntary System of Accountability (VSA)
  • Initiative by public 4-year universities to
    supply basic, comparable information on the
    undergraduate student experience to important
    constituencies through a common web report the
    College Portrait.

Sponsored by APLU AASCU. Funding from Lumina.
VSA Goals
  • Provide a useful tool for students during the
    college search process
  • Assemble information that is transparent,
    comparable, and understandable
  • Demonstrate accountability and stewardship to
  • Measure educational outcomes to identify and
    enhance effective educational practices

Multiple Audiences
  • Prospective students and families
  • High school, community college counselors
  • Policymakers and elected officials
  • Campus faculty and staff
  • Accrediting associations, institution and state
  • Taxpayers and public-at-large

VSA Participation
  • 329 institutions from 49 states and Puerto Rico
  • 63 of 520 APLU AASCU institutions
  • 250 College Portraits posted on

VSA Context
  • Increasing disinvestment in higher education
  • Perceived lack of useful and transparent data
    prevents institutions from demonstrating
    accountability and contribution to public good
    (Spellings Commission)
  • Policy-makers and employers demand evidence of
    educational outcomes particularly broad
    transferable skills

A Community Effort
  • Concept outlined in APLU white papers, refined
    through feedback from members
  • Strengthened by AASCU endorsement
  • Structure developed by task forces - presidents,
    provosts, student affairs, IR
  • 82 members from 70 institutions
  • Input from associations and larger HE community

Result The College Portrait
Free web report that communicates user-friendly,
consistent information about college campuses.
  • Student Family Information
  • Student Experiences Perceptions
  • Student Learning Outcomes

No rankings, no spin ... just the facts!
Student Family Information
  • Institution description, highlights
  • Student characteristics
  • Cost of attendance, net cost calculator
  • Success progress rate
  • Degree programs, class sizes
  • Post graduation plans

Introduction to Campus Community
Menu links take users to each section and data
Campus Defined Buttons
Success Progress Rate
  • Developed as alternative to IPEDS graduation rate
  • Focuses on student success in higher education
  • Tracks student progress across 2-year and 4-year
    institutions using NSC data
  • New cohort query available to all Student Tracker

(No Transcript)
VSA College Cost Calculator (new version late
fall 2009)
Estimate of costs tuition/fees, housing choice,
books, etc
Estimate of awards Grants, student loans, work
Student Experiences Perceptions
  • Selected senior responses from one of 4 surveys
  • Snapshot of student experiences on campus within
    6 common areas
  • group learning
  • interaction with faculty
  • institutional commitment
  • active learning
  • diversity experiences
  • student satisfaction

6 Common Constructs
Selected responses from Seniors
Student Learning Outcomes
  • Pilot project to measure learning gains in broad
    cognitive skills at institution level
  • Skills critical thinking, analytic reasoning,
    problem solving, written communication
  • Institutional assessment as one part of
    comprehensive assessment plan
  • Combined with other types of assessments
    courses, programs, general education
  • Combined with other outcomes measures
    professional licensure exams

Student Learning Outcomes
Learning gains in critical thinking, analytic
reasoning, written communication measured at
institution level
Reported in a common format using CAAP, CLA or
FIPSE Test Validity Study
  • Findings

  • For VSA reporting we needed to measure
    value-added for the core learning outcomes of
    critical thinking and written communications.
  • Because many provosts strongly objected to using
    a single test to measure these outcomes at all
    universities, VSA taskforces identified three
    tests that measured the outcomes. The VSA
    Presidential oversight taskforce decide to permit
    participating universities to choose from the
    three the test that best suited their campus
  • The three companies that produce the tests agreed
    to adopt the same value-added computation and
    reporting technique and format so that reporting
    would be standardized across universities while
    controlling for entering student quality.

The FIPSE study
  • What remained was to ensure that the three tests
    actually produced consistent scores.
  • The only way to be sure of this was to have a
    large number of students take each of the three
    tests and observe the results.
  • Funding from the Fund for the Improvement of
    Post-Secondary Education made the study possible.

Study Parameters
  • 13 tests administered to 1,100 students at 13
  • 13 tests components of CLA, CAAP, MAPP
  • 4 tests of critical thinking
  • 2 tests of reading
  • 2 tests of mathematics
  • 4 tests of writing
  • 1 test of science

Participating Universities
  • University of Michigan, Ann Arbor
  • University of Minnesota, Twin Cities
  • University of Texas, El Paso
  • University of Vermont
  • University of Wisconsin, Stout
  • Alabama AM
  • Arizona State
  • Boise State
  • California State-Northridge
  • Florida State
  • MIT
  • Trinity College
  • University of Colorado-Denver

Sample Parameters
Construct Validity
  • Construct Validity is very high when the school
    is the unit of analysis
  • All tests rank schools similarly, regardless of
    the construct or the response format
  • Mean correlation .92 for 9 multiple choice
  • Mean correlation .84 for 4 constructed response
  • Mean correlation .85 for multiple choice test
    and constructed response measure of the same

Senior Freshman Score Differences
  • Score gains between class levels indexed to
    common measure of effect size to control for
  • score distribution differences across tests
  • differences in average ACT/SAT scores
  • Larger effect sizes indicate greater differences
    in freshman versus senior scores
  • Seniors had higher mean scores than freshmen on
    all tests except the CAAP mathematics exam

Effect Sizes
  • Effect sizes not systematically related to
    constructs tested, response format or test
  • Effect sizes across the 12 tests range from .25
    to .5 standard deviation (CAAP math excluded)
  • Effect sizes
  • ACT .33 (excluding math test)
  • CLA .31
  • ETS .34

  • Reliability is score consistency
  • if there is a 100 chance a student will receive
    the same score on different forms of a test, the
    reliability score is 1.0
  • if there is a 0 chance, the reliability score is
  • When the school was unit of analysis, score
    reliability varied from .75 to .87 across the 13
  • Conclusion Score reliability is not a major

Overall Conclusions
  • Validity and Reliability are high across tests
    and should not affect choice of measure
  • Factors affecting adoption
  • Acceptance by students, faculty, administrators
    or other policy makers
  • Trade-offs in cost, easy of administration, etc
  • Utility of the test for other purposes such as
    supporting campus activities and services or
    providing guidance on improving learning

The Premise on Which VSA Core Outcomes Evaluation
was Premised is Sound
  • While the TVS study did not measure Value-Added
    using the VSA technique, it did find that the
    effect size was approximately the same regardless
    of the exam utilized.
  • The premise that portions of CLA, CAAP and MAPP
    can be used interchangeably to produce roughly
    consistent value-added learning outcomes
    measurements for universities with similar
    students is consistent with the VSA finding.
  • Thus VSA will continue to permit universities to
    choose to use either CLA, CAAP or MAPP.

Final Report
  • Two Reports are available on the VSA website
  • The complete (and technically complex) TVS full
    report as well as
  • An interpretative (and less technical) report
    Christine Keller and I authored especially for
    VSA participating schools can be found at
  • http//
Write a Comment
User Comments (0)