How to Get Started with Learning Analytics - PowerPoint PPT Presentation

1 / 28
About This Presentation

How to Get Started with Learning Analytics


Those organizations that have an imperative to begin the ... Challenge - Data cleanliness confounds correlative ability. Need to understand impacts on data ... – PowerPoint PPT presentation

Number of Views:273
Avg rating:3.0/5.0
Slides: 29
Provided by: jimeve


Transcript and Presenter's Notes

Title: How to Get Started with Learning Analytics

How to Get Started with Learning Analytics
  • Jim Everidge

Who Will Benefit?
  • Those organizations that have an imperative to
    begin the process of proving the value of
  • Those professionals that are trying to formulate
    a strategy around Measurement and need additional
    data to develop that strategy
  • Vendors that have training products that are
    charged with helping their customers connect the
    value of the products to the customers business

1. Background Approaches
2. Learning Analytics Maturity
3. Developing a Vision
4. Elements of Project Success
5. Project Evolution
6. Case Studies
Learning Analytics Approaches
  • Opinion-based data
  • Relies on collecting data about perceptions of
    impact of learning
  • Challenge - Hawthorne Effect
  • Harder to defend when the result is a change in
    business process
  • Operations-based data
  • Focuses on the correlation of learning data and
    business data
  • Challenge - Data cleanliness confounds
    correlative ability
  • Need to understand impacts on data

Business Intelligence
  • "The oft-quoted example of what data mining can
    achieve is the case of a large US supermarket
    chain which discovered a strong association for
    many customers between a brand of babies nappies
    (diapers) and a brand of beer.
  • The explanation goes that when fathers are sent
    out on an errand to buy diapers, they often
    purchase a six-pack of their favorite beer as a

Financial Times of London February 7, 1996
If you cant measure it, you cant manage it.
Nolan Norton Consultants Founders of the
Balanced Scorecard
Driven by demand for rapid content deployment,
plus growing interest in value-added modules like
training analytics and competency management, the
market for e-learning infrastructure systems from
U.S.-based vendors is expected to grow 12 in
2004 to 529.4 million
Simba Information 1/9/2004
History of Business Intelligence
  • Originally conceived as Data Warehousing Data
  • Now called Business Intelligence (coined by
    Howard Dresner of Gartner Group, 1994)
  • Dresner defined BI A generation of software
    that allows corporations to accelerate the rate
    at which managers can physically process
  • Traditionally expensive and hard to do
  • Today available to everyone - Techniques, Best
    Practices, Tools
  • Includes data integration, analysis, reporting,
    and data visualization

Current Practices
  • Better Techniques
  • Technology allows for data integration
  • Purchase pre-defined configurations
  • Best Practices
  • Focus on business metrics
  • Group and filter data
  • Drill up and down
  • Create visually expressive charts
  • Better Tools
  • Microsoft Business Intelligence Platform
  • 9 Tools one of which is Microsoft Office

  • On Line Analytical Processing
  • Allows the user to interact with the data
  • Multi-dimensional analysis
  • Drill up or down through various dimensions
    characteristics of the data that you are looking
  • Contrasted with
  • Standard SQL Reports
  • One time setup
  • Choose parameters
  • Static results at a moment in time

Why Do Learning Analytics?
  • Turn Data into Information
  • Measure Learning Effectiveness
  • Learning Activity
  • Catalog Effectiveness
  • Total Cost of Learning
  • Manage Compliance
  • Understand Business Impact
  • Focus Strategic Alignment Initiatives
  • Using Business Metrics as your guide
  • Allows alignment of learning with strategy

Expected Results
  • Correlation of business data and learning
    intervention data
  • Use correlations to driver operational changes
  • Incremental skills in getting the learning team
    to understand operational data
  • Incremental skills in getting the operations
    group to understand learning data

What NOT to Expect
  • Individual prescription based on individual
  • Contrast with Performance Management where
    individual performance contributes to the greater
    performance metrics
  • Absolute certainty cause effect

Correlation of Business Learning Data
Effectiveness of Learning Experience
Learning Experiences
Learning Analytics Maturity
  • Level 1 - Influence Individual Action
  • Just starting to collect information about
    Learning Experiences
  • Requires tools like LMS or equivalent
  • Need to template Business Questions so that the
    right data is collected from the outset
  • Level 2 Understanding the Business
  • Acquired tools to collect data but early in the
  • Trying to understand what Business Metrics have
    value in the organization
  • Begin to draft Vision for Learning Analytics
  • Level 3 Questioning Effectiveness
  • Has collected learning data for 6 months have a
    sense of the Business Metrics that have a
    reliable correlation to Learning Experiences
  • Ready to implement Vision for Learning Analytics

Documenting the Vision
  • Business Opportunity
  • Provides context for the initiative(s)
  • Includes a Vision Statement
  • Benefits Analysis
  • Solutions Concept
  • Roadmap for initiative(s)
  • Analysis -gt Risk, Feasibility, Usability,
  • Solutions Design
  • Proposed Technical Architecture
  • Initial Project Scope
  • Provides range of features/functions
  • Defines out of scope
  • Criteria for success

Vision Template
  • Send an email request for document
  • Commitment to provide feedback to first cut at
    Vision document

Elements of Project Success
Business Sponsors
Cross Functional
Business Reps
Lots of Data
Learning Analytics Projects
Meta- Data
Skilled Staff
Iterate Projects
Clean Data
Business Analysis
Computerworld White Paper Shaku Atre, Atre Group,
Inc. 2004
Project Team Composition
  • Business Executives
  • Customers
  • External business partners
  • Learning
  • Finance
  • Marketing
  • Sales
  • IT
  • Operations

Involving Business Sponsors/Execs
  • Understand the value of the project remove
    political barriers
  • Focus the initiative to a specific set of
    business questions manage the scope
  • Initiate a data-quality campaign within their
  • Periodic project reviews

Iterate Projects
  • Develop a Clear Vision
  • Go through a Readiness Assessment Exercise
  • Operationalize your Learning Analytics
  • Integrate Business Analytics

Readiness Assessment
  • Focus is to define initial cut at Business
  • Identify Business Owners and Involvement
  • Identify Process Outputs and Users
  • Identify logical Data Sources and availability
  • Identify iterative projects
  • Prioritize iterative projects
  • Develop SOW for Operationalizing Learning
    Analytics Engine

Operationalizing Learning Analytics
  • Identify Learning questions
  • Identify sources/uses of data
  • Validate data integrity
  • Install analytics server
  • Validate ETL Schema adjust as necessary
  • Set up Template Reports that address initial
  • Weekly reviews of reports and opportunities

Integrating Business Analytics
  • (Assumes Learning Analytics engine is operational
    and OLAP Analysis on learning data is being done)
  • Identify Business questions
  • Identify sources/uses of different data sets
  • Validate data integrity
  • Determine ETL schema adjust as necessary
  • Validate ETL schema
  • Set up Template Reports that address initial
  • Weekly Reviews of reports and opportunities

Uses of Information
  • Adjust Program Design
  • Improve Program Delivery
  • Influence Application Impact
  • Enhance Reinforcement for Learning
  • Improve Management Support for Learning
  • Improve Satisfaction with Stakeholders
  • Recognize Reward Participants
  • Justify or Enhance Budget
  • Develop Norms or Standards
  • Reduce Costs
  • Market Learning Programs

Phillips, Phillips, Hodges Make Training
Evaluation Work ASTD, 2004
Case Study Learning Analytics
  • Profile
  • Packaging Shipping
  • 1100 Retail Centers
  • 18,000 Employees
  • 250,000 Hours Training for new business
  • Intervention
  • 3 different certifications assigned by job role
  • 12-14 modules for each certification
  • Had to be complete in 8 weeks
  • Results
  • 1100 Concurrent users of the learning content
  • 300 of these were simply running reports
  • Allowed for self-service reporting analysis
  • Challenges
  • Had to teach some level of application
    proficiency to Retail Store managers
  • Field support went to regional HR managers (not
    IT Help Desk)

Case Study Business Analytics
  • Profile
  • Telecomm
  • 16 Call Centers
  • New product rollouts happening quickly
  • Customer defections increasing
  • Intervention
  • Monitored training activity in 6 call centers
  • Obtained business data from the same call centers
  • Results
  • Data indicated that training impacted sales
  • Four call centers needed to significantly
    increase training
  • Challenges
  • Use of correlated data is marginal justification
    for significantly altering business operations
  • Setting up the right environment with the right
    data set for analysis is challenging

Case Study Unintended Results
  • Profile
  • Retail
  • 1254 Store locations
  • Product training defined monthly based on
    seasonal merchandise
  • High turnover in personnel
  • Intervention
  • Monitored training completions at the store level
  • Utilized store sales results as the business
    operations benchmark
  • Results
  • Negative correlation between training and store
  • Further inquiry revealed inappropriate
    application of training
  • Challenges
  • Broadly promoting results in advance of
    understanding the data and what is driving it
  • Getting complete data sets from large audiences
    without investments in management technology

QA - Discussion
Jim Everidge, President Rapid Learning
Deployment, LLC (770)874-1190 x
Write a Comment
User Comments (0)