Using  Data  to  Improve  Student  Success

Thursday,  June  4,  2015  Jason  Levin,  WGU  

Bruce  Clayton  &  Chris@  Amato,  Sinclair  Community  College  Dena  Laney,  Bellevue  College  

       

This  product  was  funded  in  part  by  a    grant  awarded  by  the  U.S.  Department  of  Labor’s  Employment  and  Training  Administra@on.  The  product  was  created  by  the  grantee  and  does  not  necessarily  reflect  the  official  posi@on  of  the  U.S.  Department  of  Labor.  The  Department  of  Labor  makes  no  guarantees,  warran@es,  or  assurances  of  any  kind,  

express  or  implied,  with  respect  to  such  informa@on,  including  any  informa@on  on  linked  sites  and  including,  but  not  limited  to,  accuracy  of  the  informa@on  or  its  completeness,  @meliness,  usefulness,  adequacy,  con@nued  availability,  or  ownership.  

Western  Governors  University  Salt  Lake  City,  Utah

Data  Feedback  Cycles  at  WGU  Ins@tu@onal  Research,  06.04.2015    

About  WGU  The  principal  mission  of  Western  Governors  University  is  to  improve  quality  and  expand  access  to  post-­‐secondary  educa;onal  opportuni;es  by  providing  a  means  for  individuals  to  learn  independent  of  ;me  and  place  and  to  earn  competency-­‐based  degrees  and  other  creden;als  that  are  credible  to  both  academic  ins;tu;ons  and  employers.  

•  59,000  students,  growing  at  20%  annually  •  Online,  non-­‐profit  •  Regionally  and  na@onally  accredited  •  76%  of  students  from  underserved  category  (Rural,  Minority,  Low  Income,  First  

Genera@on  College  Student)  •  All  students  are  full  @me  •  24%  Masters,  76%  Bachelors  •  Four  colleges:  

•  Business  (35%)  •  Health  Professionals  (19%)  •  Informa@on  Technology  (17%)  •  Teachers  College  (29%)    **  Only  NCATE  accredited  online  college  **  

•  Six  month  terms,  new  term  star@ng  on  the  first  of  every  month  •  Tui@on  is  $2,890  per  term  •  Competency  based,  self-­‐paced,  asynchronous  

Student  

Course  

Ins@tu@on  

Program  

Faculty  

PDM,  Eval,  CM  

Provost,      ND  

President,      Provost  

4  

Provide  differen@al  treatments  to  students  based  on  risk  classifica@on  

Create  proac;ve  vs.  reac;ve  protocols    Use  all  available  data  about  a  student  at  any  given  @me  to  classify  student  risk  based  on  the  probability  making  term  OTP    Classifica@on  of  students  into  high,  medium,  and  low  risk  categories  is  dynamic  and  influenced  by  recent  behavior    

Student  –  Faculty  Feedback  Student  Risk  Classifica@on  and  Interven@on  

5  

Above  Average  Average  Below  Average  

Leaning  Green  Leaning  Red  

Show  data  for  past  six  weeks  

7  

Course  Development  Feedback  Loop  MYC1  Case  Study  

8  

Course  Level  Diagnos@cs  

9  

Program  Feedback  Loop  –    MSISA  Case  Study  

10  

MSISA  Outcomes  

11  

Ins@tu@onal  Feedback  Loop  Harris  Interac@ve  Survey  of  Alumni  

 

  WGU  (A)  

National  (B)  

Public  4  Year  (C)  

Current  employment  status   %   %   %        Total  employed     89%

 B   84%   86%  

         Employed  full-­‐time     78%   74%   77%            Employed  part-­‐time     8%   6%   5%            Self-­‐employed     3%   4%   4%                    Total  not  employed     11%   16%   14%            Not  employed,  but  looking  for  work     7%   8%   7%            Not  employed  and  not  looking  for              work/retired/homemaker/student   4%   8%

A   7%

A  

           Employed  in  same  field  as  degree  earned  (among  currently  employed)   83%   81%   83%  

  Graduates  2006-­‐2009   Graduates  2010-­‐2012  

 WGU  (A)   National  (B)  

Public  4  year  (C)   WGU  (D)   National  (E)  

Public  4  year  (F)  

NET  CHANGE  in  income  from  prior  to  earning  degree  to  present    

+  $18,600   +  $12,400   +  $16,600   +  $9,000   +  $8,500   +  $8,400  

 Source:  WGU  Harris  Study,  2012  

12  

Sinclair  Community  College  Dayton,  Ohio

CBE4CC

Ins@tu@onal  &  Programma@c  Context

•  Large  size.    Sinclair  serves  approximately  20,000  students  per  fall  and  spring  term.  

•  Significant  non-­‐tradi3onal  student  popula3on.      

• CBE  program:  self-­‐paced  and  online.  

CBE4CC

Challenges

•  Suppor@ng  large  numbers  of  students  with  limited  student-­‐support  resources  

• Providing  adequate  academic  support  to  students  who  have  been  away  from  the  classroom  and  who  have  compe@ng  demands  on  their  @me  

• Compensa@ng  for  an  instruc@onal  model  that  limits  interac@ons  between  students  and  faculty  and  access  to  on-­‐campus  student-­‐support  services  

CBE4CC

CBE  Learner  Support  Model

CBE4CC

Student  Performance  Repor@ng:    Essen@al  Requirements • Caseload  based  •  Sensi@ve  to  known  predic@ve  indicators  • Real-­‐@me,  ac@onable  course-­‐performance  informa@on  • At-­‐a-­‐glance  usability  •  Fully  automatable  genera@on  and  delivery  

CBE4CC

Performance  Repor@ng  Within  LMS

• Basic  course-­‐sec@on-­‐level  repor@ng  •  Course-­‐sec@on  specific  •  Informa@on  segmented  across  mul@ple  reports  •  Restricted  access  •  Poten@ally  insensi@ve  to  key  performance  metrics  

•  Enhanced  repor@ng  through  LMS  analy@cs  packages  •  More  flexible,  but  ojen  limited  to  pre-­‐defined  report  elements  

•  Ojen  available  as  an  LMS  add-­‐on  at  addi@onal  cost  

CBE4CC

Moving  Beyond  the  Standard    End-­‐User  Experience •  Establish  the  structure  of  a  useful  student-­‐performance  report  •  Repor@ng  “wish  list”    

•  Iden@fy  the  data  systems  and  sources  that  contain  the  relevant  informa@on  •  Student  informa@on  system/ERP  •  Case  management  system  •  Learning  management  system  

•  Leverage  available  resources  to  assemble  and  deliver  final  report  •  Office  of  Research,  Analy@cs  &  Repor@ng,  staffed  by  team  of  report  writers,  data  analysts,  researchers,  and  database  administrators  

•  Data  warehouse  and  sophis@cated  data  management  sojware  

CBE4CC

DMS  

SIS   CMS  

LMS  

DW  

CBE4CC

Student  Informa3on   Course  Informa3on   Risk  Indicators  

Last  Name  

First  Name  

Tartan  ID   Course   Term   Enrollment  

Status   Week   Term  Progress  

Course  Progress  

Mid-­‐Term  Exam  Status  

Average  Grade  

Final  Grade  

Unsuccessful  ALempt  On  Record  

Mul3ple  Unsuccessful  ALempts  On  Record  

Logins  Last  7  Days  

Submissions  Last  7  Days  

Risk  Level   Comments  

Hilton   Davis   0652665   BIS-­‐1120   15/SP   Enrolled   7   35%   0%   N/A   .     .   No   No   5   0   High   No  Submissions  In  Week  1  

Rasor   Robert   0564225   MAT-­‐2170   15/SP   Enrolled   7   35%   0%   N/A   71.00%   .   No   No   1   0   High   Very  Slow  Progress  

Raston   Eric   0133647   CIS-­‐1510   15/SP   Enrolled   7   35%   52%   Pass   98.78%   .   No   No   2   0   Moderate   Low  Ac@vity  

Raston   Eric   0133647   CIS-­‐2640   15/SP   Enrolled   7   35%   36%   .   96.90%   .   No   No   1   0   Moderate   Low  Ac@vity  

Varney   Lisa   0369845   MAT-­‐2170   15/SP   Enrolled   7   35%   0%   N/A   .     .   No   No   1   0   High   No  Submissions  In  Week  1  

Vinton   Samantha   0975632   BIS-­‐1120   15/SP   Enrolled   4   21%   0%   N/A   .     .   No   No   0   0   High   No  Submissions  In  Week  1  

Vinton   Samantha   0975632   CIS-­‐1111   15/SP   Enrolled   4   21%   3%   .   100.00%   .   No   No   0   0   Moderate   Low  Ac@vity  

Vinton   Samantha   0975632   CIS-­‐1130   15/SP   Enrolled   4   21%   5%   .   90.00%   .   No   No   0   0   Moderate   Low  Ac@vity  

Vinton   Samantha   0975632   CIS-­‐1107   15/SP   Enrolled   4   21%   13%   .   100.00%   .   No   No   2   2   Low      

Aston   Noah   0512897   CIS-­‐1140   15/SP   Enrolled   7   35%   18%   N/A   90.00%   .   No   No   6   0   Moderate   Low  Ac@vity  

Barton   Emma   0221554   CIS-­‐1510   15/SP   Enrolled   7   35%   66%   Pass   96.82%   .   No   No   0   0   Moderate   Low  Ac@vity  

Bosco   James   0398521   CIS-­‐1140   15/SP   Enrolled   7   35%   18%   N/A   107.50%   .   No   No   1   0   Moderate   Low  Ac@vity  

Bosco   James   0398521   CIS-­‐2416   15/SP   Enrolled   7   35%   70%   N/A   99.50%   .   No   No   13   14   Low      

Brunswik   Sylvia   0021056   CIS-­‐2426   15/SP   Enrolled   7   35%   37%   N/A   99.43%   .   No   No   2   0   Moderate   Low  Ac@vity  

Clancy   Thomas   0648875   CIS-­‐1411   15/SP   Enrolled   7   35%   24%   N/A   99.62%   .   No   No   0   0   Moderate   Low  Ac@vity  

Clancy   Thomas   0648875   CIS-­‐1140   15/SP   Enrolled   7   35%   73%   N/A   95.31%   .   No   No   7   4   Low      

Colton   Daniel   0021557   CIS-­‐1107   15/SP   Enrolled   7   35%   25%   .   100.00%   .   No   No   0   0   Moderate   Low  Ac@vity  

Davis   Benjamin   0364987   CIS-­‐2212   15/SP   Enrolled   7   35%   44%   .   92.53%   .   No   No   1   0   Moderate   Low  Ac@vity  

Hanson   Holly   0864135   CIS-­‐1111   15/SP   Enrolled   4   21%   24%   .   100.00%   .   No   No   1   0   Moderate   Low  Ac@vity  

Hanks   Kendra   0215477   CIS-­‐2640   15/SP   Enrolled   4   21%   17%   .   93.31%   .   No   No   0   0   Moderate   Low  Ac@vity  

Harris   Stephanie   0677954   CIS-­‐1510   15/SP   Enrolled   .   .   100%   .   94.81%   A   .   .   .   .   .   .  

Accelerate  IT  Caseload  Performance  Report  Academic  Coach:    Chris@na  Amato  

CBE4CC

Lessons  Learned

• Develop  a  full  partnership  between  student-­‐support  and  IR/IT  personnel  

 • Maximize  the  capabili@es  of  exis@ng  data  systems  in  ways  that  enhance  and  transcend  standard  end-­‐user  applica@ons  

 •  Secure  faculty  buy-­‐in    • Know  and  engage  other  impacted  stakeholders  

Bellevue  College  Bellevue,  Washington

CBE4CC

About  Bellevue’s  Competency  Based  Program •  Launched  6  month  pilot  project  in  January  2014  for  our  Business  Sojware  Specialist  cer@ficate  program    •  Consists  of  6  courses  +  1  required  orienta@on  course  completed  in  a  quarter  system    •  Accepts  both  majors  and  non-­‐majors  who  need  the  course  as  part  of  another  program.    •  Program  con@nued  ajer  pilot  project  –  averaging  about  100  student  enrollments  per  quarter  and  growing  

CBE4CC

Navigator  Student  Support

•  Handles  enrollment  interviews  •  Provides  entry  codes  for  our  students  to  register  for  courses  

•  Teaches  the  orienta@on  course  •  Provides  support  and  checks  in  with  students  throughout  the  quarter  

•  Collects  lots  of  data  stored  in  many  different  places    •  SmarterMeasure  Test  Results  •  Interview  Notes  •  Entry  Codes/Interest  Lists  •  Progress  Reports  from  Students  •  Progress  Updates  from  Course  Learning  Management  System/Instructor  

CBE4CC

Faculty  Student  Support

•  Routes  inquiries  to  Navigator  for  enrollment/interview  • Works  with  Navigator  to  issue  entry  codes  •  Provides  support  and  checks  in  with  students  throughout  the  quarter  

•  Collects  lots  of  data  stored  in  many  different  places    •  Canvas  Learning  Management  System  (par@cipa@on,  page  views,  assignment  comple@on,  interac@on  with  Instructor,  last  logged  in)  

•  Other  course  sojware  such  as  MyITLab  progress  •  Feedback  from  Student  Navigator  

CBE4CC

The  Problem:  Managing  our  Data

Ajer  the  first  quarter  of  our  pilot  project,  we  knew  we  had  to  get  a  handle  on  the  enormous  amount  of  data  that  we  were  collec@ng  to  ensure  our  students  were  on  track  and  would  be  successful  comple@ng  the  course.    

CBE4CC

Navigator  Data?  Many  files  in  OneDrive

*this  is  only  about  1/3  of  the  files  (mostly  

excel)  from  the  past  17  months!    

CBE4CC

Faculty  Solu@on:  Lots  of  hand  wriUen  notes! My  first  idea  to  track  progress  was  to  keep  lots  of  notes  which  quickly  became  a  problem.    

They  weren’t  very  prety.    Or  very  func@onal.      But  they  helped  me  iden@fy  the  students  that  were  not  on  track.    

CBE4CC

Learning  Management  Analy@cs A  tradi@onal  online  course  analy@cs  –  Note  my  Sunday  deadlines  

A  competency  based  course  with  varying  start  dates  –  no  official  deadlines  

CBE4CC

Communica@on  Breakdown

There  was  a  communica@on  breakdown  between:  •  Faculty  and  Student  •  Navigator  and  Student  •  Faculty  and  Navigator  •  Everyone  and  data  

Too  much  @me  was  spent  trying  to  answer  simple  ques@ons  –    •  “Is  this  student  enrolled  in  mul@ple  competency  based  courses?”  •  If  so,  are  they  performing  well  in  those  courses?      

CBE4CC

Implemen@ng  a  Solu@on

We  needed  a  solu@on  that  helped  us  track  all  of  this  data  • Without  spending  hours  trying  to  locate  the  spreadsheet  or  progress  report  on  the  OneDrive  • Without  trying  to  locate  my  hand  writen  notes  or  my  own  self  created  progress  spreadsheets  • Without  having  to  email  constantly  back  and  forth  with  the  navigator  to  check  in  on  a  student  • Allows  for  a  quick  lookup,  progress  and  note  tracking  

 

CBE4CC

Implemen@ng  a  Solu@on

I  created  a  web  based  tool  to  help  manage  our  data.        

CBE4CC

Basic  Student  Data  &  Search

     

CBE4CC

Detailed  Student  Data,  Notes  &  Course  Plan

     

CBE4CC

Visual  Representa@on  of  Course  Road  Map

     

CBE4CC

Notes  on  Student  Progress  &  Communica@on

     

CBE4CC

Track  Student  Progress  Per  Class

     

CBE4CC

Other  Data  Tracked •  Interested  Student  Contact  Informa@on  &  Notes    •  File  Upload  for  SmarterMeasure  test  results  &  Orienta@on  assessment    • Progress  Report  Web  Based  Form  to  allow  students  to  quickly  update  us    • Automated  Progress  Report  Notes  from  Student  

     

CBE4CC

Wish  List/Pending  Development

•  Import  data  from  Course  Learning  Management  Sojware    •  Import  data  from  external  tools  (MyITLab,  Cer@fica@on  Prac@ce  Exams)    •  Student  Portal  for  feedback,  entry  code  request  &  course  road  map  view    •  Extensive  analy@cs/averages  per  class  to  iden@fy  problems    •  Automated  student  alert  system  for  progress  updates/concerns  

     

CBE4CC

What  we  have  learned • Managing  student  data  is  a  MUST  for  a  CBE  program  to  support  the  faculty,  student  navigator  and  the  student’s  overall  success!  •  As  the  program  grows,  it  is  very  challenging  to  keep  track  of  which  students  are  coming  and  going  •  Having  a  tool  in  place  from  the  beginning  will  save  a  lot  of  data  entry!    •  Being  able  to  u@lize  the  tool  saves  a  lot  of  @me  searching  for  informa@on  and  follow  up  between  Faculty  and  Student  Navigator  and  Student.    •  Allows  faculty  to  be  informed  when  communica@ng  with  students  and  see  the  big  picture.  •  Everyone  can  be  on  the  same  page!          

Considera@ons  and  Q&A

CBE4CC

 Considera@ons • What  problem  or  problems  can  data  help  your  ins@tu@on  solve?  

• Who  at  your  ins@tu@on  needs  to  be  involved  in  formula@ng  solu@ons  from  data?  

CBE4CC

Ques@ons  &  Answers