a 'generic' eportfolio: the newcastle experience background aims design implement...

A 'generic' ePortfolio: the Newcastle experience

Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Design, implementation and evaluation of a Design, implementation and evaluation of a 'generic' ePortfolio: the Newcastle experience.'generic' ePortfolio: the Newcastle experience.

Simon Cotterill, Tony McDonald, Paul Drummond, Geoff Hammond

School of Medical Education DevelopmentUniversity of Newcastle, UK


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

SST (Student Support & Tutoring)

FDTL-4 project• ePortfolios in Medicine• Evidencing outcomes• PDP

1st Newcastle-Nottingham PARs project, 1998-2000

2nd Newcastle-Nottingham PARs project, 2000-02

Internet-PARs• DfES funded • PDP focus

BackgroundBackground


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

• An FDTL-4 project developing ePortfolios for undergraduate Medicine (2002-2005):

• Newcastle University (lead site)

• Leeds University

• Sheffield University

• Dundee University (for consultancy)

Project funded by

BackgroundBackground


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

AimsAims

1. Personal Development Planning (PDP)

"a structured and supported process undertaken by an individual to reflect upon their own learning, performance and/or achievement and to plan for their personal, educational and career development”

Quality Assurance Agency for Higher Education, 2001.


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

a) Towards a life-long learning record

School HE CPD

AimsAims

2. Supporting Life-Long Learning (LLL)

b) Encouraging the development of LLL Skills: • Evidencing Learning Outcomes• Transition to Independent learning• Preparation for professional requirements:

- Appraisals- Assessment- Revalidation


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Aims (technical)Aims (technical)

3. Design a ‘generic’ ePortfolio

-transferable / useful in different contexts

4. Integration with VLEs


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Context-specific tools

Common tools

‘Generic’framework Common contents structure

• sharing• cross-referencing• searchable• integrated action planning

Customisation• select tools by course/year• outcomes / skills sets• nomenclature• graphics / layout• simple tool editor

• CV• Outcomes / skills log• PDP• Reflective learning diary • Meetings log• SWOT

Integrated into thecommon contents structure

ArchitectureArchitecture

Developed with Open Source products: Zope, MySQL


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

• create context-specific tools via simple Web forms

Student view

Selecting tools by Course and Year

Course Admin view

‘‘Generic’ framework: CustomisatonGeneric’ framework: Customisaton


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Configuring learning outcomes / skills sets

Course Admin view

‘‘Generic’ framework: CustomisatonGeneric’ framework: Customisaton


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

‘‘Generic’ framework: SharingGeneric’ framework: Sharing


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Piloting & Implementation

The ePortfolio developed at Newcastle has been applied in the following contexts:

2003/4 Undergraduate MedicineContract Research Staff (CPD)Generic Portfolio (project demonstrator)

2004/5 Dentistry (Undergrad + Vocational Training)Undergraduate BiosciencesSt Andrew’s University (Medicine)


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Generic ePortfolio:Generic ePortfolio:Public DemonstratorPublic Demonstrator http:www.eportfolios.ac.uk


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Medicine at Newcastle:Medicine at Newcastle:Integration with a bespoke VLEIntegration with a bespoke VLE


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Common Tools: Common Tools: Learning Diary (Medicine – Newcastle)Learning Diary (Medicine – Newcastle)


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Context-specific Tools: Context-specific Tools: PDP for Year 4 Student Selected ComponentsPDP for Year 4 Student Selected Components


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Common Tools: Common Tools: Record courses (CRS)Record courses (CRS)


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Common Tools: Common Tools: Learning Outcome / Skills (CRS)Learning Outcome / Skills (CRS)


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Common Tools: Common Tools: SWOT (Bioscience prototype)SWOT (Bioscience prototype)


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Common Tools: Common Tools: CV (St Andrew’s University)CV (St Andrew’s University)


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Common Tools: Common Tools: Tutor meetings (St Andrew’s University)Tutor meetings (St Andrew’s University)


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Context-specific Tools:Context-specific Tools:DentistryDentistry

Project involves 6 Dental schools and 2 Postgraduate Deaneries


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Promoting Continuity & LLL in Dentistry

Context-specific Tools: Context-specific Tools: DentistryDentistry


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Integration #2: Blackboard Integration #2: Blackboard (via a ‘tab’)(via a ‘tab’)


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Integration #2: Blackboard Integration #2: Blackboard (via (via the tool barthe tool bar))


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Evaluation #1: Evaluation #1: Medicine (Medicine (Years 1&2Years 1&2))

Cotterill SJ, Sarma S, McDonald AM, Bradley P

• Ethical approval / informed consent

• Focus groups + Questionnaires

• Key issues:• Generally well received but what motivation ?

• not assessed• not viewed by tutors (staff interface completed later)

• Need better clarity of purpose• Wanted a more quirky / fun design !• Learning diary - useful at first, less so over time• Good to browse the Learning Outcomes


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

• Ethical approval / informed consent

• Questionnaires (before and after using the ePortfolio)• Change in awareness of intended learning outcomes ?• Factors influencing use of the ePortfolio• Attitudes and perceptions of using the ePortfolio• Student perception of impact of using the ePortfolio• Evaluation of technical features and ‘usability’

• 186 students completed the portfolio (~100% compliance)

• 105 questionnaires completed (56% of cohort) at 30/06/2004

Evaluation #2: Evaluation #2: MedicineMedicine (Year(Year 4 4)) PDP for PDP for student selected componentsstudent selected components

Cotterill SJ, McDonald AM, Bradley P, Robinson R, Hammond GR


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Evaluation - positiveEvaluation - positive

“It encouraged me to really give thought to what I wanted to achieve during the option, which was especially useful as this was my first option. As a result of the portfolio I think I got much more out of the option than I would have otherwise.”

“It made me concentrate on creating aims at the start of the option and allowed me to plan the option with my supervisor in a defined way. Overall it made my learning for the option more organised and focused.”


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Evaluation - barriers Evaluation - barriers && problems: problems:

“Most of the things I learned couldn't be 'measured/quantified' so I felt it was of little value to try and invent a way in which they could.”

“Access to computer as option took place in Barnard Castle, records may have been more detailed if completed straight after meetings etc.”


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Building the ePortfolio was a useful learning experience

0 10 20 30 40 50

Strongly Disagree

Weakly Disagree

Agree

% responses

80% thought it wasa useful learning experience

0 10 20 30 40 50

Strongly Disagree

Disagree

Weakly Disagree

Weakly Agree

Agree

Strongly Agree

% responses

Having clearly defined intended learning outcomes influenced the way in which I approached the option

72% felt that the LOsinfluenced their approach


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

0 10 20 30 40 50

Strongly Disagree

Disagree

Weakly Disagree

Weakly Agree

Agree

Strongly Agree

% responses

93% reflected on their learning after their SSC

At the end of the option I spent time considering what I had learned from this option.


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Conclusions #1Conclusions #1

The ‘generic’ ePortfolio

• Highly configurable

• Can be readily adapted for different contexts

• Existing tools have an emphasis on PDP

-but the portfolio framework can serve other purposes

Integration in VLEs

• Fully integrated into a VLE for Medicine

• Initial work to integrate with Blackboard has been successful


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Evaluation (in Medicine)

• ePortfolios implemented / piloted in 2003/4

• Demonstrated: Feasibility & Acceptability

• Most students perceive these as beneficial

• However, unless assessed / mandatory the motivation to complete is reduced.



Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

A systematic map and synthesis review of the effectiveness of personal development planning for improving student learning.EPPI-Centre (2002) http://eppi.ioe.ac.uk

Educational Objectives (including PDP)

• Some evidence that PDP had a positive impact on planning and approach to learning (Year 4 SSC)

• Encouraged reflection in some students

• Good preparation (and practice) for LLL & CPD

• However, long-term impact on learning is unknown

• Further research is essential

• Some (limited) evidence that PDP improves learning:



Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Additional Developments:

• ePortfolios Extension Toolkit (ePET)

-A JISC funded project to develop a ‘Web services’

interface for the portfolio (by 04/2005)

http://www.eportfolios.ac.uk/ePET

• IAMSECT ‘Shibboleth’ Single Sign-On project

- JISC funded project

http://iamsect.ncl.ac.uk

Further information on the ePortfolio:

http://www.eportfolios.ac.uk

Simon Cotterill ([email protected])


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

Implementation of ePortfolios

- Humans are important too !

• Buy-in of key stakeholders:

Learners Clarity of purpose ? Sense of Ownership ?

Tutors & Admin staff Extra work ?

Senior Curriculum staff

• Integration with existing processes

• Training (all)

• Monitoring and evaluation


Background

Aims

Design

Implement

Evaluation

Conclusions

www.epor

tfolio

s.ac.u

k

InstitutionalInstitutionalDataData

Portfolio for Presentation

Portfolio for Application

(job / promotion)

Portfolio for Assessment

Portfolio for Accreditation/Revalidation

PDP(shared)

PDP / Reflective(private)

Portfolio for Appraisal

Learner’sLearner’s‘‘repository’repository’

Central data:TranscriptMIS/ HR data

Programme data:Granular assessment dataOutcomes / skills sets

Selecting sub-sets of portfolio data for different purposes:

a 'generic' eportfolio: the newcastle experience background aims design implement...

Documents

pdp focus background

learning outcomes skills

customisaton slide

mysql slide

revalidation slide

vles slide

project eportfolios

courseyear outcomes