learning analytics: what are we optimizing for?

httptwittercomsbskmi

Learning Analytics what are we optimizing for

Simon Buckingham Shum Knowledge Media Institute The Open University UK simonbuckinghamshumnet

edfuturenet MOOC on CurrentFuture State of HigherEd

1

Knowledge Media Institute

edX ldquothis is big data giving us the chance to ask big questions about learningrdquo

2

Will the tomorrowrsquos educational researcher be

as helpless without an analytics infrastructure as

a geneticist without genome databases or a physicist without CERN

the planhellip

joined-up multi-layer analytics an analytics ecosystem

are analytics (r)evolutionary

3

the convergence of analytics layers

4

Micro individual user actions

(and hence cohort)

MacroMesoMicro Learning Analytics

Meso institution-wide

Macro regionstatenationalinternational


(and hence cohort)




Will institutions be dazzled by the dashboards or know what

questions to ask at each level

For examples of each level of analytichellip

7 Buckingham Shum S 2012 Our Learning Analytics are Our Pedagogy Keynote Address Expanding Horizons 2012 Conference Macquarie University Sydney httpwwwslidesharenetsbsour-learning-analytics-are-our-pedagogy

The VLEmdashBImdashITS convergence

8


(and hence cohort)

As data migrates up it enriches higher layers normally accustomed to sparse data



Aggregation of user traces enriches meso + macro analytics with finer-grained process data


(and hence cohort)

hellipwhich in turn could enrich lower layers mdash local patterns can be cross-validated




Breadth + depth from macro + meso levels could add power to micro-analytics

anatomy of an analytics ecosystem

11

A learning analytics ecosystem

12

learners

educators


13

learners

educators


14

learners

educators


15

learners

educators

data curators translators

dashboard design team

data capture design team

Where did the data come from

16

learners


17

learners

researchers educators instructional designers

theories pedagogies

assessments tools


18

learners


theories pedagogies

assessments tools

technologists

Data Intent

The map is not the territory Analytics are not the end but a means The goal is to optimize the whole system

19

learners


theories pedagogies

assessments tools

desi

gn feedback

intent

outcome

Optimize the system for what

20

Same outcomes but higher scores

Learning Analytics as

Evolutionary Technology

bull more engaging bull better assessed bull better outcomes

bull deliverable at scale

21

New outcomes we couldnrsquot assess before


Revolutionary Technology

bull learner behaviours quantifiable bull interpersonal networks quantifiable

bull discourse quantifiable bull moods and dispositions quantifiable

22

ldquoWe are preparing students for jobs that do not exist yet that will use technologies that have not been invented yet in order to solve problems that are not even problems yetrdquo

ldquoShift Happensrdquo httpshifthappenswikispacescom

23

Learning analytics for this


ldquoWhile employers continue to demand high academic standards they also now want more They want people who can adapt see connections innovate communicate and work with others This is true in many areas of work The new knowledge-based economies in particular will increasingly depend on these abilities Many businesses are paying for courses to promote creative abilities to teach the skills and attitudes that are now essential for economic successhelliprdquo

All our Futures Creativity culture amp education May 1999 24


25

Creativity Culture and Education (2009) Changing Young Lives 2012 Newcastle CCE httpwwwcreativitycultureeducationorgchanging-young-lives-2012

Think about the analytics products and initiatives reviewed above ndash where would you locate them on these dimensions

Expert-led enquiry

Student-led enquiry

Expert-led teaching

Student-led revision

Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge

Teacher agency Student agency

Repetition Abstraction Acquisition

Authenticity Agency Identity

Teaching as learning design

The Knowledge-Agency Window


Ruth Deakin Crick Univ Bristol Centre for Systems Learning amp Leadership ldquoPedagogy of Hoperdquo httplearningemergencenet20120921pedagogy-of-hope

analytics grounded in the principles of good

assessment for learning

(not summative assessment for

grading pupils teachers institutions or nations)

27

Assessment for Learning

28

httpassessment-reform-grouporg Few learning analytics are

currently able to take o board the richness of this

original conception of assessment for learning


29

httpassessment-reform-grouporg


30


To what extent could automated

feedback be designed and evaluated with

emotional impact in mind


31


Can analytics identify proxies

for such advanced qualities


32


How do we provide analytics feedback

that does not disempower and de-motivate struggling

learners

analytics for learning conversations

33

Socio-cultural discourse analysis (Mercer et al OU)

bull  Disputational talk characterised by disagreement and individualised decision making

bull  Cumulative talk in which speakers build positively but uncritically on what the others have said

bull  Exploratory talk in which partners engage critically but constructively with each others ideas

34 Mercer N (2004) Sociocultural discourse analysis analysing classroom talk as a social mode of thinking Journal of Applied Linguistics 1(2) 137-168


bull  Statements and suggestions are offered for joint consideration

bull  These may be challenged and counter-challenged but challenges are justified and alternative hypotheses are offered

bull  Partners all actively participate and opinions are sought and considered before decisions are jointly made

bull  Compared with the other two types in Exploratory talk knowledge is made more publicly accountable and reasoning is more visible in the talk

35


Mercer N (2004) Sociocultural discourse analysis analysing classroom talk as a social mode of thinking Journal of Applied Linguistics 1(2) 137-168

Analytics for identifying Exploratory talk

36

Elluminate sessions can be very long ndash lasting for hours or even covering days of a conference

It would be useful if we could identify where quality learning conversations seem to be taking place so we can recommend those sessions and not have to sit through online chat about virtual biscuits

Ferguson R and Buckingham Shum S Learning analytics to identify exploratory dialogue within synchronous text chat 1st International Conference on Learning Analytics amp Knowledge (Banff Canada 27 Mar-1 Apr 2011)

Defining indicators of Exploratory Talk

37

Category Indicator Challenge But if have to respond my view Critique However Irsquom not sure maybe Discussion of resources

Have you read more links

Evaluation Good example good point Explanation Means that our goals Explicit reasoning Next step relates to thatrsquos why Justification I mean we learned we observed Reflections of perspectives of others

Agree here is another makes the point take your point your view

Extract classified as Exploratory Talk

38

Time Contribution 242 PM I hate talking -P My question was whether gadgets were just

basically widgets and we could embed them in various web sites like Netvibes Google Desktop etc

242 PM Thanks thats great I am sure I understood everything but looks inspiring

243 PM Yes why OU tools not generic tools

243 PM Issues of interoperability

243 PM The new SocialLearn site looks a lot like a corkboard where you can add various widgets similar to those existing web start pages

243 PM What if we end up with as many appsgadgets as we have social networks and then we need a recommender for the apps

243 PM My question was on the definition of the crowd in the wisdom of crowds we acsess in the service model

243 PM there are various different flavours of widget eg Google gadgets W3C widgets etc SocialLearn has gone for Google gadgets

-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory

Discourse analytics on webinar textchat

Sheffield UK not as sunny as yesterday - still warm Greetings from Hong Kong Morning from Wiltshire sunny here

See you bye for now bye and thank you Bye all for now

Extensions to Ferguson R and Buckingham Shum S (2011) Learning Analytics to Identify Exploratory Dialogue within Synchronous Text Chat Proc 1st Int Conf Learning Analytics amp Knowledge Feb 27-Mar 1 2011 Banff ACM Press Eprint httporoopenacuk28955

Given a 25 hour webinar where in the live textchat were the most effective learning conversations Not at the start and end of a webinar but if we zoom in on a peakhellip


-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag

Wei amp He extensions to Ferguson R and Buckingham Shum S (2011) Learning Analytics to Identify Exploratory Dialogue within Synchronous Text Chat Proc 1st Int Conf Learning Analytics amp Knowledge Feb 27-Mar 1 2011 Banff ACM Press Eprint httporoopenacuk28955

Classified as ldquoexploratory

talkrdquo

(more substantive for learning)

ldquonon-exploratoryrdquo

Given a 25 hour webinar where in the live textchat were the most effective learning conversations

De Liddo A Buckingham Shum S Quinto I Bachler M and Cannavacciuolo L Discourse-centric learning analytics 1st International Conference on Learning Analytics amp Knowledge (Banff 27 Mar-1 Apr 2011) httporoopenacuk25829

KMirsquos Cohere a web deliberation platform enabling semantic social network and discourse network analytics

Rebecca is playing the role of broker

connecting 2 peersrsquo contributions in meaningful ways

analytics for scholarly writing

42

Discourse analysis (Xerox Incremental Parser)

BACKGROUND KNOWLEDGE

Recent studies indicate hellip

hellip the previously proposed hellip

hellip is universally accepted

NOVELTY

new insights provide direct evidence

we suggest a new approach

results define a novel role

OPEN QUESTION hellip little is known hellip hellip role hellip has been elusive

Current data is insufficient hellip

GENERALIZING emerging as a promising approach Our understanding has grown exponentially growing recognition of the

importance

CONRASTING IDEAS hellip unorthodox view resolves hellip paradoxes hellip

In contrast with previous hypotheses

inconsistent with past findings

SIGNIFICANCE studies have provided important advances

Knowledge is crucial for understanding

valuable information from studies

SURPRISE We have recently observed surprisingly

We have identified unusual The recent discovery suggests intriguing roles

SUMMARIZING The goal of this study Here we show

Altogether our results indicate

Detection of salient sentences in scholarly reports based on the rhetorical signals authors use

Aacutegnes Saacutendor amp OLnet Project httpolnetorgnode512

De Liddo A Saacutendor Aacute and Buckingham Shum S Contested Collective Intelligence Rationale Technologies and a Human-Machine Annotation Study Computer Supported Cooperative Work 21 4-5 (2012) 417-448 httporoopenacuk31052

Human and machine analysis of a text for key contributions

19 sentences annotated 22 sentences annotated 11 sentences same as human annotation


Document 1

Document 2

httptechnologieskmiopenacukcohere20120109cohere-plus-automated-rhetorical-annotation De Liddo A Saacutendor Aacute and Buckingham Shum S Contested Collective Intelligence Rationale Technologies and a Human-Machine Annotation Study Computer Supported Cooperative Work 21 4-5 (2012) 417-448 httporoopenacuk31052

analytics for intepersonal networking

45

Semantic Social Network Analytics


Visualizing and filtering social ties in SocialLearn by topic and type

Visualising Social Learning in the SocialLearn Environment Bieke Schreurs and Maarten de Laat (Open University The Netherlands) Chris Teplovs (Problemshift Inc and University of Windsor) Rebecca Ferguson and Simon Buckingham Shum (Open University UK) SoLAR Storm webinar Open University UK httpbitlyUaFhbL













Closing thoughts

54

55

ldquoThe basic question is not what can we measure

The basic question is

what does a good education look likerdquo

(Gardner Campbell)

httpchroniclecomblogstechtherapy20120502episode-95-learning-analytics-could-lead-to-wal-martification-of-college httplak12wikispacescomRecordings

56

Our analytics promote values pedagogy and assessment regimes

Are we clear which master

our analytics serve Are we happy to be judged by them

57

Will learning analytics merely turbocharge the current educational paradigm

mdash which is so often declared

not fit for purposehellip

58

hellipor will learning analytics reflect what we now know about designing authentic

engaged learning developing the new qualities that a

complex society demands

edX ldquothis is big data giving us the chance to ask big questions about learningrdquo

2

Will the tomorrowrsquos educational researcher be

as helpless without an analytics infrastructure as

a geneticist without genome databases or a physicist without CERN

the planhellip



3


4


(and hence cohort)





(and hence cohort)









8


(and hence cohort)






(and hence cohort)







11


12

learners

educators


13

learners

educators


14

learners

educators


15

learners

educators





16

learners


17

learners


theories pedagogies

assessments tools


18

learners


theories pedagogies

assessments tools

technologists

Data Intent


19

learners


theories pedagogies

assessments tools

desi

gn feedback

intent

outcome


20






21






22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58




the planhellip



3


4


(and hence cohort)





(and hence cohort)









8


(and hence cohort)






(and hence cohort)







11


12

learners

educators


13

learners

educators


14

learners

educators


15

learners

educators





16

learners


17

learners


theories pedagogies

assessments tools


18

learners


theories pedagogies

assessments tools

technologists

Data Intent


19

learners


theories pedagogies

assessments tools

desi

gn feedback

intent

outcome


20






21






22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





4


(and hence cohort)





(and hence cohort)









8


(and hence cohort)






(and hence cohort)







11


12

learners

educators


13

learners

educators


14

learners

educators


15

learners

educators





16

learners


17

learners


theories pedagogies

assessments tools


18

learners


theories pedagogies

assessments tools

technologists

Data Intent


19

learners


theories pedagogies

assessments tools

desi

gn feedback

intent

outcome


20






21






22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





(and hence cohort)





(and hence cohort)









8


(and hence cohort)






(and hence cohort)







11


12

learners

educators


13

learners

educators


14

learners

educators


15

learners

educators





16

learners


17

learners


theories pedagogies

assessments tools


18

learners


theories pedagogies

assessments tools

technologists

Data Intent


19

learners


theories pedagogies

assessments tools

desi

gn feedback

intent

outcome


20






21






22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





(and hence cohort)









8


(and hence cohort)






(and hence cohort)







11


12

learners

educators


13

learners

educators


14

learners

educators


15

learners

educators





16

learners


17

learners


theories pedagogies

assessments tools


18

learners


theories pedagogies

assessments tools

technologists

Data Intent


19

learners


theories pedagogies

assessments tools

desi

gn feedback

intent

outcome


20






21






22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58







8


(and hence cohort)






(and hence cohort)







11


12

learners

educators


13

learners

educators


14

learners

educators


15

learners

educators





16

learners


17

learners


theories pedagogies

assessments tools


18

learners


theories pedagogies

assessments tools

technologists

Data Intent


19

learners


theories pedagogies

assessments tools

desi

gn feedback

intent

outcome


20






21






22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





(and hence cohort)






(and hence cohort)







11


12

learners

educators


13

learners

educators


14

learners

educators


15

learners

educators





16

learners


17

learners


theories pedagogies

assessments tools


18

learners


theories pedagogies

assessments tools

technologists

Data Intent


19

learners


theories pedagogies

assessments tools

desi

gn feedback

intent

outcome


20






21






22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





(and hence cohort)







11


12

learners

educators


13

learners

educators


14

learners

educators


15

learners

educators





16

learners


17

learners


theories pedagogies

assessments tools


18

learners


theories pedagogies

assessments tools

technologists

Data Intent


19

learners


theories pedagogies

assessments tools

desi

gn feedback

intent

outcome


20






21






22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





11


12

learners

educators


13

learners

educators


14

learners

educators


15

learners

educators





16

learners


17

learners


theories pedagogies

assessments tools


18

learners


theories pedagogies

assessments tools

technologists

Data Intent


19

learners


theories pedagogies

assessments tools

desi

gn feedback

intent

outcome


20






21






22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





12

learners

educators


13

learners

educators


14

learners

educators


15

learners

educators





16

learners


17

learners


theories pedagogies

assessments tools


18

learners


theories pedagogies

assessments tools

technologists

Data Intent


19

learners


theories pedagogies

assessments tools

desi

gn feedback

intent

outcome


20






21






22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





13

learners

educators


14

learners

educators


15

learners

educators





16

learners


17

learners


theories pedagogies

assessments tools


18

learners


theories pedagogies

assessments tools

technologists

Data Intent


19

learners


theories pedagogies

assessments tools

desi

gn feedback

intent

outcome


20






21






22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





14

learners

educators


15

learners

educators





16

learners


17

learners


theories pedagogies

assessments tools


18

learners


theories pedagogies

assessments tools

technologists

Data Intent


19

learners


theories pedagogies

assessments tools

desi

gn feedback

intent

outcome


20






21






22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





15

learners

educators





16

learners


17

learners


theories pedagogies

assessments tools


18

learners


theories pedagogies

assessments tools

technologists

Data Intent


19

learners


theories pedagogies

assessments tools

desi

gn feedback

intent

outcome


20






21






22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





16

learners


17

learners


theories pedagogies

assessments tools


18

learners


theories pedagogies

assessments tools

technologists

Data Intent


19

learners


theories pedagogies

assessments tools

desi

gn feedback

intent

outcome


20






21






22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





17

learners


theories pedagogies

assessments tools


18

learners


theories pedagogies

assessments tools

technologists

Data Intent


19

learners


theories pedagogies

assessments tools

desi

gn feedback

intent

outcome


20






21






22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





18

learners


theories pedagogies

assessments tools

technologists

Data Intent


19

learners


theories pedagogies

assessments tools

desi

gn feedback

intent

outcome


20






21






22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58




Data Intent


19

learners


theories pedagogies

assessments tools

desi

gn feedback

intent

outcome


20






21






22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





20






21






22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58









21






22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58









22



23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58






23






25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58








25



Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58




Expert-led enquiry

Student-led enquiry

Expert-led teaching


Kno

wle

dge

co

-gen

erat

ion

an

d us

e

Pre-

scrib

ed

Kno

wle

dge












27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58








27


28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





28





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





29



30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





30






31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





31





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





32




learners


33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





33











35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58














35




36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





36





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





37






38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





38










-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58




-60

-40

-20

0

20

40

60

80

92

8 9

32

93

6

94

0

94

1

94

6

95

0

95

3

95

6

10

00

10

05

10

07

10

07

10

09

10

13

10

17

10

23

10

27

10

31

10

35

10

40

10

45

10

52

10

55

11

04

11

08

11

11

11

17

11

20

11

24

11

26

11

28

11

31

11

32

11

35

11

36

11

38

11

39

11

41

11

44

11

46

11

48

11

52

11

54

12

00

12

03

12

04

12

05

Average Exploratory







-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





-100

-50

0

50

100

92

8 9

40

9

50

1

000

1

007

1

017

1

031

1

045

1

104

1

117

1

126

1

132

1

138

1

144

1

152

1

203

Averag



talkrdquo









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58









42






NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58









NOVELTY







importance

















Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58







Document 1

Document 2



45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58





45

















Closing thoughts

54

55




(Gardner Campbell)


56




57




58




















Closing thoughts

54

55




(Gardner Campbell)


56




57




58




55




(Gardner Campbell)


56




57




58




56




57




58




57




58




58


