using student data to inform support, pedagogy & curricula: ethical issues & dilemmas
TRANSCRIPT
Presentation at the Milpark Business School (MBS)
Research Colloquium 25 June 2016Image credit: Image compiled and adapted from an image retrieved from- https://pixabay.com/en/binary-code-man-display-dummy-face-1327498/
Using student data to inform support, pedagogy & curricula: ethical issues & dilemmas
By Paul Prinsloo (University of South Africa, Unisa)
Acknowledgements
I do not own the copyright of any of the images in this presentation. I therefore acknowledge the original
copyright and licensing regime of every image used.
This presentation (excluding the images) is licensed under a
Creative Commons Attribution-NonCommercial 4.0 International
License
Overview of the presentation
• The broader context of research on students in higher education
• Student data as Medusa – mapping the field, the tools, the actors, the dimensions and the implications
• Looking away: Pointers for consideration• (In)conclusions
Higher education should…
• Do more with less• Expect funding to follow performance rather than precede it• Realise it costs too much, spends carelessly, teaches poorly, plans
myopically, and when questioned, acts defensively(Hartley, 1995, p. 412, 861)
We also cannot & shouldn’t underestimate the impact of the dominant models of neoliberalism and its not-so-humble servant – managerialism – on higher education (Diefenbach, 2007)
The broader higher education context
Image credit: http://commons.wikimedia.org/wiki/File:Mcdonalds_logo.png
• Changes in funding and audit regimes –evidence-based policy versus research-led…
• Increasing concerns regarding student retention and dropout
• International ranking systems, increased competition in higher education
The broader higher education context (2)
(See: Murphie, A. (2014). Auditland. PORTAL Journal of Multidisciplinary International Studies, 11(2), 1-41.)
The broader higher education context (3)• How do national,
institutional, disciplinary contexts support or frustrate efforts to remove barriers?
• What are the issues of costs and scalability in erasing inter-generational inequality?
• What data do we need in order to move towards more just, caring and compassionate access, teaching and learning?Image sources: https://twitter.com/urbandata/status/695261718344290304
https://za.pinterest.com/barbaralley/fair-is-not-equal/
• Looking for sustainable business models #FeesMustFall
• The algorithmic turn and quantification fetish in higher education
• The increasing digitisation of learning and teaching, and access to students’ digital shadows
• The gospel of technosolutionism in higher education
• The lure of Big(ger) data
The broader higher education context (4)
(Student) data as Medusa
Higher education is mesmerized and seduced by the potential of the collection, analysis and use of student data. If only we know more…
Image credit: http://en.wikipedia.org/wiki/Medusa
There is an increasing need for data/evidence
We have access to increasing amounts and granularity of student data
We have increased capacity & technologies for analysis and visualisation
The impact of impotent, static, & obsolete legislation, policies and guidelines
And a lack of oversight and enforcement
Image credit: Retrieved from https://www.flickr.com/photos/timrich26/3308513067
We need to ensure the sustainability of higher education in the light of• funding constraints• increased competition• the socioeconomic
downturn• student needs• increased need for
efficiency/effectiveness• audit & quality
assurance regimes• #FeesMustFall
The fiduciary duty of higher education to• care• create supportive,
appropriate and effective teaching and learning environments
• ethical collection, analysis and use of student data
• transparency
A balancing act
See: Prinsloo, P., & Slade, S. (2014). Educational triage in open distance learning: Walking a moral tightrope. The International Review of Research in Open and Distributed Learning, 15(4), 306-331. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1881/3060
So…, who has access to and use student data to inform student
support, pedagogy & curricula; and under what conditions and who
provides oversight
See: Willis, J. E., Slade, S., & Prinsloo, P. (in review). Ethical oversight of student data in learning analytics: A typology derived from a cross-continental, cross-institutional perspective. Submitted to special issue of Educational Technology Research and Development (Exploring the Relationship of Ethics in Design and Learning Analytics: Implications for the Field of Instructional Design and Technology), guest edited by M. Tracey and D. Ifenthaler.
When the collection, analysis and use of student data have an internal focus• Departmental/institutional reports
& planning• Scholarship of teaching and
learning• Provide appropriate and effective
student support• Allocation of staff/resources
When the collection, analysis and use of student data have an external focus• Reporting to a range of
stakeholders, e.g. government, industry, etc., and for a range of purposes, e.g., funding
• Conference presentations• Journal articles• Monographs & edited volumes• Popular press• Marketing
Institutional Research• Often located in a
designated department• Staffed by data
scientists, analysts• Inform strategy and
policy• Use student data
already ‘gifted’ during application/ registration process and from Learning Management System (LMS)
• Specific data collection• Often blanket ethical
clearance
Research (capital ‘R’)• Mostly faculty, but
increasingly support and professional staff• Varying skills and
understanding• Chasing outputs, h-
index, citations• Results mostly not used
to inform teaching and learning
• Use primary and secondary student data
• Oversight provided by Institutional Review
Boards (IRBs)
Emerging forms of research• Mostly faculty, but increasingly
support and professional staff• Varying skills and understanding• Not produced for formal outputs
eg publication, but to inform pedagogy, assessment, personalisation, departmental reports
• Often use student data already ‘gifted’ during application/ registration process and from Learning Management System (LMS) or personal synchronous or asynchronous communication
• No ethical review/oversight
Academic & learning analytics
Type of analytics
Level or object of analysis
Who benefits?
Learning analytics
Course level: social networks, conceptual development, discourse analysis, “intelligent curriculum”
Learners, faculty
Departmental: predictive modelling, patterns of success/failure
Learners, faculty
Academic analytics
Institutional: learner profiles, performance of academics, knowledge flow
Administrators, funders, marketing
Regional (state/provincial): comparisons between systems
Funders, administrators
National and International National governments, education authorities
Siemens, G., & Long, P. (2011). Penetrating the Fog: Analytics in Learning and Education. EDUCAUSE review, 46(5), 30-40. Retrieved from http://er.educause.edu/articles/2011/9/penetrating-the-fog-analytics-in-learning-and-education
“learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs.”
1st International Conference on Learning Analytics and Knowledge, Banff, Alberta, February 27–March 1, 2011. In Siemens and Long (2011)
(1)Humans perform
the task
(2)Task is shared
with algorithms
(3)Algorithms
perform task: human
supervision
(4)Algorithms
perform task: no human input
Seeing Yes or No? Yes or No? Yes or No? Yes or No?
Processing Yes or No? Yes or No? Yes or No? Yes or No?
Acting Yes or No? Yes or No? Yes or No? Yes or No?
Learning Yes or No? Yes or No? Yes or No? Yes or No?
Human-algorithm interaction in the collection, analysis and use of student data
Danaher, J. (2015). How might algorithms rule our lives? Mapping the logical space of algocracy. [Web log post]. Retrieved from http://philosophicaldisquisitions.blogspot.com/2015/06/how-might-algorithms-rule-our-lives.html
We know where you are. We know where you’ve been. We can
more or less know what you're thinking
about
(@FrankPasquale, 2016)
Image credit: https://en.wikipedia.org/wiki/Surveillance
Imagine what we could learn if we put a tracker on everyone and everything (Jurdak, 2016)
Image credit: https://www.flickr.com/photos/jeepersmedia/13966485507
Page credit: http://insider.foxnews.com/2016/01/31/oklahoma-college-forcing-students-wear-fitbits
Page credit: https://dzone.com/articles/are-university-campuses-turning-into-big-brother
Page credit: http://www.theguardian.com/higher-education-network/2015/nov/27/our-obsession-with-metrics-turns-academics-into-data-drones
Page credits: http://www.ft.com/cms/s/2/634624c6-312b-11e5-91ac-a5e17d9b4cff.html#slide0
‘how much is enough data to solve my problem?’
(Adryan, 2015)
Image credit: https://www.flickr.com/photos/uncle-leo/1341913549
How much (more) student data do we need?
… has become saturated with data – ranging from automatically collected, analysed and used, purposefully collected, analysed and used and volunteered on social media and in exchange of (perceived) benefits despite concerns about privacy, the uncertainty of how the data will be used (and combined with other sources of data) downstream and in the context where our trust in the collectors of data is often misplaced, irrational or wishful thinking (See Kitchen, 2013, pp. 262-263)
How do we think of the collection, analysis and use of student data in a world that…
Image credit: https://commons.wikimedia.org/wiki/File:Big_Hand_-_geograph.org.uk_-_644552.jpg
• Knowing• Not knowing• Knowing what we don’t know• Knowing what we may never know• Knowing more
The solution is not only (or necessarily?) in knowing more, but ensuring that once we know, we respond in ethical, caring,
discipline and context-appropriate ways
We need to critically consider the ethical implications of …
Pointers for a way forward• Students’ digital lives are but a minute part of a bigger whole – so we
should not pretend as if our data represent the whole• The data we collect are never ‘raw’, ‘uncontaminated’, or just ‘scraped’…
Our samples, choices, timing and tools change and impact on data. “Data are in fact framed technically, economically, ethically, temporally, spatially and philosophically. Data do not exist independently of the ideas, instruments, practices, contexts and knowledges used to generate, process and analyse them” (Kitchen, 2014, p. 2)
• Data have contexts. To re-use data outside of the original context and purpose for which it was collected impacts on the contextual integrity.
• Knowing ‘what’ is happening, does not necessarily tell us the ‘why’…• Education is an open, recursive system (Biesta 2007, 2010) where multiple
variables not only intersect but often also constitute one another. Let us therefore tread carefully between correlation and causation…
Caught between correlation and causation
Image credit: http://www.tylervigen.com/spurious-correlations
Caught between correlation and causation (cont.)
Image credit: http://www.tylervigen.com/spurious-correlations
Using student data and student vulnerability: between the devil and the deep blue sea?
Students (some more
vulnerable than others)
Generation, harvesting and
analysis of data
Our assumptions, selection of data and algorithms
may be ill-defined
Turning ‘pathogenic’ – “a response intended to
ameliorate vulnerability has the paradoxical effect of exacerbating existing
vulnerabilities or generating new ones”
(Mackenzie et al, 2014, p. 9)
Adapted from Prinsloo, P., & Slade, S. (2015). Student vulnerability, agency and learning analytics: an exploration. Presentation at LAK15, Poughkeepsie, NY, 16 March 2015 http://www.slideshare.net/prinsp/lak15-workshop-vulnerability-final
• boyd and Crawford (2011) point to the fact that just because we have access to increasing amounts and granularity of personal data, does not mean that we have to and need to collect, analyse and use this data
• While research participant involvement in Research (with a capital ‘R’) is governed by institutional review boards and policies, the (automatic) collection, analysis and use of individuals’ digital data in emerging forms of research (small ‘r’) often fall and take place outside of these policies and review boards (Willis, Slade & Prinsloo, in review)
Just because we can, does not mean we have to, and if we do, who will provide oversight?
Collecting, analysing and using student data: towards an ethics of care
1. Do no harm. Repeat after me. Do no harm2. They have a right to know. If not, then this research
resembles surveillance and spying, and not research3. Make it clear what data are collected, when, for what
purpose, for how long it will be kept and who will have access and under what circumstances
4. Provide students access to information and data held about them, to verify and/or question the conclusions drawn, and where necessary, provide context
5. Provide access to a neutral ombudsperson(See Prinsloo & Slade, 2015)
Collecting, analysing and using student data: towards an ethics of care (2)
6. Context matters. Downstream use for purposes other than the original purpose for the collection of data compromises the contextual integrity of data
7. Involve students in the meaning-making. They are not data points on a PowerPoint at a conference. They have contexts, histories. They are infinitely more than their data.
8. Who will we hold accountable for algorithms? 9. What are the benefits for students? For you? For the
institution? Be transparent.(See Prinsloo & Slade, 2015)
(In)conclusionsI am not your data, nor am I your vote bank,I am not your project, or any exotic museum object,I am not the soul waiting to be harvested,Nor am I the lab where your theories are tested,I am not your cannon fodder, or the invisible worker,or your entertainment at India habitat centre,I am not your field, your crowd, your history,your help, your guilt, medallions of your victory,I refuse, reject, resist your labels,your judgments, documents, definitions,your models, leaders and patrons,because they deny me my existence, my vision, my space,your words, maps, figures, indicators,they all create illusions and put you on pedestal,from where you look down upon me,So I draw my own picture, and invent my own grammar,I make my own tools to fight my own battle,For me, my people, my world, and my Adivasi self! ~Abhay Xaxa
Source: http://www.adivasiresurgence.com/i-am-not-your-data/
THANK YOUPaul Prinsloo Research Professor in Open Distance Learning (ODL)College of Economic and Management Sciences, Office number 3-15, Club 1, Hazelwood, P O Box 392Unisa, 0003, Republic of South Africa
T: +27 (0) 12 433 4719 (office)T: +27 (0) 82 3954 113 (mobile)
[email protected] Skype: paul.prinsloo59
Personal blog: http://opendistanceteachingandlearning.wordpress.com
Twitter profile: @14prinsp
Bibliography and additional reading
Biesta, G. (2007) Why “what works” won’t work: evidence-based practice and the democratic deficit in educational research, Educational Theory, 57(1),1–22. DOI: 10.1111/j.1741-5446.2006.00241.x.
Biesta, G. (2010) Why ‘what works’ still won’t work: from evidence-based education to value-based education, Studies in Philosophy of Education, 29, 491–503. DOI 10.1007/s11217-010-9191-x.
Booth, M. (2012, July 18). Learning analytics: the new black. EDUCAUSEreview, [online]. Retrieved from http://www.educause.edu/ero/article/learning-analytics-new-black
Boyd, D., & Crawford, K. (2013). Six provocations for Big Data. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1926431
Chamayou, G. (n.d.). Every move will be recorded. [Web log post]. Retrieved from https://www.mpiwg-berlin.mpg.de/en/news/features/feature14
Ball, N. (2013, November 11). Big Data follows and buries us in equal measure. [Web log post]. Retrieved from http://www.popmatters.com/feature/175640-this-so-called-metadata/
Bergstein, B. (2013, February 20). The problem with our data obsession. MIT Technology Review. Retrieved from https://www.technologyreview.com/s/511176/the-problem-with-our-data-obsession/
Bertolucci, J. (2014, July 28). Deep data trumps Big Data. Information Week. Retrieved from http://www.informationweek.com/big-data/big-data-analytics/deep-data-trumps-big-data/d/d-id/1297588
Booth, M. (2012, July 18). Learning analytics: the new black. EDUCAUSEreview, [online]. Retrieved from http://www.educause.edu/ero/article/learning-analytics-new-black
Boyd, D., & Crawford, K. (2013). Six provocations for Big Data. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1926431
Brock, A. (2015). Deeper data: a response to boyd and Crawford. Media, Culture & Society, 37(7), 1084-1088.Chamayou, G. (n.d.). Every move will be recorded. [Web log post]. Retrieved from
https://www.mpiwg-berlin.mpg.de/en/news/features/feature14 Citron, D.K., & Pasquale, F. (2014). The scored society: Due process for automated predictions.
http://ssrn.com/abstract=2376209 Crawford, K. (2013, April 1). The hidden biases in Big Data. Harvard Business Review. Retrieved from
https://hbr.org/2013/04/the-hidden-biases-in-big-data/ Crawford, K. (2014, May 30). The anxieties of Big Data. The New Inquiry. Retrieved from
http://thenewinquiry.com/essays/the-anxieties-of-big-data Danaher, J. (2014, January 6). Rule by algorithm? Big Data and the threat of algocracy.[Web log post]. Retrieved
from http://philosophicaldisquisitions.blogspot.com/2014/01/rule-by-algorithm-big-data-and-threat.html Danaher, J. (2015, June 15). How might algorithms rule our lives? Mapping the logical space of algocracy. [Web
log post]. Retrieved from http://philosophicaldisquisitions.blogspot.co.za/2015/06/how-might-algorithms-rule-our-lives.html
Deleuze. G. (1992). Postscript on the societies of control. October, 59 pp. 3-7. Diakopoulos, N. (2014). Algorithmic accountability. Digital Journalism. DOI: 10.1080/21670811.2014.976411 Diefenbach, T, 2007, The managerialistic ideology of organisational change management, Journal of
Organisational Change Management, 20(1), 126 — 144. Eubanks, V. (2014, January 15). Want to predict the future of surveillance? Ask poor communities. The American
Prospect. Retrieved from http://prospect.org/article/want-predict-future-surveillance-ask-poor-communities
Floridi, L. (2012). Big data and their epistemological challenge. Philosophy & Technology, 1-3.
.
Bibliography and additional reading (cont.)
Gitelman, L. (Ed.). (2013). “Raw data” is an oxymoron. London, UK: MIT Press.Hartfield, T. (2015, May 12 ). Next generation learning analytics: Or, how learning analytics is passé. [Web log post]. Retrieved from http://timothyharfield.com/blog/2015/05/12/next-generation-learning-analytics- or-how-learning-analytics-is-passe/ Hartley, D. 1995. The ‘McDonaldisation’of higher education: food for thought? Oxford Review of Education, 21(4), 409-423.Henman, P. (2004). Targeted!: Population segmentation, electronic surveillance and governing the unemployed in Australia. International Sociology, 19, 173-191Johnson, J.A. (2015, October 7). How data does political things: The processes of encoding and decoding data are never neutral. [Web log post]. Retrieved from http://blogs.lse.ac.uk/impactofsocialsciences/2015/10/07/how-data-does-political-things/ Kitchen, R. (2013). Big data and human geography: opportunities, challenges and risks. Dialogues in Human Geography, 3, 262-267. SOI: 10.1177/2043820613513388 Kitchen, R. (2014). The data revolution. London, UK: SAGE. Kitchen, R., & McArdle, G. (2016). What makes Big Data, Big Data? Exploring the ontological characteristics of 26 datasets. Big Data & Society, January-June, 1-10. DOI: 10.1177/2053951716631130 Knox, D. (2010). Spies in the house of learning: a typology of surveillance in online learning environments. Paper presented at Edge, Memorial University of Newfoundland, Canada, 12-15 October.Lagoze, C. (2014). Big Data, data integrity, and the fracturing of the control zone. Big Data & Society (July-
December), 1-11.
Bibliography and additional reading (cont.)
Leonhard, G. (2014, February 25). How tech is creating data "cravability," to make us digitally obese. Retrieved from http://www.fastcoexist.com/3026862/how-tech-is-creating-data-cravability-to-make-us-digitally-obese?utm_content=buffer643a8&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer
Lupton, D. (2015). The thirteen Ps of Big Data. This Sociological Life. Retrieved from https://www.researchgate.net/profile/Deborah_Lupton/publication/276207564_The_Thirteen_Ps_of_Big_Data/links/5552c2d808ae6fd2d81d5f20.pdf
Mayer-Schönberger, V. (2009). Delete. The virtue of forgetting in the digital age. Princeton, NJ: Princeton University Press.
Mayer-Schönberger, V., Cukier, K. (2013). Big data. London, UK: Hachette.Morozov, E. (2013a, October 23). The real privacy problem. MIT Technology Review. Retrieved from
http://www.technologyreview.com/featuredstory/520426/the-real-privacy-problem/ Morozov, E. (2013b). To save everything, click here. London, UK: Penguin Books. Morozov, E. (2013). To save everything, click here. London, UK: Penguin Books. Napoli, P. (2013). The algorithm as institution: Toward a theoretical framework for automated media production
and consumption. In Media in Transition Conference (pp. 1–36). DOI: 10.2139/ssrn.2260923Nissenbaum, H. 2015. Respecting context to protect privacy: Why meaning matters. Science and engineering
ethics. Retrieved from http://link.springer.com/article/10.1007/s11948-015-9674-9 Pasquale, F. (2015, October 14). Scores of scores: how companies are reducing consumers to single numbers. The Atlantic. Retrieved from http://www.theatlantic.com/business/archive/2015/10/credit- scores/410350/
Bibliography and additional reading (cont.)
Bibliography and additional reading (cont.)
Pasquale, F. [FrankPasquale]. (2016, February 19). "We know where you are. We know where you’ve been. We can more or less know what you're thinking about.” http://www.theatlantic.com/technology/archive/2016/02/google-cute-evil/463464/ … #Jigsaw [Tweet]. Retrieved from https://twitter.com/FrankPasquale/status/700473628605947904 Pasquale, F. (2015). The black box society. Harvard Publishing, US.Prinsloo, P. (2015, September 3). The ethics of (not) knowing our students. Presentation at the University of
South Africa, Pretoria. Retrieved from http://www.slideshare.net/prinsp/the-ethics-of-not-knowing-our-students-52373670
Prinsloo, P., & Slade, S. (2014). Educational triage in open distance learning: Walking a moral tightrope. The International Review of Research in Open and Distributed Learning, 15(4), 306-331. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1881/3060
Prinsloo, P., & Slade, S. (2015, March). Student privacy self-management: implications for learning analytics. In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp. 83-92). ACM. Retrieved from http://dl.acm.org/citation.cfm?id=2723585
Prinsloo, P., Archer, E., Barnes, G., Chetty, Y., & Van Zyl, D. (2015). Big (ger) data as better data in open distance learning. The International Review of Research in Open and Distributed Learning, 16(1).
Rosen, J. (2010, July 21). The web means the end of forgetting. New York Times [Online].Selwyn, N. (2014). Distrusting educational technology. Critical questions for changing times. New York, NY:
RoutledgeScharmer, O. (2014, July 18). From Big Data to deep data. Huffington Post. Retrieved from
http://www.huffingtonpost.com/otto-scharmer/from-big-data-to-deep-dat_b_5599267.html
Bibliography and additional reading (cont.)
Shacklett, M. (2015, January 6). Thick data closes the gaps in big data analytics. TechRepublic. Retrieved from http://www.techrepublic.com/article/thick-data-closes-the-gaps-in-big-data-analytics/
Slade, S., & Prinsloo, P. (2013). Learning Analytics: Ethical Issues and Dilemmas. American Behavioral Scientist 57(1) ,1509–1528.Slade, S., & Prinsloo, P. (2015). Student perspectives on the use of their data: between intrusion, surveillance and care. European Journal of Open, Distance and Elearning. (pp.16- 28).Special Issue. http://www.eurodl.org/materials/special/2015/Slade_Prinsloo.pdfTene, O. & Polonetsky, J. (2013). Judged by the Tin Man: Individual rights in the age of Big Data. J. on Telecomm.
& High Tech. L., 11, 351.Totaro, P., & Ninno, D. (2014). The concept of algorithm as an interpretive key of modern rationality. Theory
Culture Society 31, pp. 29—49. DOI: 10.1177/0263276413510051 Uprichard, E. (2013). Big data, little questions. Discover Society, 1 October. Retrieved from
http://discoversociety.org/2013/10/01/focus-big-data-little-questions/ Wagner, D., & Ice, P. (2012, July 18). Data changes everything: delivering on the promise of learning analytics in
higher education. EDUCAUSEreview, [online]. Retrieved from http://www.educause.edu/ero/article/data-changes-everything-delivering-promise-learning-analytics-higher-education
Wang, T. (2013, January 20). Why Big Data needs thick data. Medium. Retrieved from https://medium.com/ethnography-matters/why-big-data-needs-thick-data-b4b3e75e3d7#.4jbatgurh
Watters, A. (2013, October 13). Student data is the new oil: MOOCs, metaphor, and money. [Web log post]. Retrieved from http://www.hackeducation.com/2013/10/17/student-data-is-the-new-oil/
41
Watters, A. (2014). Social justice. [Web log post]. Retrieved from http://hackeducation.com/2014/12/18/top-ed-tech-trends-2014-justice
Wigan, M.R., & Clarke, R. (2013). Big data’s big unintended consequences. Computer,(June), 46-53. Willis, J. E., Slade, S., & Prinsloo, P. (in review). Ethical oversight of student data in learning analytics: A typology
derived from a cross-continental, cross-institutional perspective. Submitted to special issue of Educational Technology Research and Development (Exploring the Relationship of Ethics in Design and Learning Analytics: Implications for the Field of Instructional Design and Technology), guest edited by M. Tracey and D. Ifenthaler.
Bibliography and additional reading (cont.)