computer science students perceptions of learning formal reasoning methods

INT. J. MATH. EDUC. SCI. TECHNOL., 1993, VOL. 24, NO. 5, 749-759

Computer science students perceptions oflearning formal reasoning methods

by P. FUNG and T. O'SHEACentre for Information Technology in Education,

Open University, Milton Keynes, England

D. GOLDSON, S. REEVES and R. BORNATQueen Mary and Westfield College, London, England

(Received 3 August 1992)

We outline the rationale, put forward by those concerned with softwareengineering, for adopting a formal methods approach to programming. Anempirical study was undertaken with second and third year computer scienceundergraduates at Queen Mary and Westfield College, University of London.Students were invited to analyse retrospectively the difficulties which theyperceived they encountered in learning formal reasoning methods. We presentthe results of the study and consider the pedagogical implications of students'perceptions of formal reasoning methods as reflected in those results. We note theproblem that can arise caused by the sometimes conflicting expectations ofcomputer science students first encountering a formal reasoning approach toprogramming and those engaged in teaching the principles involved. Weconclude by indicating the steps which we are currently taking in order to addressthe issue of mismatched expectations and to overcome the difficulties reported bythose students interviewed.

1. IntroductionThe current trend towards improving the reliability of computer software has led

to interest in adopting a more rigorous approach to computer programming. It isgenerally acknowledged that computer software design would benefit from in-sistence on a high standard of reliability. A major concern, discussed in detail byGries [1], is the need to develop a scientific as opposed to an operational approach toprogramming and software design. Employing formal methods for programverification is seen as an essential step in achieving a precision and clarity of programdesign [2]. Those supporting this view believe that if software engineering is tobecome a legitimate branch of 'engineering' then it must develop the same rigorousapproach to design and planning that is taken for granted in traditional engineeringprojects. They feel that the high standard of reliability and quality demanded ofsophisticated software systems today can be better satisfied if the specification anddesign of those systems are underpinned with verification of a mathematical rigour.Both in industry and universities, formal reasoning methods are increasingly seen asa means of reducing the likelihood of errors in computer software products byestablishing the correctness of programs at an early stage in the design process. Agrowing number of universities are integrating this approach to programming intotheir undergraduate computer courses. This, however, raises issues related to theprocess of learning formal reasoning methods and the difficulties which many

0020-739X/93 $10-00 © 1993 Taylor & Francis Ltd.

750 P. Fung et al.

computer science students experience on such courses. The study described in thispaper is part of a broader collaborative project between Queen Mary and Westfieldcollege (QMW) and the Open University (OU) which attempts to address theseissues.1 The following section is a very brief outline of the project.

Two major cycles of empirical studies are being undertaken by the OU (O'Sheaand Fung), while at QMW members of the computer science department (Bornat,Reeves and Goldson) are undertaking a project which will result in the developmentof a number of prototype tools. The goal of the first cycle of empirical studies is toinvestigate the difficulties students experience in learning formal methods ofreasoning and to look closely at the nature of those difficulties. Feedback from theseinvestigations is playing a formative role in tool development. QMW are introducinga selection of computer based tools (both commercially available packages and theirown 'in-house' developed tools) into their first year computer science courses. Bymaking it easier and more intuitive for students to manipulate formal expressions,the tools should help to reduce difficulties students encounter in learning how toreason formally. In tandem with the introduction of these tools our second cycle ofempirical studies will begin. This will provide a summative evaluation of theeffectiveness of using such tools to address students' difficulties with formalreasoning methods and provide a basis for further development and refinement ofthe 'in-house' tools.

As an initial step in our first cycle of empirical work we undertook a preliminarystudy in spring 1991. Data was obtained from questionnaires and interviews. In thefollowing sections we outline the study and in the light of the results look at thefactors which may contribute to the difficulties some students experience. Weconclude by considering the implications of this for computer science teaching.

2. The studyThe computer science department at QMW specifically introduces formal

reasoning about programs into the core of its first year curriculum [3]. Successfulteaching of formal reasoning techniques and the principle which underlies their use,i.e. that programming and software design is a considered and formally provableprocess rather than an ad hoc 'try it and see' affair, is seen to have long-termadvantages both for the study of computer science and practically for the studentsthemselves. Gries [1] summarizes what he sees as the advantages for the latter:

one learns that the shape of the formalization can itself lend insight intodeveloping a solution. One acquires the urge to clarify and simplify, to seek theright notation in which to express a problem. One acquires a frame of mind thatencourages precision and rigor. This frame of mind can have a strikinglybeneficial effect on what work one does later as a professional in computing.

The benefits of using a formal approach to software development are indeedrelatively easy to appreciate when one considers the growth of software applicationsdesigned to be used in circumstances which demand a high level of reliability.However, in common with computer science department staff in other universities,it is recognized at QMW that learning to use formal methods is not without itsproblems for undergraduate students. In the study described here, current second

1 The Calculator Project, funded by the Joint ESRC, MRC, SERC Committee, CognitiveScience/HCI initiative (ref. no. 9019558).

Perception of learning formal reasoning methods 751

and third year computer science students were questioned retrospectively abouttheir attitudes to learning formal reasoning techniques and the difficulties they had,if any, with these techniques during their first year courses. From the results wehoped to gain, at the outset of the project, a first insight into students' perceptions ofthe values and the difficulties of learning formal methods.

2.1. Courses using formal reasoning methodsFour courses, in each of which learning about formal reasoning techniques

formed a significant part of the work, were chosen as the basis, both in questionnairesand interviews, for our investigation of students' experiences. Two of these courseswere taken in the first semester of the academic year:

(i) Programming I (PI)(ii) Introduction to discrete structures (IDS)

PI was a pre-requisite for most of the computer science courses and was based onteaching language-independent concepts of programming, with emphasis on theproof of programs against information specifications. However, mathematicalexperience beyond O-level (GCSE) was not required for the course.

IDS covered the basic repertoire of formal notions used in computer science,describing programming languages, constructing relational databases and in devel-oping correct programs. It covered topics such as equivalence relations and partialorders, sets, functions and relations, induction, graphs and trees and lambdanotations. The remaining two courses were taken in the second semester:

(i) Introduction to logic (ITL)(ii) Programming II (PII)

ITL introduced the basic ideas and notions of classical propositional calculus, withemphasis on the notions of proof and validity. This logic was then extended toclassical predicate calculus and the notion of proof considered in more detail. Thecourse is designed to help students realize how closely computer science and logic arelinked and that much of logic is inseparable from the notion of an effective algorithm[4]. PII was concerned mainly with the use of data structures and followed on fromPI of the first semester. In this course the programming language Pascal was taughtand used to implement concepts learned in PI.

2.2. ParticipantsParticipation, both in answering questionnaires and attending interviews was

voluntary and unpaid. Marginally over 50 out of the 150 second and third yearstudents who were contacted completed a questionnaire. As a follow up to thesequestionnaires 12 students were interviewed individually and each given theopportunity to talk in more depth about their first year experiences of learningformal methods of reasoning. Data collected, unless otherwise stated, refers to the 58students who actively participated, i.e. 51 who returned completed questionnairesabout the courses and 12 who volunteered to be interviewed personally (five studentsboth completed questionnaires and volunteered to be interviewed). Of those whohad completed questionnaires, 31 had given their names, while 20 of the forms werereturned anonymously.

752 P. Fung et al.

Overall 2nd. year results

n*79

good passes

comfortable

struggling

deferred/withdrawn/transferred

31

27

15

6

Overall 3rd. year degree results

n=81

firsts:

upper twos:

lower twos:

thirds:

passes:

deferred/ftq:

5

28

19

13

8

8

2nd.yr named questionnaires)

good passes

comfortable

struggling

9

3

1

3rd.yr named questionnaires

n=13

firsts:

upper twos:

lower twos:

thirds:

pass

2

6

2

2

1

2nd. yr (interviewed)

n=3

good passes

comfortable

struggling

1

1

1

3rd.yr (interviewed)

n=9

firsts:

upper twos:

lower twos:

thirds:

1

6

1

1

Figure 1. Summary of end of year examination results.

Although this number of replies is obviously not large enough to use as the basisof any generalizations, the replies were interesting and proved useful in bringing toour attention some of the factors which students see as contributing, adversely orotherwise, to their progress. It was possible, using end-of-year examination results,to check the academic progress of the participating students whose names we knew.In light of their end of year results, it seems reasonable to presume that commentsfrom these students were not unduly coloured by the experience of basically findingthe whole undergraduate programme very difficult.

Comparing their results with the overall levels of attainment for their year(Figure 1), those named participants who completed questionnaires show a nearaverage spread of attainment, perhaps biased towards the upper levels of achieve-ment. The same is true of those who volunteered to be interviewed personally.

2.3. The questionnairesThe tick box format of the questionnaires required the students to rate each of the

four courses described above, in terms of how easy they had found it, how relevant totheir computer studies they had found it, how interesting it had been. Students werealso encouraged to make additional comments on the courses, or their ratings ofthem, as they wished.

Did you find this course interesting? {please tick appropriate box)

very Q fairly • not very • not at all Q


2.4. The interviewsThe structure of each interview allowed the interviewee to discuss what it was he

or she had found hard about the four courses, if any difficulties had been encounteredand if so, how these difficulties could be minimized. The 'interest', 'relevance' and'easiness' aspects used on the questionnaires, were a suggested framework, butinterviewees were free to develop the conversation in other directions if they wished.The goal of undertaking in-depth interviews with a selection of students was toinvestigate in more detail the outline picture obtained from the questionnaire ratingsand to look for any commonality in the difficulties they may have encountered or theattitudes they held towards the courses.

3. ResultsOn looking at the data, an interesting feature of students' perceptions of

computer science soon became apparent. One of the primary aims of the first yearcourses concerned was to instil principles which encourage the development ofprograms which are 'reliable' 'correct' and 'consistent with the specification', all ofwhich are qualities demanded of software in the 'real world'. Yet the answers weobtained from questionnaires and feedback from the interviews indicated that for anumber of students, in their initial perceptions of computer science there exists a gapbetween the academic view of computing and computing 'in the real world'. Beforewe look at these student responses more closely and discuss the implications whichthey may have for computer science teaching, we give a brief overview of the dataobtained from this preliminary study.

3.1. Data from questionnairesFigures 2, 3 and 4, summarize students' responses to the questions relating to

how easy or difficult they had found a course, how interesting they had found it andwhether or not they had found it relevant.

From the data summarized in Figure 2 we can conclude that the majority ofstudents found PI either 'not very easy' or 'not at all easy' while students' perceptionof I T L indicate that on the whole more found it 'fairly easy' or 'very easy' than 'notvery easy' or 'not at all' easy. The data indicates clearly that a majority of studentsperceived PII as either 'easy' or 'fairly easy'. Answers relating to the IDS courseindicate that while students were evenly divided in viewing the course as either'fairly easy' or 'not very easy', at the more extreme ends of the ratings few studentshad perceived it as 'not at all easy'.

I logic

very fairly not very not at all

ratings chosen by students to describe level of easiness I

Figure 2. The easiness factor.

754 P. Fung et al.

logic


ratings chosen by students to describe level of interest I

Figure 3. The interest factor.

a2018

number of 14students g


• logic

S pi;

Sp2,

Oids.

ratings chosen by students to describe level of relevance I

Figure 4. The relevance factor.

Viewing the data at this level, it is relatively easy to pull out which courses wererated more interesting than others, which seemed to present most difficulty to mostof those students and which these students considered most relevant. This howeverraises the 'why' questions—why did the interest rating for IDS peak above that of theother courses?

Why did the relevance rating behave similarly for the course P2? Why did somany students find PI not very or not at all easy, yet also rate that course to be fairlyor very relevant? We attempted to gain insight on these aspects from analysing themore detailed questionnaire comments and through the in-depth interviews, inwhich we investigated what criteria students used in rating the courses.

4. Interpreting the dataIn the event, not unexpectedly, there were no straightforward answers to many of

the questions, but across the number of students interviewed, it was possible todiscern certain recurring themes. These concerned factors which the studentsinterviewed saw as contributory to their ability to succeed on the individual coursesand indicated how those students perceived the relevance of the courses to theircomputer science studies. In summarizing these themes it is convenient to classifythem broadly as occurring at a general level, a subject level and, in relation to theperceived relevance of courses, at a personal level. This classification is to a certainextent an arbitrary one, since the relationship between the factors which affectstudents' learning, or their perception of their learning, is not one which can inpractice be so easily unravelled.


4.1. At a general levelStudents saw the transition from school to university as an important factor in

their progress. For many of them this meant adapting to a very different work style.Some saw this as a challenge. For others it posed problems. For them the change hadnot been easy. It had entailed transition from a relatively regimented framework ofstudies at school, to an environment where the responsibility for organizing one'sown work schedule, attending lectures, meeting deadlines etc., ultimately lay withoneself. At this level, the lecturing skills of those taking first year courses were mostkeenly appreciated. Interest in a lecture or a course seemed to be stimulated by a clearpresentation, the use of relevant examples where appropriate, the ability of a lecturerto 'communicate' with the students (as some interviewees phrased it), and theconstant setting of each lecture in the wider context of the semester's or year'sstudies. Interest generated in this way seemed to have helped a number of thosestudents interviewed to persevere at points where they would have categorized thecourse as 'not very easy'.

4.2. At a subject levelIn many cases, students perceived that the difficulties they had experienced on

courses were due to their lack of 'background' knowledge. Computing or program-ming experience was not a pre-requisite for this computer science degree course. Anumber of students who had no previous computing experience felt disadvantaged,however, in relation to those students with computing experience and saw this ashaving been a source of their difficulties. These students felt that the time which theyhad to spend familiarizing themselves with the computing environment left themconsiderably less time to come to grips with the formal reasoning techniques towhich they were being introduced.

Another source of difficulty at this subject level, as perceived by the studentsinterviewed, was that given the time constraints of the first semester, the system offormal notation used in program proofs was too complex for them to manipulateconfidently. While they were able to follow the steps, albeit with some difficulty,when these were explained to them, they found that carrying them out for themselveswas a slow process with a high possibility of error given their lack of familiarity withthe notation.

4.3. At a personal levelStudents who reported that formal methods of reasoning were difficult to learn,

found, in their first year, that this was compounded by the perceived irrelevance ofthose methods. Most students who choose to study for a computer science degree doso in the expectation that the course will primarily involve computer programming.Many students come to university with programming experience, usually throughschool courses, occasionally in the workplace. For these students their firstintroduction to formal reasoning about programs can be tantamount to experiencingsevere culture shock. For them, accepting the principles involved and learning howto apply the formal techniques used to verify programs is counter-intuitive to theirestablished conception of programming. In conversation it became clear that for themajority of those interviewees with pre-university experience of programming, theirview of the activity had been shaped by that experience. Presented with an approachto programming which advocated prior reasoning about the correctness of a program

756 P. Fung et al.

in relation to its specification, they had felt frustrated. Time not spent sitting at thekeyboard actually working out a program by trial and error had seemed to them, atthat stage, time wasted. For these students, this latter approach was what theyexpected to find on a programming course. Their initial reluctance to employ formalreasoning techniques was perhaps expressed most clearly in the comment 'I don'tbelieve programmers in the real world do'. For them, coding in a first pass at asolution to the programming problem and then adjusting this program until itproduced a satisfactory answer, was what 'real' programming was about. Otherstudents had no prior programming experience and thus were spared this initial'culture shock'. This did not necessarily mean that for those without programmingexperience it was easier to see the relevance of using formal techniques. In somecases, these students, though not previously part of any 'computer culture', foundthe relevance of formal reasoning methods equally difficult to perceive. According toone such student, 'it was difficult to grasp abstract and theoretical aspects withouthaving any of the basic computing concepts'. For them a significant difficulty was thesense of frustration engendered by having to accept the principles of a formalapproach to programming before they have sufficient expertise to appreciate themotivation for that approach.

5. Students' changing perceptionsCuriously enough, this inability to perceive the relevance of learning formal

methods is not obvious in the data. If we look for example at the data showing theratings given to the course PI (Figure 5) indicating how easy (or not) that studentsfound the course, what we see is that a relatively high number of students found thecourse 'not very' or 'not at all' easy, as was confirmed by the information we gatheredfrom interviews. However, the chart also shows that the relevance ratings for PI werequite high.

This apparent anomaly was explained in the course of the interviews and thatexplanation was supported by additional comments on some of the questionnaires.Many of the students remarked that they had modified their initial perceptions astheir understanding of computer science had increased.

A remark, often repeated, common to most of the interviewees, as well asappearing in some 'additional comments' slot on the questionnaires, was that therelevancy of PI had become apparent only in subsequent semesters. It would seemthat many of those students rating the relevance of PI as high on their questionnaires,

E3 Easiness

I Relevance

very fairly not very not atall

Easiness and relevance ratings - coarse PI I

Figure 5. Ratings for course PI.


were doing so with hindsight. From the intervieweees, it became apparent that theformal reasoning approach to programming, introduced in PI in their first semester,had not only seemed difficult, but at at the time, it had been hard to appreciate therelevance of the concepts being taught.

Nevertheless in spite of this, students' perceptions of the relevance of learningformal reasoning techniques had modified in the course of their studies. Of 12students interviewed, 10 mentioned that they had not found it initially easy toappreciate the relevance of the techniques they were being taught, but explained thatit was only by a later stage, in most cases near the end of the first year, they were ableto appreciate the value of formal techniques learnt in their first semester.

6. Some implicationsThis study was undertaken as a first pass at viewing the teaching of formal

methods, from the perspective of those students who had followed courses whichincorporated this approach to programming. The interest of the study lay in gaininginsight into the difficulties which the students themselves felt they had encounteredin learning formal methods; in hearing from them whether or not they felt thoseinitial courses had influenced their own conception of what 'programming' was; inlearning from their experiences in which ways they could most effectively be helpedto appreciate and to put into practice this particular approach to programming.

There is a number of areas in which the study has been useful. At a most generallevel, information gained reinforces and justifies the need to approach first yearundergraduate teaching with the enthusiasm and careful preparation which thosestudents interviewed had so obviously appreciated. Long-term, our data raisesinteresting questions concerning the approach taken in schools and sixth formcolleges to learning in general and to computer science in particular. How does thatgap which exists between many students' expectations of university courses anduniversity 'learning styles' (for want of a better term) come about? What can be doneto bridge it, or better still, to prevent it appearing?

At a level more specifically related to the formal reasoning contents of the coursesin question, the information obtained raises several more questions. In which way isit possible to help computer science students appreciate the extent to which theacquisition of initial knowledge and techniques (which are possibly unfamiliar andcounter-intuitive to them) will make subsequent studies and projects easier? Howbest to avoid the situation, where, as one student commented, 'you only realize toolate, that the initial work is so very useful. But it's very difficult to realize therelevance of something for the next step, when you do not know what the next stepis'.

At a practical level, the study has shown that there is a need for a means ofautomating some of the processes involved in formal reasoning methods. It hasshown that students lack confidence in manipulating formal expressions. Develop-ing and introducing on-line tools which make those manipulations more transparentand reduce the likelihood of students' mistakes in performing operations on thoseexpressions is a principal goal of the current research project, currently beingundertaken at Queen Mary and Westfield College.

It is of course a necessary condition for the success of any on-line tools that theydo not make excessive demands upon the student, that they must be supportive andnot a distraction. For a novice using a computing environment fot the first time, there

758 P. Fung et al.

are already distractions enough. Neither do we see the tools we are planning to use asteaching accelerators. They may well, however, mean that problems which arise dueto the pace of teaching being too fast for some students, may be ameliorated in thatthey could allow new ideas to be picked up more easily. They should help students tocover the same material but more effectively.

Most importantly, as we have seen from our study, a perceived lack of relevancecan give rise to a real lack of motivation and it is here that we hope our tools will havethe biggest impact. The study has shown previous programming experience to be amixed blessing which sometimes creates a powerful clash of expectations betweenstudent and teacher. Formal reasoning is intended to instil a discipline ofprogramming. This discipline may at first appear unduly rigorous to those who haveexperienced a more ad-hoc approach to programming, but it proves to be beneficiallater on in the students' studies when they are called upon to manage and engineerlarger-scale programs. The danger is that only later will they see the reward of thisprogramming discipline, by which time it might be too late to change attitudes andperspectives. Attempts have been made in various ways to bridge this gap ofexpectations [5,6]. Our belief is that even if we cannot adequately show the relevanceof the formal approach at the outset, we can hope to motivate students' learning moreeffectively if we put at their disposal tools which are of immediate practical help andfun to use. Recall the student who found it 'difficult to grasp abstract and theoreticalaspects without having any of the basic computing concepts'. This suggests a role fortools, not only as concept reinforcers but also as devices for concept acquisitionwhere new concepts are learned more easily by easier acquisition of the skillsrequired to manipulate those concepts. The problem of failing to comprehend keyabstractions because of a prior failure to absorb basic concepts has also beenidentified by Barwise and Etchemendy [7]. They are addressing this problem inmuch the same manner as we are, by developing tools which support a pedagogicphilosophy of learning skills by solving problems [8]. It is our belief that byintroducing a selection of graphical computer-based tools which exemplify keyconcepts, students should find it easier to acquire the basic concepts needed. Inaddition the tools should serve to reinforce the principles underlying a formalmethods approach to teaching programming, thus alleviating some of the difficultiesstudents experience in their initial encounter with formal reasoning techniques.

In conclusion, while we accept that on-line programming tools are in no way a'deus ex machina' we are convinced that they do offer a way of overcoming some of theproblems faced by novice programmers in learning how to program using formalmethods. We hope too that using such on-line help tools may also encourage anearlier modification of students' initial perceptions of programming in the realworld.

References[1] GRIES, D., 1991, Calculation and discrimination: a more effective curriculum. Commun.

ACM, 34, 50-55.[2] JONES, C, 1986, Systematic Software Development using VDM, series ed. C.A.R. Hoare

(London: Prentice-Hall International (UK) Ltd).[3] BORNAT, R., 1987, Programming From First Principles, series ed. C. A. R. Hoare

(London: Prentice-Hall International (UK) Ltd).[4] REEVES, S., and CLARKE, M., 1990, Logic for Computer Science (Wokingham, England:

Addison-Wesley Publishing Co.).


[5] COOKE, J., and STONE, R., 1991, Formal methods: Education before training. Proceedingsof the First National Conference on Software Engineering in Higher Education.Southampton Institute of Higher Education, pp. 30-53.

[6] COULOURIS, G., 1992, Motivating abstraction design through application design. Proceed-ings of the Conference on Developments in the Teaching of Computer Science, Universityof Kent, April, pp. 17-25.

[7] BARWISE, J., and ETCHEMENDY, J., 1990 The Language of First-Order Logic (Stanford,USA: Centre for the Study of Language and Information).

[8] GOLDSON, D., and REEVES, S., 1991, The Language of First-Order Logic. Technical Reportno. 546, Department of Computer Science, QMW, University of London.

computer science students perceptions of learning formal reasoning methods

Documents