fas ashfield internal audit

8/8/2019 Fas Ashfield internal audit

1/19

Str ict ly Private & Confid ential

INTERNAL AUDIT REPORTAshf ie ld Computer Tra in ing

Ref.: SPE121/JR/2008

To : Ciaran ConlonAnna Kelly Director, Training Services,Director Curriculum & Quality Assurance

CC: Patricia CurtinRay KellyEamonn McFaddenJim Wadden

ADG, Corporate Policy & Standards,Regional Director - North EastManager Financial StandardsRegional Director Dublin

4 February 2009


2/19

1. Executive SummaryThe audit was initiated in response to a request from Training Services in the North East regionwhich raised concerns regarding aspects of the assessment process on contracts run byAshfield Computer Training (ACT). Internal Audit examined some of the assessmentshighlighted by the region as v/ell as assessments completed on contracts run by ACT in otherFAS regions.The key issue of the audit was to determine the integrity of theassessment process in thesecontracts and by extension the integrity of the certification processed by FAS and issued by thevarious certifying bodies such as FETAC and City &Guilds.Before detailing the main conclusions of the audit, two important issues arose that InternalAudit was not in a position to examine in detail but need to be highlighted and which warrantfurther consideration and action by FAS. The first of these matters was that CurriculumSections within FAS training centres adopted different approaches to the verification ofassessments earned out by contracted trainers. The Curriculum Section in Dundalk supervisedindependent verifications of contracted courses, in much the same way as they did for in-centrecourses. On the other hand, Baldoyle Training Centre did not carry out these independentverifications. Such a divergence of approach between training centres is a matter for concern inthe context of FAS' role in regard to the certifying bodies.The second matter that arose during the audit was the question of shortcomings in the set-upof contracted training courses which involve a significant amount of IT training and certification.The issues that arose in relation to the ACT contracts included outdated and unreliablecomputers, the lack of adequate numbers of printers or networked printers during assessmentsand computer disks being re-used or unavailable. These issues were raised both by traineesinterviewed by Internal Audit and by ACT tutors, by way of explanations as to why certainassessment tasks such as printing and saving files to disk did not happen within the allottedassessment timeframe. While contracted training locations are often by their nature temporaryand not as well equipped as permanent locations such as FAS training centres, the issuesraised in regard to ACT contracts suggest that they fall below a minimum acceptable standardin relation to the provision of equipment. The matter was not examined in detail but needs to beexamined in the wider context of how contracted training is delivered generally.In regard to the key issue of the audit; the integrity of the assessment process in the sample ofACT contracts, Internal Audit examined a total of 7contracts. It is important to place theInternal Audit findings in an overall context. In all the cases examined Internal Audit found thatthe assessment event did take place and in the case of 3contracts Internal Audit found that thesampled assessments had been processed correctly by ACT. The issues identified with theremaining 4 contracts sampled by Internal Audit varied in significance. However, the findingspoint to a number of instances of the manipulation of assessment material in contractsdelivered by ACT on behalf of FAS. The manipulation of material initially came to light on aCAD contract run by ACT in the North East Region. ACT accepted that the assessmentmaterial had become "unreliable" through the actions of the tutor on that contract This wasaccepted in writing by ACT in April 2007.In the view of Internal Audit, two further ACT contracts completed in the North East Regionduring 2007 also showed evidence that assessment material had been manipulated. It is notclear to Internal Audit the extent to which, if any, learners had any involvement in manipulating

r . . i . r 1 A


3/19

assessment material. However, the responsibility for ensuring the integrity of the assessmentsas they occurred on the ground lay primarily with ACT and their tutor. In the light of ACT'sacceptance that assessment material had been manipulated on the CAD contract in the NorthEast there was an obvious onus on ACT to ensure that the same did not occur on thesubsequent contracts; this in the view of Internal Audit was not done.Issues raised by Internal Audit in relation to the two contracts in Dublin North (contracts 0826and 1240) were likewise referred to ACT for consideration. ACT was able to providesatisfactory explanations to some of the concerns raised by Internal AuditIn both of these contracts Internal Audit identified weaknesses in the way that multi-choiceassessments were completed. In both contracts the supplied answer sheets were not used andanswers were written in pencil on the question sheet. In the case of the Word Processing 2module for contract 1240 there was a very high number of errors (40) in the marking of theassessment The explanation supplied by ACT and the tutor was that 'on reflection, thereseems to have been a systems failure in relation to the exam correction processHowever,when Internal Audit applied the appropriate City & Guilds marking template to this assessment,the correct outcome would have been that 6 learners would have passed the assessmentwhile 5 others should have been marked as having failed (at the time all 11 learners weremarked as having passed the assessment). In the view of Internal Audit the marking wasprobably conducted with a view to ensuring that all the learners passed.In regard to other modules on both the Dublin North contracts Internal Audit identified a numberof issues including missing assessment material (both disks and printed material), inconsistentdates and times on assessment material and multiple copies of assessment files. In a numberof instances ACT stated that they had approval from the Curriculum Department in Baldoyle tosave assessments to disk rather than print material as required by the City & Guildsassessment guidelines. Internal Audit was unable to independently confirm this with FASBaldoyle. In the case of contract 0826 Internal Audit was unable to find any supportingassessment material (either on disk or printed) for a significant part of one of the assessments.The explanation provided by ACT was that there had been "big problems with the disks for thisassignment". In the view of Internal Audit the obvious answer to technical problems that mighthave occurred on the day of an assessment would be to postpone the assessments to a laterdate and sort out these problems. The absence of any assessment material for theseassessments and the ACT explanations were less than satisfactory.In relation to contract 1240, Internal Audit concluded that the tutor's actions had assisted somelearners in achieving a pass mark that they would otherv/ise not have achieved and that thisassistance took place after the assessment was completed.In overall terms the assessment material examined for the four contracts has shown a numberof examples where Internal Audit is of theopinion that the assessment material had beenmanipulated and the certification process has been undermined. It was not possible for InternalAudit to fully determine the motivation of tutors working for ACT but there does appear to beevidence, in a number of instances, of tutors "helping" learners to achieve the required gradesin the way that assessments were managed and marked. In the North East Region ACT hadaccepted that one of their tutors had manipulated assessment material on the CAD contract.Following on from that acceptance it would have been expected that ACT would expend asignificant effort in ensuring that all the assessments for their subsequent contracts in 2007would be conducted with integrity and meet the clear standards set down by the certifying


4/19

bodies. Internal Audit is of the view that the material highlighted in this report shows that thisdid not happen.Internal Audit considers that a number of the findings of this report to be significant andrecommends that FAS considers the matters that have been raised in this report.Internal Audit would like to acknowledge the assistance of management and staff inboth theDublin and North East regions for their assistance in completing this audit

Joe RoeManager IT Audit/Support- Internal AuditPatrick KivlehanDirector CorporateGovernance/InternalAudit

r\ - A r -i R\


5/19

2. BackgroundIn January 2008 Internal Audit was contacted by FAS Training Services in the North EastRegion in regard to apparent irregularities on two training contracts run by the AshfieldComputer Training (ACT) within the region. The apparent irregularities highlighted by the regionrelated to assessments completed by the trainer on the following contracts;

Contract 0769 - CAD - 24 January 2006 to 1December 2006 Contract 1032 - Office Administration - 9 October 2006 to 20 June 2007In the case of the CAD course a number of matters of concernwere identified by the region inrelation to the course assessments. Computer disks appeared to have been accessed after thedate of the assessments and test data manipulated. When interviewed by FAS management inMarch 2007, the tutor used by ACT to deliver this course accepted that he had manipulated theassessment material. On 25 April 2007 ACT wrote to Training Services in the North East andaccepted that the results of the CAD assessments were8unreliable" and added that thecompany was 'totally unaware o f th e events in this case until brought to our attention by you,an d we have taken the necessary steps to avoid any recurrence. We also no longer use theservices of the person involved. In the latter part of 2007 the Curriculum Section in DundalkTraining Centre then completed a certification verification of a second ACT contract within theRegion (contract 1032) and the issues this raised led, in turn, to the contact with Internal Audit.Internal Audit commenced an examination of the assessment material for contract 1032 in lateFebruary 2008. The initial examination of this material confirmed a number of concernsregarding the assessments for this contract and it was decided that the assessments for asample of other sample contracts run by ACT would also be examined. The sampled contractsincluded a number in the Dublin Regions as well as a further contract in the North East region.In response to the issues identified in contracts in the North East Region, Training Services inDundalk with the assistance of Curriculum & Quality Assurance has applied a significantamount of resources in meeting with trainees and tutors, organising and providing re-training,liaising with the certifying bodies and ensuring that where necessary learners completedassessments3. ScopeThe audit examined sample assessments for the following contracts;

Dubl in Nor th I Contract 08261DN430 Computer Appl ica t ions and Office Ski l ls -June 2006 to February 2007 Dubl in Nor th I Contract 12401DN430 Computer Appl ica t ions and Office Ski l ls -May 2007 to December 2007 Dubl in North I Contract 13371 Compu ter Appl ica t ions and Office Ski l ls - June2007 to March 2008 Dubl in South West I Contract 11301 Manual & Compu ter ised Accounts - March2007 to June 2007 Dubl in South West I Contract 13251 Manual & Compu ter ised Accoun ts -September 2007 to November 2007 North East I Contract 10321 Office Ad min ist ra t ion - October 2006 to June 2007


6/19

North East I Contr act 12271 Off ice Adm inistrat ion April 2007 to December 2007For the sampled assessments from the above contracts, the audit compared the written,printed and saved data (disk) from the assessment material against the relevant requirementsspecified by either FETAC or City & Guilds. Areas of concern that arose from this comparisonwere raised with ACT. An initial meeting was held between Internal Audit and the Director ofAcademic Affairs with ACT. This meeting took place in late May 2008. Further meetings thentook place in Baldoyle and Dundalk Training Centres where the actual assessment materialfrom four of the above contracts was reviewed in detail. Present at these reviews, which tookplace on 18 and 27 June 2008, were ACT's Director of Academic Affairs and an ACT tutor whohad worked one of the four contracts. Internal Audit was assisted at these meetings by amanager from the FAS Curriculum & Quality Assurance Department Arising from thesemeetings on the assessments for the four contracts, Internal Audit wrote in early July 2008 toAshfield Computer Training and supplied the company with a detailed schedule of the issuesthat had been raised at the meetings and for which Internal Audit sought further comment Iclarification. ACT responded to Internal Audit on 18 August 2008.Internal Audit also met with trainees from one of thecourses: Contact 1032, in early July andlate August 2008 to discuss details of theassessment processes. Internal Audit again met withrepresentatives of ACT in September 2008 and arising from these meetings supplied ACT withan updated schedule of the queries raised previously. In this updated schedule Internal Auditeither noted the responses made by ACT and in a small number of instances accepted theexplanations provided by ACT. The company was also asked whether they wished to add anyfurther comments. ACT responded by email to Internal Audit on 24 September 2008. Thecompany did not add any further comments but referred to a Microsoft technical article on thesignificance of changes to file dates and times.4. FindingsInternal Audit examined a number of assessments from the seven sampled contractscompleted by Ashfield Computer Training (ACT). For three of thecontracts examined InternalAudit found that the assessment material met the City & Guilds or FETAC requirements and noissues were therefore raised with ACT. In the case of the remaining four contracts, InternalAudit identified a number of matters in relation to some or all of the modules examined andthese were brought to the attention of ACT. The findings section details the issues raised byInternal Audit, the responses from ACT and any conclusions made.During the examination of the assessment material a number of other issues were highlightedto Internal Audit regarding the running of contracted Training courses and the process ofcertification between contractors, FAS and the certifying bodies. While these items are notexplored in detail within this report they are important matter that the organisation as a wholeneeds to examine.

4.1 Contracted Training and assessment ver i f ication.The audit highlighted the different approaches taken by Curriculum sections in different FASregions to the question of independent verification of contracted training assessments. Theissues in relation to ACT arose originally from verifications completed in the North East Region


7/19

under the auspices of the Curriculum section in Dundalk Training Centre. When Internal Auditexamined the assessment material held by FAS for the ACT contracts in Dublin North andDublin South I West, it became apparent that neither region had undertaken verifications of theassessments completed as part of theexternal training contracts. The checks carried out inDublin North and Dublin South I West centred on ensuring that all the necessary assessmentmaterial was returned and that the overall certification assessment results agreed with those forthe individual modules. These constituted completeness checks rather than verifications.4.2 Contracted Train ing equ ipment resourcesA feature of the work conducted during the audit was the number of times the issue ofinadequate computer equipment was raised. This applied to both the learners interviewed forone of theACT contracts and to the explanations provided by ACT to explain issues whichwere queried by Internal Audit. These issues included

Old computers which sometimes did not allow assessment material to be savedproperly. Typical setups where only one printer was available for the whole class and whichwere not on any network, thus forcing assessments to be printed off one at a time. A lack of computer disks leading to the constant re-use of older and sometimesunreliable disks.

While the issue of obsolete or inadequate equipment was not examined as part of the audit, itv/ould be important that this matter be followed up through Training Services.4.3 Speci f ic Co ntracts and Modules exam ined.In broad terms, the issues raised with ACT fell under a number of headings;

Issues where material that was deemed to be mandatory by the certifying body for alearner's assessment to be passed was absent from the assessment material. Issues where a number of learners' assessment material contained the same patternof errors. Issues where the dates of computer files on disks used as part of learnersassessments were inconsistent with the dates and times of when the assessmentswere held. Issues where dates and times recorded on physical assessment material appeared todiffer within individual learner's assessments. Issues where handwriting on disk labels used as part of assessments were not thoseof learners. Issues where multiple answer sheets were marked incorrectly.

The s ign i f icance o f c ompu ter fi le dates and t imesThe issue of differences between computer file creation and modification dates and times arosein a number of the contracts examined in this audit. Normally computer file creation andmodification dates operate in a predictable manner. Many City & Guilds assessments require


8/19

the learner to create and name a new computer file (Word document Excel spreadsheet, etc).When the file is first saved, the "date created" is established based on the time of the computerclock. This "date created" is normally the earliest date and time and any subsequent changesand saves to the document alter the "date modified" rather than the "date created". It isgenerally unusuai therefore to have a date modified for a file which is before a date created.However, this does occur where a file is copied from one location to another and moreparticularly from one device to another device (e.g. from disk to disk). In all cases however, thedate modified is the last date that the file was actually updated and saved by a user and isnever altered by subsequent copying.

4.3.1 North East I Contract 1032 / Off ice A dm inistrat ion - October 2006 to June 2007This contract was run in Ballivor, Co Meath and was part of the regional response to theclosure of the N.E.C. computer plant. Internal Audit raised a number of issues with ACT inregard to the following City & Guilds modules for this contract;

Presentation Graphics (7262-01-006) Word processing 2 (7262-02-022) Spreadsheets 2 (7262-02-023) Integrated Applications (7262-02-029)

4.3.1.1 Contract 1032 - Presentation Graphics (7262-01-006)Three learners on this contract had 2 essential files on disk with a creation date of 15 February2007 (H0LIDAYBILBA01 and H0LIDAYBILBA02), while a third essential file(H0LIDAYBILBA03) had both a creation and modification date of 27 February 2007. Thisessential file had both creation and modification dates 12 days after the assessment date and 7days after their assessments were marked on 19 February 2007.These learners were awardeddistinctions by the tutor for this assessmentThe response from ACT and the tutor was that"due to the number of times that the disks wereaccessed by Ash field, initially to correct the exams, the number of times the disks wereassessed by Ashfield/ FAS in verification, the incorrectly formatted date on a machine and theinconsistency of floppy disks could lead to these errors' (Response 1)Four other learners were missing an essential file from both their printouts and from theircomputer disks (H0LIDAYBILBA03) and were awarded distinctions.The response from the ACT and the tutor was thatBall mandatory and optional elements werepresent when correcting exams; no bias was shown to any candidate' (Response 2)4.3.1.2 Contract 1032 - Word processing 2 (7262-02-022)The date that the assessment took place was 5 March 2007. For one learner, the printedassessment material for this module was also dated 5 March 2007 (as would be expected), theassessments were marked by the tutor on 13 March 2007 and the learner was deemed to haveachieved a distinction mark. However, data files created as part of theassessment showed


9/19

both creation and modification dates of 5 April 2007, over 3 weeks later. ACT responded - seeResponse 1above.Six learners were missing two essential files from their disks (PETLET) and (PETMERGE), onelearner was missing one essential file (PETMERGE) and another was missing an essentialprintout (NEWSLETTER). These learners were marked as having achieved a distinction markin the assessment. ACT responded - see Response 1above.The NEWSLETTER printout for 6 learners (an essential point) was found to be identical andcontained the same 3 mistakes., the exact same layout and the same choice of clip-art. Threeof the files had a creation date of 28 August 2008: approximately two months after the contractwas finished. All 6 learners were awarded distinctions. ACT responded - see Response 1andalso"due to teaching method used (mock examples of tests provided to practice) that lead tostudents following a system they were comfortable with - similar mistakes were common",4.3.1.3 Contract 1032 - Spreadsheets 2 (7262-02-023)The date of this assessment was 8 March 2007. Many of theessential files on learners disksalso show a modified date of 8March 2007 (as expected) but a creation date for the files of 4April 2007, approximately 2 weeks after the assessments were graded by the tutor on 20March 2007. All the disk labels for this module were completed in the handwriting of the tutor.One learner screenshot for this exercise were created on 4 April 2007 but modified on 7 March(a day before the assessment). The learner was awarded a distinction by the tutor.Another learner awarded a distinction for this module on 20 March 2007 had essentialassessment files modified on 21March 2007 and created on 30 March 2007.In the case of a third learner's assessment printouts displayed large differences in the times;14.09 (2.09 pm) on one sheet and 02.47 (2.47 am) on a second. While in the case of a fourthlearner there were also large variations in times on printouts that matched neither otherprintouts nor the times on disks; 1.06 am and 1.54 pm.Three of these learners were awarded a distinction while the fourth received a credit. ACTresponded -"see Response 1" above.Eleven learners failed to print two forms onto a single portrait sheet (an essential point)Two learners failed to print PRINTOUT1 in landscape (essential point) and one learner did nothave produce PRINTOUT1 (an essential point). ACT responded - "see Response 2" above.4.3.1.4 Contract 1032 Integrated Applications (7262-02-029)The date of this assessment was 13 March 2007. Many of theessential files on learners disksalso show a modified date of 13 March 2007 (as expected) but a creation date for the files of 3April 2007. approximately 4 days after the assessments were graded by the tutor on 30 March2007.


10/19

One learner's files for this assessment were created on 3 April 2007 but modified on 12 April2007. Dates on printouts for the assessment corresponded to the date the assessment lookplace; 13 March 2007.Another two learners had files on disks that were both created and modified on 3 April 2007.The labels for all of the learners' disks had their names entered in the handwriting of the tutor.When interviewed by Internal Audit, the learner's maintained that they had always written theirnames on their disk labels themselves and that they had never been supplied with disks withlabels completed by the tutor. ACT responded -"see Response V above.One learner's printed assessment material as well as the material on disk appeared to becopies of another learners material (including the name of thesecond candidate).Almost all learners' assessment material contained a copy of thesame incorrect pie-chart forthis module. The learners were asked in the assessment to produce a pie-chart based on thecombined sales for 4 salesmen for the first quarter of the year. The pie-chart submitted in eachcase was based on that for the same individual salesman only. The same incorrect pie-chartappeared many times in learner's assessment material, often up to 8 times per learner. Alllearners were assessed as having attained a distinction for this assessment. ACT responded -"I can t explain ho w this would happen, no bias was shown to any candidate'4.3.1.5 Contract 1032 conclusionsMissing essential elementsThe audit confirmed that there were a number missing essential elements from the assessmentmaterial for the modules examined in this contract. There was material missing from both theprinted assessment material and from disk. In the view of Internal Audit, ACT was unable toprovide a reasonable explanation as to how learners had passed these assessments in theabsence of such essential material (indeed, in many cases learners were awarded distinctionsfor modules where essential elements were missing).Inconsistent dates and times on contract 1032In the case of the three learners highlighted in the Presentation Graphics module above, it isnot clear why two essential files should have creation dates of 15 February 2007 and the thirdessential file have both a creation date and a modification date of 27February 2007,12 dayslater and 8 days after the assessments were graded by the tutor. The most likely explanation isthat someone accessed the disk at the later date and created the third file. In such a case theintegrity of the assessment process would seem to have been undermined.In the case of the Spreadsheets 2 assessment, all learners' disks had labels completed in thehand of the tutor and files on those disks had a creation date of 4 April 2007, almost one monthafter the assessment took place. It is likely that these files were copied onto the disks at thelatter date. As mentioned above, the learner's maintained that they only ever used disks withlabels completed by themselves. The explanation provided by ACT was that the disks wereaccessed on a number of occasions, that machine dates were possibly incorrect and that therewas generally an Inconsistency of floppy disks" do not explain what occurred. This isparticularly the case where there were unexplained differences between the times appearing


11/19

on the same learners printed assessment material and where, in two cases, both the datecreated and date modified characteristic of the assessment files were later than both theassessment date and the date the assessment material was graded by the tutor.Likewise, the explanations provided by ACT were insufficient to explain the differences in datesand times for files on disks in the other two modules examined; Word Processing 2 andIn tegrated A ppl ica t ions.One explanation suggested by ACT was that the dates and times on the learner's computerswere not correct at the time of the assessments and that this therefore lead to incorrect datesand times being applied to the learner's files created as part of the assessments. Wheninterviewed, the learners stated that they were conscious of ensuring that computer clock datesand times were correct, that they were aware, during the course, of these values having beencorrect and that the tutor had given the group a briefing on how to ensure that computer datesand times were maintained accurately. Perhaps more significantly is the likelihood that if thelearners' computer dates and times were incorrect then it would be so consistently. In otherwords, that certain learner's file dates and times would consistently be incorrect across therange of assessment they completed. This was not the case, and learners with inconsistentdates and times for one assessment often had correct dates and times for files relating to theother assessments.Finally, no explanation was provided by ACT as to why the date created for the NEWSLETTERdocument in the Word Processing 2 module was 28 August 2007; two months after the coursefinished. As mentioned above, 6 learners had the same pattern of errors in their suppliedversion of this particular file. While all of the assessment material had been delivered toTraining Services in Dundalk soon after the contract finished in late June 2007, all theassessment material was given back to ACT on August 2007 in order to allow them to addressshortcomings in the assessment material.Consis tent error pat terns in assessmentsThe two most significant instances of the same errors repeated in learner's assignmentsoccurred in the Word Processing 2 and Integrated Appl ications modules. In the case of theNEWSLETTER document produced in the Word assessment there were 6 candidates whosedocuments were identical and contained a series of errors that suggest the document wascopied. In the Integrated Applications the consistent use of the same incorrect pie-chart alsosuggested that the assessment material had been manipulated. The fact that all disks labels forboth these assessments were completed by the tutor and that the files on the disks hadcreation dates subsequent to both the assessment dates and the date the assessments weremarked, points to culpability on the part of the tutor rather than the learners.In the view of Internal Audit some of the assessment material for this contract was manipulatedto ensure that learners who had not achieved a sufficient level in the assessment were deemedto have passed. This view is based on the absence of mandatory elements from sampledassessments, unexplained date and time differences within assessment material and thepresence of consistent errors in assessments which could not have arisen randomly.

T .


12/19

4.3.2 North East / Contract 12271 Off ice Adm inistrat ion - April 2007 to December 2007This contract was run in Navan, Co Meath and the ACT tutor used was the same as that for thecontract 1032 in Ballivor. Internal Audit raised issues in regard to the following City & Guildsmodules for this contract;

4.3.2.1 Contract 1227 - Email 1 (7262-01-007)One learner was missing two essential elements of theassessment (PRINTOUT1 andPRINTOUT2). Two other learners were each missing one essential element for thisassessment. Three further learners were missing a number of essential elements from theassessment and this was described by the tutor as being "due to a technical fault problem". All6 learners were awarded distinctions. While the assessment took place on 21September 2007and was marked on 24 September, each of these three learners had blanks printouts dated 27September 2007 included with their assessment material. It is not clear why these learnerswere not required to re-sit the assessment once the technical fault was overcome.For one learner two disks were supplied for this assessment one in the learners' handwritingand another in the tutors handwriting. The first disk contained no files whole the secondcontained files with a creation and modification dates of 25September 2007 - 4 days after theassessment was completed and a day after the assessment was marked by the tutor, Thelearner was awarded a distinction. The possibility that an incorrect date setting on the learnerscomputer may have caused this difference can be largely discounted as the dates on theprinted assessment material for this learner, which are also sourced from the computer s dateand time setting, were correctly dated as 21September 2007 (the date of the assessment)The assessment required that the tutor send an initial email to each of the learners emailaddresses and that the learners in turn respond to the tutor via email. As stated above, theassessment took place on 21 September 2007 and the learners should have been allocatedtwo hours to complete the assessment. However the tutor sent theemail to each of thelearners email addresses two days prior to the assessment on 19 September 2007. Thisappeared at odds with restricting the assessment to the specified two hour timescale. ACT orthe tutor did not supply an explanation for this approach.4.3.2.2 Con tract 1227 - Word pro cess ing 2 (7262-02-022)As was the case for contract 1032, some but not all of the assessments for this contract haddisks with labels completed in the handwriting of the tutor. In the case of theWord Processing2 module the assessment took place on 20 June 2007 and the learners' files had a modifieddate of the 20 June, suggesting that was when the material was last accessed. The datecreated for the learners material was 21August 2007 which was 2 months after theassessment and the tutor completed the disk labels. The explanation provided by ACT for tutorhaving completed the labels himself was that "in some instances the tutor prepared the disks(labels an d description) due to the fact some students were incapable of following basic pre-

Email 1 Word processing 2 Integrated Applications(7262-01-007)(7262-02-022)(7262-02-029)


13/19

exam instructions and keeping to exam timing guidelines". No explanation was provided toexplain the need to copy files to these disks some two months after the assessment date.4.3.2.3 Contract 1227 - Integrated Applications (7262-02-029)The assessment for the Integrated Applications module took place on 6 July 2007.All learners for this assessment had four mandatory files on their disks. Each of these files hada creation date of 3July 2007 and a date modified of 6July. In addition, all saved screen shotson disk also had the same dates created and modified as the four files. This suggested that thefiles were created 3 days prior to the assessment and accessed, and updated, on the day ofthe assessment.For two of the learners the automatic date fields from their spreadsheet printouts displayed adate of 17 September 2007, over 3 months after the assessment took place. The date of 17September 2007 appears on 3-4 separate pages of the printed material of their assessments,while the automatic date fields printed on some of theother pages of thesame assessmentwere the same as the sate of the assessment; 6 July 2007. However, like all the other learners,the dates of the files on their disks showed a creation date of 3July 2007 and a modified dateof 6July. For some reason the assessments for this module were not marked until 14September 2007This assessment required that learners produce a pie-chart based on the total sales figures for4 salesmen for the first quarter of theyear. Ten of thecandidates had the same incorrectversion of apie-chart as part of their assessment. The values in this common pie-chart wereincorrectly based on the unit price of four items for sale. Each learner had used the incorrectpie-chart a number of times in their assessment The pie-chart is an essential element with theassessment All ten learners were awarded distinctions when the tutor marked theassessments.In response, ACT and the tutor stated that adu e to teaching method used (mock examples oftests provided to practice) this lead to the students following a system they were comfortablewith - similar mistakes were common. Due to the large number of people who made the similarmistake it is possible the tutor followed the mistake as correct production of the pier-charf.4.3.2.4 Contract 1227 con clus ionsMissing essential e lementsThe audit confirmed that there were a number missing essential elements from the assessmentmaterial for the modules examined in this contract. There was material missing from both theprinted assessment material and from disk. In the view of Internal Audit, ACT were unable toprovide a reasonable explanation as to how learner's had passed these assessments in theabsence of such essential material.Inconsistent dates and t imesACT and the tutor provided little by way of explanation for the differences between datesrecorded on files and printed assessment material highlighted above in each of the sampledmodules. Despite this, it is feasible that for some reason, a tutor may have copied all of the


14/19

learners' file from disks or a drive to floppy disks many weeks after the assessments (thusensuring that disks had file creation dates well after last modified dates). However, the datesfor the Integrated Applications assessment files strongly suggest that these were all created anumber of days in advance of theassessment While the assessment had an allocated time of5 hours and could potentially be split over two or more days, the fact that the file creation dates,modification dates and screenshots for all parts of theassessment were consistent would notsupport this.Likewise, the date and time differences on the E-mail module tend to suggest that someassessment material was created both after the dates of theassessment and after the date theassessments were graded by the tutor.Consistent er ror pat terns in assessments.In the view of Internal Audit, ACT and the course tutor failed to give a reasonable explanationas to why the same incorrect pie-chart appeared in ten of the learners' assessments for theIntegrated Applications module. Their explanation centred on possible confusion, initially by thelearners and then by the tutor, between mock tests or assessments and the official tests.The learners were provided with sales figures of four salesmen for four separate products overa number of quarterly sales periods. They were asked to develop a pie-chart based on thesales total for the first quarter of the year. However, the pie chart produced by the ten learnersin question was based on the unit price of each of the four product lines sold. A pie-chart basedon product unit price is quite different from one based on sales totals for the first quarter andwhile it is possible that one or two of the group might make such an error in an assessment, itis highly unlikely than ten learners would independently make this error. In addition, theincorrect pie-chart appeared a number of times in each of the learners' assessment material;on average, the incorrect pie-chart was used 4 times by learners in their assessments.In the view of Internal Audit, some of the assessment material for this contract was manipulatedto ensure that learners who had not achieved a sufficient level in the assessment were deemedto have passed. This view is based on the absence of mandatory elements from sampledassessments, unexplained date and time differences within assessment material and thepresence of consistent errors in assessments which could not have arisen randomly.In addition to the verbal and written responses received fromACT and the tutors, Internal Auditmade a number of requests for a meeting with the person who was the tutor on both contract1032 and contract 1227. The tutor's employment with ACT was terminated in early September2008 and when contacted subsequently by Internal Audit, he stated that he was not in aposition to meet with Internal Audit

I I 5 1 I r t


15/19

4.3.3.2 Contract 0826 Spreadsheets 2 (7261-422)As with the Database 2 module above, the Spreadsheets 2 module also contained a multi-choice assessment. The finding was again that the learners did not use the supplied answersheet and completed their answers in pencil.The practical assessments completed for this module were PA01, PA02 and PA06. The PA01assessment was structured in three parts. The first part involved spreadsheet design Thesecond and third parts of the PA01 assessment involved the creation of the actual spreadsheetbased on the design and the candidates were requested to save data files to disk and to haveprinted out material based on different aspects of thesaved spreadsheets.However, Internal Audit was not able to find any printed material for parts two and three of theassessment nor was it possible to locate any disks for the PA01assessment. The word "pass"was handwritten on the majority of thecandidates question sheets which accompanied the part1 material. In response, ACT and the tutor stated that there were "big problems with the disksfo r this assignment These may have been done on another disk'. A subsequent examinationof all of the disks for the contract conducted jointly between ACT and FAS Internal Audit, didnot locate the files for parts two and three of the assessment.The other two assessments completed for this module; PA02 and PA06 were likewise atvariance with the City & Guilds guidelines in that there was no printed material produced aspart of the assessments. ACT and the tutor stated that this material was uno t printed but savedto disk as agreed with FAS Curriculum Baldoyle". Internal Audit was unable to confirm whetherthere was agreement on this matter between ACT and FAS Baldoyle,

4.3.3.3 Contract 0826 - conclusionsMul t i -cho ice quest ionsFor both the Database 2 and Spreadsheet 2 modules the designated answer sheets were notused for the multi-choice assessments. While there was no evidence that the assessmentswere conducted in a less than fair manner, the use of pencil rather than ink for the assessmentwas not in accordance with the City & Guilds guidelines and was less than satisfactory.Missing mandatory assessment mater ia lThe missing assessment material for the PA01 assessment in the Spreadsheets 2 module isproblematic. It was not possible to locate either the printed assessment matenal for parts twoand three of this assessment or any computer disks or files. Without this material being presentcandidates should have not been deemed to have passed the assessment If there was atechnical problem with computer disks on the day of the assessment (as stated by ACT) thenthe problems should have been addressed and the assessment completed properly at a laterstage.Internal Audit has been unable to confirm with FAS Baldoyle, that in the case of this contractagreement had been reached to save files to disk in the Spreadsheets 2 module, rather thanprinting out as well as saving to disk, as the City & Guilds guidelines required at the time.

r


16/19

4.3.3 Dubl in North I Contrac t 08261 Comp uter Apps and Off ice Ski l ls - June 2006 toFebruary 2007This contract was run in Mulhuddart Co Dublin and Internal Audit raised issues in regard to thefollowing City & Guilds modules for this contract;

Database 2 (7261-421) Spreadsheets 2 (7261-422)4.3.3.1 Contract 0826 Databases 2 (7261-421)In relation to the Database 2 module, Internal Audit raised a number of queries regardingmandatory City & Guilds assessment requirements which did not appear to accompany thelearners' material. According to the learner's instructions and tutors marking guidelines forassessment PA01, a number of files must be saved to disk (NEWARD, HOSP2 and PA01), aswell as mandatory printouts such as

List of patients with sore throats List of patients born in 1940's List of National Health patients Screen input form.ACT and the tutor responded that the City & Guilds requirements "are not specific to MicrosoftAccess, so the interpretation of the tutor was to create these items as tables in the samedatabase'. Likewise, rather than printing out lists the tutor asked the learners to save querieson the database. ACT and the tutor stated that both these actions were done with theagreement of the Curriculum Department in Baldoyle Training Centre. Internal Auditsubsequently confirmed that the material queried was in fact saved to the learners' databasesfor each of the learners involved.Unlike the 7262 City & Guilds e-Quals certification which was used in contracts 1032 and 1227above, the older 7261 qualification used in this contract and contract 1240 below, used multi-choice questions and a standardised answer sheet. Learners were provided with a questionrelating to the module and four possible answers. The assessment instructions requested thelearners to use the answer sheet provided and to tick the appropriate selection against therelevant question using black or blue ink.In relation to the Database 2 module for contract 0826, none of the learners used the standardanswer sheet provided. Instead they used a pencil to indicate their selection on the questionsheet which was handed back to the tutor. In response to why the City & Guilds guidelineswere not used, ACT and the tutor stated that "as the exams were being corrected manually thetutor felt that it was no t necessary for the students to complete the answer sheet as this isnormally used for scanning purposes in correction and is not always provided by FAS with theexams".


17/19

4.3.4 Dubl in North I Contr act 12401 Com puter Ap ps and Off ice Ski l ls - May 2007 toDecember 2007This contract was also run in Mulhuddart, Co Dublin and Internal Audit raised issues in regardto the following City & Guilds modules for this contract;

Word Processing 2 (7261-420) Spreadsheets 2 (7261-422)

4.3.4.1 C ont ract 1240 Word Processing 2 (7261-420)The multi-choice assessment for the Word Processing module was examined as part of theaudit According to the original marking conducted by the tutor all 11 learners were deemed tohave passed the assessment However on examination it appeared that there were 40incorrect markings made across the assessment The majority of the incorrect markingoccurred when incorrect answers were marked as correct but there were also a number ofother cases where correct answers supplied by the learner were marked as incorrect Whenthe City & Guilds marking template was applied correctly to the assessment material for thiscontract 6 learners would have passed the assessment and 5 failed. The explanation suppliedby ACT and the tutor in this case was that aon reflection, there seems to have been a systemsfailure in relation to the exam correction processThe PA03 assessment for this module occurred on 6 September 2007. The disk labels for thePA03 assessment were all completed in the handwriting of the tutor. The tutor stated to InternalAudit that this v/as done to save time.In general the files for the PA03 assessment were created the day after the assessment (7September 2007) and the tutor confirmed that the files had been saved to the trainee's harddisks on the day of the assessment and copied to floppy disks the following day. One of thelearner's assessment disks was unusual in that it contained folders with the assessmentmaterial for four other learners. In addition, each of these four learners had their own individualdisks as part of the assessment material. ACT and the tutor explained that "those four studentshad initial problems saving to disk so as a back up I saved twice and forgot to delete" and thelearner whose name was on the diskh a d missed nearly all of the module and I did not think hewould be able to complete the exam ...so had no disk for h im...and the only disk I hadavailable was the backup one. I should have deleted the backups of the other students bu t wasunderpressure somewhat then to include him".The explanation provided by ACT and the tutor failed to fully explain this unusual situation. It issuggested that the learner who arrived unexpectedly for the assessment used a disk thatalready had been used to backup files created by four other learners. As these backup filesv/ere created by the learners during the various parts of the assessment, this suggested thatthe unexpected learner completed the assessment after the four learners. This is not supportedby the fact that the learner's signature is one of the first on the official test signature sheet andthat the creation times of his files are similar to the other four learners. In addition, one of thefour learners whose assessment material is on the disk has files with both a date created anddate modified of 7 September 2007 despite also signing the official test signature sheet on 6September 2007.


18/19

The PA05 assessment for the Word Processing 2 module was completed on the morning of 6October 2007. The learners1 mandatory files., created as part of theassessment had timesmodified of the morning (am) and times created of the afternoon (pm). Again the handwnting onthe disk labels was the same. The tutor confirmed that she had completed the labels to speedup the process and had copied the files to disk in the afternoon, hence explaining the creationtime of the afternoon on 6 October 2007. However, for 7 of the learners, one mandatory file(BROWSING) had a modification time of the late afternoon of 6October 2007. A modificationdate and time suggested that the file was last saved in the afternoon rather than in the morning.Four of the times saved were after 4 pmwhen it is unlikely any learners were present.4.3.4.2 Contract 1240 Spreadsheets 2 (7261-422)The date of the assessment for Spreadsheets 2 assessment was 6 November 2008. However,the learners' files showed a variety of dates created and modified between the 6,7 and 8November 2008. The explanation provided by ACT and the tutor was that "there was one inkjetpnnter in the classroom 16 students. If the exam was before lunch I let the students print outafter lunch as this took 2-3 hours to complete...if theexam was in the afternoon I let thestudents print out the next day". The files produced as part of theassessments could havebeen accessed for printing at a later time without changing the date modified - by saving thefile, but this was not done.4.3.4.3 Contract 1240 conclusionsMul t i -cho ice quest ionsNo explanation, other than a reference to a "systems failure", was available for why the markingof the multi-choice assessment for Word Processing 2 resulted in 5 learners being marked ashaving passed, when a correct application of the marking would have failed them. The possibleexplanation that the person marking the assessment had used an incorrect marking template isunlikely given that the same answers to certain questions were marked as both correct andincorrect for different learners. The net effect of the marking that occurred was that all 11candidates were deemed to have passed. In the view of Internal Audit the marking of thisassessment was conducted in a manner that was less than satisfactory.Dates for assessm ent f i lesWhile ACT and the tutor did provide explanations that explained apparent differences betweenthe dates and time of assessment files, not all matters have been explained to the satisfactionof Internal Audit.The explanation for the instance where one learner's disk also contained the assessmentmaterial for four other learners in the Word Processing 2 module is lacking on a couple ofpoints. The dates and times of assessment files for the learner for whom a disk had to befound, do not support the explanation that four other learners assessment material had beenbacked up previously before he received the disk. In addition, his signature on the assessmentattendance sheet is earlier than some of the four learners on the disk and one of those haddates for files the following day after the assessment.


19/19

The findings in relation to the PA06 file (BROWSING) suggest that this file was updated wellafter the assessment was completed on 6 October 2007 for 7 of the learners and, in 4 cases, itappeared that the file was last updated after the learners had left for the day (after 4pm).in summary, it is the view of internal Audit that efforts were made on the part of the tutor toensure that learners, who had not achieved a sufficient level in some assessments, wereassisted in achieving a pass level after the assessment was completed.

fas ashfield internal audit

Documents