Final Report to QCA on the eVIVa Project 2002 - 2004

by Malcolm Moss — last modified Saturday Mar 4, 2006 20:46

Final Report to QCA on the eVIVa Project 2002 - 2004

(August 2004)

Project Team

Lesley McGuire

Gill Roberts

Malcolm Moss

Contents Page ( images have been removed)

Executive Summary 3

1. Introduction 9

2. Background and Philosophy 9

3. Research Approach and Methods 12

4. The eVIVA Process: 13

Getting started 13

Self-assessment and question selection 14

The e-portfolio 15

Feedback 16

The eVIVA assessment 17

The Teacher area 17

Process diagram 18

5. Findings 19

Pupil response and motivation 19

Teacher support and feedback 21

Lessons learned 23

Improving assessment 25

Demonstrating ICT capability 28

System manageability and workload 29

Technical issues 30

6. Conclusion 32

Fair and fit for purpose 32

Using e-portfolios for assessment 33

The Future 34

Bibliography 36

Appendices:

Appendix 1: Table showing eVIVA record of activity 38

Appendix 2: Collated teacher responses to the main research questions 39

Appendix 3 Collated teacher responses to the additional questions 50

Appendix 4 Pupil online exit questionnaire – 2003 58

Appendix 5 Pupil online exit questionnaire - 2004 62

Appendix 6 Appendix 6 – Student feedback emails 67

Executive Summary

Introduction (1)

eVIVA is an innovative “blue skies” pilot project which uses mobile phones, voice recognition technology and the Internet to support formative and summative assessment. The two-year project came to an end in July 2004 and this report examines the development of the eVIVA process, the responses of pupils and teachers to this new approach and the implications of the project for e-assessment.

Background and Philosophy (2)

Ultralab were contracted to carry out a feasibility study for the Qualifications and Curriculum Authority into the development of an online assessment tool for Key Stage 3 Information and Communication Technology. The first phase of the study ran from June 2002 until July 2003 and involved ten schools across the country. The project was extended for a second year with five of the original ten schools being involved in the second phase of the study.

In developing eVIVA emphasis was placed on using the technology to support the assessment, not on automatically generating it. The main focus of the assessment was on formative assessment or “assessment for learning” based on the work of Black and Wiliam (1998). The design of the process was informed by constructivist theory so the emphasis was on investing the power of the technologies in the learners allowing them to actively construct, rather than passively receive their knowledge.

Ultralab’s aim was to design an assessment tool that demonstrates “internal fairness,” by taking children’s “individual differences into consideration” (Smith 2001). Consequently, the assessment model or paradigm that ultimately influenced the design was the Ipsative paradigm (Mabry 1999), which looks at the individuality of each learner and how the individual progresses in comparison to her/ himself. This is at the heart of eVIVA - hence the name, which stands for “electronic virtual ipsative valid assessment”.

Research Approach and Methods (3)

The research approach used in this study is that of an interpretative, naturalistic, practitioner enquirer. The research findings were used to guide the iterative development of eVIVA as a tool for assessment for learning. To achieve triangulation feedback was collected from teachers, pupils and facilitators over a two-year period from June 2002 until July 2004. Teachers and pupils were co-researchers in the project, with Ultralab facilitators acting as both researchers and facilitators of learning.

The eVIVA process (4):

The eVIVA assessment process aims to empower and enable pupils to reflect on their work over time, share their thinking and early drafts of their work, receive meaningful feedback from their teacher and their peers, and provide their teacher with a variety of evidence to support their judgements.

Pupils have access to the eVIVA website where they begin by setting up an individual profile of system preferences and recording an introductory sound file, on their mobile or land phone. After this pupils carry out a simple self-assessment activity by selecting a series of simple ‘I Can’ statements designed to start them thinking about what they are able to do in ICT.

The website has a question bank from which the pupils are asked to select 4 or 5 questions for their telephone viva or assessment carried out towards the end of their course, but at time of their own choosing. Pupils are guided in their choice by the system and their teacher.

Pupils have their own e-portfolio web-space in which they are asked to record significant ‘milestone’ moments of learning, and to upload supporting files as evidence. Each milestone is annotated or described by the pupil to explain what they have learnt or why they are proud of a particular piece of work.

Once milestones have been published teachers and pupils can use the annotation and the messaging features to engage in dialogue with each other about the learning. Pupils are encouraged to add comments to their own and each other’s work and the annotations can be sent via phone using SMS or voice messages.

When ready, pupils dial into eVIVA either by mobile or land phone, and record their answers to their selected questions. This gives pupils the opportunity to explain what they have done and reflect further on their work. Their answers are recorded and sent to the website as separate sound files.

The teacher makes an holistic assessment of the pupil’s ICT capabilities based on the milestones and work submitted in the e-portfolio, pupil reflections or annotations, the recorded eVIVA answers and any written answers attached to the questions and classroom observations.

The Findings (5)

Pupil response and motivation

Feedback on eVIVA from pupils was very positive, with the majority of pupils saying they enjoyed using the system. Several pupils said that they thought it was a better way of taking “a test” and that it was good to know the questions in advance. It was clear that they found using the system stimulating and many mentioned the fun of recording the voice files and messaging.

There were a small number of pupils, however, who were not quite so positive in their comments saying that the system was challenging and one or two pupils mentioned finding the voice recording difficult and “scary” While these pupils were in the minority it is interesting to note that these comments serve to highlight the fact that this system is genuinely challenging for pupils and not an easy option. One pupil claimed it was “a bit too adventurous for now” while another thought the questions “a bit hard and should have been easier for kids of our age”.

It was clear that pupils were motivated and engaged by the text and voice messaging features of the system but their responses showed that they also valued the feedback provided by both their peers and their teachers.

“When other people left comments on my work it helped me to improve my work and change bits so it sounded better. It also helped when people said my work was good because it made me feel that my work was ok.”

When asked if they would like to see more or less comments on their work from their teachers 81% of pupils were in favour of more comments with some even saying they placed more value on the feedback from the teacher because it was likely to be more honest.

“I would like to see more comments from teachers because comment from friends are not always a help cause most off them will just say they thought it was good whether they thought it was or not”

Teacher support and feedback

When asked if being involved in the project had been useful the response in both phases was overwhelmingly positive, with one teacher believing it was such a valuable experience that he asked incredulously, “Are you kidding?”

Responses to the project usefulness primarily related to the impact on pupils and listed factors such as:

· Increased motivation of pupils,

· Increased self-esteem,

· Awareness of audience,

· Pupils taking responsibility for their own learning and becoming independent learners,

· Improved teacher-pupil relationships

· Recognition of the value of oracy.

Several teachers also mentioned that the project promoted a sense of online community and developed communication skills.

Significantly one of the teachers observed that assessment is no longer “a bolt-on but is now integrated into the teaching process.”

Lessons learned

According to one teacher,

“It is interesting that most of the things I have learned as a result of doing the project are about how children see learning – I didn’t expect that to be the focus. I expected it to be about the electronic nature of the activity.”

Another observed that as a result of working on this project, instead of thinking about what she is teaching, she now finds herself thinking about what the pupils are learning.

Teachers reported that the discussions surrounding the self-assessment led to a much greater awareness on their part about pupils’ ICT experience and capabilities. While for pupils it led to a much greater understanding of the criteria, by which they were being assessed. Also pupils are motivated and empowered by sharing their work in an online space. Exhibiting their work in an e-portfolio appears to give pupils a sense of audience and serves to lift their expectations.

Teachers also noted that pupils need to be taught new skills in the art of reflection and peer review particularly mentioning older pupils who seemed to find the idea of opening their work up to criticism quite intimidating.

“Students find it difficult to be independent learners. Current system not geared to this. With e viva they were suddenly being empowered but they need more help and support than I thought they would need… In a normal lesson they are told what they will learn … here they have to think it out for themselves.”

Improving assessment

All of the teachers involved in phase 2 stated that eVIVA has helped their assessments because of the insights into pupil thought processes and the ability to use online dialogue for clarification. The pupil reflections allow them to show their “capability at a higher level than their work would suggest”, and also help teachers make “inroads into differentiated assessment”.

“ It provides more evidence, particularly where process is concerned. It compels the pupils to analyse their own methodology and the evidence this gives is possibly unavailable in any other way. It has given information that I wouldn’t otherwise have got”

Demonstrating ICT Capability

Feedback from teachers was that using eVIVA made pupils much more aware of what they were doing “and why they were doing it”. Knowing that their work was going to be seen by others has a positive effect and “raises their game”. Certainly teachers reported increased self-confidence in pupils. Also it could be argued that simply by using eVIVA pupils demonstrate “competence in use of Internet, uploading/ downloading, searching”.

Teachers also noted that the showcase element of eVIVA allows the pupils to show progression over time, it encourages them to try harder, and seeing the work of others encourages them to review and amend their own work. “Publishing to an audience is more significant in this environment.”

System manageability and Teacher Workload

Many of the teachers identified benefits such as getting to know the children and their capabilities much better, and argued that, once pupils and staff were familiar with the system and the processes, it would save time.

4 of the 5 teachers stated categorically that using eVIVA hadn’t changed their workload because they would be doing the marking, commenting and assessments anyway.

“No big impact on assessment time – checking work on server had been built into assessment routines: eVIVA achieves what I was doing before, but in a neater way.” (School 2)

It was also noted that eVIVA gives heads of department, an overview of what other teachers and pupils in the department are doing, and helps them in supporting non-specialists, “because it pulls all assessment tools into one place and combines it with student portfolios.”

Conclusion (6)

Tomlinson (2004) in his interim report on the 14-19 Curriculum argues that an assessment system should be “fit for purpose”.

Fair and fit for purpose

When asked if eVIVA is fit for purpose and a fairer way of assessing ICT one teacher said it was a “different way,” one which allows progression, accessibility and equality of opportunity. According to another,

“Is ‘Fair’ best term? Would say it is ‘better’ because of peer reflection, self evaluation as part of assessment process”

Also the oral element potentially offers greater fairness. This is particularly true now it has been extended to allow pupils to add voice annotations. It seems that for the teachers involved in the project one of the most significant features seems to have been the “ipsative” nature of the process.

“I am not a fan of levels, I prefer to record achievements and routes to improvement – this is what eVIVA does! (Ipsative)”

Using e-portfolios for assessment

According to Sue Walton of QCA (2004),

“The use of eVIVA as an assessment tool and its particular emphasis on formative assessment has represented a major cultural shift for teachers and their pupils. The use of the on-line system has been a new experience, but so too has been the use of assessment techniques such as self and peer assessment and annotation.”

The teachers needed much more support from the facilitation team to keep on track than anticipated. The need for support appears to have been as much to do with the change in classroom practice as the technical aspects of the project.

Feedback and evidence from both phases of the project certainly suggests that e-portfolios are effective tools for supporting both formative and summative assessment in the classroom. Teachers clearly value having everything related to their assessments in one place, with that place being easily accessible. It also seems clear that e-portfolios have the potential to engage and motivate pupils.

The Future – What next?

Funding this project involved a certain element of risk-taking on the part of QCA. The challenge now is that the risk appears to have paid off, and the project findings suggest that the eVIVA assessment process actually works. Ultralab believes that eVIVA should now move into a new phase to discover whether the potential that it offers of a fairer, better, more effective way of assessing learning can be fully realized. To do this we would argue the need for the following:

Trialling on a much larger scale;
Exploration within a different school phase or education sector;
Development in a different subject area;
Creation of an open source, publicly licensed version for wide distribution;
Integration into other projects, which aim to develop online community, new learning, assessment, CPD and awareness in government agencies in connate ways.

Final Report on the eVIVa Project

1. Introduction

When we think about assessment using digital technologies, we often assume this means the introduction of computer-based tests. There are, however, other ways in which technology can [LM1] [LM2] play a useful and important role in assessment for learning.

This paper will discuss an innovative project, eVIVA, which uses mobile phones and the Internet to support formative assessment. The paper, reporting on the findings of the pilot project, which came to an end in July 2004, will examine the development of the process, the responses of pupils and teachers to this new approach, and the implications of the project for e-assessment.

In a recent news interview about this project the interviewer asked why it was necessary to use innovative digital technology to assess pupils’ work. Why couldn’t the teacher just sit down, talk to the pupils and mark their work just like in the ‘good old days’? It was a good point.

Anyone who has taught in a classroom, particularly teaching ICT (Information and Communication Technology)[LM3] , knows that the opportunity to talk with one pupil, in a class of thirty or more, about their learning is not easy. It is impossible be aware of [LM4] every group discussion. It is also very difficult to judge the thinking processes and effort behind pieces of work submitted for assessment, purely on the end result. How many teachers will also recognise the scenario of returning assignments to pupils covered with comments and feedback, only to see the pupils turn straight to the final grade or percentage, ignoring the feedback in favour of the final mark.

Any assessment tool or process which aims, to enable the pupils to reflect on their work over time, allow them to share their thinking and the early drafts of their work, give them meaningful feedback from their teacher and their peers, empower them and provide the teacher with a variety of evidence to support their judgements surely has to be worth a consideration.

“It is often said that we assess too much, that is, have too many formal examinations. It is not said often enough that we assess too little, that is, assess too narrow a range of human abilities and skills by far too limited methods, mainly pencil-and-paper tests. More creative, blue skies work by researchers on what can be assessed and how it might be better assessed would be very welcome. “ (David Hargreaves 2001)

2. Background to the eVIVA Project

Ultralab was contracted to carry out a feasibility study for the Qualifications and Curriculum Authority into the development of an online assessment tool for Key Stage 3 Information and Communication Technology. The intention was to develop a tool that would allow both formative and summative assessment of pupils’ work as well as facilitating constructive dialogue between pupils and teachers.

The project was initially set up to run for a year, and was part of a series of assessment projects created by the QCA to look at the online assessment of ICT at KS3. Phase 1 ran from June 2002 until July 2003 and involved 10 schools. The project was then extended, with Phase 2 running until the end of July 2004 and involving 5 of the original schools.

Pupils and their teachers were co-researchers, with Ultralab facilitators, in the project. The schools were each asked to involve approximately 20 pupils in the pilot, and one teacher within the school, in most cases the ICT Co-ordinator, worked with an Ultralab facilitator to report their findings.

Pupils were asked to compile an online portfolio to provide annotated evidence of their ICT learning milestones. Once their "digital portfolio" was ready, pupils were expected to participate in a telephone viva in which they answered a number of pre-selected questions, chosen by the pupils themselves, about both their working processes and their learning journey.

The aim of the study was to use eVIVA to encourage reflection and dialogue, both teacher-to-pupil and pupil-to-pupil, about the learning that had taken place, and to enable the assessment of higher levels of attainment, creativity and understanding rather than simply testing a body of knowledge. It was also hoped that teachers would engage in dialogue with each other, not only about the project, but also about the assessment and learning in their classrooms.

Ultralab contracted a New Zealand software company to develop the software, and the telecommunications company Orange offered assistance with the design of the website and the SMS interface. They also provided a freephone number for the phone calls.

2.1 Philosophy

From the start of the project the emphasis has been on using the technology to support the assessment, not to automatically generate it. The role of the human assessor is seen as vital and the technology is seen as offering new ways to make that contribution light, viable and appropriate. The intention was to build on the work of Black and Wiliam (1998) with the main focus of the assessment on formative assessment or “assessment for learning”.

“All those activities undertaken by teachers, and by their students in assessing themselves, which provide information to be used as feedback to modify the teaching and learning activities in which they are engaged.”

Jonassen (1994) advocates investing the power of the technologies in the learners. “Power to the people, so to speak.” Certainly the design of eVIVA, as with other Ultralab online spaces, is informed by constructivist theory (Brown, Collins & Duguid, 1989. Bruner, 1986, Fosnot, 1996) and aims to empower the participants to actively construct, rather than passively receive their knowledge.

A starting point for the development of the eVIVA process was a consideration of the issue of fairness in assessment. Smith (2001) argues that although assessing all learners in a similar way, as in a test marked according to a fixed answer sheet, might be demonstrating ‘external fairness’, by ignoring children’s individual differences, it fails to demonstrate ‘internal fairness’,

“The child behind the test paper is of no importance; the contents of the test paper is the only concern…With children who are developing…who are going through a process of learning about themselves, their talents and abilities do we not need to take their differences into consideration?”

Mabry (1999) identifies three paradigms of assessment (see Figure 1 below):

· The Psychometric Paradigm, which serves the purpose of external fairness

· The Contextual Paradigm, which serves the purpose of internal fairness and looks at the individuality of the group in the context of teaching

· The Ipsative or Personal Paradigm, which also serves the purpose of internal fairness and looks at the individuality of each learner and how the individual progresses in comparison to her/ himself.

Figure 1. Mabry’s model of assessment paradigms

*Psychometric*	*Contextual*	*Personal or Ipsative*
Standardised in content and administration	Curriculum sensitive and group sensitive	Student sensitive: content setting and time vary
Objective items and formats	Objective and subjective items and formats	Subjective items and formats. Student involved in selection
External marking (machine)	Teacher marking	Teacher marking
No Self-assessment	Self-assessment important	Self-assessment essential
Summative - no feedback beyond score	Formative use of results. Can be used summatively.	Formative use of results. Can be used summatively.

In developing the eVIVA software and assessment process Ultralab’s aim was to design an assessment tool that served the purpose of internal fairness. Therefore, it was the Ipsative Paradigm that influenced the design and indeed this is reflected in the name eVIVA, which stands for ‘electronic virtual ipsative valid assessment’.

The following features, associated with good classroom practice in assessment for learning, were identified as essential components to the eVIVA system:

Dialogue between teachers and pupils about the learning process
Reflective review of their work by pupils
Self-assessment by pupils
Peer-assessment of work
Teacher feedback to enable pupils to progress and improve their work

The key research questions identified were:

What have we learned during the project regarding assessment of ICT?
What evidence, if any, is there that eVIVA improves teacher assessment of ICT activity by pupils
What are the ways in (and extent to) which eVIVA enables pupils to demonstrate their ICT capability
What technical issues arose during the project and what are the implications for national rollout?
Is this a fairer way to assess ICT?

3. Research Approach and Methods

The research approach used in this study is that of an interpretative, naturalistic, practitioner enquirer carrying out evaluative research. This ongoing evaluation was used to guide the iterative development of eVIVA as a tool for assessment for learning. To achieve triangulation feedback was collected from teachers, pupils and facilitators over a two-year period.

As stated in the previous section, teachers and pupils were co-researchers in the project, with Ultralab facilitators acting as both researchers and facilitators of learning, based on experience gained from earlier Ultralab online projects such as SchoolNet 2000 and Talking Heads.

Ultralab facilitators visited the pilot schools initially once a term, and the teachers co-researching the project were interviewed for their feedback. A face-to-face meeting was also held each term at which feedback was collected from all those involved in the project.

Facilitators compiled a report following each visit, which was shared with and edited by the teacher concerned. At the end of the year, teachers worked with their facilitators, at the last face-to-face meeting, to produce a final report.

Pupil feedback on the project was collected in a variety of ways:

An email address was set up to allow pupils to email free text comments directly to the project team.
A small group of pupils was interviewed regarding the project.
The questions from the pupil interview served as a pilot for a simple exit questionnaire, which was set up online, to collect pupils’ anonymous responses.

Some issues identified in the initial pilot year, resulted in some changes in emphasis and approach for the second year.

It became apparent during the first year that, if schools are left to their own devices, they sometimes lose momentum. Day-to-day demands take precedence and timelines slip. It must also be remembered that schools volunteered to be involved in this project it was not compulsory. Over the course of the first year, facilitator involvement was found to be crucial to support, encourage and keep the schools on track. During phase 2 the role of the facilitator was expanded to include online facilitation within the web space. Facilitators were asked to be more proactive online, to set themselves up as a teacher within the school online space, to annotate pupil milestones, model good practice and monitor the school’s progress from within the system.

It was also clear from feedback at the end of the first year that some teachers were still not as familiar with the software and the eVIVA process as they could have been, in spite of regular updates, handouts and sessions with their facilitators. So it was decided to change the way in which the termly Ultralab meetings between teachers, facilitators and the project team were used during phase 2.

Consequently the first session concentrated on the software, took teachers and facilitators through all stages of the eVIVA process and all aspects of the application. Together they identified the bugs, ‘must haves’ and ‘nice to have’ features, which were then translated into the software work programme for the year.

Recognising a professional development need for teachers in relation to “Assessment for Learning”, successive sessions were used to create opportunities for teachers to share ideas and good practice about how to encourage pupils to reflect on their own learning, and annotate work done by their peers.

Facilitators increased, and timetabled, their visits into schools to monitor progress, support the process and help to keep the school on task. Although facilitators continued to report back on these visits, more responsibility was placed on the teachers for the production of the final report on progress within their school, to enable the capture of their ‘authentic voice’. A large part of the final two face-to-face sessions was devoted to this reporting process.

4. The eVIVA Process

The eVIVA system comprises a secure, online website within which pupils can post their work and comments and receive feedback from their teacher and other pupils.

The initial design called for each pupil to have a space on this website for these objects:

A personal profile area, including an introductory sound file
A profile compiled from “I CAN” statements
The questions they have selected to answer in their final telephone viva
The date and time for their final viva
The number of the mobile phone they will be using (for authentication)[LM5]
An e-portfolio for uploaded files of work
Annotations on all of the above, by themselves, or by others
The recording of their final telephone viva

Over the two years of the pilot eVIVA became technically more sophisticated, allowing pupils to annotate their own, and each other’s work using the website, SMS messaging and telephone voice recordings. A help system was also developed, supported by sound files, to cater for a range of learning styles.

Throughout the pilot, findings from teacher and pupil co-researchers informed each stage of software development. The current eVIVA process is outlined below and summarised in figure 8:

4.1 Getting started:

Pupils are registered, by their teachers, as users of the eVIVA online space and assigned a user name and password. The eVIVA website sends them a welcome message, instructions on how to proceed and an email containing their unique PIN number, user id and password.

The first time they log into the website pupils are prompted to complete a pupil profile, setting up their messaging preferences choosing whether to receive their messages solely on the website, or to have them copied to their mobile phones (see figure 2)[LM6] .

Figure 2. The Home Page

Pupils are then asked to prepare and record a ‘voice postcard’ introducing themselves to other pupils within the system. They dial the eVIVA free-phone number to record their voice postcard. The resulting voice file is posted to their profile area on the eVIVA website. (N.B. This file serves a dual purpose as it can later be used to authenticate the final VIVA voice file)

4.2 Self-assessment and question selection:

Once they have completed their profile pupils are asked to carry out a simple self-assessment activity by selecting a series of ‘I Can’ statements, based on the national curriculum level descriptions. This process is designed to start pupils thinking about what they are able to do in ICT and gives them an idea of the assessment criteria their teachers will be using. (See figure 3)

Figure 3. I can statement selection

Pupils, with guidance from their teacher, select 4 or 5 questions from the question bank on the website for their final oral assessment. This selection can be amended at any time during the project until the final eVIVA. (See figure 4)

Figure 4. Questions selected from the question bank for the final Viva

4.3 The e-portfolio:

Throughout the duration of the project, pupils use the portfolio feature of the website to record significant ‘milestone’ moments of learning, and to upload supporting files of work. Each milestone is annotated or described by the pupil to explain what they have learnt or why they are proud of a particular piece of work. (See Figure 5)

Pupils have extensive publishing rights over their work and can determine their audience (teachers, pupils etc.) Pupils can link their milestones to their chosen questions and can draft written answers to these questions on the website in preparation for the VIVA.

Figure 5. Adding a milestone

4.4 Feedback:

Once milestones have been posted and published teachers can use the annotation tool and the messaging feature to engage in dialogue with pupils about their learning. (See Figure 6) Pupils can also use the annotation tool and the messaging feature for peer review and reflection, and are encouraged to add comments to their own and each other’s work.

Figure 6. A teacher annotation

Teacher 5

11/03/04

4.4 The eVIVA assessment:

At the end of the project pupils are notified by SMS or email when it is time for their telephone eVIVA. Pupils dial the eVIVA phone number, either by mobile or land phone, and record their answers to their selected questions. This gives pupils the opportunity to explain what they have done and reflect further on their work. The questions are generated by VXML, which, in Phase 1 activated pre-recorded sound files, and in Phase 2 activated a robotic voice. The answers are recorded and sent to the website as separate sound files which are then attached to each of the questions previously selected by the pupils.

The teacher makes an holistic assessment of the pupil’s ICT capabilities, based on the milestones and work submitted in the portfolio, pupil reflections or annotations, the recorded eVIVA answers and any written answers attached to the questions and classroom observations. The teacher enters their summative assessment level into the system as the teacher assessed level or TAL. (See Figure 7)

4.5 The Teacher area:

In addition to the aspects of the system described above there is a teacher area in which teachers manage the use of the system with their pupils. The system sends a notification to the teacher whenever a pupil portfolio is updated or a recording has been made.

The teacher can also see the status of their class in one view enabling them to see, for example, which pupils have posted milestones, what questions have been selected, who has completed their eVIVA, their “I can” suggested level (ICSL) and so on. This information can be exported in CSV format, which can then be imported into a spreadsheet or reporting package. Teachers can also generate graphs showing question coverage and ‘I can’ recommended levels. (See Figure 7)

Figure 8. The eVIVA process

Pupil completes profile and records ‘voice postcard’

Pupil self-assessment using

‘I can’ statements

Questions selected to work towards and for the eVIVA

Date and time for eVIVA chosen (if not done when profile completed)

Pupil adds learning milestones to their portfolio and uploads supporting files of work

Pupil takes eVIVA on a phone of their choice

Pupil logs on

Teacher and other pupils annotate work

Pupil can refer to these in their portfolio at any time

Teacher uses portfolio contents to inform judgement and enter final assessed level

5. The Findings

The original intention was that eVIVA should be a one-year research project. At the end of the first year however it became clear that one year just wasn’t enough. During this pilot phase teachers found themselves concentrating on embedding the technology and the eVIVA process into schemes of work, and helping pupils understand and internalise the system. It was not until they had experienced the process from start to finish that teachers and pupils really began to understand how eVIVA worked.

As a result, the benefits of the iteration, and the technological developments were only just becoming apparent as the project was drawing to an end. Pupils and teachers were just beginning to explore the use of the annotation, and the messaging tools in relation to peer review but pupils needed more time to develop their skills in this area. The whole process took much longer than initially expected and, disappointingly, in some schools pupils were unable to complete their final telephone vivas due to time constraints.

Although positive feedback on eVIVA was received from both teachers and pupils, any claims for the system were based more on “gut feeling” than evidence. Therefore the main recommendation of the end of project report was that the pilot project should be extended to allow more time to explore some of the issues and to collate more data. After due consideration QCA agreed to extend the project for a second year but scaled the scope down to involve only five of the original ten schools.

The findings presented below are drawn from both phases but the main emphasis will be on the more recent feedback from the second year of the pilot.

5.1 Pupil response and motivation

As stated earlier pupil feedback was collected through an online questionnaire, selected interviews, and an email address for free text comments. In the first year of the project 25 pupils emailed free text comments, while 30 pupils responded to the online questionnaire. In the second year 58 pupils took part in the questionnaire but no emails were received from pupils involved in the pilot, perhaps because they had online access to facilitators, and more opportunities for feedback within eVIVA itself.

Feedback on eVIVA from pupils in both phases was very positive, with the majority of pupils saying they enjoyed using the system. Several pupils said that they thought it was a better way of taking “a test” and that it was good to know the questions in advance. It was clear that they found using the system stimulating and many mentioned the fun of recording the voice files and messaging. One pupil even went so far as to predict,

“I think the whole of eviva is good because it would probably change the future of schools and the way we do tests and I think that it was a good idea”.

While another gave the system 9/10 and said,

“ I think eviva is a good way to show off what your ICT skills are shown as. It's also good because you can see what other people have been doing. Messages are a totally cool idea!”

There were a small number of pupils, however, who were not quite so positive in their comments, mentioning that it was hard work, it was boring, that they had to write too much and do extra tests, and that they found the questions and ‘I can’ statements difficult to understand. One or two pupils mentioned finding the voice recording difficult,

“My eviva experience was scary because I’ve never experience anything like it, the good bits were the messages, voice postcard and posting things on the site. I didn’t like the questions and the I CAN statements because I thought that was hard. The voice postcard was scary because I sound really weird on the phone and I kept on doing it over and over again because I thought I wasn’t doing it right.”

While these pupils were in the minority it is interesting to note that these comments serve to highlight the fact that this system is genuinely challenging for pupils and not an easy option. One pupil claimed it was “a bit too adventurous for now” while another thought the questions “a bit hard and should have been easier for kids of our age”.

During phase 1 the feature most enjoyed by pupils was the voice recording. In phase 2 it was overwhelmingly the messaging features, perhaps because of the new developments over the second year. In phase 1 the questions and ‘I can’ statements caused the most difficulty while, in phase 2 pupils said they found the milestones more difficult. Again this may well have been because in the first year pupils and teachers found the language of the questions and ‘I cans’, inaccessible. As a result a lot of time was spent with the teachers improving the wording. Certainly when asked directly if they found the ‘I can’ statements difficult or easy, the majority in the second year said they found them easy to use. Interestingly some pupils also indicated in their responses an awareness of progression within the “I cans”.

“I found them reasonably easy but then again up to level 5-6 it started to get a bit more complex and above my personal level. They were all similar to a certain extent but each level they involved another part to it, If you get what i mean”

Similarly, a lot of work was done to explain what was meant by the term ‘milestone’ in the first year and although pupils in phase 2 indicated some difficulty it was not with the concept of a milestone, but rather with the number of different elements involved in the milestone area, particularly attaching questions to milestones.

Although when asked what they would change about the system, the majority said they would not change anything, the changes that were mentioned included appearance and layout, the method of uploading milestones, the wording of the questions, and inappropriate messaging. One pupil asked for a games or fun page to be added to the site and this was echoed by one of the pupils in the face-to-face interviews. A dyslexic pupil interviewed face-to-face during the first year suggested that the text on the website should also be available as a sound file for pupils with reading difficulties and this was in fact implemented in phase 2.

Also in phase 1, many of the pupils who sent in emails mentioned the issue of having to pay for their phone calls when using a mobile phone not on the Orange network. This was obviously a sore point particularly in relation to taking the final assessment or viva. This was addressed by offering schools top up cards and by the second year this seems to have ceased to be such an issue, possibly because teachers had been more explicit about call and messaging costs.

A few technical problems were encountered by pupils during the first year e.g. difficulties logging on, recording voice postcards, etc. and one pupil reported receiving a couple of unpleasant messages from other pupils. In the second year fewer technical difficulties were reported, although one pupil reported receiving an “insulting”’ peer annotation.

Rather disappointingly in the first year just under half of the pupils report that they had not looked at the work of other pupils. Only just over half of those who did, left any comments. A lot of work was done by teachers during phase 2 to encourage peer annotation, and by the end of the second year this figure had improved enormously, with 76% reporting visiting other pupils’ work and leaving annotations, and 84% reporting receiving annotations or messages about their work.

The exit questionnaire was amended for phase 2 to find out if pupils found these annotations helpful or desirable and the majority said that they had.

When asked if they would like to see more or less comments on their work from their teachers 81% of pupils were in favour of more comments. Sadly a number of the responses suggested that some pupils are not used to teacher feedback

“More - because I don’t really know what he thinks of my work”

“A few more because you do not get comments from the teacher very often.”

Interestingly while some pupils observed that they wanted feedback from peers as well as teachers,

“I'd like to see more comments from not only the teacher but the pupils as well to give me more ideas on how to improve my work! “

Others indicated they placed more value on the feedback from the teacher because it was likely to be more honest.

5.2 Teacher support and feedback

Teacher feedback was collected through school visits, face-to-face sessions, and annual reports based on the research questions. When asked if being involved in the project had been useful the response in both phases was overwhelmingly positive, with one teacher believing it was such a valuable experience that he asked incredulously, “Are you kidding?”

Initial responses to the project usefulness related to the impact on pupils and listed factors such as:

· Increased motivation of pupils,

· Increased self-esteem,

· Awareness of audience,

· Pupils taking responsibility for their own learning and becoming independent learners,

· Improved teacher-pupil relationships

· Recognition of the value of oracy.

Several teachers also mentioned that the project promoted a sense of online community and developed communication skills.

“Promotes sense of e-learning community because of the way it encourages communication amongst all members.” (Teacher - School 2)

In relation to themselves teachers identified useful factors such as: raising staff awareness of formative assessment, offering a different ‘non didactic’ approach, allowing for ‘anywhere anytime learning’, providing an opportunity to interact with pupils and each other, and opening up a range of exciting professional development opportunities.

“For a professional development it has been fantastic, opened eyes and make me think about things I hadn’t thought about or encountered before.” (Teacher – School 4)

However, in spite of the enthusiasm and positive responses to the project it must be noted that it proved much harder to get the schools started than expected. Admittedly there were some delays with the software development, and some schools appeared to be holding on for the technical ‘bells and whistles’, such as the voice postcards and SMS messaging, before starting even though the website was up and ready. However, it soon became apparent that the real issue was that most of the schools needed considerably more support than had been anticipated. Originally the plan was for one visit per term into the schools plus one face-to-face session at Ultralab. As the project progressed it became clear that Ultralab facilitators needed to go into schools on a more regular basis, and in some cases to work alongside teachers in introducing the project. The facilitators helped schools register their pupils and offered support when pupils were doing their self-assessments and voice recordings. This not only gave the teachers more confidence to get going but also served to maintain momentum when day-to-day pressures would otherwise have pushed the project onto the back burner.

In Phase 2 of the project regular face-to-face meetings between teachers and their facilitators were timetabled in from the start of the year to ensure schools had enough support and continued to make progress. Also, as was stated earlier, the role of the facilitator was changed to include online facilitation within the web space. Facilitators took a much more pro-active role working alongside the teachers in the online space, commenting on pupils work and monitoring progress. This had the added bonus of making visits into school more relevant to the pupils as they were able to meet their Ultralab facilitator virtually and face-to-face.

Another surprise was how difficult it was to communicate with the teachers. Although most of the teachers involved in the project were ICT co-ordinators they did not actually respond well to the various ICT based means of communication set up for the project management. An online discussion forum was set up, initially using an Ultralab platform and later moving into Think.com, which several of the teachers were familiar with, but with three exceptions most failed to use it and the forum was reluctantly abandoned. Many teachers were also slow to respond to emails and phoning the schools meant getting past the gate keeper in the office, not always easy. Instead, in most instances, facilitators found that communication via mobile phone was more effective.

When asked what impact the project had on their schools, all of the teachers in both phases reported that the project had a profound impact on the pupils involved in the pilot. Many also added that other pupils outside the pilot groups had expressed an interest in being involved.

“The ability to add work and have an audience is a prime motivator for the children. Anything involving mobile phones is seen as fun and the domain of children … Children in pilot group very keen to use eViva – find it intrinsically enjoyable to use.” (Teacher – School 2)

In phase 1, apart from raising interest amongst colleagues, teachers reported that the project had very little impact on staff or senior management because it was at too early a stage of development.

In phase 2 two teachers mentioned that the project had impacted on other staff in their schools, one received a very positive response after demonstrating eVIVA during an inset session on ‘formative assessment’, the other ‘spread the word’ through the pupils.

“Pupils who were using eviva in ICT were talking, and enthusing about it in their geography and D&T lessons, which led to the teachers to want to find out more about it.” (Teacher – School 3)

The other three teachers commented that while staff and senior management are aware of eVIVA, they are not yet making the link as to how it could impact on their work.

However, having said that, all but one of the teachers reported using the information from eVIVA to inform their assessments this year. The one who didn’t is not actually teaching the group but commented that if she were, she would give the URL to parents so they could view their children’s work.

One teacher reported using the information to check that his initial judgements were accurate, to inform his preparation and prompt his classroom questioning. Another said he used the uploaded work and the pupil comments to “more fully inform the assessment process, reinforcing and adding evidence of strategies employed by the pupils in compiling the work.” Similarly the third teacher reported using the milestones to help decide on pupil levels for their end of year reports. The fourth teacher plans to compare the levels achieved through the existing in-house system with the levels resulting through use of eVIVA. Significantly one of the teachers observed,

“No longer is assessment a bolt-on but is now integrated into the teaching process.” (Teacher – School 3)

5.3 Lessons learned

In phase 1, when asked what had been learned about the assessment of ICT capability, one teacher’s response was that “It’s proved really difficult.” This was a recurring theme through the first year.

As noted earlier pupils found the “I can” statements and questions difficult to understand. There was an expectation that teachers would help pupils to interpret the statements but it quickly became clear that teachers also found them difficult to understand. The discussions surrounding the “I can” statements and the eVIVA questions brought the shortcomings of the existing system into sharp relief, and made clear how much confusion there is about National Curriculum levels and how to interpret them. On the positive side the teachers reported that the discussions surrounding the statements led to a much greater awareness on their part about pupils’ ICT experience and capabilities. It also led to a much greater understanding of the criteria by which they were being assessed on the part of the pupils.

Another difficulty highlighted during phase 1 was that many pupils failed to do themselves justice in annotating their work and their comments show that they clearly found it “scary”, “hard” and even “adventurous” to be asked to identify their moments of learning. What was also noticeable was that when asked about what they have learned, pupils invariably talk about what they have been taught, not the same thing at all! Teachers identified the need for more ongoing teacher annotation to prompt pupils to reflect more effectively, as well as to give formative feedback on work.

It was also clear from teacher feedback that the issue of “peer review” and the use of annotation was an area needing further exploration and development. Pupils need to be taught new skills in the art of reflection and peer review. Although attention was focused on this area during the final milestone of the project, many of the schools found it took time to achieve, and even encountered some early resistance on the part of some older pupils who seemed to find the idea of opening their work up to criticism initially quite intimidating. There was, however, considerable improvement in annotation in phase 2.

Teacher and pupil feedback from both phases indicated that pupils are motivated and empowered by sharing their work in an online space. Exhibiting their work in an online portfolio appears to give pupils a sense of audience and serves to lift their expectations and performance.

In phase 2 rather than talking about difficulties, teachers talked about what they had learned about learning through using eVIVA.

Teachers reported looking more closely at the process of skills acquisition and progression in pupils’ learning and that this had clarified the steps involved. They felt they had a clearer understanding of what they had to do to address some of the learning issues such as drawing attention to the learning in lessons, fostering oracy, prompting and scaffolding pupils, and developing independent learning skills. One teacher suggested the need for a more formal induction period looking at the “aims and ethos of assessment”. Another observed that as a result of working on this project, instead of thinking about what she is teaching, she now finds herself thinking about what the pupils are learning.

“Students find it difficult to be independent learners. Current system not geared to this. With e viva they were suddenly being empowered but they need more help and support than I thought they would need… In a normal lesson they are told what they will learn (objectives of the lesson) – here they have to think it out for themselves. They start to become reflective practitioners.” (Teacher – School 4)

Also for the first time in the project there was mention of gender difference, perhaps because until the end of the second year very few pupils managed to complete their final vivas.

“Girls seem more task focused, mature, communicate orally better. More aware of learning aspects - Boys interested in ‘playing’ with the technology.” (Teacher – School 5)

5.4 Improving assessment

During phase 1 the teachers clearly indicated that they felt eVIVA had improved assessment of ICT but that this belief was based on “Gut feeling that it does” rather than on supporting evidence. Teachers claimed that eVIVA had made them think more about the assessment criteria and process because they had to talk to pupils about them. As a natural extension to this there was a strong feeling that pupils had gained a far better understanding of the assessment process as a result of the dialogue about the “I can” statements and questions.

It was clear at the end of the first year that in most schools the pupils self-assessments had tended to be unrealistic, with many of them awarding themselves levels that were too high. Much of this was due to the difficulties over the accessibility of the “I can” statements as has been mentioned in the previous section. In the second year a lot of work was done by facilitators and teachers to make the statements much more accessible, and this seems to have resulted in more realistic levels in those schools where teachers worked directly with their pupils. In some schools the pupils worked on the project during lunchtimes, or in a special session run by a classroom assistant, so the teacher support was more limited and mainly online. In these cases there were still instances of high self-assessments, however, it is important to note that one of these groups was a gifted and talented group so perhaps their high levels were justified.

It was also clear from the phase 1 feedback that pupils find annotating their own and others’ work difficult, and, as noted earlier, teachers identified a need for more pupil guidance in this area. In phase 2 teachers reported improvements but noted that in some instances pupils were seen to be reacting to teachers’ comments rather than being proactive and reviewing work done by peers. However, the number and quality of pupil and teacher annotations increased and improved considerably over the second year, and most teachers predicted what one teacher referred to as ‘incremental implementation’, as pupils continue to use the system.

Teachers employed different strategies to encourage pupils to annotate, such as asking the pupils to identify one thing that was good about the work and one thing that needed improving.

“I used annotation to motivate, focus, and encourage use of the system … but annotations by children have been more difficult as this is new and in many ways alien to them.’” (Teacher – School 2)

It was also clear that teachers could see the benefits of peer review and that being asked to annotate the work of their peers has sparked off a lot of useful dialogue amongst pupils.

“Although a little reluctant to commit comments to the system, a lot of oral dialogue took place about pieces of work what was good about them and what they could improve. The next step is to encourage more pupils to get their comments online.” (Teacher – School 1)

All of the teachers involved in phase 2 stated that eVIVA has helped their assessments because of the insights into pupil thought processes and the ability to use online dialogue for clarification. One claimed that eVIVA has helped him to refocus his attention on the children and keeps him realistic about their capabilities. Another commented,

This is supported by two others, who claim that the pupil reflections allow them to show their “capability at a higher level than their work would suggest”, and also help teachers make “inroads into differentiated assessment”.

Feedback from two of the teachers suggests that the communication elements of the system need to be extended to include an online discussion place or forum where the pupils can seek clarification on the eVIVA process rather than on their work. One of these teachers actually set up such a forum using www.think.com (Oracle software). He then used the forum to upload help files on aspects like the voice postcard and to offer a question and answer discussion called “Ask Me” for pupils. (See Figure 9 below)

Interestingly although he also set up a discussion specifically about the technology, the majority of pupils seem to have focused their questions much more on the assessment process. Certainly this has served to highlight a definite area for development in the software, as there is currently nowhere in the system to facilitate and capture this kind of dialogue other than the messaging which is 1:1 rather than 1:many.

Figure 9. Think.com eVIVA forum

5.5 Demonstrating ICT Capability

Feedback from teachers was that using eVIVA made pupils much more aware of what they were doing “and why they were doing it”. Knowing that their work was going to be seen by others had a positive effect and “raises their game or expectations” and certainly teachers reported increased self-confidence in pupils as they realized,

“They are better than they thought! It has been motivating for students – positive reinforcement”.

Also it could be argued that simply by using eVIVA pupils demonstrate “competence in use of Internet, uploading/downloading, searching” etc.

However it was also clear that pupils in the first year didn’t do themselves sufficient justice with their milestones, often introducing their work with comments such as “Here is my presentation – I hope you like it!” With inadequate annotation the files they uploaded as evidence became much more important and screenshots not at all helpful.

‘Ofsted told me I woefully underestimate what has been achieved but when I look at the files that have been uploaded I’d probably assess them even lower! Snapshots really do need commentary. I guess that we have not yet uploaded enough evidence to give a true picture. It’s a little like trying to get an accurate impression of an elephant by viewing bits of it through a tube. Whole files (whether accessible or not) and commentary are essential.’ (Teacher – School 1)

During phase 2, as a result, pupils uploaded complete files as evidence instead of screenshots, and milestone reflection or annotation significantly improved. However, one teacher observed that milestone comments were still not good enough and that teachers themselves need to be secure in their understanding of ICT capability so that when “minimalistic” work is published they can tell pupils what “ICT capability” is.”

Teachers also noted that the showcase element of eVIVA allows the pupils to show progression over time, it encourages them to try harder and be seen to be improving, it allows them to focus on newly developed skills and highlight them in their work, it allows them to demonstrate their capabilities in different contexts of their own choice, and seeing the work of others encourages them to review and amend their own work. “Publishing to an audience is more significant in this environment.”

During phase 1 only one teacher managed to get all his pupils through to their final viva. Most of these pupils gave thoughtful and well-considered responses and had obviously prepared well for their vivas, probably scripting their answers.

Several teachers expressed concern about less able pupils and problems they might have with the “I can” statements, the questions, the reading level of text used on the system and the emphasis on text for the annotations. It was suggested that the oracy element could be extended to the annotation facility and this feature was implemented in the second year. It was clear from discussions at the teacher conference that the questions, although they have been simplified, are still too complex and long-winded. They need to be improved if we are to avoid finding ourselves in a situation where those pupils with weaker literacy skills will be disadvantaged because of a need to script their answers.

During phase 2 most of the schools managed to get a reasonable percentage of their pupils to the point of taking their viva. Results varied from school to school, with pupils from the school using the discussion forum producing some excellent answers, again obviously well thought through and prepared. There was considerable discussion at the face-to-face session immediately preceding the vivas about whether pupils would really be demonstrating oracy skills if they pre-scripted their answers. Different strategies were explored such as using bullet points or other prompts to help scaffold pupils. It was obvious when listening to the files which pupils had scripted or planned their answers and which were speaking completely “off the cuff”. Anecdotal evidence from teachers suggested that boys were more likely to adopt a spontaneous, unscripted approach than the girls.

Unfortunately, due to the timing of the vivas, which seem to naturally fall into the second half of the summer term, very few teachers were able to comment on their usefulness at the final face-to-face session. However, those who did, recommended that the questions be broken down into smaller parts and simplified to make it easier for pupils to draw out important points. It was also suggested that there should be two stages to the viva, an initial read through and then a repeat of the questions. Finally the point was made that, as with annotations, pupils would improve over time as they became familiar with the process.

“Even the viva can be viewed as formative! If this was the real deal and started at year 5 as they get older and face interviews in the real world they understand where the questions are coming from.” (Teacher – School 1)

5.6 System manageability and Teacher Workload

It was an important part of the initial brief that the eVIVA system should not increase teacher workload. In the first year teacher feedback on this issue was mixed. Three of the teachers claimed the system was manageable, or had minimal impact on their workload, with one going so far as to say,

“Found it easy – prepared to be critical, but it was dead easy… Impact on teacher workload – a lot easier as no homework to take home – at least in the form of books to mark. Around 70% of children have computers at home and 50 – 60% have Internet access. System straightforward so not a long time to learn – children found it easy.” (Teacher, School 4)

One teacher mentioned the time implications of listening to sound files particularly if scaled up but no one else picked up on it and, in contrast, another teacher observed how much better it was assessing portfolios and online work than marking books.

The remainder focused on the issue of manageability and the time required to introduce the system, particularly explaining the “I can” statements, trying to encourage annotations and milestone uploads etc and the way in which this ate into time needed to cover schemes of work and lesson content. However, balanced against this, many of the teachers identified benefits such as getting to know the children and their capabilities much better, and argued that, once pupils and staff were familiar with the system and the processes, it would save time.

In the second year the tension between working with pupils on eVIVA and finding time to meet the requirements of the curriculum remained an issue for teachers. However, rather than viewing this in a negative light, solutions were offered to reduce the problems. It was suggested that breaking the project down into stages, introducing different features over time would help, rather than trying to do it all in one year. It was also noted that, as pupils used the system across the Key Stage, then each year would require less input and time as pupils built on their experience of the previous year.

One teacher raised the issue of mentoring or supporting pupils in their use of the system as being potentially time consuming. His solution was to introduce a mentoring system involving older pupils and parents, overseen by the teacher, and an online community space in which the mentoring could take place. Another teacher also mentioned having successfully used a ‘buddy’ system this year with last year’s pupils acting as the buddies for this year’s group.

Two teachers commented on the need to extend the communication aspects of the system to allow for more online discussion about the use of the system and the eVIVA process, rather than about the work. In the case of School 5 the teacher actually set up an additional online discussion forum where the pupils could ask questions about technical or assessment issues relating to their use of eVIVA. Interestingly this teacher was not working directly with the pupils, a gifted and talented group being supervised by a classroom assistant, so the online forum provided the space for the additional dialogue needed to explain the system.

4 of the 5 teachers stated categorically that using eVIVA hadn’t changed their workload because they would be doing the marking, commenting and assessments anyway. Although it was suggested that it would be helpful if the system could more effectively distinguish annotations already addressed from those not yet dealt with.

“No big impact on assessment time – checking work on server had been built into assessment routines: eVIVA achieves what I was doing before, but in a neater way.” (School 2)

The voice files are mentioned but only to say that listening to them has not been too onerous.

“Listening to voice postcards has replaced something else but no idea what! (Can’t have been very important because no one has said anything!)” (School 1)

One teacher describes as ‘ideal’ the potential eVIVA offers her, as head of department, of an overview of what other teachers and pupils are doing, as well as the ability to add her own comments. Another mentions that the system helps him in his head of department role of supporting non-specialists, “because it pulls all assessment tools into one place and combines it with student portfolios.” (School 3)

5.7 Technical issues

During phase 1 there were a number of technical issues, which slowed the development of the software. The ambitious and innovative nature of the project, particularly in relation to VXML or voice recognition technology, caused some complications. The main problem revolved round the eVIVA call flow, which was actually beyond the capabilities of the voice gateway (Orange) we originally planned to use. Some time was wasted trying to make the two systems talk to each other before a gateway was identified that could cope. Once this problem was resolved, the remaining elements of eVIVA, such as the text messaging, fell neatly into place.

There was very little system down-time during the first year of the project, the only real problem was when an electrical storm took the voice server offline for a few days, and none of the teachers reported the problem until a facilitator visited one of the schools. Once it was known the problem was easily solved. During the second year there were a few minor versioning problems but overall the system worked extremely well. As one teacher observed,

“Overall, I am amazed at how efficiently the software works given its complexity and use of sound files…There have been some technical problems but as a pilot system I am impressed with the effectiveness of the system. My pupils were obviously frustrated at times by technical difficulties but were enthusiastic to persist.”

In phase 2 a number of changes were made to the software and the system, based on phase 1 feedback from pupils and teachers. The look and feel of the website was changed in response to feedback relating to pupils with literacy problems, with the aim of making it more accessible. Accordingly the text size was increased, the amount of text on the home page was reduced and simplified, and a pop-up help system was introduced, with accompanying voice files, so that pupils could listen to the help or read it. Also to help those with literacy problems, the messaging system was amended to allow pupils to leave voice-recorded messages, using the eVIVA freephone number.

The teacher view generated a simple CSV file of the pupil information page, which could then be imported into a school management information system. In the second year this reporting feature was improved, to allow more sophisticated reporting and graphing of information.

In the planning stage of the project, there was debate over whether to have a real person interacting with pupils during the viva or to use a pre-recorded or computer-generated voice. It was decided not to use a real person, so that pupils would feel comfortable recording and re-recording their answers until they were satisfied with them. In phase 1 a pre-recorded voice was used but feedback indicated that the voice was too ‘posh’ and not young enough. So, in phase 2, a computer-generated, classless voice was introduced, this had the added benefit of making it easier to make changes to the questions without having to re-record relevant voice files.

The navigation of the eVIVA call flow was also simplified to allow voice navigation instead of keying in numbers. Feedback made it clear that being asked to think about pushing a key on the keypad when ready to start recording distracted pupils from thinking about their answer, voice navigation makes the process easier. There was only one instance of a problem with this with one pupil who had such a strong local accent the system could not identify her. In the end she had to resort to keying in her PIN number for identification rather than her voice.

One question often asked is about the security of the system. “How can we be sure that the person taking the final viva is actually the child in question?” Currently, the assessment is not a high stakes assessment and the purpose of the system is to inform the judgment of the class teacher, who should be familiar with the pupil’s voice anyway. By recording a voice postcard at the start of the project pupils are also providing a voice file for comparison should there be any doubt about the authenticity of the recording. However, if the project were to roll out nationally, it is important to consider whether the use of the voice postcard and PIN number would be sufficient to ensure that the correct candidate was taking the telephone viva. It would be advisable for any system introduced to use caller log identification, and possibly some simple voice recognition software.

Another question often asked is “Who pays for the phone calls and text messages?” As mentioned earlier Orange set up a free-phone eVIVA number but unfortunately the mobile phone industry does not recognize free-phone numbers across all providers. All Orange calls or calls from landlines were free, but for any call from a mobile not on the Orange network there was a charge. Although teachers and pupils were made aware that calls from a land phone were free, and Orange offered to reimburse students for their calls, it took a while for the message to reach all teachers and students. As can be seen from some of the feedback, many pupils were very indignant at the thought of being expected to pay phone charges to take an assessment. Obviously this issue has implications for national rollout since either a solution has to be found if the oracy element is to be retained. Some schools in the pilot addressed this situation by making provision for pupils to use a school phone but if the system were to be rolled out nationally this would be an impractical solution. The cost of setting up a free-phone number would need to be explored and factored into any system. The best solution would seem to be to negotiate a free-phone number that dials out, rather than the system used during the pilot where pupils have to dial in. Pupils would merely send a short text message to indicate that they are ready for their viva and the system would then dial the appropriate phone number specified in the online student profile.

6. Conclusion

It is interesting to note that Tomlinson (2004) in his interim report on the 14-19 curriculum, highlights the failure of the current system to

“Equip young people of all abilities with the generic skills, knowledge and personal attributes they will need for future learning, employment and adult life.”

Tomlinson’s report espouses an assessment system which amongst other things: builds upon existing strengths and good practice; enriches learners’ experience by using a variety of types of assessment; provides formative feedback on progress; avoids placing undue burdens on learners, teachers and institutions; embraces the potential benefits of e-assessment: makes appropriate use of the professional judgement of teachers and is ‘fit for purpose’. These are all elements which have been incorporated into the eVIVA system and which have been discussed throughout this report. This conclusion will consider whether eVIVA is ‘fair and fit for purpose’, it will also discuss the use of portfolios for assessment and possibilities for the future of eVIVA.

6.1 Fair and fit for purpose

When asked if eVIVA is a fairer way of assessing ICT one teacher said it was a “different way” which allows progression, accessibility and equality of opportunity. He did qualify this by saying that because it requires pupils to select which work they make public it does rely on pupils being able to identify which is the best work to select. According to another teacher,

“Is ‘Fair’ best term? Would say it is ‘better’ because of peer reflection, self evaluation as part of assessment process”

Certainly, teachers at the end of the second year were very positive about the system mentioning: its potential for moderation across schools; its transparency; the ease of access through a web browser; the fact that it provides evidence to support assessments made and offers continuity and progression across a Key Stage; its involvement of pupils in the process and the fact that it allows them to elaborate on their answers through annotation. For the teachers involved in the project, and teachers who have attended presentations on the project, the most significant word, encapsulating the whole eVIVA process, has been “ipsative”,

“I am not a fan of levels, I prefer to record achievements and routes to improvement – this is what eVIVA does! (Ipsative)” (Teacher – School 1)

During an ICT lesson there are lots of technical demands on the teacher, which make it difficult to have much quality dialogue with individual pupils about their work or their learning. The opportunity to have more reflective dialogue and extend relationships with pupils is another positive outcome that teachers have highlighted throughout the project.

Although the language of the viva questions still needs further revision to make the questions more accessible, and pupils need help to develop and improve their oral skills to enable them to “talk with purpose”, the oral element potentially offers greater fairness. This is particularly true now it has been extended to allow pupils to add voice annotations, making it easier for those with writing difficulties, or different learning styles, to enter their comments through speech rather than text.

It is clear that eVIVA has the potential to be a fairer or ‘better’ system. It is also clear that pupils are not used to working in this way and, if they are to become independent learners, they need to be supported in developing the reflective processes that the system promotes. Teachers have also found that they need to make changes to their normal way of working in order to maximize the formative nature of the eVIVA process and make assessment an integral part of the learning process, rather than a bolt on activity. This was clearly illustrated by the number of pupils who said they wanted more feedback on their work from their teacher.

6.2 Using e-portfolios for assessment

According to Sue Walton of QCA (2004),

As mentioned earlier, teachers needed much more support from the facilitation team to keep on track than anticipated. The need for support appears to have been as much to do with the change in classroom practice as the technical aspects of the project.

Feedback from both phases of the project, and evidence currently on the eVIVA website, certainly suggests that e-portfolios are effective tools for supporting both formative and summative assessment in the classroom. Teachers clearly value having everything related to their assessments, including the evidence to support their judgements, in one place, with that place being easily accessible through a web browser and the Internet. It also seems clear that e-portfolios have the potential to engage and motivate pupils, although it is worth emphasizing that this may well depend on who is at the centre of the system, the teacher or the pupil. In eVIVA our aim has been to put the pupil at the centre of the system through the self-assessment and reflective tools, the privileges system and publishing rights.

It was also abundantly clear that pupils particularly enjoyed using the mobile phone and text messaging elements of the e-portfolio, particularly given that mobile phones are very much part of their everyday lives but are usually banned in school. Their responses showed that the messaging introduced a “fun” element and really engaged and motivated pupils.

“It is fun to comment on other peoples work because you can tell them how well they are doing.”

“I enjoyed the messaging to your friends or to other people it made the work fun and enjoyable.”

6.3 The Future

Now that the eVIVA project has come to an end it begs the question “What next?”

Hargreaves (2001) identifies that a feature of “blue skies work” is “the degree of risk-taking involved” and it could be argued that funding this project has involved an element of risk for QCA because of the implications that, along with teachers and pupils, the educational establishment also needs to change its practice. The challenge now is that the risk appears to have paid off, and the project findings suggest that the eVIVA process actually works.

The project has attracted an enormous amount of media attention and has been featured in several newspaper articles, at least three television and two radio broadcasts. Presentations on the project have been given at a range of conference venues in the UK and overseas. Teachers from different sectors, phases and subject areas have expressed an interest in the project, and we have been inundated with requests to be involved in any further trials or pilots. It is interesting to note that teachers of foundation stage pupils felt that the oral element had much to offer their pupils, while, at the other end of the scale, lecturers at a further education conference wanted to be able to use the system with their students.

Different aspects of eVIVA have also attracted particular attention because of the perceived potential apparently offered in certain areas. Some of the suggestions have included developing it as a modern languages tool, an e-portfolio system, or as an online moderation community for teachers. While it is tempting to consider some of these possibilities there is a danger that in focusing too closely on one aspect of the process the assessment for learning emphasis would be lost.

Phase 2 feedback seems to indicate that developing an integral community aspect to the eVIVA website would foster not only dialogue between teachers and pupils about the learning, but would also facilitate the sharing of ideas and best practice by teachers. It is interesting to note that, as originally hoped, their involvement in this project has engendered extensive discussion amongst the teachers about assessment and learning. It has been suggested that eVIVA would make an excellent professional development tool for teachers, with the online portfolio offering an ideal way to record their activities for CPD and accreditation. Certainly this would appear to match a need identified by Black and Wiliam,

“ What they (teachers) need is a variety of living examples of implementation, by teachers with whom they can identify and from whom they can both derive conviction and confidence that they can do better, and see concrete examples of what doing better means in practice” (Black and Wiliam 1998)

As part of the original brief for the eVIVA pilot Ultralab was asked to consider the issue of scalability and any technical issues that might have implications for national rollout. As mentioned earlier in this report the software has proved amazingly robust and reliable particularly when considering the complexities of the system. The issues relating to system security and the cost of calls to pupils do not appear to be insurmountable problems. Voice recognition software combined with caller log identification could solve the problem of user authentification, and a simple notification mechanism could be put in place to activate an eVIVA call to the pupil rather than the other way round.

However, it is still a big jump from this relatively small-scale pilot straight to national rollout given that there are still things we have yet to discover. There are several technical developments that we would want to trial as part of the scaling up process.

The first of these could be to develop the software as open source. This would mean that schools could download the software to their school intranet, personalize the “I can” statements and questions and store work files on their local server. The benefit of this would be that teachers would have a better understanding of the assessment criteria if they had a hand in developing them. The challenge would be to see if it would still be possible for them to work online with other schools across the country while still maintaining their own system.

In addition to the move to open source changing to using the WebDAV protocol (Web-based Distributed Authoring and Versioning), a set of extensions to the HTTP protocol, which allow users to collaboratively edit and manage files on remote web servers, would allow users much more flexibility in editing the website than currently exists. This would mean, for instance, that the help files could be edited more easily and new ‘skins’ could be offered to allow users to change the look and feel of the site etc. The system could also be extended to accept multimedia messages as well as text.

As stated earlier there are still things we have yet to discover. The main thrust of our research so far has been about possibilities, potential, and proof of concept. The main focus has been to show that eVIVA can work it has not been about proof of effectiveness. We believe that eVIVA should now move into a new phase to discover whether the potential that it offers of a fairer, better, more effective way of assessing learning can be fully realized. To do this we would argue the need to trial on a much larger scale that previously and would suggest working within a different school phase or education sector and across different subject areas. The creation of an open source publicly licensed version of the software would also enable wider distribution across schools. We would also advocate integrating the use of eVIVA into other projects, which aim to develop online community, new learning, assessment, CPD and awareness in government agencies in connate ways.

Bibliography

Black, P. and Wiliam, D. (1998) Inside the black box King’s College London

Black, P. et al (2002) Working inside the black box King’s College London

Hargreaves, D (2001) The Nuttall Memorial Carfax Lecture – BERA 2001

Jonassen, D (1994) Technology as Cognitive Tools: Learners as Designers - IT Forum Paper 1

Mabry, L. (1999) Portfolio Plus – A critical guide to alternative assessment Thousand Oaks, Corwin Press Inc

McGuire, L. (2003) Milestone 4 Report - Presentation to QCA of the final report on the eVIVa Project - July 2003

Smith, K. (2001) Children’s rights, assessment and the digital portfolio: Is there a common denominator? In IATEFL Brighton Conference Selections (p.55-58)

Tomlinson, M et al (2004) 14-19 CURRICULUM AND QUALIFICATIONS REFORM – Interim Report of the working group on 14-19 Reform www.14-19reform.gov.uk

Walton, S (2004) eVIVA: Assessment for learning: using e-portfolios in the classroom. IAEA Conference Paper, Philadelphia, 2004

Appendix Documents

Appendix 1: Table showing eVIVA record of activity 36

Appendix 2: Collated teacher responses to the main research questions 37

Appendix 3 Collated teacher responses to the additional questions 48

Appendix 4 Pupil online exit questionnaire – 2003 58

Appendix 5 Pupil online exit questionnaire - 2004 61

[LM1]

[LM2]

[LM3] Ict rather than computer room

[LM4]aware of

[LM5]Make clear their phone number

[LM6] Explain messaging preferences

Sections

Personal tools

Final Report to QCA on the eVIVa Project 2002 - 2004