Too Hard To Measure: On the Value of Experiments and the Difficulty to Measure Lesson Quality

Interesting article in The Guardian (from some time ago, I'm a slow reader) about the overblown importance attributed to doing experiments during science lessons.

The article reminds me of my experience in Cambodia, where experiments are also frequently espoused as proof of a student-centred lesson.


In reality experiments in Cambodian classrooms are often a very teacher-centred activity:

• the teacher demonstrates and students (at best) trying to observe what happens. 
• students do the experiment in large groups, by adhering to a strict series of steps outlined in a worksheet. 
• students work in large groups, in which usually only one or two students do the work, The others are merely bystanders. 
• the procedure, observations and interpretation of the experiment are laid down in detail beforehand.

The article touches upon two interesting elements.  First, there is the questionable educational value of many experiments in science classes.  secondly, there is the challenge to measure lesson quality beyond 'ticking off' the occurrence of activities such as experiments.

The article refers to 'The Fallacy of Induction' from Rosalind Driver.  Her book 'Making Sense of Secondary Science' is an excellent book on misconceptions in science education and has been an important inspiration for me.  

Driver doesn't dismiss practical work in science, but argues that 'Many pupils do not know the purpose of practical activity, thinking that they ‘do experiments’ in school to see if something works, rather than to reflect on how a theory can explain observations.' (Driver et al, 1993, p.7).

She raises two main arguments.  First, practical activities are often presented to students as a simulation of 'how science really works', collecting data, making observations, drawing inferences and arriving at a conclusion which is the accepted explanation.  It's simplistic, and pupils happily play along, following the 'recipe' in the 'cookbook', checking whether they have 'the right answer'.  In reality, science rarely works this way:

For a long time philosophers of science and scientists themselves have recognised the limitations of the inductivist position and have acknowledged the important role that imagination plays in the construction of scientific theories.' (Driver, 1994, p.43)

The second argument is that pupils don't arrive in class with a blank slate, but with a whole range of self-constructed interpretations or 'theories' on how natural phenomena work. These 'preconceptions' require more than an experiment to change, as children tend to fit observations within their own 'theoretical framework'.

Observations are not longer seen as objective but influenced by the theoretical perspective of the observer. ‘As Popper said, ‘we are prisoners caught in the framework of our theories.’ This too has implications for school science, for children, too, can be imprisoned in this way by their preconceptions, observing the world throught their own particular ‘conceptual spectacles.’ (Driver, 1994, p.44)

Misconceptions can be changed if they are made explicit, discussed and challenged with contradicting evidence.  After this 'unlearning' phase, children may adopt a different framework.  Driver concludes: 'Experience by itself is not enough. It is the sense that students make of it that matters' (Driver et al, 1993, p.7).  

Discussion activities, in which pupils have the opportunity to make their reasoning explicit and to engage with and try out alternative viewpoints, including the 'scientific one', need to be central (cognitive conflict). Practical activities can be complementary to these discussions, instead of the other way around, when discussion and conclusion are quickly reeled off at the end of the practicum.

However, the love for experiments while neglecting the question whether and what students are actually learning also touches upon the difficulty to measure adequately lesson quality.  Limited time and resources result in a focus on outward and visible signs. However, these:

• deny the complexity of teaching and learning;
• deny the individuality of students' learning and understanding;
• steers teachers and programme staff towards focusing on these outward signs, as they know they will be evaluated on these criteria. 

Collecting valid and reliable data on lesson quality is hard.  Self-assessment instruments are notoriously prone to confirmation bias. Lesson observations don't give a reliable everyday picture of lesson practice.  They suffer from the fact that teachers pull out special lessons when visitors appear for announced (or unannounced) visits.   Conversely, as Cuban describes beautifully, other teachers tremble and panic when an evaluator walks into their classroom and the lesson becomes a shambles.

Evidence-based evaluation is often touted as the way forward for development projects.  Randomized trials in health have been useful to collect a body of knowledge on what works and what not. In a randomized trial a group of students where teachers received pedagogical training is compared with a group of students where teachers didn't receive training.  Comparisons can be made with test scores, student satisfaction or drop-outs.

However, test scores are unsuitable as exams are notoriously prone to cheating and questions focus on recollecting factual knowledge, the opposite of what we want to achieve.  A self-designed test could be a solution, but there's the risk that programme activities will focus more on the test than on improving teaching skills.  Student satisfaction scores are prone to the aforementioned confirmation bias.  Drop-outs are hard to use as they are influenced by many interrelated factors such as geography, economic growth and government policy.

Ownership by the direct target group on the evaluation is part of the solution in my opinion, as well as using a variety of data sources.  In future blog posts I plan to write more on how we try to measure lesson quality.

------------------------

For more detail see this available study from Prof. James Dillon (pdf) on the value of practical work in science education.

 
Dri­ver, R. (1994) 'The fal­lacy of induc­tion in sci­ence teach­ing', in Teach­ing Sci­ence, ed. Levin­son, R., Lon­don, Rout­ledge, pp.41–48.

Driver, R., Squires, A., Rushworth, P. and Wood-Robinson, V. (1993) Making Sense of Secondary Science, Routledge.

Improving Physics Education in Cambodia: Beyond the Workshop

Last week we’ve been organizing a workshop on physics education for lower secondary teacher trainers in Cambodia at the regional teacher training centre in Kandal province.  All Cambodian physics teacher trainers were present.  That makes around 20 people.  The workshop lasted 5 days.   Each day we discussed a different part from the curriculum.  There were days we focused on sound, mechanics, pressure, optics and electricity and magnetism.  The last day participants collaboratively made a lesson plan using materials they’d learned.   There was a strong emphasis on low-cost experiments, but there’s also attention for simulations and animations, and student-centred approaches.

The underlying concept of the workshop – and actually the whole programme – is the TPACK concept (Mishra and Koehler,2006; Koehler and Mishra, 2007; Abbitt, 2011), an extension of Shulman’s idea of pedagogical content knowledge, this is knowledge of pedagogy that is applicable to the teaching of specific content.  

Read further here (new blog address)

Student-Centred Approaches for Science Teacher Trainers

This week we organize the first in a series of workshops on student-centred approaches for science lecturers and Ministry of Education officials in Cambodia.Integrating student-centred approaches has received support from the Ministry of Education in recent years.  It has been integrated in the revised curriculum for teacher training and teacher trainers are regularly stimulated to move away from rote-learning and adopt a more active approach.Below I include the presentation I gave during the opening session, which provides some background on the concept of student-centred learning as an introduction to the more practical sessions of the workshop.

Introduction to Student Centred Approaches

View more presentations from Stefaan Vande Walle

A main point in the presentation is that student-centred learning is not the same as working in groups. It means that students take more responsibility over their learning.  Some techniques that are introduced in this workshop focus on self-assessment, critical thinking, identifying misconceptions and active reading & writing, all techniques that can be done as individual activities.This workshop is only the first step in a change process.   Curriculum and assessment remain overtly content-oriented.  Follow-up activities are necessary to guide and stimulate teacher trainers, but are time-costly.  School directors, ministry officials and inspectors need to be convinced of the benefits as well, in particular in a hierarchical society as Cambodia.  

Why a bus system in Phnom Penh doesn't make sense

A family vehicle

Traffic is undoubtedly a major annoyance for expats (and many locals) in Phnom Penh.  My daily ride to Takhmao, 10 km outside Phnom Penh, resembles a live video game dodging packed motorbikes, "remorque motos" full with garment workers, tortoise-like Lexus SUV's and fume-belching Hyundai trucks. But, rejoice, because the city of Phnom Penh is considering introducing a bus system with assistance (meaning donations of discarded buses) from international donors.


There are many reasons to get frustrated by Phnom Penh's traffic, but a lack of public transport is not one of them.  True, there are no buses in Phnom Penh - except for long distance buses and the ubiquitous minivans, plying the routes to neighbouring towns, and neither is there a tram or metro system.  No, Phnom Penh has a demand-based, flexible, cheap and door-to-door transport system.  It consists of 'tuk-tuks' and 'motodops'. Motodops are motorbike drivers at street corners taking passengers. Tuk-tuks can take between 1 and 6 (for locals: 15) passengers, but can double up to transport motorbikes, furniture or about anything else.

Phnom Penh's famous public transport system

Tuk-tuk sir?

I don't understand the fixation on bus systems.  They offer no escape to congestion and, having experienced many bus trips in Cambodia, I wouldn't call them safe either. Phnom Penhites don't like walking to bus stops, which is understandable given the walking-unfriendly climate - with blistering heat (March -June) or torrential downpours (July - October) - and the absence of pavements (and shade).  No, the way forward is a better regulation of motodops and tuk-tuks, nurturing a convenience that is impossible to match with a standard bus or tram system.  


Phnom Penh's emerging congestion is caused by a switch to cars by an increasingly affluent middle class. Chances are slim they would trade their newly acquired luxury for daily bus rides.  Buses would inevitably become the transportation mode of the poor, a stigma that western countries are desperately fighting.


The focus on tram and bus systems betrays lazy thinking and imposing western policies in a context that is culturally and environmentally very different. As in education, it makes sense to try understanding what people think, how existing capacities can be strengthened and how a city can be made more liveable for all its inhabitants.

Adult Learning and Professional Development in Cambodia

Workshops and trainings are favourite instruments of donor organisations to develop capacity with their target groups.  Also VVOB Cambodia regularly organizes workshops, usually trainings on pedagogy and multimedia geared towards teacher trainers and government staff.  Measuring their success is notoriously hard.

Read further here (new blog address)

Vocational training in Cambodia

Technical and vocational training has received renewed attention by donors after a strong focus on general education in the 1990s.  The author focuses on the specific Cambodian context.  In Cambodia the formal vocational education (VE) component is very weak.  Only 0.7% of the country's labour force comes from publicly-provided VE institutes.  There's a widening skills gap in the country's two main economic sectors, garment industry and tourism.  Its post-conflict society counts a large number of vulnerable youth, who dropped out of formal education and lack supporting family structures.


In a successful VE model should not only focus on employability, but also on social and psychological needs.  Hsuan stresses the need to empower students, working on their confidence, critical citizenship, moral values, identity and learning attitude.  He suggests a three-tier approach to create such an empowering learning environment.

It's a strong article that compares and generalizes the approaches of 9 NGOs active in VE in  Cambodia.  It proposes a valuable model of how vocational education (not vocational training) could be strenghtened and moved beyond the sole focus on employable skills, taking into account the post-conflict and cultural background of Cambodia.  It remains to be seen though, whether public authorities share the underlying vision that all students should be educated to empowered and critical citizens, regardless whether they are in vocational or general education.


* Cheng, I-Hsuan (2010) 'Case Studies of Integrated Pedagogy in Vocational Education: A Three-Tier Approach to Empowering Vulnerable Youth in Urban Cambodia', International Journal of Educational Development, 30, pp. 438-446.

Environmental life skills?

Today was the last day of our nation-wide workshop in Cambodia on biodiversity and environmental life skills in education. Unfortunately not everyone has moved up to practice mode yet.

Thinking positively

In Cambodia VVOB focuses on student-centered approaches in science education in teacher training institutes.  Teacher trainers are guided towards adopting approaches to teaching that involve students, stimulate them to think and engage instead of passively noting down and regurgitating what the teacher declaims.


Some student-centred approaches focus on stimulating students' writing skills.  Writing and the skills that accompany it (collecting thoughts, filtering, structuring, creativity, conciseness...) are arguably important 21st century skills.  To extend writing in science lessons beyond copying teachers' notes we introduced techniques such as 3-2-1 sheets, 2-minute papers and creative writing (link to my presentation on these).  These techniques aim at stimulating students to write about what they have learned in their own words.

There may be unintended benefits as well

Interestingly, New Scientist refers to a study from David Creswell from Carnegie Mellon University that revealed that letting students write about what they have learned, stimulating qualities such as creativity and independence, gives them 'self-affirmation' that enables them to perform better.  


'Compared with a control group, students who 'self-affirmed' in this way had lower levels of adrenaline and other fight-or-flight hormones in their urine on exam day.'  (Health Psychology, vol 28, p 554, quoted in New Scientist, 27 August 2011)


Apart from letting students re-interpret what they have learned, these writing exercises may also - if well designed - contribute to improving students' sense of self-worth.

#H800 Technology's broader impact on education

Technology arguably has an impact on education which extends beyond the teacher and the learner. The broader impact of technology on education has been described in various reports. In week 22, we had a look at the Horizon Report (2011 edition) from the New Media Consortium, the National Science Foundation (NSF) Cyberlearning Report (2008) and the Becta Report (Harnessing Technology: Next Generation 2008-14). I compared the key elements of these reports with the Cambodia Master Plan for ICT in Education 2009-2013 (Third Draft version, 2009).

All reports consider technology as a major contributor to improving education and to meeting upcoming challenges.  In Cambodia challenges centre on dealing with an increasing influx of students as a result of demographics, reduced drop-outs and increased enrolments. In the US and the UK, they focus on globalisation and competition in a global workforce.

Both recognize the importance of aligning education with digital literacy and employability requirements for graduates. They regard technology in education as a prerequisite for improving the quality of education.

While reforming the public school system is well beyond the scope of our present task force, positive effects on schooling would certainly result from invigorating and inspiring learners through the rich new environments made possible by the Internet and developments in cyberinfrastructure.(NSF, p.12) 



One might say the reports suffer from a light to severe form of ‘techno-utopia’, regarding technology as the solution for all educational ills.  In this over-reliance on the role of technology in education, there is a lot of writing about ‘potential’ and ‘best practices’, but very little in the way of actual impact in real life contexts. There is hardly any discussion on the impact on students, teachers (apart from increasing training needs) and institutions. The Cambodian plan states the introduction of e-learning and a national Open University to reach remote areas and non-traditional students, but doesn’t mention deeply ingrained suspicion of e-learning and less teacher-dependent forms of education with students and the population. The NSF report considers all learners as ICT proficient ‘Millennials’ and brushes over indications that a deep digital divide persists, also in developed countries. 

Ensuring effective use of technology in education requires that also content and pedagogical knowledge are sufficiently developed. Merely focusing on technology provision leads to teachers letting students copy their notes into a Word document instead of a notebook or reading from a PowerPoint presentation instead from a book. There is no strategy in the Cambodian Master Plan how teachers will be trained from basic skills up to efficiently integrating technology, content and pedagogy, through an intensive in-service programme. Lack of content knowledge leads often to a teacher-centred approach with little room for experimentation, discussion and exploration, since these activities may expose the teacher’s alleged weaknesses.

There are some striking differences between the Cambodian Master Plan and the other plans though. The NSF report stresses the need for ‘cyberlearning’. This is learning mediated by networked computing and communications technologies. Students could experience more authentic learning by using large amounts of data, like climate data sets or databases generated by sensors. They could learn finding meaningful patterns in the datasets, tweaking parameters and ways of representing the data patterns. These kinds of activities require multidisciplinary approaches and networking between educators and scientists. The NSF report refers to the relevance of the ‘long tail’ in markets for education, allowing catering for students’ interests.

The global scope of networked educational materials, combined with “recommendation engine” software, helps individuals find special, niche content that appeals to their needs and interests.(NSF, p.16)

The Cambodian ICT Master Plan on the other hand stresses the alignment with the country’s curriculum. Software, textbooks and all ICT resources are to be fully in Khmer and explicitly approved by the Ministry of Education. The focus is more on an ‘isolated’ desktop model of ICT where learners access carefully selected resources, in contrast with a ‘web 2.0’ approach to technology-enhanced learning, let alone stimulating them to create Personal Learning Environments (PLEs).

All plans are surprisingly vague in pointing out evidence or strategies in how technology may improve learning outcomes. Monitoring is focused on measuring outputs, whereas improved outcomes on learning as a result of technology seem to be taken for granted. The NSF and Becta reports argue that better data collection can allow for better individual tracking of learning progress and better information for parents and educators.
Technology has an important role to play in ensuring greater participation by parents in children’s learning. It enables schools and colleges to report to parents on their child’s progress as it develops, rather than at a few fixed points in the year, and offers parents new opportunities to engage in dialogue with providers. Patterns in achievement, attendance and behaviour become easier to identify, so teachers and other professionals can make more timely and effective interventions (Becta report, p13).

Finally, the Cambodian plan is heavily focused on infrastructure, which seems logical given the low penetration rate of ICT equipment and internet connectivity. There is very little mention of mobile technologies however, although these may offer more chances to get large numbers of students connected to the Web than desktops in computer classes.  The NSF and Horizon plans put heavy emphasis on these mobile devices, including not only smartphones, but also tablets and e-readers.

References:

Becta (2008) Harnessing Technology: Next Generation Learning 2008–14, Becta report on behalf of the Department for Children, Schools and Families and the Department for Innovation, Universities and Skill, Coventry, Becta; 

National Science Foundation (NSF) (2008) Fostering Learning in the Networked World: The Cyberlearning Opportunity and Challenge, A 21st century agenda for the National Science Foundation, report of the NSF Task Force on Cyberlearning, Arlington, VA, National Science Foundation

New Media Consortium (NMC) (2009) Horizon Report, report from the New Media Consortium and the Educause Learning Initiative, Austin, TX, The New Media Consortium

Ministry of Education, Youth and Sports (MoEYS), 2010, MASTER PLAN FOR INFORMATION AND COMMUNICATION TECHNOLOGY IN EDUCATION 2009-2013 , available online at http://www.moeys.gov.kh/ict_master_plan_2009-2013.php