How do we know if it works?

Overview

In this chapter we’ll be taking a critical look at innovative practices in pedagogy and learning technology and trying to sort out what actually works and what doesn’t.

Why is this important?

It’s important that whatever choices we make in terms of designing learning experiences, and then implementing those choices (teaching), that we are basing our decisions in evidence and what actually works for learners.

Key Readings

There are no key readings for this chapter, but you are encouraged to check out the studies below and to dig deeper in the research surrounding the previous two chapters on innovative pedagogies and technologies, this way you can get a deeper understanding of what these research findings mean for teaching and learning, and the implications for real world contexts.

Oliver, M. (2000). An introduction to the evaluation of learning technology. Journal of Educational Technology & Society, 3(4), 20-30.

Webster, M. D. (2017). Philosophy of technology assumptions in educational technology leadership. Journal of Educational Technology & Society, 20(1), 25-36.

Wilson, G., & Randall, M. (2012). The implementation and evaluation of a new learning space: A pilot study. Research in Learning Technology, 20(2), n2.

Learning Science and the implications of research

A field in educational research, one that is related to psychology is called Learning Science. This area of research is devoted to the science of learning, that is the inquiry into learning processes and practices to explore how learning happens and what makes learning effective. Studies in learning science are primarily based on measurable outcomes such as test scores, so there is evidence that a new technology or a new pedagogical approach actually worked to improve learning.

The challenge with any research though, is the ‘generalisability’ of this research result. What this means is that if we have a study that involved 30 female students in rural China who achieve a high score by using a new practice, we cannot generalise this to a larger population – in other words we can’t confidently say that this would also work for a group of male students in urban Australia. This applies to many aspects of the people who were studied (participants) and means that age, location, prior understanding of a topic, the circumstances they were in when they participated in the experiments and other factors can mean it’s hard to assume these findings will work in other circumstances. For this reason, it’s very common to conduct ‘replication studies’ in science – to do the experiment again, either exactly the same or with a very slight change – to ensure the findings are accurate and can apply to other populations of people.

When it comes to research on educational technology and surrounding pedagogies, this is something we need to be mindful of, but don’t get discouraged, it just means we might need to take these findings with a grain of salt, and when we think of the implications for our teaching and learning practice, we must allow space in our heads to consider what may or may not work when we implement different things.

Skepticism is good

Whenever a new technology comes along, there is always the potential for it to improve our lives in amazing ways, but there is also the potential for it to have other effects, so being critical is a good thing.

With the advent of many new tools at our disposal like Learning Analytics, Machine Learning and AI, we can marvel at what new affordances these tools give us both to improve teaching and learning and in the case of VR and AR to engage our learners. It’s very important, however, to understand as much as we can about a pedagogy and a technology so that we are implementing it both ethically and responsibly.

Never use technology for technology’s sake.

When someone asks you why you’re using a certain tool, you should always have a good pedagogical reason.

While our intention may be in the best interest of our learners, it is nevertheless important to consider the effects of using technology, especially with those who may be less versed with that technology. For example, if we are using learning analytics tools without informing students of their value, the collection and use of that data may be misinterpreted, leading to potential mistrust of the instructor. The same is true for VR – with many VR experiences, disclaimers for motion sickness, dizziness and other physiological effects should be provided due to these potential effects the experience may evoke.

If we think back to the tenets of Open Educational practices, with the goal of ensuring equal and equitable access to technologies and learning experiences, then consideration of the pros as well as the cons of specific technologies and practices is something to keep in mind. Again, we must consider the Spiderman rule and listen to Aunt May / Uncle Ben: “With great power, comes great responsibility”

Taking this into consideration, any implementation of a learning technology as well as our justification for using that technology should include both the opportunities and considerations that we might need to take into account, and this means doing some research beyond our desire to justify doing something ‘cool’.

Case Studies in Learning Technologies

A quick google scholar search of ‘evaluation learning technology‘ or ‘case study learning technology‘ or any derived terms will give us a good place to start when critiquing innovative pedagogies and technologies. We can even be specific when looking for case studies, using the specific technology type or pedagogical approach we wish to explore (e.g. ‘case study virtual reality’). By looking at examples of how others have evaluated their practice we can learn more about how to do it in our own teaching and learning environments.

Case studies are types of research studies that focus on a particular set of circumstances and cover the details of those circumstances so we have a window both into what actually happened and how the researchers justified their choices and approaches to trying something out. When developing a pilot test of new technologies or approaches, using case studies and other research to explain our choices and what we need to consider shows our leadership that we’ve thought this thing through, and our plans are more likely to get support.

Evaluation of our Practice

As we engage in the work of designing learning experiences and then implementing there are a number of ways we can evaluate what we’ve done. Thinking about what you’ve read so far in the research, what ways do you already know of that would be useful to you?

There are a number of different ways we can collect information about a learning experience from our students. Check out the video below – while it does cover non-educational fields, the same rules apply when we’re exploring educational settings.

So some common ways to find out if something worked are outlined below:

Another thing we can think about is the technology itself – did it work as intended? How reliable was it? As a teacher, was it easy to use? Easy for your train the students how to use?

While you are not going to be a fully fledged researcher (unless you eventually want to finish a PhD or Masters with research), the idea here is that you know a little bit about these processes so you can learn about your own practice and the environment in which it occurs.

If you think a new learning technology or pedagogical approach is valuable in your workplace or learning context, then there needs to be a way to 1) prove that it’s good and 2) learn how to improve it from the start. By engaging in this sort of mini-research, you can gain an understanding of how to self-evaluate and improve your students’ learning experiences.

Key Take-Aways

• We must always be able to answer the question of ‘Why’ we are using innovative pedagogies and practices
• It’s important to be able to understand the real benefit that using innovative practices brings to enhance learning
• By knowing some basics of research practices, we can begin to reflect on our own work to determine if something is working for our own students or not, and work to improve where possible.

Revisit Guiding Questions

Consider the answer to the question that this chapter poses. Think of a lesson you have planned or designed that uses learning technology to enhance it. In your head, or in explaining the lesson to someone else, start by saying “I know this approach will benefit my students because…” and continue on from there, and no, “because I thought the technology is cool” is not a good answer 👎

Further Reading

Phillips, R., Kennedy, G., & McNaught, C. (2012). The role of theory in learning technology evaluation research. Australasian Journal of Educational Technology, 28(7).

Young, S., & Nichols, H. (2017). A reflexive evaluation of technology-enhanced learning. Research in Learning Technology: the Journal of the Association for Learning Technology/ALT-J.