Assessment, Evaluation & Learning Analytics

Overview

In this chapter, the basics of assessment and evaluation are reviewed from the perspective of technology-enabled and online learning, with a brief overview of the field of Learning Analytics, and how it is related to Assessment and Evaluation. This chapter will form a foundation for the rest of this book, and as subsequent chapters will detail how educational data and its role and its emerging role can inform these practices.

Why is this important?

When we think about how the development of learning experiences occur, much of the work we do is based upon our own experience and may not, all of time, be based on objective evidence. Having an awareness of the difference between assessment and evaluation, including their purpose and how we can best go about these practices, can help us to ensure that everything we do, from teaching to implementing new technologies have built in mechanisms to ensure they’re effective.

Key Readings

Biggs, J. (2001). The reflective institution: Assuring and enhancing the quality of teaching and learning. Higher education, 41(3), 221-238.

Garira, E. (2020). A proposed unified conceptual framework for quality of education in schools. Sage Open, 10(1), 2158244019899445.

Kevin Ashford-Rowe, Janice Herrington & Christine Brown (2014) Establishing the critical elements that determine authentic assessment, Assessment & Evaluation in Higher Education, (39)2, 205-222, DOI: 10.1080/02602938.2013.819566

On Terminology

At the outset, it must be said that the english language is a flexible thing and allows for different interpretations of the same word in different situations, amongst different populations of people and to convey different intentions. With that being said, the terms ‘Assessment’ and ‘Evaluation’ may be completely different or they may be the exact same thing in different educational settings. They may be frame and applied differently in a report vs. an educational manual and they may be used interchangeably in other situations.

For the purposes of this chapter, Assessment is used to describe the observation and measurement of whether learning has occurred within a student and to what level, whereas Evaluation will refer to the process of critically looking at something (e.g., new technology, new process, existing process) to make sure it is effective, so that the findings of the evaluation can be used to inform subsequent changes for improvement.

Assessment Frameworks

For a review of basic assessment approaches check out the ‘Assessment’ section of How do we Facilitate Learning with Technology?

---

After reviewing the above overview of Assessment, the first thing to be know is how assessment is framed in different educational contexts.

Outcomes-based Assessment

This usually refers to how students are assessed based upon outcomes (or objectives or goals, depending on the terminology used). Outcomes are simply what you’d like your students to be able to know or do at the end of any learning sequence or lesson. As these outcomes can be framed in many different ways they can either focus on specific knowledge acquired, the ability to think and write critically about new concepts they’ve learned or even demonstrate their ability to complete tasks and solve problems. Due to the inherent variation in how outcomes are written, assessment in this form can range from very abstract (e.g., write an essay that primarily focuses on building of knowledge through theoretical understanding) to very concrete (e.g., solving a real world problem by developing and building a product). In this form of assessment there are usually varying to degrees to which a learner can be ‘successful’, which is why we see terms in rubrics like ’emerging’, ‘developing’ and ‘proficient’.

In schools / K-12 systems, state and national levels of departments of education usually establish these and include outlines of what should be taught, how it should be taught and how it should be assessed.

• The NSW Dept of education has principles of assessment strategy, based on how students should be assessed, however this is framed through the proficiencies that teachers need to demonstrate in the form of Graduate Standards published by the NSW Education Standards Authority (NESA). They even consolidated much of this information for those teaching online during the COVID-19 pandemic.
• Other states and country’s educational systems will have their own set of standards that may or may not align with national standards.

In higher and tertiary education settings, this is the same.

• The Australian Tertiary Education and Quality Standards Agency (TESQA) has a Standards Framework that establishes all practices in how the academic side of these institutions should be run, including assessment

Competency-based assessment,

This form of assessment is very much based on performance of a skills and an almost binary means of measuring whether these skills can be demonstrated or not. This ensures that key skills required by a job whether it be in the trades or in the military are achieved and that the learners can actually apply and DO something to the level of proficiency required in the workforce.

For the Military, which is usually more Competency-based, phrases such as ‘performance assessment’, ensuring that soldiers can demonstrate the performance of skills, knowledges as well as attitudes that link what they’re learning in the classroom or online to what they’ll be doing in their deployments (see Assessment section of Full Time Soldier Training for more). In this context, emphasis is placed on authentic assessment, meaning what soldiers are trained on and assessed on should definitely be linked to real world situations – how to accomplish this is sometimes difficult to achieve, however.

For polytechnic and trades training, the same is true, though a mix of Comptetency and Outcomes-based approaches may be applied. See the TAFE NSW Assessment procedures document for more.

When it comes to corporate training and professional learning, this is very much based on the aims of the company and what it requires of its employees – it could be competency-based or it could be outcomes-based.

Common Ground

The contrast between different approaches and goals of learning and assessment is sometimes discussed through the lens of “Is there a difference between ‘education’ and ‘training’?”, with a common joke being “I would rather my doctor be trained to perform surgery, instead of being educated on it” – again, see the note about English above 😉

What is clear is that assessment, whether for online or classroom settings, may not differ greatly in their approaches. Different requirements and mandates for how assessment should occur, how it should measure learning and how it should be reported will be the same for online and face-to-face environments for the most part. Since the beginning of online learning, however Exam procedures have differed greatly for online settings, with he prevalent assumption that students will cheat if completing an exam online still existing today.

For more on Writing effective Learning Objectives, see What is the process for designing an experience?

Online v. Face-to-Face

One of the common myths about fully online learning is that you cannot conduct assessment effectively compared to face-to-face environments. Given the advances in technology over the last 20 years, one could argue that this is no longer the case – that there are now actually more varied opportunities for students to demonstrate their learning than was possible in the classroom. Additionally, with the advent of technologies such as smartphones that allow for photo, video and location sharing, there are so many more opportunities to demonstrate learning in different ways. Check out the Assessment section of the page linked above for more

Alternative Approaches to Assessment

In the video below, some Canadian researchers give a fantastic summary of all the ways we can think about assessment, marking (grading) as well as some alternatives to how we might do this that is less punitive and more flexible for our learners.

Evaluation Frameworks

While assessment is about measuring the individual learner’s ability to master certain learning objectives, evaluation is usually at a much broader scale, and is all about making sure educational practices, ranging from how a degree or training program might meet professional standards, or how the classes within a degree effectively prepare students for the workforce. In a nutshell, evaluation is all about making sure what we’re doing works the way we think it should, usually based on an established standard. Many times this is related to something called Quality Assurance (QA), so you may find lots of evaluation techniques and approaches that overlap.

“Evaluation is the collection of, analysis and interpretation of information about any aspect of a programme of education or training as part of a recognised process of judging its effectiveness, its efficiency and any other outcomes it may have.” – Mary Thorpe in Ellington, Percival, and Race (1993)

Just like Assessment, depending on the context where you work or study, different organisations may have different frameworks for evaluation as well (e.g., The NSW Dept of Education and Communities 2014 Evaluation Framework)

As evaluation has a much broader application, what is being evaluated and how it can be evaluated varies greatly. A quick Google Scholar search on Evaluation on shows the variety of different ways researchers have explore this topic. Commons means of evaluation in education include, but are not limited to:

• Staff performance evaluations
• Student evaluations of their learning experiences
• Evaluation of teaching effectiveness
• Evaluation of the form of assessment in its ability to demonstrate learning outcomes for students
• Evaluation of graduate job placements
• Evaluation of programs and degrees and how they meet government or professional standards (AKA Accreditation)
• Pilot testing and evaluation of instructional methods, ways of presenting learning materials, etc.
• Evaluation of Learning Design Processes

For more on evaluating learning design and designed learning experiences see How do we ensure effectiveness?

As evaluation is all about making sure something is effective, most things within any educational setting can be evaluated.

One example of a framework is Biggs’ model for evaluating and improving teaching and learning (2001), which describes 3 categories that can help those doing evaluations to think about how to frame their work.

1. Quality Model (QM): What theory of learning does the target of the evaluation align to?
2. Quality Enhancement (QE). The organisation should have intentionally designed and built-in means by which they can gather information to inform enhancement.
3. Quality Feasibility (QF). Are changes feasible? In other words are there are barriers to actually implementing the changes that the QE area tells us we must implement? (e.g., technology, policy, etc.)

Analytics

You may have heard the term analytics before, there are many types of analytics. Analytics simply refers to the practice of using computers to collect and analyse data – sets of information that tell us about a particular phenomena. You may have heard of Google Analytics – this product allows owners of websites to collect and analyse information about website usage, including who visits, where they are, what time they visit and many other data points. From this information, you can start to see trends in the data and make decisions about how to improve your website, and this is the primary reason to use analytics – to inform change.

Wherever we go online, our interactions with many of the websites, web tools and apps we use is recorded by the companies and owners of those systems. Usually this is not done for nefarious purposes, but simply as a way for these parties to evaluate how their products are being used. Using this information can then tell a story about individuals, as well as groups of people and their behaviours, including any trends or patterns that emerge – this is what analytics is about – looking through lots of data to identify patterns and make judgements about a particular phenomena.

The two types of analytics you need to know about in this chapter are Business Analytics and Learning Analytics.

Business Analytics

Business Analytics or sometimes referred to as Business Intelligence, or in educational institutions, it’s even called Institutional Research refers to the collection, analysis and reporting of data related to the business operations of a company or educational organisation. While this may relate to learning in some instances, in educational settings it usually has to more to do with enrolment reporting, graduation rates, budgets, and KPIs (Key Performance Indicators) that define how well the company / institution is doing at meeting pre-established goals.

Learning Analytics

Learning Analytics, on the other hand is defined as “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs” – defined by SoLAR (The Society of Learning Analytics Research.

There may be some confusion in educational settings and some mislabelling of certain practices or information as Learning Analytics, when it actually is not, so it’s important to make this distinction early on. The field of Learning Analytics is quite new (15-ish years old), so you may meet people who have different viewpoints on what constitutes Learning Analytics and what doesn’t.

Educational Data

Educational Data simply refers to any information that can be used in an educational system. In terms of learning technologies and online learning contexts, educational data usually has to do with the data that these systems collect, including student engagement, time-on-task, devices used, location and other data points. Outside of this, grades / marks, enrolment history, student demographics and classroom locations can all be examples of educational data that can tell us something about how an educational system works.

Generally, access to this sort of information is limited based on a few factors – our roles, the technology’s ability to collect and report on data, and existing policies and practices related to its use. Data Governance is the term used to describe the processes and practices of managing data in any context, including where it’s stored, who should have access to it and other ‘rules’. Your organisation may have a Data Governance team or committee, along with a Data Governance policy. These people and documents typically outline the details of who has access to what, what exists, and how the data should be managed from a technical perspective.

Key Take-Aways

• Beyond the design of assessments, there are often frameworks and standards for assessment that we need to be aware of and create our assignment and assessments around.
• Evaluation is the process of exploring whether something does what it is designed to, almost akin to research.
• Learning Analytics and the use of educational data is an emerging area of research and practice, and can help us understand and respond to the needs of our learners.

Revisit Guiding Questions

If someone asked you ‘What’s the difference between Assessment and Evaluation?’ how would you answer this question. Try answering it in your head, as if responding to a colleague.

What about defining Learning Analytics? Try to explain it to someone who’s never heard of it.

References

Ashford-Rowe, K. (2020) Assessment within the Military Environment. Retrieved from https://cove.army.gov.au/article/assessment-within-the-military-environment on Aug 18, 2021.

Ellington, H, Percival, F, and Race, P (1993) “Handbook of Educational Technology”, Kogan Page, London

Li X., Shi X., He H., Zhao H., Men N., Ye C. (2018) The Study of the Performance Assessment in Military Training. In: Long S., Dhillon B. (eds) Man–Machine–Environment System Engineering. MMESE 2017. Lecture Notes in Electrical Engineering, vol 456. Springer, Singapore. https://doi.org/10.1007/978-981-10-6232-2_80