Rebecca Frankum - NZ Vocational Education and Training Research Forum (NZVETRF) master class - on school leaving certificates

 Attended a 'masterclass' or fireside chat organised by the NZVETRF presented by Rebecca Frankum, Transitions in Upper Secondary School Education, OECD Secretariat. A recent report 'the theory and practice of upper secondary certification' authored by Hannah Kitchen and Rebecca, forms the basis for the discussion. 

Notes taken from the presentation/discussion:

Josh Williams from Skills facilitates the session. He began with an overview of NZVETR, the background to this session (the change of NZ school leaving qualifications) and introduce Rebecca - who presents from Paris.

The presentation worked through the report, with an emphasis on some of the implications and applications within the NZ context.

- Began with the importance of upper secondary certficates as it accreditates them towards the next step in their lifes. 

- Study looked into how upper secondary certificates were structured. did they incorporate and assess a broad range of skills? reflect the diverse skills and strengths of all students? enable progression to the students' next steps.

- Analysis of upper secondary certicates (71 certificates, 38 systems) on nature of assessments, who marked, what did they include, was there activity within unseen questions/tasks? allow for natural occuring evidence.

- 3 categories - certificates that include external exams, but no internal assessment; certificates include internal but no external exams; and certificates that include both (NZ NCEA is an example).

- exams still have an important role as they assess complex skills like analysis, evaluation and creating and include a range of problems, sources and multimedia materials.

- Higher education entrance exams - same exam papers, consistent marking, consistent standard of difficult, same exam conditons. Useful as 'gate keeper' - certify knoweldge, understanding and skills and facilitate selection.

- Wider range of exam formats support validity but can be hard to design and some skills (practical, social. emotional, higher order ectc) hard to assess.

- Balancing assessment approaches (fairness, credibility, relevant, manageable) supports robustness of certificates.

- Most systems (usually 3 - 2 vocational and 1 general) have separate certificates for vocational education - NZ is an outlier with one to fit everything.

- Models for designing upper secondary education include personalised systems (significant choice) (Australia, NZ, US of A), intermediate (Estonia, Poland, Denmark), structured (limited choice) (Austria, Germany, Netherlands, Switzerland).

- how vocational certificates are assessed shapes pathway opportunities. All external, all internal or both. Vocational certificates are generally all internal whereas general programmes are mostly all external. General certificates usually set by teachers and assessed externally. Whereas VET set by teachers and 'boards' and may be assessed both externally and internally.

- Achieving 'parity' across vocational and general certificates is not about just assessing the same skills and providing the same pathways, Vocational upper secondary certificates need to balange authentic assessment of practical skills and also providing pathways to higher education. Efforts fo create equitable pathways can result in close alignment to general certificates, masking the unique value and skills of vocational programmes.

- To meet the parity challenge, need to involve a range of stakeholders and assessment formats. Assess occupational skills through practical exams and workplace assessments. Involve professionals and employers through local juries or assessment boards to ensure vocational certificates represent the skills and knowledge of industry. 

Q & A followed. 

Balance between vocational and general is a challenge, fit for purpose and enable learners to be recognised for what they have developed through education. 

Modern certificates tend to recognise a wider range of skills, knowledge and attributes, beyond just the academic. Curriculum reforms tend to focus on how to accredit across the wide range. 

In NZ, autonomy from students and teachers is high. This is not the case internationally. Innovations and ways to meet the needs of learners by schools, means the consistency across NCEA is difficult to manage. Retaining this flexibility is still important. 

There is room for both external (exams) and internal assessments. The purpose of the assessments, and what is to be assessed should be the key to making the decision. 

Recognised the complexities of VET learning :) 

Good update on what is happening in the school completion certificates. There is no right/wrong approach, many challenges, and working through these require clear understanding of the purpose of these qualifications. 

- 

- 

AI-generated assessments for vocational education and training - webinar

 Here are notes from the webinar on the ConCove Tūhura project AI-generated assessments for VET, 

The report provides the literature scan and details of the process undertaken to identify appropriate AI to undertake the task, and the processes to ensure that the AI- generated assessments would meet moderation requirements (quality assurance) for use for assessing VET standards. 

The work was undertaken by Stuart Martin from George Angus Consulting and Karl Hartley from Epic Learning. Both present in the webinar which begins with an introduction by Katherine Hall (CE for ConCoVE Tūhura) and by Eve Price (project manager at ConCoVE).

In Katherine's introduction, the rationale for the project was shared along with some of the journey taken by the project to break new ground.

Eve Price provided the background of the project. Most projects focus on integrating AI into ako or the prevention of AI for assessment. This project wanted to help support the time consuming 'back room' processes including resource and assessment development.

Karl ran through the approaches to the product. The evaluation/review processes could not really keep up with the speed at which assessments can be developed when it is supported by AI. 

Stuart shared reflections on how the process evolved and the various processes put in place, were reflected on and were then reintroduced into the AI-generation project. Explained how various quality pointers were met to ensure the efficacy of the process.

Eve detailed the need to be specific with what needed to be achieved - assessment, feedback, etc. Selection the correct AI is also important. Prompts are detailed in the project report. Important to evaluate at each step.

The bigger picture with micro-credentials, skills standards and AI-generated assessments all add innovations to the VET ecosystem. Understanding the policies and processes used by WDCs and NZQA need to always be part of the process, so that various quality points are met.

Stuart summarised some of the challenges and how the project worked through these. 

Karl talked on the importance of people in the process when AI is generating the assessments. Firstly, important to understand some of the mechanics of AI - what is under the hood. Secondly, quality assurance must be focused on the concepts, not so much the grammar/spelling etc. Thirdly, need to make sure assessment purpose is clear. 

Next, academic integrity and ethics were discussed. Important to ensure that there is understanding the impact of AI on privacy and data sovereignty (including indigenous perspectives). Important to train the AI to understand tieh Aotearoa context. Claude AI was selected due to its stance on human rights, ethics etc. 

Findings included: assessments did not meet moderation but improved the opportunities for inclusiveness and personalisation of learning. Failing moderation added to the learnings from the project. The items involved too many questions, answers being at too long and at too high a level. 

Eve reiterated the need to 'define what good looks like' to the AI, so that human objectives/ perspectives are taken into account. Important to ensure principles of ethics etc are maintained as it is important to 'keep humans at the centre'.

Karl's learning include AI drawing in novel content through its hallucination. The AI included assessor approaches into its assessment and this caused him to consider the learner information that should be included to provide direction. The U S of A standardised approaches to writing assessments, seemed to permeate the assessments produced by AI. This had to be superseded through careful prompting.

Flexibility to allow for personalisation to industry (example safety unit standard customised to a range of work roles/ disciplines); and learners (for ESOL, neurodiverse learners etc.). 

 Q & A followed 

The webinar was recorded. 

Discussions revolved around practicalities, challenges and solutions.

All in, good sharing that adds to everyone's learning about the roles of AI to support teaching and learning, integration of practice/practical and cultural contexts, the need to be aware of the fish hooks' in using AI, how quickly AI is developing to meet user needs, and the need to continually learn to ensure that the understanding of AI / ethics etc. form the foundation for working with AI. 

Assignments in the AI era

 In light of this article from Radio NZ, whereby some universities in Aotearoa are no longer checking assessments using AI tracking platforms, a summary of ways to think about assessments in the AI age is of importance. There has been much discussion on how assessments in higher education need to be evaluated and re-thought, given the infiltration of AI into our work and study. This article in Times Higher Education, distills many of the main discussion points in academia on how AI affects academic writing.

The work undertaken at my institute is focused around holistic / programme wide assessment design, rather than on individual courses. The term 'programmatic assessments' is sometimes used to describe this approach

Some of the other strategies we have used, are summarised in this blog - NavigateAI (Dr. Ryan Baltrip)  In summary, to place greater weighting on recording the evidence of learning, rather than the product of learning. Therefore, portfolios and similar assessments are more useful than one off invigilated exams, or assignments. 

In Aotearoa, Otago Polytechnic's Bruno Balducci, have introduced the concept of AI safe design, a framework for the design of assessments which take into account the influences of AI. These are useful as a way to help educators work through the many pitfalls involved in redesigning assessments that will be authentic and relevant, but will not tempt learners into using AI to complete them.

The other concept we have used to help our teachers work out how to structure assessments in an AI age is the 'two lanes' assessment structure.  Here, lane 1 assessments are used to as assessments OF learning - or summative, higher stakes assessments. Lane two are the assessments FOR learning, taking on formative approaches to inform learners as they progress to the course.

Therefore, it is important to not just assume that current assessments will be appropriate but to undertake a stock take to understand the purposes of each assessment, and to put in place relevant assessments that will meet the purposes of each assessment i.e. evidence that the learner has met learning outcomes.  

Enacting assessment reform in a time of AI - Tertiary and Standards Quality Agency (Australia)

 The Australian Tertiary and Standards Quality Agency (TESQA) has published a report to provide guidance on assessment reform in the age of AI. 

The main approaches are:

- taking a programme wide approach to assessment reform i.e. across the entire degree programme

- assuring learning in every "unit/subject"- i.e. across a course

- implementing a combination of the above - by ensuring assessment mapping is constructively aligned between learning outcomes and assessments. 

The strengths and challenges of each of the above are presented and discussed. 

Guides from Deakin University - Gen AI in work-integrated learning

The Centre for Research in Assessment and Digital Learning (CRADLE) at Deakin University has published a series of guides on Gen AI in work-integrated learning.

The resources are for Higher Education/University staff, students and workplace industry partners. There are four sets of guides:

- for ensuring academic integrity and assessment security with online delivery

- assessing work-integrated learning programmes - a guide to effective assessment design

- inclusive assessments

- reimaging exams - workshop resources.

All in there are good recommendations. Although pitched at the higher education sector, the work-integrated and inclusive assessment sections are easily generalisable to other contexts.

ASCILITE - transforming assessment site

This year's ASCILITE (Australasian Society for Computers in Learning in Tertiary Education) conference is being held in Christchurch in early December. Some of my colleagues will attend.

ASCILITE itself maintains good resources to informvarious pedagogical and technology enhanced learning endeavours. 

Of note is their site on assessments  on 'transforming assessments'., which also archives webinars on the topic and has a collation of past events as well. 

Of interest, is one on 'guiding the use of Gen AI for assessment' head in September. 

Assessment in the age of AI - NZQA symposium - notes

 The New Zealand Qualifications Authority (NZQA) convened a symposium on the implications of AI on assessments. The main event held at Te Papa with online attendance by many. The event is also supported by the Ministry of Education, Universities NZ and the NZ Assessment Institute, NZ Council for Educational research, Post Primary Teachers Union, Tertiary Education Union and Network for Learning. 

Notes from the online /streamed sessions follow:

The day begins with mihi whakatua, and introduction. Lee Kershaw Karaitiana MCs along with offering the karakia and welcomes. 

DrSimon McCallum, Senior Lecturer in Software Engineering, Victoria University of Wellington opens with the first keynote on ‘the dawn of AI’. Begins with his pepeha (Māori introduction). Has been teaching game programming since 1999 and every year, there is something new and the students coming in change each year. Went through what is Generative AI - what are are Large Language Models (LLMs) which is the training of a machine to translate. Much of language relies on our experiences. Explained how word vectors work to help machines understand words and how these contribute to ChatGPT unravelling the nuances of language. Then explained how ChatGPT works to answer the prompts it is given and the importance of prompt engineering. Provided the principles of prompt engineering including how ChatGPT learns as the process of prompting continues. Currently, other AI platforms - ChatGPT 4 etc. has added guardrails and other 'agent-based' systems to try to provide more authentic outputs. Explained the many processing methods used to evaluate what the output will be. 4 is much more advanced and able to provide less stilted outputs, and the scholar plug in generates real citations - cost US$20 a month - so there is an equity issue. AutoGPT (cost $20 per complex problem) uses Python to create a plan with the ability to write code to solve the problem. Warning on privacy issues as AutoGPT able to make a plan with access to all the items in your (Google) account! Provided examples for AI image generation - Dall-E 2, stable fusion, Nvidia AI playground etc. Photo generation is now very blurry, given images can be 'enhanced', sometimes without our knowledge (Samsung phones often provide a better version of a photo you take!).

Note - AI understands language but not actual words. Assessments often draw on learners use of language as a way to assess critical thinking etc. However, now AI able to do similar, making it a challenge to how we assess students. Observation of groups of low capability students have high use of AI but then do not learn :( High ability students learning AI progress quickly though. Improving understanding is the key, not just using it to replace the work learners have to do. Posits that presently, ChatGPT able to complete assessments at Level 3 but Bard and Bing able to meet Level 7 to 9 in some areas. Argues that all work is now group work. Need to assess learners' contribution to the group :) 

Challenged us to think about how we prepare learners? AI can be used to 'augment' so the combined AI and human effort requires assessment. Suggests assessments as 'motivational' which are agentic, intrinsic, relevant and covert - works with small groups of highly motivated learner. Authentic assessments must connect task/time to assess complex reasoning/thought. How do we roll out a new approach to assessment, especially when the future in the world of AI is still unknown. Encouragement to use AI as a tutor, supporting personalised learning 24/7 able to translate concepts to different levels, attain customised explanations and form chains of thought. If AI now a co-author, then author statements require being clear as to who had done the work and justification of not using AI now required!! We need to be the 'rider' of AI. Suggests flipped exams (it is the prompts, not the answers), AI to triage the work and rethink of what authentic assessments will look like. Finished with some thoughts on what may happen into the future (pessimistic). Shifting from clever words to caring people, need to be aware of the apathy epidemic (people who no longer have to think!). 

 The keynote is followed by a short presentation by Dr. George Slim, consultant advisor to the Prime Minister's Chief Science Advisor who speaks on 'a science policy review'. Provided a Aotearoa science perspective on how AI has changed (increased/accelerated) research - biology (DNA, viruses etc etc). Panel being assembled to bring together a report as to how to address the many challenges presented. Resources also being provided to archive contemporary thinking as the technology moves on. Government is just beginning work on implications and response. Do we ban it (Italy), leave it to the market (US of A). NZ Privacy Commission has begun to work on guidelines and resources. Important understand and manage AI. 

 

After morning tea, online presentation  from Dr Lenka Ucnik, Assistant Director Higher Education Integrity Unit, Tertiary EducationQuality and Standards Agency (Australia) provides the Australian context. Provided context and background of TEQAS - does not regulate on vocational education though but for higher education. The key messaging on AI is that it is here to stay. Can be an assistive tool for students (especially for those with disabilities), research and teaching. The main premise is to implement risk analysis management to maintain academic integrity. AI affects academic integrity and there are discipline specific processes. Important to ensure learners/students attain the skills to work with AI (see learner guide). Encouraged participants to think beyond the immediate and evaluate /plan / strategise towards the future. There are opportunites but also important to mitigate risks! and the ongoing work required to ensure the integrity of education. What is the most important objective of education and how can the affordances from AI contribute. 

 

Professor Cath Ellis from the University of New South Wales then presents on ‘the link between cheating and assessment’. Shared an observation from a student, generating a presentation using ChatGPT and attaining a good mark. Currently, learning is assessed with an artefact/performance - a proxy. Learning is embodied :) Assessments pitched at being 'just good enough'. At the moment, ChatGPT moved from producing work which as from just enough to good. What needs to be done and who does it now?  'Cheating is contextual and socially constructed' - example of ebook for commuting (good) but in the Tour D France (bad). There is a plethora of sites which allow support essay writing. 

We still need to ensure the authenticity of assessments, whose work is it when AI is available. We need to focus on finding evidence that learning has occurred, not why cheating has occurred. Do we need to assess some things many times?? Education's role is really about making sure our learners are able to weed out 'hallucinations' generated by AI. Conceptual frameworks on academic integrity and assessment security needs to be discussed. We need to champion those learners who are able to work and willing; and not criminalise students who are unable or unwilling. Bulk of energy needs to be in championing, not so much in criminalising. Encourage to focus on metacognitive rather than with content. Call for placing importance on critical AI studies. Check critical AI. 

 

Following on is ProfessorMargaret Bearman, Centre for Research in Assessment and Digital Learning, Deakin University (Australia) who presents on ‘generative AI – the issues right here, right now’. Presented on the short term implications and moved on to future. Defined assessments as both graded and non-graded, not necessarily marked by teachers. Assessments should not only assure learning but also promote learning. Educational institutions response to AI can be ignore, ban, invigilate, embrace, design around or rethink. Uncertainties around Ai include legal, ethical, and access. 

 Design around is probably the best option for the moment. Shared possibilities through ways of making knowledge requirements more specific - knowing your students, specifically requiring the assessment task to reference something that happened in class, designing more authentic assessments. Design to cheat proof assessments. Invigilation is costly, stressful, tests capabilities unrelated to task, narrow band capabilities and cheating still goes on. Rethinking invigilation may be one option - move towards oral, assessment of learning outcomes across tasks, invigilate only the common sorts of knowledge/skills. Need to rethink the curriculum to account for AI. 

Then Associate Professor Jason Stephens from the University of Auckland on 'achieving academic integrity in academia: the aspirations and its obstacles'. Covered what does it mean to achieve with integrity and why it is important, the obstacles to achieving integrity. Being honest is not (always) easy! students need help (somethings a lot) to achieve with integrity. Educators are obligated to design environments that mitigate dishonesty. Defined achievement of integrity as being hones and having strong moral principles and the state of being whole and not divided. Shared model of moral functioning in academia. Survey across 7 institutions in 2022 (before ChatGPT) reveals 15.1% of students use AI. Obstacles to achieving integrity include thinking as being costly, modern society moves fast and has high expectations, a culture of cheating is sometimes seen to be supported, through contagion effects and 'the power of the situation'. Harmful for well-being when students afforded opportunities for cheat, so important to maintain academic integrity. 

A local school based perspective is presented by Claire Amos, Principal | Tumuaki, Albany Senior High School and Kit Willett from Selwyn College. Claire briefly went through context of the school which has a well-established, well-embedded innovative curriculum model - tutorial, specialist subjects and impact projects. Approach to AI is to embrace its potential, rethink assessments. Use AI to reduce workload, support UDL and support learner agency and self-direction. Working through addressing ethical issues, teaching critical thinking and addressing plagiarisms. Shared examples of how teachers used AI - maths to generate practice tasks, photography to look for connotation and denotations in images, create quick worksheets etc. Also shared how students use AI - support design learning approaches and spending time to discuss how to use AI for good and not for cheating but as a coach :) Shared concerns of increase in more additions into a busy curriculum, the compounding of digital equity, and the need to support students to use AI in a critical manner. Summarised reflections - for example, what happens when we do not assess/rank/grade students?

Kit shared that there has been more plagiarism this semester than across the last few years. Students were briefed about the consequences but many still did not take the advise. Kit works in a school with more traditional approaches including using invigilated assessments. Shared challenges a teacher has to undertake to meet NZQA requirements. A more traditional approach! Shared how teachers could use AI to help lower their workload as well. 

 

Panels and forum occur after lunch.

 The first with perspectives on AI, convened by A. P. Jason Stephens,student association representative (four high school ākonga and two university ākonga), Claire Amos and Kit Willett. All acknowledged knowledge of AI, usually comes up more when assessments are handed out. Did not report on conversations with teachers as to how AI should be/or not used. High school students are cognisant of AI capabilities and will use it as a resource but know of others who use it to plagiarise. Student association representatives from higher education wanted better utilisation of AI to support equity in education and fairness with regards to invigilated assessments. Image-based disciplines need to really work on how to assess when there is so much available. AI use in creativity needs to be clarified - is AI augmenting or doing all the work? 

Banning AI will only make 'forbidden fruit' more attractive. People who want to cheat, will do it. Inequities are acerbated as students able to afford AI still advantaged. 

Good points brought up by all the ākonga. They are pragmatic. AI can support learning, however assessments are still a grey area. Interesting discussion ensued around what is learning, the role of technology in supporting learning, and assessment philosophies. Call to look at updating an archaic education and assessment system to reflect the technology affordances and what is the present and future social /work / industry environments. 

The second is the AI forum with the Aotearoa NZ perspective with Gabriela Mazorra de Cos as convenor and Professor Michael Witbrock and Dr. Karatiana Taiuru. Micheal overviewed 'Where are we going with AI?' As a country, we respond well to new developments. Summarised history of AI from 1940s and the current rapid improvement in its usability. Ran through pluses and minuses of AI. automate everything?? to free humans from mundane work. AI may be in the from of a organisational/consolidated form rather than as an individual form. Integration of natural and artificial intelligences with existing and new kinds of organisational intelligences need to be considered. Education will be about how to help learners become the best humans :) 

Karatiana covered 'how do we embed Kaupapa Māori ethics and culture from the outset?' Spoke about the opportunities to turn back effects of colonisation. Digital technologies and now AI confer affordances to support the revitalisation and increase in Te Reo and Mātauranga Māori. Argues all datum has Mātauranga Māori threaded through it, it is a taonga and must be used to empower Māori. AI is no exception. Still much to be done to ensure the integrity and ethics of how Mātauranga Māori is used. Important to plan towards the future to ensure ākonga are educated about Māori ethics. Also proposed the deployment of AI as personal learning assistants/tutors to assist with the shortage of Māori experts, but must be developed in association with Māori. 

Q & A ensued covering future possibilities as we are a small, generally well-educated country, to leverage off AI for the betterment of all. 

Then a provider response panel Associate Professor Jenny Poskitt from Massey University chaired by with Dr. Mark Nichols from the Open Polytechnic / Te Pūkenga, Kit Willett, Dr. Kevin Shedlock Victoria University and Sue Townsend from Le Cordon Bleu,  the Private Training representative. Mark posited that education is a way to treat ignorance and AI may enhance understanding. Both of these must be addressed with assessment. Four techniques, video practice, videoed randomised questions through interactive oral assessments, viva voce and then use of AI tutors. Video is now more commonplace and allow for interpersonal assessments. Interactive oral assessments are useful as one approach.

 Kevin argued that 'grey' aspects of life, where there are no 'correct' answers is something Māori find normal. The head of the fish can set the direction, but the tail and the body must also follow. Therefore, important to be collaborative when working with AI. 

Sue ran through context of Le Cordon Bleu. Impact of AI seems to be similar to schools and universities'. Three main areas, AI will impact on the types of work available; change inevitable for assessment practices and support facilitators to shift; flexibility and equity for learners have access. 

Kit(secondary school context)  reiterated that tamariki need skills going into the future. Spoke on personal growth, curiosity, intentionality and the challenges of assessing these. Rethinking assessments is a key - reducing assessments and ensuring they are more focused. 

Q & A followed. 

The workshop closes with a panel on ‘reflections’ with AP Poskitt and Dr. Grant Kinkum, CE of NZQA. Jenny summed up the day's discussions with the rise of AI and its many implications. Education which engages is truly a great experience. Rethinking, redesigning are required to leverage off AI to address equity, inclusion. Human being requires reciprocity, empathy and relationships. Ethical challenges are posed by Ai and to move forward, dialogue is required to create new ways of doing. 

Grant thanked presenters and participants and encouraged the conversations to continue. A whole of education required to reap the advantages of AI and meet the challenges. Collaborative work between ākonga and kaiako and the system at large required. Summarised the important themes across the day. Learning content, skills etc. less important than ensuring our ākonga attain the cognitive, evaluative and critical thinking to be agile/flexible as AI continues to evolve. The purpose of education is just as important as assessment design. Encouraged ongoing work as we move through into the future. 

Lee closes the symposium with karakia. 

ChatGPT and assessments

 One of the main challenges facing educators when digital tools can be used for a host of writing tasks, is that of assessments. How can the integrity of assessments be safeguarded when students are able to use nefarious means to complete their assessments.

Over the last two months, there have been a large number of articles discussing the issue. 

- FENews proposes the use of a variety of question types, effective proctoring systems and different testing types e.g. using oral presentations or practice-based assessments instead of written exams.

- The conversation advices the need to rethink assessments. In particular, to ensure they are authentic i.e. connected to personal context or the course specific material. Project-based, group and inquiry/problem based learning are recommended.

- workhe.com recommends that educators trust their students and to take on the possibility of discussing with students, the pros and cons of using AI to augment, or scaffold their work.

- Pulse in the context of teacher education reminds us that learning is a process, not a product or artefact. Hence authentic assessments are the way to go.

- uts provides good ideas on how students and teachers can draw on AI to support their work. Good ideas include asking students to use ChatGPT to generate a response to the assessment and then set up the criteria to critique the response and provide feedback to improve on it. 

In all, it is important to include context into assignments and to ensure students take ownership and responsibility for their learning. AI in the form of Chatbots able to provide responses which mimic those of humans, are here to stay. They will also improve rapidly as they learn from the feedback to responses provide by users. Therefore, it is important to ensure educators understand the implications and work through the consequences. The opportunity is now provisioned to improve learning and assessment processes, so that assessments are for learning and not of learning. 

Assessment in the age of AI - open access journal article

 An interesting and timely article published Computers and Education: Artificial Intelligence. A relatively new journal with articles across the year compiled into an annual volume.

This article in volume 3 (2002) discusses the challenges of assessment in the age of AI and proposes some ways to leverage assessment practices which draw on AI approaches. The article critiques current assessment practices. The ways AI can be useful include automating assessment construction, AI assisted peer assessments, and using AI generated text to conduct 'writing analytics'. The challenges and caveats for introducing and using AI to support assessments are also presented with a focus on how AI may disfranchise educators from assessment processes.

A good reference article to use for supporting discussions on the impact of AI on assessments.

Principles of assessment and aromatawai - draft for consultation from nzqa

 NZQA has released draft principles for assessment and aromatawai for consultations. 

The consultation process closes at the end of May. The principles will cover all educational sectors / NZQA qualifications from school to tertiary. Consultation feedback will be used to develop the final guidance documents and develop contextualised resources.

Of note is the inclusion of aromatawai, the teaching, learning and assessment approaches based on Maori values, beliefs and aspirations. The six underpinning kaupapa are -  Rangatiratanga, Whanaungatanga, Manaakitanga, Pūkengatanga, Kaitiakitanga, Te Reo.

The five principles of assessment are that they are valid, reliable, informative (i.e. with an emphasis on the formative), equitable and authentic.

All in, good synergy between aromatawai and the usual assessment principles. Addition of equitable allows for more personalised assessments to be conducted. 

Visible learning a way to end exams

 Here is a good overview of the current focus on re-thinking the ways education 'measure' learning. The short blog also includes the video by Toby Morris 'on the plat' summarising the challenges for those with lower social capital to attain equity within the current educational system.

The above, reiterates the overall them of this year' s Ulearn coference - see the notes on the first keynote setting the tone of - success for all and how 'measurement' of success needs to be shifted. 

For over a decade, I have advocated removal of 'summative assessments' from our course descriptors, as a way to move emphasis to formative assessments. Due to the current NZ Qualification Authority guidelines, it is not possible to remove summative assessments as such. So we are still 'teaching to the assessment' in many courses, as student success relies to 'passing the course'. Regardless of where the learner starts on their learning, they still have to meet 'standards'.

When I was teaching, about 1/2 the class would 'pass' the assessment, given slight 'steering' to establish a contextualisation of what they already knew, to the topic to be 'taught' and assessed. These students, just needed a few deliberate practice sessions, to polish skills, review theory and had the 'right' dispositions to do well. For the 'other' 1/2, it would take much effort on their part, to step up and my job was to help them, often, to just get to the start point (i.e. to scrape through). Yet, if they were willing to put in the effort, they would succeed in work. So, in the long run, it is the formative and the attainment of individualised learning goals which are more important, than 'passing the exam'. Many, in the 'other' 1/2, gave up along the line, as they saw the 'exam' as too difficult :(  Yes, they were making some progress through formative assessments. Many, feared 'failure' as that had been their experience, throughout school. Yet, formative assessments were looked forward to, as the feedback, provided the guide to improving, little by little, their confidence and skill. As confidence increased, so did their attitudes to learning. Always a win-win, as far as I was concerned. 

Therefore, my long 'crusade' to remove summative assessments. They are next to meaningless for many learners, measuring performance within a very narrow bandwidth. A means to stratify people into the 'cans and the cannots' when society needs everyone to contribute towards solving the 'wicked' challenges of humanity. So, more work still on promoting a better understanding of the role of assessments, in a world which is not 'black/white', where there are no 'right/wrong' answers and where 'out of the box' thinking may be the only way to ensure our planet is still livable for my grandson's generation.

CRADLE 2020 - Day two 'morning' presentations

 

Day two begins with presentations on assessment in a post digital world. The presenters are Associate Professor Phillip Dawson, Associate Professor Rola Ajjawi and Professor Margaret Bearman (researchers from CRADLE at Deakin). The discussant is Professor Michael Henderson from Monash University.

The symposium began with Michael acknowledgement of Country. Each of the presenters shared their work – which are chapters in the book – reimagining university assessment in a digital world.

Rola begins by presenting on ‘assessment as portrayal and the strategic negotiation of persona’. Began with the impact on digital technology on work – permeates everything, blurring professional and private life, careers becoming more fluid and greater portrayal of digital identities. Assessments are for certification, guide learning, help with employability and learning how to make evaluative judgements. Completing the qualification recognised with award and transcript but does that portray who the graduate is?? Proposed assessment-as-portrayal to enable students to portray their achievements in public ways, communicate directly with others. Examples include the use of graduate learning outcomes through portfolios/credentials etc. Another pathway may be through persona studies used by celebrities. Persona can be considered as a strategic identity (Marshall & Henderson, 2016). Five dimensions of persona include the public, mediatised, performative, collective and intentional value and reputation. Reimagining assessments using persona studies involves introduction of a subjective dimension to assessment practices, a rethink of how we judge the quality of portrayal for future work; and encouragement of student agency. Concluded with some important questions on how portrayal may work.

Margaret covered the topic ‘assessment for an AI-mediated world’. What does AI do well? – identify patterns more accurately than humans, faster and at scale. However, this is a myth. Automated essay grading is available but restricted to appropriate topics and tasks. The systems are bounded by the quality of the original data set! If a machine is able to grade a task, then it can do the task. Therefore assessments need to be centred around unique human capabilities – evaluative judgements, deployment of meta-intelligence (i.e. knowing about how you know). Assessments that promote EJ is not just about identifying but the ability to construct their own sense about quality withing a unique practice environment. Assessments can promote EJ through cycles of self, peer and educator assessments. Assessment should develop personal epistemology. Deliberately call attention to the EJ process. Use example rubrics to form powerful epistemic artefacts. Ask learners to question the role of rubrics (Bearman & Ajjiwi, 2019). Assessment tasks should prompt students to answer questions on – what qualities make this a good piece of work? If I were to set the criteria for success, what will it look like? How do we know we have achieved?

Phillip presented on ‘surveillance and the weaponization of academic integrity’. At the moment, Higher ed (HE) assessment has a surveillance culture, this may be OK but we should fear the weaponization of surveillance on academic integrity’. Drones, webcams and other tools collect data on student learning. Argued that presently, a surveillance culture pervades society and this is seen as acceptable by many ‘for our own good’. Surveillance culture’s effect on assessments include the creation of a culture of distrust; frames students as being in need of monitoring; students and their data become a product; academic integrity becomes a game of submitting to or subverting surveillance; and there eventuates an acceptance of the surveillance. There needs to be a balance between surveillance and academic integrity. Assessment security may be adversarial, punitive and evidence based but academic integrity is positive, educative and supportive.

 

Session one DAY 2: Learning through and for work is with Associate Professor Gloria Dall’Alba from the University of Queensland. She presents on ‘toward a pedagogy of responsive attunement in educating for the professions’.

Rhea Liang introduces the session.

Placed the presentation in the current context, where ‘fake news’ is of concern. Society relies on truthfulness. When fake news disrupts the standard discourse, it causes frustration and disillusionment. Proposes responsive attunement as a means to ensure professional are able to evaluate information in a studied and evidence based approach. Therefore, it is important to ensure our ‘way of being-in-the-world’ (Heidegger, 1962) is anchored in the ability to evaluate and respond to information. Advocated for ‘the capacity to care’ (Noddings, 2005) as a mark of personhood. Need to listen to the multiplicity of perspectives, and to be able to assess their importance. Learning incorporated what we know and do (epistemological) but also how we are learning to be (ontological). Being attuned allows professionals to work through complexities and ethical dimensions. Responsive attunement is responding on the basis of tuning in to others and things, with reflexivity. Responding includes sensate, motile, emotional – i.e multimodally. Education needs to begin the process so professionals are prepared for the challenges posed at work. Implications summarised as – highlighting tuning in and responding, focus on programme goals, reward in assessment, address embodied learning, have well-integrated program and ensure epistemological and ontological dimensions addressed.

Then a session with Dr. Emma Scholz from Charles Sturt University on ‘Professional identity development : Implications for sustainable assessments’. Presents on work with Franziska Trede.

Emphasis on professional identity development in the workplace and the bringing in of Bloom’s taxonomy to assist the process. Presented the concept of locating professional identity in a practice theory framework – in the context of first year of practice for new graduate veterinarians. Research focus needs to be grounded in ‘individuals in their social practice’. Detailed sequential analysis process to thematically analyse interviews and field notes. Themes than taken back to the data to construct key stories. These underwent dialogic analysis to provide for the narratives. Theme on practice capability, the learning professional, striving to become a capable practitioner, and mobilising criticality within capability, shared with interview vignettes. A screen shot of a key story presented. Proposed implications for practice.

CRADLE conference 2020 - Session presentations

 

Today’s sessions for CRADLE conference 2020.

Most of the sessions have two presentations. I attend the following:

Then the first session themed ‘learning in a digital world 1’ is with Dr. Adam Brown and Emily Wade from Deakin. They present on ‘Serendipity, imperfection and vulnerability: Harnessing live video for ‘authentic’ teacher performance.

Adam began with providing the theoretical background and approaches.

Proposed the undertaking of ‘the meddlers in the middle’ as the role of the educator as opposed to ‘stage on the sage’ or ‘guide on the side’. Meddlers in the middle reposition the teacher and student as co-directors and co-editors of their social world.

Need to be ‘authentic’ and that authenticity is an effect, not an essence in the context of self-presentation as being performative.

Emily introduced collaborative autoethnography and how digital platforms like Periscope may be used to enable collaborative work. Explained how this was used for their own teaching context and shared reflections on the process.

Very much an academic look into their teaching and a critical reflection into how video impacts on, supports and extends their teaching practice.

 

Session 2: Learning in a digital world 2.

This session chaired by Simon Knight.

On ‘Development of educational tools that enable large-scale ethical empirical research on evaluative judgement' with Dr. Hassan Khosravi from the University of Queensland. Presented on work carried out with others – George Gyamfi, Dr. Barbara Hanna and Dr. Jason Lodge.

Presented on evaluative judgement (EJ), educational tools, conceptual models and examples.

EJ is the capability to make decisions about the quality of work of oneself and others. This skill helps students use feedback effectively, develop expertise in the view and attain autonomy.

 EF revolves around rubrics, self assessment exemplars, feedback, reflection and peer assessment – mostly theoretical.

Educational tools include peer grading and evaluation systems. Most build without the aim of supporting research and they do not allow data harvesting to be undertaken or the set up of controlled experiments. Draws on work of Associate Professor Paul Denny (University of Auckland) on PeerWise and with Professor Neil Heffernan on the ASSISTments ecosystems.

This presentation on the conceptual framework of RIPPLE. Conceptual model described to help promote both EJ and to undertake research on it. System developed for adaptive learning, learnersourcing and peer grading and feedback with EJ strategies of rubrics, self and peer assessment and exemplars. Connected to metrics, experiment design and ethical guidelines and a data repository.

Provides details of the RIPPLE platform.

Shared one study – Can students create high-quality resources? Walked through the process of students creating and evaluating each other’s questions.

The shared a case study and reflections. In general, students’ EJ improved through iterations. Students tended to provide higher ratings when compared to instructors. Found that the rubric criteria may not suit the ‘assessment’ and re-tested new rubric. 

Shared plans for new investigations and encouraged other researchers to make contact to use the tool.

Followed on with George Gyamfi, also from UQ on ‘The effect of rubrics on evaluative judgement: A randomised controlled trial'. As per above presentation, George’s work is completed with Dr. Barbara Hanna and Dr. Hassan Khosravi.

Defined EJ as per previous presentation. Shared work on rubrics (Reddy & Andrade, 2010 and several others) and how these may be used to enhance student learning. These studies tend to be mostly theoretical and does not bring in the perspectives of students.

Overviewed the research methodology and how RiPPLE was used for the study. Participants were undergraduate students (n= 354) learning database principles. The study involved having a control group which completed peer assessments without a rubric and the treatment group which completed peer assessments with a rubric.

Findings indicate rubrics can be a way of influencing how students attend to quality and can impact students’ judgement in assessing the quality of learning resources. However, construct of the rubric is the key! Even without rubrics, learners are able to make judgements but rubrics provided better clarity.

Session three: Assessment of learning 1 :

Session chaired by Zi Yan.

Dr. Sin Wang Chong from the Queen’s University Belfast presented on ‘Student feedback literacy as an ecological construct’. A conceptual look at feedback literacy covering feedback orientation to feedback literacy, feedback literacy in and beyond the classroom and a reconceptualization of the concept. Presentation based on paper published earlier this year.

Shared study on feedback orientation (London & Smither’s 2002) – utility, accountability, social awareness and feedback efficacy. Also Kremmel & Harding (2020) on language literacy – using a spider chart to provide each learners’ literacy. Then Sutton (2012) in conceptualisation of feedback literacy as epistemology (knowing), Ontological (being) and practical (acting). Carless and Boud (2018) appreciate, make judgments, manage affect and take action on feedback. Most recent Carless and Winstone (2020) looking into both teacher and learner feedback literacy.

Feedback literacy is not only a product (Hattie & Timperley, 2007) but also a process (Interpersonal and intrapersonal (Carless & Boud, 2018).

Chong reconceptualised using a ecological perspective (2020 paper) to include the context within which feedback is undertaken (interpersonal, textual, instructional, sociocultural) and the learner’s beliefs, goals, experiences and abilities. Therefore the engagement dimension (as proposed by Carless and Boud – understand feedback, manage affect, make judgment and take action) has added to it the context and individual dimensions(

Introduced the ecological systems theory (EST) as nested or networked EST with micro, meso, exo, macro and chrono systems. Used example from a paper in preparation – process of scholarly peer-review – to explain the framework. Proposed actor-network theory as a means to understand better the inter-connections/inter-relationships of ESTs.

Concluded with implications including how the framework may be applied towards better understanding the contextual and individual contributions to understanding feedback literacy.

The next session is with Dr. Akilu Tadesse from the University of Bergen on ‘Scaffolding feedback in complex dynamic system context: Effect of online interactive learning environments’. Presents on work undertaken with Professors Pai Davidsen and Erling Moxnes. 

 Presented on the 'problem', covered the notion of scaffolding feedback, application of this study and findimgs.

People, even experts, have difficulties in understanding and communicating their understanding of complex dynamic systems. Also difficult to measure improvements in this domain when CDS are difficult to understand.

The study looks into how to enhance students learning of CDS by developing educaitonal feedback to scaffold feedback that supports students learning.

Shared the notion of scaffolding feedback as a continual spiral of building knowledge. Gaps of knowledge are 'filled either by students' own efforts are through support from 'external agents'. 

Used this scaffolding feedback notion to integrate into a personalised and adaptive online learning environment. Learner is presented with a CDS and supported to progress through a sequence of learning activities to attain CDS. 

Shared a 'case study' to help explain the concept of how the platform works. Then presented the research questions - to find out if scaffolding feedback would reduce the gap between existing high and low performing students. Averaged results across 5 tasks. In general, the gap did close as tasks (increasing in complexity) were worked through. 

Overall, a good range of presentations showing the ways digital technologies may be useful in supporting assessments of learning. 

 

 

 

 

 

Kick-off session #12 - Powerful and authentic assessments

 Today's lunch time kick-off session was facilitated by Jane Bates, one of our team of educational developers at Ara and hosted by Marion Hale. The topic covered was on powerful and authentic assessments, in particular the transfer of learning to work. 

Jane provided an overview before several lecturers (Michelle Thompson from nursing, Dr. David Weir from computing and Dennis Winter from architecture) provided examples of their assessment practices. 

Jane began with key points towards ensuring assessments enhance students' learning and work readiness. Reviewed the concepts of 'work ready' vs 'work ready plus' graduates; and the Ara teaching and leanring framework. Work-ready plus draws from the work of Emeritus Professor Geoff Scott. Work readiness includes generic/transferable skills and role and discipline specific skills and knowledge required for effective practice. These could be thought of as competencies. Work ready plus also includes capabilties like personal (self awareness, decisiveness, commitment), interpersonal and cognitive skills and knowledge. 

Summarised the Ara teaching and learning framework which takes us to 2022. Key attributes or capabilties include graduates being critical thinkers, able to act sustainably, communicate effectively, innovate, be cuturally intelligent and responsible. Provided link to access the framework.

Emphasised the need to ensure assessments are embedded into teaching and learning processes. Referred to presentation by Arun Pradhan who referred to 21st century requirements including digital fluency, learning to learn skills, problem solving, working collaboratively and empathy and the T shaped graduates with both disciplinary and broad soft and cross-functional skills.

Defined power assessments as high quality assessment practices that address key employment competencies and personal, interpersonal, and cognitive capabilities.

Assessments have to be authentic and demonstrate what people can do, are problem based and directly related to what is being learnt. Graduates need to have industry skills and knowledge but also the personal, interpersonal and cognitive capabilities to adapt to the challenges of the future of work.

Michelle shares her example, first year nursing students reinforcing communications skills as practiced within the health context. Used scenarios and students practice with each other to hone these skills. Used videos to assist with formative feedback and eventual completion of the summative assessment. Nursing have been refining this process for several years.

Dennis shared the architecture year 3 project. The projects are supported with a comprehensive brief as per architectural practice. Students present their project as it progresses with 'crits' from tutors, peers and industry, again reflecting industry practice. An exhibition of students' work and presentation along with portfolio of work also required to evidence their learning. 

David shared the project-based year 3 capstone computing course objectives and results. Year 3 computing students work on an industry led project through a semester. Their results are reported through a report, poster and a presentation. Important to ensure computing students are provided the opportunity to learn key communication and presentation skills.

All the three examples use authentic assessments to elicit students' reflective learning on not only the core knowledge and skills of the discipline but also the occupationally specific dispositions required to be effective professionals. Lecturers model the industry ways of doing and thinking and provide important 'checks' as students progress through the curriculum.

Active discussions ensued after each example is presented.

Jane provided summary - a powerful asessment checklist and examples of powerful assessments- before thanking all presenters and closing the session. 

Reimagining university assessment in a digital world - book overview

 Browsed through this ebook from the Ara library over the last week or so. Bearman, M. Dawson, P. Ajjawi, R. Tai. J. & Boud. D. (2020). Re-imagining university assessment in a digital world. Springer. 10.1007/978-3-030-41956-1

The eidtors are based at the Centre for Research in Assessment and Digital Learning (CRADLE), Deakin University,  Melbourne, Australia. 

Notes are brief as I have access to this book through Ara library and able to look up content when required :) 

After an introductory chapter by the editors, there are 19 chapters sectioned into 3 parts. The introductory chapter provides a brief rational and overviews of the chapters. 

A second introduction with New directions for assessment in a digital world by M. Bearment, D. Boud and R; Ajjawi. Here, the chapter uses a ‘programmatic portfolio’ to example how technology can shift assessment beyond the usual. Included is a call to include co-construction of assessments into HE; assessments should be one avenue to help provide graduates with the digital skills required in the workforce; and the inclusion of social media profiles and digital profiles of students into assessments so that they are useful beyond under graduate studies.

Part 2 – The changing role of assessment in the digital world

Digitally mediated assessment in HE: ethical and social impacts M. Bearment, P. Dawson and J. Tai.

This chapter sets out the many social, ethical and pedagogical implications of conducting assessments in the digital world. The themes covered include ethical issues, feedback, AI and influences of digital practices and portrayals.

Cognitive off-loading and assessment with P. Dawson introduces the concept of cognitive offloading into notes, calculaors, spell checkers etc. and their effect on the underpinning expectations of assessments. Some ideas on how to incorporate cognitive offloading are suggested including issues of transparency, programmatic assessment, evaluative judgement and authentic assessment.

Preparing university assessment for a world with AI: Tasks for human intelligence with M; Bearman and R. Luckin. Overviews the uses of AI in assessment. Idenitifies the things humans do well – personal epistemology and evaluative judgement as key. Uses critical appraisal assessment as a means to focus on human aspects. 

Repositioning assessment-as portrayal: What can we learning from celebrity and persona studies? With R. Ajjawi, D. Boud and D. Marshall,. Introduces the interesting concept of applying authentic industry or discipline norms to assessments in the form of persona studies.

Towards technology enhanced dialogic feedback by E. Pitt and N. Winstone. Updates on the aspect of technology-enhanced feedback via video, audio and screencast. Uses Carless’s old vs new paradigm to unpack the advantages and pitfalls.

Catering for diversity in the digital age: reconsidering equity in assessment practices by L. R. Harris and J. Dargusch. An overview of the challenges of digital equity with regards to assessments. 

Assessments as and of digital practice: building productive digital literacies by M. O’Donnell. Argues for the use of assessments not only as a means to judge learners’ work but also to scaffold, develop and assess the digital skills integrated into the assessment.

Part 3 the role of big data in re-imaging assessment

Augmenting assessment with learning analytics by S. Knight. Introduces learning analytics and how this can transform the way assessments are developed and implemented and assessment practices. Suggesions are provided to improve how LA may be instrumental in supporting better assessment processes.

Are assessment practices well aligned over time? A big data exploration with J. Rogaten, D. Clow, C. Edwards, M. Gaved and B. Rienties. Uses big data to look at 20 years of assessments across 2 studies to find out if student assessments/grades were well aligned with the qualifications. Recommendations are derived to improve the longitudinal alignment of grading trajectories by ensuing consistemt grading policies are followed.

The bi-directional effect between data and assessments in the digital age by A. Pardo and P. Reimann. Digital tech affects assessments and feedback. Data collected important to be translated to yield actions and design aspects to further positive effects on student experiences.

Standards for developing assessments of learning using process data by S. Milligan. Provides background on shift from just testing knowledge to incorporation of ‘soft’ skills and competencies important to graduate attainment of graduate outcomes.

Part 4 practical exemplars

Re-imaging peer assessment in self-paced online learning environments with L. Corrin and A. Bakharia. Example of self-paced online learning environment which is flexible, customisable and scalble peer assessment and feedback.

The future of self and peer assessment: are technology or people in the key? By J. Tai and C. Adachi. Theoretical frameworks provided along with how digital could provide for better pedagogical goals and best practice self and peers assessment. 

Using technology to enable a shift from marks to outcomes-based assessments by A. Cain, L, Tubino and S. Krishnan. Introduces the web application ‘Doubtfire’ to allow for a task-orientated portfolio-based approach to assessment to scaffold and evidence different standards of achievement. 

Challenges with designing online assessment to support student task understanding with T. Apps, K. Beckmand and S. Bennett. Cases study of open-ended online assessment.

 Re-iimagining assessment through play: a case study of metarubric With Y. J. Kim and L. Rosenheck. Uses MetaRubric, a playful assessment design tool which creates rubrics for teachers and students and allows them to learn about assessments.

Sharing achievement through digital credentials: are universities ready for the transparency afforded by a digital world? With T. J. de St Jorre. Discusses the topic and the advantages/disadvantages of digital credentials.

Part 5 conclusion

Concluding comments: reimagining university assessment in a digital world by P. Dawson and M. Bearman, draws the book to a close, reiterating the need to not just replace current assessment practices with a layer of digital support, but to actually examine the actual purposes of assessments and to envisage how digital technologies may be useful in transforming assessment practices.

 

 

 

 

Reflection - a week of conferencing

Two conferences last week provided some time away from the usual busy work routine. Importantly, the week allowed for time to catch up with others practitioners, passionate about helping learners. Always energised after a week away by presentations on applying precepts of good learning, to various approaches and strategies to assist learners.

Things that would be helpful for my own practice as an educational developer and researcher include:

- need to understand the exigencies of teaching from the experiences of teachers and students. For teachers, is to be empathetic with time-pressured and resource lean situations. To build good relationships with teachers and to provide possible solutions which are doable. Thinking through, together (teacher and ed.developer) to agree on a goal and to work towards the objective in small achievable steps. The 'inquiry cycle' as small interventions, each informing another cycle, has been a major plus for the e-assessment project.
For students, it is important to 'make the learning visible'. Too often, students do not know WHY they are having to engage in a learning activity or assessment. Learning outcomes require iteration throughout a course, not just at the start when the course outline or equivalent is waved in from of them, or they are told that the course outline is to be found on the institution's learning management system! Students are time jealous and will only do what is required to 'pass', but many do not actually learn, let alone change behaviours, attitudes or perceptions.

- There is still limited understanding across the ITP and ITO sectors, of the implications of NZ qualifications being graduate profile based. To some, the graduate profiles just add another layer to a complex schedule of atomised and siloed assessments! Moderation, in particular post - moderation of assessments, is still seen to be the checking of content covered :( Hence 'consistency arrangements' whereby qualification deliverers have to rationalise how their graduates meet the graduate profiles, are seen to be another assessment moderation process (aargh).

- Still confusion as to WHAT are assessments FOR learning. Calling them formative may not always be correct. Requirements to have summative assessments for courses, makes it difficult, in time poor courses, which are filled with content, to 'fit in' assessments for learning. There needs to be more work done, to help teachers understand how to 'design' and develop assessments for learning which provide benefit to learners. Exemplars across various discipline areas may be helpful.

So, much work still to be done. However, above provides a tighter framework to report on the e-assessments for learning project. the project 'guidelines' will need to provide:
- connection between assessments for learning and qualification graduate profiles
- examples of assessments for learning across several discipline areas
- comparison of assessments for and of learning for these discipline areas
- approaches to learning appropriate to required knowledge, skills and attitude learning
- links the above to constructivist (intra-psychological) and socio- cultural / socio -material (inter-psychological) learning
- templates for decision making  / design of assessments for learning as connected to approaches to learning
- Learning 'activities' suited to meeting holistic attainment of graduate profiles i.e. problem/inquiry- based, projects, portfolios etc.
- how to match these with appropriate technology to enhance student learning

Above provides a way forward for thinking through over the summer :)

Assessing Learning Conference, DAY 3

Day 3 dawns fine and warm. The weather across the entire week has been very summery. Hopefully a prelude to a good summer.

Begins with supporting colleagues Maaike Jongerius, John Delany and Lyn Williams from the Academic Division at Ara Institute of Canterbury, presenting the ‘assessment health check tool’. This is a moodle resource to support Ara tutors with ensuring their assessments are constructively aligned. Rationalised the pedagogical frame for undertaking the development for the moodle resource. If assessment drives learning then improvement of assessments will be a core objective. The resource had to cover the principles of assessment but not be too basic for staff who have completed teaching qualifications recently. The integrated activities in the health check can be completed online or as part of a facilitated workshop. The moodle site was brought up and examples of various worksheets / exemplars and the reasons and background on how they are used. Evaluations of the resource, the likes and dislikes, also shared. Presented on what Ara is committed to progress work on assessment practices.

Then Dr. Salome Meyer and Nancy Groh, educational advisors in the education development centre from Eastern Institute of Technology / Napier on ‘the changing conversation about early diagnostic assessment’. Outlined background, original premise / benefits and evolution of LNAAT. The tool is one of several developed to support the NZ government strategy to raise the capability of the workforce. Rationalised the need to change the approach to using diagnostic assessments. Matched literacy and numeracy demands in various occupations – what reading or calculation is required everyday at work? Provided a guide to tutors to better integrate literacy and numeracy within situated learning off-job. Addressed the issue of international students and their distinct needs. Developed academic inquiry course(non-credit bearing) to assist international students to orientate to the NZ educational demands. Developed a revised view of literacy diagram to summarise the different concepts.

Last session is a panel with Geoff Scott, Shaima Al Ansari and Tracey Bretag on ‘What will you do on Monday?’ Panel presented their takes on – what is the single key message you will take away? What single thing will you do, or do differently? What would you tell your boss they need to do? A question and answer session followed.

All in a good opportunity to achieve several things. One was the affirmation of my own understanding and application of the principles of learning -centred assessments. The various attended, all provided some templates, exemplars, concepts and tools useful in both my educational developer and researcher roles. Thankfully, many of the sessions I selected, focused on assessment FOR learning, although there was still a thread running through on summative assessments, prevention of plagiarism etc. Many presentations were on problem / inquiry / project based learning but not many examples from the vocational education / trades learning context. Therefore, as always, there is still a need for more ‘structured’ inquiry and study to build an evidence base of how to assist trades learning.

Assessing learning conference - DAY 2

A full day starting a 9am.

First up, Keynote panel – on the student voice facilitated by  Dr. Alistair Shaw with 4 students. As always a very valuable session. In short, students did not know about learning outcomes and how they connect to the assessment. Students preferred authentic assessments  which reflected real-world practice. Each institution has culture of practice and differen priorities. Not all have ability to provide authentic learning but assessments may be a means to bring authenticity into courses.

Andrew Kear with a team from the BCITO with ‘assessment in the workplace: principles for on-job assessment’. Gave out copies of publications relevant to NZ context. Shared the BCITO guiding principles and how they connected to Karen Vaughan and Marie Cameron’s good principles of workplace learning and assessments. A clear purpose for assessment is crucial. Provided overview of BCITO to apprentice support, workplace learning and assessment processes. Philosophy and approach is key with all BCITO understanding the distinct culture of BCITO. Belief in each learner is an individual. Group session to discuss how organisations may be able to support individualised learning programme. Assessments need to gather progressive evidence of learning and also be contextualised to be relevant to the learner. Learners should not be put through ‘hoops’ but have authentic evidence of learning recognised – maximising the use of naturally-occurring evidence. Evidence does not have to be written, could be video, aural etc. important to allow annotation of evidence. Moderation has to contribute to the validity and reliability of assessment decisions. Although time consuming and expensive, still has to take place and ‘communities of practice’ amongst ‘assessors’ and moderators, both taking part in the assessment. Moderation is a second opinion. Entire process requires appropriately recruited, trained and professionally developed people.

Followed by support of Faye Wilson-Hill and Niki Hannan from Ara Institute of Canterbury on their work with OneNote as an assessment tool. Provided background of the programme on why the assessment portfolio tool is used. Not only to be an assessment for learning resource but also to model to other teachers, a platform to support learning. Shared assessment principles – integrated into learning process, draws on learners’ experiences, encourages reflection and allows for multiple points for formative feedback. Moodle did not allow for all of these principles to be deployed. One note classroom notebook was selected as it allowed principles to use. Detailed process – how to start – shifting a word document into Onenote. Begin with familiar and work in the online environment first. Reflective practice has to be scaffolded. Showed example of how the notebooks used and structure of the notebook. Feedback is progressive as the course goes on so student have formative assessment for learning every 2 weeks.Feedback from tutors can be written or oral. Used a video capture (Panopto) for students to share portfolios if there is no collaborative space. Concluded with reflection on the process. Still learning but holds promise.

Then a choice of two plenary sessions after lunch. I select Dr. Eleanor Hawe’s on assessment for learning: A catalyst for student self-regulation. Defined assessments for learning and the second generation conceptualisations. In general, formative assessment research in school sector, formative feedback could be more dialogic; and the need to have explicit focus of pedagogy on preparing students to be independent learners. All assessment should support the advancement of student learning (Carless, 2015); assessment does not stand outside teaching and learning, but stands in dynamic interaction with it (Gipps, 1994); Students are no longer objects of their teachers’ behaviour but animators of their own effective learning (James & Pedder, 2006).

Second generation definition – assessment for learning is part of everyday practice by students, teachers and peers, that seeks, reflects upon and responds to information from dialogue, demonstration and observation in ways that enhance ongoing learning (Klenowski, 2009); Therefore, should be part of pedagogy – should be the formative use of assessment (not formative assessments). Aim to develop students as self-regulating learners who can monitor, regulate and control their thinking, behaviour and motivation while engaged in ‘academic tasks’. Sadler advocates that students need to know what is expected  (quality); sufficient evaluative knowledge and expertise to be able to compare current thinking / learning / performance; and a range of strategies to enable to effect improvement and further their thinking / learning / performance.

**Recommended five strategies. Promote student understanding about goals of learning and what constitutes expected performance; engineering effective discussion and activities including assessment tasks to promote and elicit evidence of learning; generate feedback (external and internal) that moves learning forward Use, notice, recognises, respond – Cowie & Bell, 1999); activate students as learning resources for each other; and activation of student ownership over and responsibility for their learning (Hawe & Dixon, 2017; Wiliam, 2011).

Provided an exemplar to illustrate the way in which 5 strategies are operationalised. Need to ensure contextualised to own discipline and practice. However, all 5 need to occur.

Then I present a short session on the eassessment project. The focus this time around, to connect assessments for learning and feedback opportunities to assisting students ‘learning to become’ as they strive and learn to meet graduate outcomes. Summarised the role of assessment for learning in assisting the student journey towards getting to the graduate profile. Details of the sub-projects and some interim findings. In particular, how to assist student to learn the many ‘qualities’ which are often difficult to describe and to work out where they are at and what they need to do to attain.

Followed by a session with Alastair Emerson from OPAIC on ‘developments in assessments for experiential, student centred and partially self-directed pedagogies’. Otago Polytechnic international based in Auckland’s student cohort tend to have only experienced chalk and talk, paint by numbers assessments and discouraged from forming own opinions. Flip learning has not worked, essays and exams are off limited use to assess capability, current assessments make passing the end goal, designed for a didactic knowledge transfer paradigm to constructive student co-created approach. Using formative assessments are incremental with worksheets and templates. Summative involves problem solving or project completion in a real world context involving actual company. Introduced experiential learning using guided self-directed discovery techniques, with diminished on texts, use templates and worksheets which suggest outcomes but do not necessarily have a set process. Need to make learning outcomes visible. Therefore move into project / problem / inquiry based learning. Provided example of worksheets and projects to encourage personalised learning.

Afternoon tea is followed by Plenary with Emeritus ProfessorGeoff Scott from Western Sydney University on assessing work-ready plus capabilities. Presented on the website flipcurric used to support the work. Advocates – good ideas with no ideas on how to implement them are wasted ideas and change does not happen but must be led, and deftly. Rationalised why bother about assessments. Learning impact is when learning design, aligned support and infrastructure and delivery intersect effectively. Summarised the 6 key components of a comprehensive, integrated HE assessment framework – check on flipcurric website. Focused on correct outcomes and assessments.

Learning outcomes – capabilities and competencies students are expected to demonstrate they have developed to a required standard by the end of a program or unit of study. Include personal, interpersonal and cognitive capabilities and the key knowledge and skills necessary for effective early career performance and societal participation. Shared his professional capability framework. Explained the subscales for each of the competences/capabilities. The Plus refers to future focus – sustainability, change savvy, creative and inventive, and clear where one stands on tacit assumptions driving current society. When through principles of powerful assessments and examples of types including key quality checks for assessment of prior learning and learning from experience.

Then Shaima Al Ansari reports on ‘the impact of PBL on employability skills development: The Bahrain Polytechnic industry project assessment case. Knowledge isn’t power until it is applied – Dale Carnegie. Explained rationale and context (business management studies). Project requires student to take on accountability, work with others as part of team. Assist with application of employability skills to attain professional identity. Summarised details of the industry project. A capstone project in the fourth and final year. Students set up a consultancy firm and are the associate consultants. Work with a real client on an ill structured / complex problem. Full time commitment with weekly 2 hour meeting with the academic supervisor who takes on the role of the HR specialist for the consultancy. Students have an orientation week before beginning. Project process detailed along with examples of guide sheets, assessment plans / schedules etc. Shared positive feedback from 'employers / clients' and  students. 

Networking session closes a busy day. Lots of reinforcement of principles we apply at Ara, the concepts underpinning the e-assessment project and some new ideas and resources to support educational development work and production of e-assessment guidelines.

Assessing Learning conference - Dunedin - DAY 1

Move across from the University of Otago to Otago Polytechnic at lunch time to begin participating in the Assessing Learning Conference. Had to miss first keynote with David Boud.

First up, a workshop with Peter Mellow on effective assessments titled: EdTech in assessment: sinner or saviour? https://tinyurl.com/yasdsxvp for slides

Presented HoTEL as a grounding framework to inform on pedagogy. However HoTEL does not indigenous knowledge, which has to be woven into the Westernised frameworks. Added the Australian dimension with examples of how indigenous peopled learnt. Also Curtin University e-resource on elearning for processes and approaches. Assessment in the 21^st century pedagogy as being the provision of timely and meaningful feedback, relevant tasks, self and peer assessments and clear, transparent goals and objectives. “Good assessment should be a learning experiences”.

Look up sinister 16 – Potter & Kustra (2012) course design for constructive alignment – A primer on learning outcomes.

Reminder on listening to students to find out what assessment strategies do students prefer? Lowest – quizzes, written papers, group projects, middle – audio recordings, open discussion, paired discussion and highest – response to video, twitter summaries, screen casts, field experiences, interviews, work samples.

Need to ensure students KNOW why they are being assessed. Promotes formative assessments as it provides feedback, have opportunities to fail and can be fun (or be a game). Learners need to know whay they are being assessed, how, what rules and the value.

Tools for assessments – organising assessments, grade centres, deployment of assessment (e.g. peer matching, multiple choice quizzes, automate feedback. New technologies not quite there but include grade/analyse/QA/authenticate assessments, automated essay scoring, block chain – authentication, badging / certification, AI, badges / gamification, Learning analytics / assessment analytics, haptics (force feedback).

Solutions to ‘cheating’ include having students pledge not to cheat, sign honour codes etc. Evidence from multimedia evidence has metadata that can be tapped to establish authenticity of data. Use learning analytics to tighten quizzes etc. on LMS – randomise, auto feedback. Revision Assistant can be used by students to obtain formative feedback on essays. Online proctoring is possible, using keyboard recognition, web camera observations and identification of students. So why not make classroom about learning and not testing?? Promoted efficacy of MOOCs – using University of Melbourne examples. Students found in video quizzes (usually questions between slides) useful.

Reommended peerwise as a tool to create a collaborative learning environment. Peermark can be an alternative to turnitin. Perusall – every student prepared for every class – allows students to annotate readings and share with others in the class.

Then support two of eassessment sub-project researchers with their presentations.

First up, Cheryl Stokes from Ara Institute of Canterbury, with ‘developing reflective practice of level 4 cookery students through sensory analysis of food.  Provided overview of her teaching context and background of the project. Especially the shift from unit standards to graduate outcomes and the shift in assessment approach to portfolio instead of exams. Rationalised the research question  - to improve student reflective learning and ability with associated vocabulary to describe the taste, texture of food. Described reflections on who teaching, process and tasting reflective skills could be improved. Discussed challenges – especially how students could improve their mobile learning practices – back up their data on the cloud to access on multiple devices or secure storage in case they lost their device. Showed how research question evolved as project progressed to meet student learning needs. Focused project on improving tasting vocabulary, find appropriate cloud based app to record photos, improve reflective writing. Introduced Mindly as a app to build mindmaps of tasting vocabulary. Allows photos to be linked to text mindmap nodes. But tutors need to refer to the app and how it can be used for students to be engaged. Used google keep to archive notes, add photos, links etc. as a collection tool for their portfolio. Works with various languages, not only in English. Able to be used in tandem with google docs and extension is available on Facebook if Chrome browser is used. Increased integration of the various bits of evidence so collection of evidence can be easier to collate portfolio. Reflected on experience as a researcher working with students in a workroom. Detailed some recommendations – especially support for tutor and timing for introduction of the tools and concepts of reflective learning.

Then James Gropp and Stuart Campbell from Nelson Marlborough Institute of Technology on ‘using reflective practice in a technical problem based learning environment’. Introduced research question and aircraft engineering background and student cohort – note more than ½ of cohort were international students from a Pacific Island airforce. Shifted from didactic pedgagogy to problem based learning approach. Most important to focus on the learning and make this visible. Set the task to repair an small airplane as the ‘problem’ to be completed. Used aircraft servicing task cards as the basis of the eportfolio. To begin, reflection was poor. 1^st cycle did not produce results as students were not taught how to reflect or think through on what they were learning. Therefore, students were task and not learning focused!! Changed questions to include ‘learning’, provided exemplars, tutors changed from engineers to teachers, honouring the learning from errors. Tutor capability developed with daily reflective sessions. 2^nd cycle revealed improvement across the board. 3^rd cycle ran without changes and evidence of students’ adoption of reflective learning and problem solving.

Next, Dr. Megan Anakin from University of Otago with  ‘constructing a developmental framework to assess reasoning skills’. Detailed background and need for 21^st healthcare practice and the challenges of teaching reasoning skills to doctors. Shared progressions used in NZ curriculum / learning maths concepts as examples of frameworks. Defined clinical reasoning, theoretical underpinnings, expert skills, involving students in teaching and how students learning it. Introduced the cognitive models – dual process – fast and slow and script theory. Senior clinicians tended to have CR as tacit and learnt through apprenticeship as modelled to them by their mentors. Developed a framework for students to unravel how the traditional framework is mapped to the real world. Provided medical students with characteristics and outlines to help them practice and select appropriate strategies. Year 2 students still had to depend on ‘scripts’, year 3 starting to realise the complexity and need to adjust their questioning. By Year 6, students able to hone in more quickly and probe deeper to try to diagnose effectively. Need to follow up on this project as there is much of relevance.