Aotearoa Tertiary AI network ATAIN - presentation from Dr. Simon McCallum

 Notes taken from presentation from Dr. Simon McCallum, Victorial University Wellington on 'Adapting to AI'.

The presentation is part of a fortnightly series organised through ATAIN which is a SIG of Flexible Learning Association of  NZ (FLANZ).

Simon began with an introduction. He has been teaching game development since 2004 but has also taught AI since 1991. Noted that Gen Ai is everywhere and we use it unintentionally, unconsciously, but also using it consciously and strategically. Productivity benefits depends on training of the AI. Ai agents work together to automate generic tasks.

Across industries, adoption is mixed, some fast, some very slow. The risks include programmers with AI automating other industries and the use of 'software / automation on demand'. 

Revised the two lane approach to assessments. Students are using AI - At UVW 66% admit using it.

Covered the following:

 AI literacy - Core / domain specific - compulsory for all students and staff, understand if and when to use AI and avoid the risk of thoughtless AI use.

Assessments need to move to testing understanding and learning rather then outputs. Test meta cognition, use oral assessments.

All non-invigilated (lane 2) work should be considered as group work, using group work assessment techniques - assess process, influence, delta, learning journey. 

Assessments can embrace AI assistance. AI selects questions, create bespoke questions, suggest oral assessment questions. Human markers determine the grade. Provided details of the process from his context. Use AI to generate questions from student submission for them to complete, to check understanding they have presented in their essay!

Increase the quantity of group/team work so students can create human connections and increase their experiences with working with others. Invrease the amount of work that is groupwork but not the groupmark.

Proposed that NZ universities fund a NZ based server to assure AI sovereignty. A more equitable approach as all students will have access to high quality models instead of having to pay for the upgraded models. 

Also the creation of a position to report to Te Hiwa (leadership group to organise and manage AI across the entire university - teaching, learning, research and professional. 

Summarised agentic AI. Moves AI from being a chatbot to actually being able to 'do things'. AI will sort out a plan, and work through it to meet the prompt objective. Swarm coding can be activated, to check the outputs from each agent. Therefore, AI is not a search engine. It is better to have the AI question us to work out what we want done! 'help me do ----' 

Summarised project on how AI is used in NZ secondary schools. Mixed across schools on strategy, current use, professional / student use and community involvement. Schools welcome clearer policies and guidelines. Challenges are similar to Universities, assessments, professional development, etc.

Shifted to a summary of the extrinsic and intrinsic purposes of learning. If the motivation is just to pass the exam, then AI is an impediment. However, if there is a less constricted 'assessment' e.g. develop a game, AI accelerated capability and leads to extended learning. 

Therefore important to engage learnings to focus on intrinsic motivation, self-reflection and etc. and to hold them to account for what they want to  learn. Teacher as NPC . Encouraged learner negotiated assessments and rubrics, giving them agency. For example, have learners establish the range of marks assigned to various aspects of an assessment. This encourages meta cognition for students to structure their learning, their strengths/ weaknesses etc. 

How do we measure metacognition - confidence is good when it is accurate, under / over-confidence is a problem and AI makes this worse. Important to operationalise and elicit accurate statements from learners.

Invitation to use the group's Discord to continue the conversation and share ideas. 

AI-generated assessments for vocational education and training - webinar

 Here are notes from the webinar on the ConCove Tūhura project AI-generated assessments for VET, 

The report provides the literature scan and details of the process undertaken to identify appropriate AI to undertake the task, and the processes to ensure that the AI- generated assessments would meet moderation requirements (quality assurance) for use for assessing VET standards. 

The work was undertaken by Stuart Martin from George Angus Consulting and Karl Hartley from Epic Learning. Both present in the webinar which begins with an introduction by Katherine Hall (CE for ConCoVE Tūhura) and by Eve Price (project manager at ConCoVE).

In Katherine's introduction, the rationale for the project was shared along with some of the journey taken by the project to break new ground.

Eve Price provided the background of the project. Most projects focus on integrating AI into ako or the prevention of AI for assessment. This project wanted to help support the time consuming 'back room' processes including resource and assessment development.

Karl ran through the approaches to the product. The evaluation/review processes could not really keep up with the speed at which assessments can be developed when it is supported by AI. 

Stuart shared reflections on how the process evolved and the various processes put in place, were reflected on and were then reintroduced into the AI-generation project. Explained how various quality pointers were met to ensure the efficacy of the process.

Eve detailed the need to be specific with what needed to be achieved - assessment, feedback, etc. Selection the correct AI is also important. Prompts are detailed in the project report. Important to evaluate at each step.

The bigger picture with micro-credentials, skills standards and AI-generated assessments all add innovations to the VET ecosystem. Understanding the policies and processes used by WDCs and NZQA need to always be part of the process, so that various quality points are met.

Stuart summarised some of the challenges and how the project worked through these. 

Karl talked on the importance of people in the process when AI is generating the assessments. Firstly, important to understand some of the mechanics of AI - what is under the hood. Secondly, quality assurance must be focused on the concepts, not so much the grammar/spelling etc. Thirdly, need to make sure assessment purpose is clear. 

Next, academic integrity and ethics were discussed. Important to ensure that there is understanding the impact of AI on privacy and data sovereignty (including indigenous perspectives). Important to train the AI to understand tieh Aotearoa context. Claude AI was selected due to its stance on human rights, ethics etc. 

Findings included: assessments did not meet moderation but improved the opportunities for inclusiveness and personalisation of learning. Failing moderation added to the learnings from the project. The items involved too many questions, answers being at too long and at too high a level. 

Eve reiterated the need to 'define what good looks like' to the AI, so that human objectives/ perspectives are taken into account. Important to ensure principles of ethics etc are maintained as it is important to 'keep humans at the centre'.

Karl's learning include AI drawing in novel content through its hallucination. The AI included assessor approaches into its assessment and this caused him to consider the learner information that should be included to provide direction. The U S of A standardised approaches to writing assessments, seemed to permeate the assessments produced by AI. This had to be superseded through careful prompting.

Flexibility to allow for personalisation to industry (example safety unit standard customised to a range of work roles/ disciplines); and learners (for ESOL, neurodiverse learners etc.). 

 Q & A followed 

The webinar was recorded. 

Discussions revolved around practicalities, challenges and solutions.

All in, good sharing that adds to everyone's learning about the roles of AI to support teaching and learning, integration of practice/practical and cultural contexts, the need to be aware of the fish hooks' in using AI, how quickly AI is developing to meet user needs, and the need to continually learn to ensure that the understanding of AI / ethics etc. form the foundation for working with AI.