AI in Oral assessment - Food and Fibre CoVE project presentation

 Updated - 26th August 

Attended part of a presentation organised through Scarlatti to present the work funded by the Food and Fibre CoVE. It is one of a range of projects on artificial intelligence.

The presentation shares work undertaken to use AI for oral assessments.A report is available, summarising the pilot and findings. The various outputs from the project can be found at this site.

The presentation showcases the AI agent for learner oral assessment and shares perspectives from users as to the AI agent's efficacy. The guide to using the oral assessment chatbot is found here.

Began with sharing of a 'case study' comparing conventional (paper-based requiring travelling to an assessment centre) to the piloted (mobile, oral, AI supported). Then a couple of warm up sessions followed by short overview of F & F CoVE. They have completed 109 projects over the past 5 years!! Also introduced Scarlatti, and the research team from Fruition, F & F Cove and the WDC. 

Then the sharing of using AI agents in Oceania followed by the demonstration of the AI agent.

AI agent oral assessments came about as written assessments were a barrier for many learners and an Ai agent needed to be tested. 

Began with sharing the AI in Education  articles - which was the way to find out what was being done. AI agents being used more by universities in Australia. Most agents created using Cogniti or custom GPTs. Most were in a text-based mode. Not as much use in vocation education or in NZ and most were focused on learning rather than assessments. 

Showcased the AI agent (VEVA) with proviso that it is still a prototype :) tested in a dairy training context. The agent does provide the answer if the assessee indicates they are unsure or unable to answer the question. Using this for formative assessment would be more pertinent. It works well if correct answers are provided but does fall over if the candidate is unprepared!

Transcripts are archived and can be used to grade the conversation. With the small number of candidates (14 + 11) the moderation of the grading, showed accuracy. 

Important finding included that an off the shelf product would not work. This was because a controlled conversation was required, precise outputs was also required, and funding/payment needed to keep the agent. 

Therefore, two agents were used. The 'examiner agent ran using Open AI Real time Voice API) and the Assessor ran on GPT 4o text mode.

A demonstration (akin to our own experiences with Ako AI) in that responses from AI can sometimes be difficult to predict. The controlled conversation was primed with standardised questions. Prompting content included course materials, assessment rubric and example answers which had been graded.  RAG (retrieval augmented generation) was used to search course information as there is too much course information to put in. Ethical guard rails were included. 

Shared the various ways in which the oral assessment AI agent could be used. These include summative and formative assessments, authentication of context and candidate, employer verification, pre-screening for readiness to be assessed and for recognition of prior learning (RPL).

Closed with where the project will move towards next. 

A panel made up for Scarlatti staff (Adam Barker, Sam COrmack, Leater Hoare) and users - Jenny Sinclair from Dairy Training and Tiffany Andrews from Fruition the answered a series of questions.

Overall, a good overview of the possibilities for using AI to support multimodal forms of assessments. 

Food and Fibre COVE - new approach to learner pathways

The Food and Fibre CoVE(Centre for Vocational Education Excellence) and the Workforce Development Council - Muka Tangata -   have developed a new skills framework for their sector. Both Muka Tangata and the CoVE supports industries across agriculture, horticulture, seafood and Veterinary services. 

The project's website also provides links to the framwork, the research report (prepared by Kathryn Koopmanschap and Dr. Nicky Murray), and the recommendation paper. 

The framework is informed by an analysis to various international frameworks and alignment with the Aotearoa NZ context. The work arose from the need to recognise the skills many people bring into the industry so that they can fast track through the industry-specific qualifications. 

The framework components include core transferable skills, core technical skills, industry-specific skills and bodies of knowledge. 

All in, a step in the right direction to help provision qualifications that are industry-led, easily understood by industry and learners, flexible, transferable and with the ability to bridge both formal and informal learning. 

NZ vocational education and training (VET) research forum - 2021 DAY 1

 This year the research forum is held online again.

The forum opens with Joe Te Rito and Alexia Tuhi from Ako Aotearoa providing the welcome. Josh Williams than provides an overview of the conference and a welcome to all the participants, from may countries and across the educational sectors. and Garry Fissenden, CEO of Skills Consulting Group, follows and reiterates the welcome to everyone. Helen Lomax, Director/Tumuaki of Ako Aotearoa also provides a welcome to the forum.

The first keynote is with Brad Olsen, Principal Economistand Director of Infometrics who presents on ‘NZ economic recovery and the NZ education system’. Provides an update on how the pandemic has altered the global landscape. It is not going away and there is a need to keep up with what the implications of the pandemic are into the future. His presentation focuses on ‘building back better means taking a different approach’. Re visited the effects of the first lockdown in NZ last year, a significant shock but also a quick recovery. Employment is well up but still over 100,000 NZers looking for jobs. We have enough people but not at the right time, right place and with the right skills. Health jobs have increased significantly but retail, hospitality etc. have dropped. Retraining may be one option but not all people need long training programmes. Upskilling is important. Regions dependent on tourism (Otago) have been most badly impacted. Tourism might never be the same again ☹

Highlighted the need to have a new way of working. As so many of us continue to work from home, groceries, home recreation and DIY – including increased spending in furniture etc. have increased. Internal tourism sees regions nearest main centres to be busiest. Accommodation and F & B have declined. Suburban centres outside of city centres also doing better with greater number of people moving out of the city to the outer suburbs. Commuting using public transport have also not revived.

Labour market is intensely tight. Instead of large numbers of NZers out of work, there is a labour shortage. Due partially to no access to skilled workers from overseas (from about 5000 a month to none – an unlikely to be as open post-pandemic). Unemployment may drop below 4% but unemployment still uneven across ethnicities. Māori and Pacific Peoples have double the rate. Young people and women also more affected. The work of Te Pūkenga is important and especially crucial to work with the Regional Leadership Groups (RSLGs).

Job advertisements well up with few people responding to these. Competition for workers therefore increased. For education, micro-credentials have become more important to fill gaps and to help people move horizontally across occupations. Integration  of the skill needs between employers and education important. Warned of inflation – soaring shipping costs filter to higher costs of consumables.

Breakout sessions then begin.

The first is with Tania Mullane (Whitereia and Weltec) on ‘tangata hourua framework’. The framework is an indigenous reciprocal research framework. It draws on Kaupapa Māori and Pacific methodologies and values. The framework upholds the rights of Māori to have their knowledge and culture embedded in research. It also reflects and represents those who self-identify as having multi-ethnicities which include Māori or Pacific Island. Important as a response to a gap in understanding and the need to address and give context to some of the tensions, conflicts, and challenges of multi-ethnic populations. As there are no existing theories, important to acknowledge and respond to the need of Generation ‘B’ - brown, brainy, beautiful, bicultural, bilingual, bold. The framework is useful when undertaking research with Māori and Pacific peoples. Important to develop indigenous methodologies and dispel the dominance of western methods and that they are the only valid /reliable ways.  The framework synthesises work on Kaupapa Māori theory (HIngangaroa Smith) and Pacific/pan-pacific approaches. Discussed the specific challenges in bringing these two important methodologies together and provided details on how the framework was build. Used both the whare and the waka as they are both shared in both cultures. Selected the double hulled canoe – as both deliberately bound together for the long sea journeys. As with a waka, tangata (people) with skills knowledge and expertise and that the waka can be upturned as a whare once the destination is reached. Provided detail of the framework including a spiral bringing together both cultures but with space to also stretch/move between the themes. Summarised ways to use the framework. Possible as a cultural ‘roadmap’ to allow for how participants may self-identify and to overlay collective group of Māori and Pacific people.

The next session is with Dr. Cherie Chu-Fuluifaga (Victoria) and Janice Ikiua-Pasi (Massey) presenting on ‘From good to great: 10 habits of phenomenal educators for Pacific learners’. Cherie introduced herself and the project. Janice then introduced herself. Shared the purpose of their project to try to turn theory into practice, develop real system change, listen to student experiences and focus on the strengths of phenomenal educators. Shared two stories of phenomenal educators. The pioneer educator and the reflective educator. Introduced ‘the Kato’ or presents - as the ’10 habits’ – summarised on Ako Aotearoa site

A panel session follows on exploring the Centres for Vocational Excellence (CoVEs) with Paul Hollings, General manager for the Foodand Fibre CoVE and Bharti Raniga, General Manager for the Construction andInfrastructure CoVE. Josh Williams introduces the panel session and provides the background and context of the creation of the CoVEs. Each GM introduces their CoVE, the purpose and direction. Bharti spoke of ConCoVE, with a Māori name forthcoming. Vision of ConCoVE is to assist the alignment of the industry and VET. Industry is not quite satisfied with current quality, type and access to training for the industry. Dialogue and co-construction (pun intended) required to increase productivity, increase diversity of current workforce, and contribute to NZ future workforce development. At present ConCoVE has been working with employers, industry, iwi, Te Pūkenga and now in the phase of focusing on the themes and recommendations arising from this consultation. Also learning much more about why and how people engage with work in the industry, when they make their career decisions, what are the learning pathways and what does technology and the future of work bring into the picture.

Paul summarised progress on Food/Fibre CoVE. Largely industry with some provider members and a board. Currently 5 staff. Reached out to their constituencies and find out what their needs. A knowledge base has been setup; a framework to evaluate how the CoVE is effective; and a literature review to find out what is going on in food/fibre education training excellence. Interest in microcredentials but important to work out how these are integrated into current qualifications framework. As with ConCoVE, working on equity issue, to bring in Māori/Pacific/women/immigrants into the industry. Also building a work-integrated learning model for the industries. Important to identify the opportunities for excellence and to leverage these. Due to the newness of the CoVEs, there were lots of questions.

The second keynote is with Olly Newton, Executive Director at Edge Foundation who presents on ‘Degree model apprenticeships’.  Provided an overview of the Edge FOundation which is working to inform educational systems to provide young people the knowledge skills and behaviours they need to flourish in their future life and work. Then provided the background on the apprenticeship reform in the UK including the work on shifting apprenticehship frameworks and standards towards apprentice endpoint assessments. An apprenticeship levy was also set up to fund work and support on apprenticeship and rationalisation and detail of this was presented. 

Shared the UK experiences with Degree Apprenticeships. Began with background as to how these are constituted - length, coverage, and delivery models. There were 13,400 starts in 2019/20 for degree apprenticeships with 55% mainly in business, administration and law. Proposed advantages and challenges. 

Summarised other examples of degree apprenticeships from Scotland (graduate apprenticeship), Norway (industry master programme), and Lithuania (create Lithuiania to attract study abroad students). 

Then covered the wider models for VET (not necessarily apprenticeship). Examples include Edge Hotel School (with University of Essex) which  is situated in a hotel and supplemented with WIL in other hotels; Wales National Software Academy (with Cardiff University) replicates a development business; Minerva University in San Francisco; and Quest University in Ontario.