AI is not the answer, it is how it is used to support learning which is the key

 Some learning from chalkbeat, reporting on why the Khanmigo chatbot has not yet met its promised goals. 

Three years ago, an AI chatbot was launched in Khanmigo. Uptake by students has been lukewarm. The Khanmigo chatbot was used by students to find answers, rather that as a 'study buddy'. The key seems to be, as always, to educate students, rather than for them to just try things out. 

A recent German study to evaluate the relationship between AI-chatbots, student assessment performance and learning outcomes in Higher Education,  The findings indicate that students may be able to complete tasks faster, but not better or lead to higher grades.

Again, it is important to plan carefully, the objectives of using AI to support students. Are the chatbot/agents used to support learning? If yes, what are the specific learning objectives? Is the chatbot/agent 'ring-fenced' sufficiently to actually support the attainment of the learning outcomes? In our current projects, we need to be very clear about how and why the chatbot/agent is being utilised. This helps delimit the range of the chatbot and to ensure that it stays on task and does not deviate from its assigned objectives. 

Castlereagh statement on AI in education (Australia) - webinar of 'soft' launch

 Dropped in to the webinar to 'soft' launch the Castlereagh Statement - a cross-sector call for action on Australian education and training in the age of AI. The statement was developed through a series of cross-sector meetings/webinars/communities of practice. 

Notes taken:

Katie Ford welcomed the audience, welcomed all to country and provided an overview of what the Castlereagh Statement is and why it has been developed.

Draws together 80+ contributors from 30+ organisations including schools, universities, VET,  industry, government and also students. Brings together the many reports and research but no necessarily leading to action. Employer and society needs are shifting and pace of AI means it is difficult for education and training to keep up. 

Came up with 3 goals (what do we value in learning, a coherent learning pathway, and every Australian engaging with AI), 6 principles and where to next (near horizon - stabilise; intermediate medium - structural; and far horizon - new foundations).

Cross sector perspectives provided:

Bridget Pearce (Brisbane Grammar School - K-12) - Returned to the 'purpose of education'. At present mismatch between the needs of communities and society - collaborators, innovative thinkers, etc. but the curriculum crowds these out with its emphasis on meeting standards. Alignment needs to now be taken between the type of education required, and how schools can help students to get there. Teachers are not resistant to change but are resistant to bad change :) 

Helen Fairweather (Engineers Australia) - Important to have every sector involved with input into the statement. All participants were passionate about contributing. Engineers Australia does not dictate to providers, what and how engineering is taught. However, they are keen to ensure that Engineers have the competencies required to practice as engineers. Consistent framing across the sectors. However, no right/wrong answers. The statement provided the opportunity to consider the future of education. Accreditation does not seat outside within industry, it is a shared project with society and education. 

Claire Field (VET) - Shared the timeline with regard to guidance on AI. The schools had developed an Australian framework for Gen AI in Schools in October 2023. However, ASQA still in the process of drafting principles. AI is not just impacting learning and assessment, it is reshaping work across all industries, and VET serves diverse students and communities. VET perhaps behind as they did not (do not still) think that it will be impacted by AI (especially in the practical area). Unlike other countries, VET students are generally not encouraged to use AI. 

Adam Bridgeman (University of Sydney) - Universities siloed and are not likely to be aware of what is happening across other sectors and also internally - not much interaction between disciplines! Authentic learning will likely now require working with AI. Values and incentives to reward what matters is important. It is now not about the product but about what learning takes place. Pedagogy and how this is understood is a key towards leveraging AI. 

Peita Davis (Business Council of Australia) - what do businesses expect from graduates? Strong AI literacies and human skills are the key - teamwork, communication, critical thinking. Young people are seen to bring AI into small businesses! However, this is not the case :( no urgency across government towards this, AI literacy for all in Australia. What can be done? The statement is good first step. Clear guidelines and next step is to have AI literacies embedded from early childhood onward.

Tim Burt (Future Skills Organisation) - Provided a higher level, governmental perspective. AI skills are being defined - what are the skills Australia to compete in the Ai economy? Do we have the skill? What is needed to mobilise the nation around AI skills? and how will we know if initiatives will be effective? Common themes between Castlereagh and the study. Human skills fundamental; coherence required across all of education and complemented by industry feedback? How can AI skills development be coordinated? 

Danny Liu presented on 'next steps'. Jason Lodge reiterated that all the sectors/industry etc. coming together provides a way forward, which is cooperative. Therefore it is a coalition of the willing.

Monthly working party will meet between now and October to create a White Paper - cross sector and sector-wide working parties to formulate specific action plans, collect and surface vignettes of practice, and make progress visible. Items include IT enablement, curriculum and alignment and credentialling and assessment.

Q & A ensued. Topics included ensuring all states included; ways to ensure industry/business included; and how to ensure what is drawn up have a voice with policy makers and government. 

 

AI in education - reading the literature with a critical eye

 The numerous publication around AI in education has made it somewhat challenging to keep up with. My Google Scholar alerts collates around 10 articles every day. Other sources of reading include recommendations via various LinkeIn posts, blogs I follow, news articles and other media, and alerts from various journals. There is therefore an avalanche of material on the topic!

However, there is still a distinct gap in work from the vocational education sector. The book 'Artificial Intelligence in Vocational Education and Training' which I edited, has had over 4000 downloads. A favourable book review in the International Journal of Training Research has helped with bringing the issue of applying and leveraging AI into the specialised context of VET. More can be done, and I have put forward another proposed book to publishers, to record how the sector rises to the challenges posed by AI. VET cannot ignore AI, given its primary goal is to prepare its graduates for the world of work. All VET needs to cover Core AI literacies to ensure that VET graduates lead their industries into the ethical, principles and careful use of AI across occupations. 

There is also a need to draw on the current corpus of work on AI in education. Code Acts in Education blog warns of being circumspect with regard to publications, often spread through the blogsphere and other social media. There are also many pre-print articles being disseminated through various channels which are in the process of peer review. Recent recantation of articles due to faulty methodology also raise flags. The blog is worth reading through for the many ways in which fast, rapid publishing, is detrimental to the credibility of sound research.

Therefore, it is important to carefully evaluate articles, be patient and wait for articles to have cleared the peer review process and be published, and keep an eye on article retractions! 

Rebecca Frankum - NZ Vocational Education and Training Research Forum (NZVETRF) master class - on school leaving certificates

 Attended a 'masterclass' or fireside chat organised by the NZVETRF presented by Rebecca Frankum, Transitions in Upper Secondary School Education, OECD Secretariat. A recent report 'the theory and practice of upper secondary certification' authored by Hannah Kitchen and Rebecca, forms the basis for the discussion. 

Notes taken from the presentation/discussion:

Josh Williams from Skills facilitates the session. He began with an overview of NZVETR, the background to this session (the change of NZ school leaving qualifications) and introduce Rebecca - who presents from Paris.

The presentation worked through the report, with an emphasis on some of the implications and applications within the NZ context.

- Began with the importance of upper secondary certficates as it accreditates them towards the next step in their lifes. 

- Study looked into how upper secondary certificates were structured. did they incorporate and assess a broad range of skills? reflect the diverse skills and strengths of all students? enable progression to the students' next steps.

- Analysis of upper secondary certicates (71 certificates, 38 systems) on nature of assessments, who marked, what did they include, was there activity within unseen questions/tasks? allow for natural occuring evidence.

- 3 categories - certificates that include external exams, but no internal assessment; certificates include internal but no external exams; and certificates that include both (NZ NCEA is an example).

- exams still have an important role as they assess complex skills like analysis, evaluation and creating and include a range of problems, sources and multimedia materials.

- Higher education entrance exams - same exam papers, consistent marking, consistent standard of difficult, same exam conditons. Useful as 'gate keeper' - certify knoweldge, understanding and skills and facilitate selection.

- Wider range of exam formats support validity but can be hard to design and some skills (practical, social. emotional, higher order ectc) hard to assess.

- Balancing assessment approaches (fairness, credibility, relevant, manageable) supports robustness of certificates.

- Most systems (usually 3 - 2 vocational and 1 general) have separate certificates for vocational education - NZ is an outlier with one to fit everything.

- Models for designing upper secondary education include personalised systems (significant choice) (Australia, NZ, US of A), intermediate (Estonia, Poland, Denmark), structured (limited choice) (Austria, Germany, Netherlands, Switzerland).

- how vocational certificates are assessed shapes pathway opportunities. All external, all internal or both. Vocational certificates are generally all internal whereas general programmes are mostly all external. General certificates usually set by teachers and assessed externally. Whereas VET set by teachers and 'boards' and may be assessed both externally and internally.

- Achieving 'parity' across vocational and general certificates is not about just assessing the same skills and providing the same pathways, Vocational upper secondary certificates need to balange authentic assessment of practical skills and also providing pathways to higher education. Efforts fo create equitable pathways can result in close alignment to general certificates, masking the unique value and skills of vocational programmes.

- To meet the parity challenge, need to involve a range of stakeholders and assessment formats. Assess occupational skills through practical exams and workplace assessments. Involve professionals and employers through local juries or assessment boards to ensure vocational certificates represent the skills and knowledge of industry. 

Q & A followed. 

Balance between vocational and general is a challenge, fit for purpose and enable learners to be recognised for what they have developed through education. 

Modern certificates tend to recognise a wider range of skills, knowledge and attributes, beyond just the academic. Curriculum reforms tend to focus on how to accredit across the wide range. 

In NZ, autonomy from students and teachers is high. This is not the case internationally. Innovations and ways to meet the needs of learners by schools, means the consistency across NCEA is difficult to manage. Retaining this flexibility is still important. 

There is room for both external (exams) and internal assessments. The purpose of the assessments, and what is to be assessed should be the key to making the decision. 

Recognised the complexities of VET learning :) 

Good update on what is happening in the school completion certificates. There is no right/wrong approach, many challenges, and working through these require clear understanding of the purpose of these qualifications. 

- 

- 

Pedagogy, then AI

 From the blog of Medkharbach, comes a call to ensure that pedagogy comes first when using AI chatbots to support learning. 

Although the above assertion is commonsense to educators, it still requires careful consideration and an inclination to work through the many challenges inherent to bringing the 'pedagogy first' rule to the fore. 

The blog recommends drawing on two references to inform the work of educators and learning designers.

McTighe, J., & Wiggins, G. (2012). Understanding by Design framework. ASCD. - with a focus on teaching and assessing for understanding and learning transfer AND using 'backward design' for curriculum structure.

Zhang, Q., Zhang, N., & Lu, C. (2026). How do pedagogical approaches affect the impact of chatbots on learning performance? A meta-analysis and research synthesis. Educational Research Review, 51, Article 100783. AKA how to utilise chatbots effectively to support problem/inquiry/project based learning and collaborative learning.