ATAIN - Human-led, tech-empowered: Evaluating educational AI tools for impact, ethics and pedagogy

 Notes taken from the monthly ATAIN session.

This month, Joan Sutherland and Associate Professor Trish McCluskey Director Digital Learning from Deakin University on 'Human-led evaluation process: learning and teaching.

Joan presented, beginning with a acknowledgement of country and emphasising the human in technology.

Asked for what tools are used and why are these used.

What are the challenges? Drew on Laurillard (2012) Teaching is Design - knowledge acquisition, collaboration, inquiry, production, discussion and practice to place emphasis on the pedagogy and not on the tool itself. This was to address what tools, when, what good practice looks like and where to get support. 

The initial evaluation surfaced the following: subjective view, lack of transparency, manual process, security risks, AI integration, workflow diversity, lack of alignment, duplication.

An explicit process required and aligned with institutional direction. Therefore, the current process was mapped; alignment and gaps identified; what does the research say? defined the ideal process; align and refine; how can automation be leveraged? 

Shared process that was initiated. A new request is raised, need analysis and alignment, research and tool selection, pilot and test, evaluation and recommendations, implementation, ongoing monitoring and support, and evaluation for license renewal. The process is visible to all staff at the university. Each stage is recorded and available so that progress through the evaluation process is shared. 

It is important to do the needs analysis is a human-led conversation to provide consistency, one source of truth, alignment, transparency and to avoid duplication. To identify gaps/duplication cover what challenges will the tool address? which activity types does the tool best align to (as per Laurillard and pedagogy approach) ? what key features expected of the tool? what specific tasks must this tool support? the tool is evaluated against a function and a feature matrix. 

AI is challenging as it is probabilistic and will not provide the same output consistency. Privacy and cybersecurity are added challenges. Cross cutting principles include privacy-by-design, data minimization, security by design and ethics and compliance. Therefore, context needs to be taking into account along with user experience, AI components (transparency, clarity, relevance, usability, integration to ensure strategic alignment and pedagogical impact.

Shared an AI tools heatmap evaluation to help users/staff understand all of the above parameters. 

Impact measured using SAMR (Puentedura, 2006), does it enhance and transform? However, it does not reflect the complexities of AI. Therefore, adding the layer of people, process, practice may help to provide a bit more representation for the wide range of AI.

A good overview of a process to evaluate technology tools that support teaching and learning. Q & A followed. 

However, my take is that perhaps we are trying to fit a problem into an existing framework. AI is especially tricky as it is what the user does, that actually defines its pedagogical role. AI is therefore not a tool but more of a 'social-cultural' partner. It can be used as a 'sage on the stage' giving answers, a 'guide on the side' to support nascent conceptualisation and learning, or 'a muddler in the middle' a socratic opponent, thinking partner etc. AI literacy for our teachers and then in turn our students is a major key to reach the potential use of AI to support teaching and learning. Learning becomes a hybrid of human+AI rather than a 'tool'. 

AI likely needs another evaluative process. 

Mysterious brain - Royal Society Te Apārangi presentation by Emeritus Distinguished Professor Cliff Abraham

 Notes from last night’s presentation organised by the Royal Society Te Apārangi on ‘Our mysterious brain: the making and breaking ofmemories’. 

The session is presented by Emeritus Distinguished ProfessorCliff Abraham - who worked for many years at the University of Otago.

After a welcome from the Royal Society and ‘housekeeping’, Professor Abraham is introduced through a background of his long research career.

Began with an overview of the parts of the brain, its structure and the function of various parts. Then the microscopic structure of axons, nodes (90 million ++) and the speed at which it works. Defined learning as the process by which experiences change the nervous system and hence behaviour (referred to as memories).

Summarised memory hyper-functionality during early development. Psychiatric disorders (psychosis, PTSD) and additions also bring about hyper-functionality. Memory hypo-functionality occurs and exampled by amnesia and dementia.

Mechanisms of memory storage through neural connections include dendrites receiving inputs and axons sending outputs. The synapse forms the connection between axon and the spines from the dendrite through neurotransmitters (glutamate, dopamine, serotonin, adrenaline, histamine) and electrical activity. Synapses are work-horses of memory, there are 1015 of them, they are metabolically active and individually modifiable. Astrocytes are also in the mix. All densely packed together.

Summarised the ‘making of memory’ through from the early 1900s. Long term potential (LTP) or depression (LTD) allows for modifiability (plasticity) of synaptic transmission. If the LTP is blocked, impairment of spatial memory formation occurs.

Therefore, does altered plasticity contribute to memory impairments in neurological disorders? Can knowledge drive new cognitive enhances and disease therapies.

Introduced Alzheimer’s disease and its effect on the human brain (lost of large amount of brain cells and inclusion of plaques and tangles). Causes neuroinflammation, synapse dysfunction / loss and nerve cell loss.

From a research and clinical perspective, understanding the process helps inform prevention/delay, lead to earlier detection, and towards more effective treatment.

Prevention involves ‘using it or losing it’. Good diet, weight control, physical exercise, good quality sleep, and keeping up with mental and social activity. Early detection of plaques with ‘PET’ scan. Blood tests being developed for early stage Alzheimer. Shared work at University Otago on ‘microRNA’ that may be a useful contribution.

Treatment – new drugs increase brain ‘tone’ – about ½ of the people taking these drugs experience modest improvement in cognition. However, effective treatment needs to be more effective, long-lasting, non-invasive, have a brain-wide delivery, cross the blood-brain  barrier, have few side effects and be affordable. Some progress presently on antibodies that remove plaques from the Alzheimer brain. Presently not widely available, very expensive and only deal to the plaque, not the tangles.

Stressed the importance of building brain resilience and maintaining cognitive reserve. Persons with brain resilience hold off dementia for longer. Shared a new treatment strategy to amplify production of neuroprotective proteins. How can these specific proteins in the brain be increased. Gene therapy approach alters cell function at a genetic level to generate a therapeutic outcome. For instance, to make more of its own neuroprotective proteins eg. sAPPa. Explained how virus-mediated gene transfer can be used. The gene encoding protein (sAPPa) is packaged into a virus shell (Trojan horse). This package is administered (injection – intracranial, intravenous). The gene enters the nerve cell nucleus, the virus shell dissolves and the protein enter the cell.

Trial of above through intracranial virus injection increased sAPPa rescued spatial memory in mice with Alzheimer. Also reduced plaque development in the hippocampus and the cerebral cortex. Future work on a human-ready virus shell. The hope is to increase neuroprotective proteins in the brain. 

An interesting foray into neurobiology and neuroscience. 

 

Future postdigital classroom - Special issue of Postdigital science and education

 The September 2025 issue of Postdigital science and education focused on the topic of the future postdigital classroom.  

The special issue was edited by I. Forsler, E. Bardone and M. Forsman.

There are 20 open access articles covering a range of topics. Each is provides interesting perspectives and provoke thought and trigger further investigations. Most are schools or higher education focused but frameworks and discussions are useful to inform VET.

- Walls come tumbling down: imaginaries and materialities of the postdigital classrooms by the editors sets the scene for the issue. The objective of the issue is to envision, discuss and critique the 'future postdigital classrooms' and on how technology impacts on teaching and learning. ChatGPT was used to generate images of a classroom without walls and found it wanting as it drew on the historical conceptualisation of classrooms with desks, chairs and whiteboards. The goals of the issues are to understand the dominant imaginings and narratives of a future classroom; how educators enact and negotiate these narrative; what methods or approaches can be used to imagine the future; can such futures be translated into the physical world; what are the boundaries of the classroom and how can the risks and promises be met or transcended?

Articles are then briefly summarised.

A 'pick and mix' of the articles for a first reading through recommended as there are many ideas introduced through the chapters. Then a focus on the pertinent ones will pay dividends. A recommended read. 

AI is not the answer, it is how it is used to support learning which is the key

 Some learning from chalkbeat, reporting on why the Khanmigo chatbot has not yet met its promised goals. 

Three years ago, an AI chatbot was launched in Khanmigo. Uptake by students has been lukewarm. The Khanmigo chatbot was used by students to find answers, rather that as a 'study buddy'. The key seems to be, as always, to educate students, rather than for them to just try things out. 

A recent German study to evaluate the relationship between AI-chatbots, student assessment performance and learning outcomes in Higher Education,  The findings indicate that students may be able to complete tasks faster, but not better or lead to higher grades.

Again, it is important to plan carefully, the objectives of using AI to support students. Are the chatbot/agents used to support learning? If yes, what are the specific learning objectives? Is the chatbot/agent 'ring-fenced' sufficiently to actually support the attainment of the learning outcomes? In our current projects, we need to be very clear about how and why the chatbot/agent is being utilised. This helps delimit the range of the chatbot and to ensure that it stays on task and does not deviate from its assigned objectives. 

Castlereagh statement on AI in education (Australia) - webinar of 'soft' launch

 Dropped in to the webinar to 'soft' launch the Castlereagh Statement - a cross-sector call for action on Australian education and training in the age of AI. The statement was developed through a series of cross-sector meetings/webinars/communities of practice. 

Notes taken:

Katie Ford welcomed the audience, welcomed all to country and provided an overview of what the Castlereagh Statement is and why it has been developed.

Draws together 80+ contributors from 30+ organisations including schools, universities, VET,  industry, government and also students. Brings together the many reports and research but no necessarily leading to action. Employer and society needs are shifting and pace of AI means it is difficult for education and training to keep up. 

Came up with 3 goals (what do we value in learning, a coherent learning pathway, and every Australian engaging with AI), 6 principles and where to next (near horizon - stabilise; intermediate medium - structural; and far horizon - new foundations).

Cross sector perspectives provided:

Bridget Pearce (Brisbane Grammar School - K-12) - Returned to the 'purpose of education'. At present mismatch between the needs of communities and society - collaborators, innovative thinkers, etc. but the curriculum crowds these out with its emphasis on meeting standards. Alignment needs to now be taken between the type of education required, and how schools can help students to get there. Teachers are not resistant to change but are resistant to bad change :) 

Helen Fairweather (Engineers Australia) - Important to have every sector involved with input into the statement. All participants were passionate about contributing. Engineers Australia does not dictate to providers, what and how engineering is taught. However, they are keen to ensure that Engineers have the competencies required to practice as engineers. Consistent framing across the sectors. However, no right/wrong answers. The statement provided the opportunity to consider the future of education. Accreditation does not seat outside within industry, it is a shared project with society and education. 

Claire Field (VET) - Shared the timeline with regard to guidance on AI. The schools had developed an Australian framework for Gen AI in Schools in October 2023. However, ASQA still in the process of drafting principles. AI is not just impacting learning and assessment, it is reshaping work across all industries, and VET serves diverse students and communities. VET perhaps behind as they did not (do not still) think that it will be impacted by AI (especially in the practical area). Unlike other countries, VET students are generally not encouraged to use AI. 

Adam Bridgeman (University of Sydney) - Universities siloed and are not likely to be aware of what is happening across other sectors and also internally - not much interaction between disciplines! Authentic learning will likely now require working with AI. Values and incentives to reward what matters is important. It is now not about the product but about what learning takes place. Pedagogy and how this is understood is a key towards leveraging AI. 

Peita Davis (Business Council of Australia) - what do businesses expect from graduates? Strong AI literacies and human skills are the key - teamwork, communication, critical thinking. Young people are seen to bring AI into small businesses! However, this is not the case :( no urgency across government towards this, AI literacy for all in Australia. What can be done? The statement is good first step. Clear guidelines and next step is to have AI literacies embedded from early childhood onward.

Tim Burt (Future Skills Organisation) - Provided a higher level, governmental perspective. AI skills are being defined - what are the skills Australia to compete in the Ai economy? Do we have the skill? What is needed to mobilise the nation around AI skills? and how will we know if initiatives will be effective? Common themes between Castlereagh and the study. Human skills fundamental; coherence required across all of education and complemented by industry feedback? How can AI skills development be coordinated? 

Danny Liu presented on 'next steps'. Jason Lodge reiterated that all the sectors/industry etc. coming together provides a way forward, which is cooperative. Therefore it is a coalition of the willing.

Monthly working party will meet between now and October to create a White Paper - cross sector and sector-wide working parties to formulate specific action plans, collect and surface vignettes of practice, and make progress visible. Items include IT enablement, curriculum and alignment and credentialling and assessment.

Q & A ensued. Topics included ensuring all states included; ways to ensure industry/business included; and how to ensure what is drawn up have a voice with policy makers and government. 

 

AI in education - reading the literature with a critical eye

 The numerous publication around AI in education has made it somewhat challenging to keep up with. My Google Scholar alerts collates around 10 articles every day. Other sources of reading include recommendations via various LinkeIn posts, blogs I follow, news articles and other media, and alerts from various journals. There is therefore an avalanche of material on the topic!

However, there is still a distinct gap in work from the vocational education sector. The book 'Artificial Intelligence in Vocational Education and Training' which I edited, has had over 4000 downloads. A favourable book review in the International Journal of Training Research has helped with bringing the issue of applying and leveraging AI into the specialised context of VET. More can be done, and I have put forward another proposed book to publishers, to record how the sector rises to the challenges posed by AI. VET cannot ignore AI, given its primary goal is to prepare its graduates for the world of work. All VET needs to cover Core AI literacies to ensure that VET graduates lead their industries into the ethical, principles and careful use of AI across occupations. 

There is also a need to draw on the current corpus of work on AI in education. Code Acts in Education blog warns of being circumspect with regard to publications, often spread through the blogsphere and other social media. There are also many pre-print articles being disseminated through various channels which are in the process of peer review. Recent recantation of articles due to faulty methodology also raise flags. The blog is worth reading through for the many ways in which fast, rapid publishing, is detrimental to the credibility of sound research.

Therefore, it is important to carefully evaluate articles, be patient and wait for articles to have cleared the peer review process and be published, and keep an eye on article retractions! 

Rebecca Frankum - NZ Vocational Education and Training Research Forum (NZVETRF) master class - on school leaving certificates

 Attended a 'masterclass' or fireside chat organised by the NZVETRF presented by Rebecca Frankum, Transitions in Upper Secondary School Education, OECD Secretariat. A recent report 'the theory and practice of upper secondary certification' authored by Hannah Kitchen and Rebecca, forms the basis for the discussion. 

Notes taken from the presentation/discussion:

Josh Williams from Skills facilitates the session. He began with an overview of NZVETR, the background to this session (the change of NZ school leaving qualifications) and introduce Rebecca - who presents from Paris.

The presentation worked through the report, with an emphasis on some of the implications and applications within the NZ context.

- Began with the importance of upper secondary certficates as it accreditates them towards the next step in their lifes. 

- Study looked into how upper secondary certificates were structured. did they incorporate and assess a broad range of skills? reflect the diverse skills and strengths of all students? enable progression to the students' next steps.

- Analysis of upper secondary certicates (71 certificates, 38 systems) on nature of assessments, who marked, what did they include, was there activity within unseen questions/tasks? allow for natural occuring evidence.

- 3 categories - certificates that include external exams, but no internal assessment; certificates include internal but no external exams; and certificates that include both (NZ NCEA is an example).

- exams still have an important role as they assess complex skills like analysis, evaluation and creating and include a range of problems, sources and multimedia materials.

- Higher education entrance exams - same exam papers, consistent marking, consistent standard of difficult, same exam conditons. Useful as 'gate keeper' - certify knoweldge, understanding and skills and facilitate selection.

- Wider range of exam formats support validity but can be hard to design and some skills (practical, social. emotional, higher order ectc) hard to assess.

- Balancing assessment approaches (fairness, credibility, relevant, manageable) supports robustness of certificates.

- Most systems (usually 3 - 2 vocational and 1 general) have separate certificates for vocational education - NZ is an outlier with one to fit everything.

- Models for designing upper secondary education include personalised systems (significant choice) (Australia, NZ, US of A), intermediate (Estonia, Poland, Denmark), structured (limited choice) (Austria, Germany, Netherlands, Switzerland).

- how vocational certificates are assessed shapes pathway opportunities. All external, all internal or both. Vocational certificates are generally all internal whereas general programmes are mostly all external. General certificates usually set by teachers and assessed externally. Whereas VET set by teachers and 'boards' and may be assessed both externally and internally.

- Achieving 'parity' across vocational and general certificates is not about just assessing the same skills and providing the same pathways, Vocational upper secondary certificates need to balange authentic assessment of practical skills and also providing pathways to higher education. Efforts fo create equitable pathways can result in close alignment to general certificates, masking the unique value and skills of vocational programmes.

- To meet the parity challenge, need to involve a range of stakeholders and assessment formats. Assess occupational skills through practical exams and workplace assessments. Involve professionals and employers through local juries or assessment boards to ensure vocational certificates represent the skills and knowledge of industry. 

Q & A followed. 

Balance between vocational and general is a challenge, fit for purpose and enable learners to be recognised for what they have developed through education. 

Modern certificates tend to recognise a wider range of skills, knowledge and attributes, beyond just the academic. Curriculum reforms tend to focus on how to accredit across the wide range. 

In NZ, autonomy from students and teachers is high. This is not the case internationally. Innovations and ways to meet the needs of learners by schools, means the consistency across NCEA is difficult to manage. Retaining this flexibility is still important. 

There is room for both external (exams) and internal assessments. The purpose of the assessments, and what is to be assessed should be the key to making the decision. 

Recognised the complexities of VET learning :) 

Good update on what is happening in the school completion certificates. There is no right/wrong approach, many challenges, and working through these require clear understanding of the purpose of these qualifications. 

- 

-