CRADLE 2020 - keynote #2 with Professor Monika Nerland

The afternoon sessions were on moderation with focus on voices of sessional academics and support for disabled assessees. Rapid orals were then presented and these covered a range of topics. I did not take notes on these when too quickly.

The closing sessions begin with a keynote from ProfessorMonika Nerland who is Professor of Education at the University of Oslo. She speaks on ‘Reconfiguring relations between higher education and work: Knowledge practices and learning in the (post)digital era’.

The presentation concentrates on how knowledge practices have changed in the workplace and in turn, this impacts on how the processes are recognised as complex work. Examples from professional learning used to explain the various perspectives. Began with an overview of HE and digitalisation. Much attention now focused on digital platforms for learning and communication. Within work, specialised digitalisation processes are now advancing rapidly. There is therefore an extension and complexity of the possible learning spaces which are distributed across activities and resources. Less attention to the digitisation and its effects on work. Yet, as work shifts, work tasks also shift and these require attention and study.

Discussed the concepts of expert communities and how they are characterised by distinct knowledge practices. Professional expertise tends to be recognised by fellow-experts. May rest on different forms of knowledge and ways of knowing that are applicable in different work contexts. Ways of developing, enacting and recognising expertise is embedded in the distinct forms within specialised communities. The actors and networks at play are variable and specific to occupational specialities.

Epistemic practises is the knowledge which is generated and shared in a given field of practice. Includes the socially and interactionally accomplished ways that  the group communicate, justify, assess and legitimise knowledge. They play a crucial role in making knowledge ‘actionable’ and are required to help assemble, translate and represent practice.

In the post digital area has increased epistemic complexity. There is more abstraction in how knowledge is visualised to allow for computation thinking and automation.  This makes it difficult to access these knowledge domains.

Challenges to professional expertise includes the working with advanced technologies and representations as demanding’ increased need for systems understanding; extended responsibilitie for mastering knowledge; and the need to also establish and manage social relations both internally and externally of organisations.

Provided examples from an ongoing project (CORPUS) to study the changing requirements in public services due to the consequences of digitisation in general and specialised occupations. For example, a study of the introduction of ‘smart care tech’ into the health sector’. How is this new process/system developed and how does it evolve as it forms their new professional roles and responsibilities and a new approach to work. Requires system understanding to bring in the many ‘actors’. Listed a few research challenges – access to observing and participating in knowledge practice as a learner; develop specialised expertise; and navigating complex information environments and constructing productive spaces for learning.

So what is the role of HE in supporting the new demands on learning. Need to enrol students in evolving professional knowledge culture – to providing access to and participation in epistemic practices help to make the profession’s ‘machinery’ of knowledge construction more transparent. With learning an ongoing process of becoming rather that predicated on one end point.

Provided case studies in legal, teacher and software engineering education through design of inquiry tasks to help learning identified epistemic practices (learning outcomes) with professional knowledge sources and the ‘transformative’ means for getting there (i.e. signature pedagogies).

Another way is to attend to students’ navigation and ways of constructing spaces for learning – i.e. moving between educational and professional environments. Also a need to keep pace with an evolving field by developing productive relations between sets of epistemic practice in education ad work and to reconfigure educational arrangements, roles and responsibilities to move beyond the education-work divide.

Conclude with a call to move beyond the education – work dichotomy and to understand that professionalism is not a given. It needs to be continually worked on.

The conference closes with the launch of the book ‘ReimaginingUniversity Assessment in a Digital World’ by Emeritus Professor Beverley Oliver (Deakin). See brief notes on book here. 

CRADLE 2020 - Day two 'morning' presentations

 

Day two begins with presentations on assessment in a post digital world. The presenters are Associate Professor Phillip Dawson, Associate Professor Rola Ajjawi and Professor Margaret Bearman (researchers from CRADLE at Deakin). The discussant is Professor Michael Henderson from Monash University.

The symposium began with Michael acknowledgement of Country. Each of the presenters shared their work – which are chapters in the book – reimagining university assessment in a digital world.

Rola begins by presenting on ‘assessment as portrayal and the strategic negotiation of persona’. Began with the impact on digital technology on work – permeates everything, blurring professional and private life, careers becoming more fluid and greater portrayal of digital identities. Assessments are for certification, guide learning, help with employability and learning how to make evaluative judgements. Completing the qualification recognised with award and transcript but does that portray who the graduate is?? Proposed assessment-as-portrayal to enable students to portray their achievements in public ways, communicate directly with others. Examples include the use of graduate learning outcomes through portfolios/credentials etc. Another pathway may be through persona studies used by celebrities. Persona can be considered as a strategic identity (Marshall & Henderson, 2016). Five dimensions of persona include the public, mediatised, performative, collective and intentional value and reputation. Reimagining assessments using persona studies involves introduction of a subjective dimension to assessment practices, a rethink of how we judge the quality of portrayal for future work; and encouragement of student agency. Concluded with some important questions on how portrayal may work.

Margaret covered the topic ‘assessment for an AI-mediated world’. What does AI do well? – identify patterns more accurately than humans, faster and at scale. However, this is a myth. Automated essay grading is available but restricted to appropriate topics and tasks. The systems are bounded by the quality of the original data set! If a machine is able to grade a task, then it can do the task. Therefore assessments need to be centred around unique human capabilities – evaluative judgements, deployment of meta-intelligence (i.e. knowing about how you know). Assessments that promote EJ is not just about identifying but the ability to construct their own sense about quality withing a unique practice environment. Assessments can promote EJ through cycles of self, peer and educator assessments. Assessment should develop personal epistemology. Deliberately call attention to the EJ process. Use example rubrics to form powerful epistemic artefacts. Ask learners to question the role of rubrics (Bearman & Ajjiwi, 2019). Assessment tasks should prompt students to answer questions on – what qualities make this a good piece of work? If I were to set the criteria for success, what will it look like? How do we know we have achieved?

Phillip presented on ‘surveillance and the weaponization of academic integrity’. At the moment, Higher ed (HE) assessment has a surveillance culture, this may be OK but we should fear the weaponization of surveillance on academic integrity’. Drones, webcams and other tools collect data on student learning. Argued that presently, a surveillance culture pervades society and this is seen as acceptable by many ‘for our own good’. Surveillance culture’s effect on assessments include the creation of a culture of distrust; frames students as being in need of monitoring; students and their data become a product; academic integrity becomes a game of submitting to or subverting surveillance; and there eventuates an acceptance of the surveillance. There needs to be a balance between surveillance and academic integrity. Assessment security may be adversarial, punitive and evidence based but academic integrity is positive, educative and supportive.

 

Session one DAY 2: Learning through and for work is with Associate Professor Gloria Dall’Alba from the University of Queensland. She presents on ‘toward a pedagogy of responsive attunement in educating for the professions’.

Rhea Liang introduces the session.

Placed the presentation in the current context, where ‘fake news’ is of concern. Society relies on truthfulness. When fake news disrupts the standard discourse, it causes frustration and disillusionment. Proposes responsive attunement as a means to ensure professional are able to evaluate information in a studied and evidence based approach. Therefore, it is important to ensure our ‘way of being-in-the-world’ (Heidegger, 1962) is anchored in the ability to evaluate and respond to information. Advocated for ‘the capacity to care’ (Noddings, 2005) as a mark of personhood. Need to listen to the multiplicity of perspectives, and to be able to assess their importance. Learning incorporated what we know and do (epistemological) but also how we are learning to be (ontological). Being attuned allows professionals to work through complexities and ethical dimensions. Responsive attunement is responding on the basis of tuning in to others and things, with reflexivity. Responding includes sensate, motile, emotional – i.e multimodally. Education needs to begin the process so professionals are prepared for the challenges posed at work. Implications summarised as – highlighting tuning in and responding, focus on programme goals, reward in assessment, address embodied learning, have well-integrated program and ensure epistemological and ontological dimensions addressed.

Then a session with Dr. Emma Scholz from Charles Sturt University on ‘Professional identity development : Implications for sustainable assessments’. Presents on work with Franziska Trede.

Emphasis on professional identity development in the workplace and the bringing in of Bloom’s taxonomy to assist the process. Presented the concept of locating professional identity in a practice theory framework – in the context of first year of practice for new graduate veterinarians. Research focus needs to be grounded in ‘individuals in their social practice’. Detailed sequential analysis process to thematically analyse interviews and field notes. Themes than taken back to the data to construct key stories. These underwent dialogic analysis to provide for the narratives. Theme on practice capability, the learning professional, striving to become a capable practitioner, and mobilising criticality within capability, shared with interview vignettes. A screen shot of a key story presented. Proposed implications for practice.

CRADLE conference 2020 - Session presentations

 

Today’s sessions for CRADLE conference 2020.

Most of the sessions have two presentations. I attend the following:

Then the first session themed ‘learning in a digital world 1’ is with Dr. Adam Brown and Emily Wade from Deakin. They present on ‘Serendipity, imperfection and vulnerability: Harnessing live video for ‘authentic’ teacher performance.

Adam began with providing the theoretical background and approaches.

Proposed the undertaking of ‘the meddlers in the middle’ as the role of the educator as opposed to ‘stage on the sage’ or ‘guide on the side’. Meddlers in the middle reposition the teacher and student as co-directors and co-editors of their social world.

Need to be ‘authentic’ and that authenticity is an effect, not an essence in the context of self-presentation as being performative.

Emily introduced collaborative autoethnography and how digital platforms like Periscope may be used to enable collaborative work. Explained how this was used for their own teaching context and shared reflections on the process.

Very much an academic look into their teaching and a critical reflection into how video impacts on, supports and extends their teaching practice.

 

Session 2: Learning in a digital world 2.

This session chaired by Simon Knight.

On ‘Development of educational tools that enable large-scale ethical empirical research on evaluative judgement' with Dr. Hassan Khosravi from the University of Queensland. Presented on work carried out with others – George Gyamfi, Dr. Barbara Hanna and Dr. Jason Lodge.

Presented on evaluative judgement (EJ), educational tools, conceptual models and examples.

EJ is the capability to make decisions about the quality of work of oneself and others. This skill helps students use feedback effectively, develop expertise in the view and attain autonomy.

 EF revolves around rubrics, self assessment exemplars, feedback, reflection and peer assessment – mostly theoretical.

Educational tools include peer grading and evaluation systems. Most build without the aim of supporting research and they do not allow data harvesting to be undertaken or the set up of controlled experiments. Draws on work of Associate Professor Paul Denny (University of Auckland) on PeerWise and with Professor Neil Heffernan on the ASSISTments ecosystems.

This presentation on the conceptual framework of RIPPLE. Conceptual model described to help promote both EJ and to undertake research on it. System developed for adaptive learning, learnersourcing and peer grading and feedback with EJ strategies of rubrics, self and peer assessment and exemplars. Connected to metrics, experiment design and ethical guidelines and a data repository.

Provides details of the RIPPLE platform.

Shared one study – Can students create high-quality resources? Walked through the process of students creating and evaluating each other’s questions.

The shared a case study and reflections. In general, students’ EJ improved through iterations. Students tended to provide higher ratings when compared to instructors. Found that the rubric criteria may not suit the ‘assessment’ and re-tested new rubric. 

Shared plans for new investigations and encouraged other researchers to make contact to use the tool.

Followed on with George Gyamfi, also from UQ on ‘The effect of rubrics on evaluative judgement: A randomised controlled trial'. As per above presentation, George’s work is completed with Dr. Barbara Hanna and Dr. Hassan Khosravi.

Defined EJ as per previous presentation. Shared work on rubrics (Reddy & Andrade, 2010 and several others) and how these may be used to enhance student learning. These studies tend to be mostly theoretical and does not bring in the perspectives of students.

Overviewed the research methodology and how RiPPLE was used for the study. Participants were undergraduate students (n= 354) learning database principles. The study involved having a control group which completed peer assessments without a rubric and the treatment group which completed peer assessments with a rubric.

Findings indicate rubrics can be a way of influencing how students attend to quality and can impact students’ judgement in assessing the quality of learning resources. However, construct of the rubric is the key! Even without rubrics, learners are able to make judgements but rubrics provided better clarity.

Session three: Assessment of learning 1 :

Session chaired by Zi Yan.

Dr. Sin Wang Chong from the Queen’s University Belfast presented on ‘Student feedback literacy as an ecological construct’. A conceptual look at feedback literacy covering feedback orientation to feedback literacy, feedback literacy in and beyond the classroom and a reconceptualization of the concept. Presentation based on paper published earlier this year.

Shared study on feedback orientation (London & Smither’s 2002) – utility, accountability, social awareness and feedback efficacy. Also Kremmel & Harding (2020) on language literacy – using a spider chart to provide each learners’ literacy. Then Sutton (2012) in conceptualisation of feedback literacy as epistemology (knowing), Ontological (being) and practical (acting). Carless and Boud (2018) appreciate, make judgments, manage affect and take action on feedback. Most recent Carless and Winstone (2020) looking into both teacher and learner feedback literacy.

Feedback literacy is not only a product (Hattie & Timperley, 2007) but also a process (Interpersonal and intrapersonal (Carless & Boud, 2018).

Chong reconceptualised using a ecological perspective (2020 paper) to include the context within which feedback is undertaken (interpersonal, textual, instructional, sociocultural) and the learner’s beliefs, goals, experiences and abilities. Therefore the engagement dimension (as proposed by Carless and Boud – understand feedback, manage affect, make judgment and take action) has added to it the context and individual dimensions(

Introduced the ecological systems theory (EST) as nested or networked EST with micro, meso, exo, macro and chrono systems. Used example from a paper in preparation – process of scholarly peer-review – to explain the framework. Proposed actor-network theory as a means to understand better the inter-connections/inter-relationships of ESTs.

Concluded with implications including how the framework may be applied towards better understanding the contextual and individual contributions to understanding feedback literacy.

The next session is with Dr. Akilu Tadesse from the University of Bergen on ‘Scaffolding feedback in complex dynamic system context: Effect of online interactive learning environments’. Presents on work undertaken with Professors Pai Davidsen and Erling Moxnes. 

 Presented on the 'problem', covered the notion of scaffolding feedback, application of this study and findimgs.

People, even experts, have difficulties in understanding and communicating their understanding of complex dynamic systems. Also difficult to measure improvements in this domain when CDS are difficult to understand.

The study looks into how to enhance students learning of CDS by developing educaitonal feedback to scaffold feedback that supports students learning.

Shared the notion of scaffolding feedback as a continual spiral of building knowledge. Gaps of knowledge are 'filled either by students' own efforts are through support from 'external agents'. 

Used this scaffolding feedback notion to integrate into a personalised and adaptive online learning environment. Learner is presented with a CDS and supported to progress through a sequence of learning activities to attain CDS. 

Shared a 'case study' to help explain the concept of how the platform works. Then presented the research questions - to find out if scaffolding feedback would reduce the gap between existing high and low performing students. Averaged results across 5 tasks. In general, the gap did close as tasks (increasing in complexity) were worked through. 

Overall, a good range of presentations showing the ways digital technologies may be useful in supporting assessments of learning. 

 

 

 

 

 

CRADLE conference 2020 - Welcome and Keynote #1 - Associate Alyssa Wise on learning analytics

 Attending the sessions for the CRADLE (Centre for assessment and digital learning) conference today and tomorrow. There is a two hour difference in time (NZ is 2 hours ahead) so It will be a late finish today 😊

The conference begins with a welcome from Professors Liz Johnson and David Boud, Deakin University. First CRADLE conference (over 900 participants!!) but the fifth year CRADLE has been running. Professor Boud provided background and rationale for the conference. Professor Johnson provided formal welcome to country and to the conference.

The first keynote (introduced by Phil Dawson) is with Associate Professor (of educational technology and learning science) Alyssa Wise from New York University on ‘The data intensive university in a post-digital world: New directions in teaching and learning.

https://steinhardt.nyu.edu/people/alyssa-wise

Defined learning analytics (LA) as the use of data science methods to generate insights into teaching and learning that lead to direct, impactful action. Important to use LA purposefully to inform how learning is designed, developed, supported, taught etc. Argued that data informed decisions are more important than ever given how online, blended, hybrid etc have moved to mainstream overnight, no longer digital vs traditional.

Shared recent survey across large number of universities. Unfortunately, pre-covid online better experienced than post-covid ‘remote learning’. However, experiences did improve (slightly) over time. Clear structure for learning important but needs to be balanced with flexibility. In general, videos of ‘lectures’ positive but online lectures themselves had mixed reviews, along with online discussions. Important to provide ‘active engagement’ for students.

How can learning analytics help? Begins with instructors design with intention to engage learners, who generate data, provide analytics, read as human insights and feed back into instructors design.

Data can be catergorised as activities, artifacts and associations. It is not only the data, but how is it analysed and applied? Sense making from analytics need to be supported with pedagogical response.

Provided a series of examples. Are my students preparing? When are my students preparing? Are my students engaging? Collected data on activity and used this to inform the  ongoing development of activities to increase engagement. Qualitative data also important – for instance to find out ‘who are my students engaging with? Where is the student spending the most time (asynchronously)? What does the student engage with – content related or non-content related discussion posts?

For instructors, important to find out how they are going. In general, professor responses which are to all levels of queries, providing coahing and supporting and social presence cues have greater engagement with professors who only respond to ‘top level queries which are straight forward.

Another example of how to find students who are struggling using diagnostic assessment with LA used to build a better model of how to identify and support students. However, relationship support (i.e. caring tutor) still most effective!

Interrupted by phone call!! So did not get to follow the last 5 minutes and the Q & A.