Evaluating various ways to deliver multiple choice questions to mobile phones

I completed a set of comparative trials with a group of full time baking students some time back and analysed the evaluations this week. Multiple choice questions were texted to students using eTxt, on AMS forms generously provided on two Nokia phones from the Kinross group and via access to Moodle quiz on their computers.

Informal feedback already indicates a preference for the flexibility of having the questions delivered to their mobile phones instead of just being available on a desktop computer via Moodle. Several of the students were keen to have more questions provided so that they could revise their lessons whenever they had a window of time. This finding confirms the learning from Peter Mellow’s studytxt trials.

Unpacking the formal evaluations using an activity theory framework revealed some important aspects of user behaviour which I did not expect to find. If we view the completion of revision questions as the object (or objective) of the activity, the things that impact on the object include the students who will use the questions, the tools they use (mobile phones) and the signs they use (SMS & the use of ‘bullets’ on AMS forms & moodle multiple choice questions). The other things that contribute to the activity include the rules (trying to get the answers right), the community (the course the students are enrolled in) & roles (students as students, students as research participants etc).

The importance of the tool & the signs used seemed to be foremost in how the students evaluated the differences between answering the same questions via eTXT (SMS messages), AMS forms (multiple choice with ‘radio buttons’ loaded on to 2 Nokia phones) & Moodle (multiple choice with buttons but had to access the questions via the web).

Students went for the familiar. They expressed a preference for using eTXT because it was simple, they understood how to use it without having to learn a new phone system, they could answer the questions by texting the answers back whenever they had free time to do so.
As the majority of students did not own Nokia phones, they were unfamiliar with the menu system on the phones & found it difficult to find the questions. Answering the questions themselves proofed to be easy although a few students (both younger & older) needed to work through a few questions before they were comfortable.

The majority of students were on prepaid phones and did not want to access the questions on Moodle via their mobile phones. They were happy to hunt up a computer at CPIT to complete the questions but the convenience, just –in – time & nomadic aspect of mlearning then became redundant. Students had completed quizzes on Moodle before and were familiar with how to access & work through the questions. Even after I demonstrated how to access the questions via my Treo, none of the students took up the opportunity to also do the same with their mobile phones.

My learning from this is that the “learn an unfamiliar activity with a familiar tool” seems to hold true. The layout of multiple choice questions on AMS forms & Moodle is vastly superior to just receiving the question as a text message. However, the students made a choice for the familiar & by their perception, the cheaper options. So whatever we do from now on needs to be simple to use BUT also familiar.

Followup on AMS forms

Had a brief meeting with Ruth Bruce from the Kinross group a couple of weeks ago, to look at enhancements made to the AMS platform for putting forms on to mobile phones. Their newest addition is to allow ‘subscriptions’ to be held on a phone so that entry of common items does not need to be continually repeated but only adjusted as required. An example will be the use of an AMS form that has a partially completed sales order, this order can be updated in the field quickly by changing already entered data rather than re-entering all the data from scratch.

Ruth has suggested that we take up a 3 month trial of AMS. Met with Nick Ford yesterday to look at the possibility of aligning AMS to Moodle. We looked at the WSDL code provided on the AMS wiki plus the various other technical information provided. With input from our elearning networking / database guru, Kristian Thornley, we discussed possibilities and challenges along with the reasons WHY we would use AMS forms instead of what we currently now use.

From our discussion, we decided to run a trial to compare the three possible ways we could sent formative assessment questions to student mobile phones. These would be using the AMS forms, using SMS via eTXT and direct web download of quizzes from Moodle. We will evaluate time taken to download, cost of downloads, usability issues and student preferences. I will work on writing up a survey to evaluate these and fund the student downloads using our CPIT Foundation grant.

Customised forms on mobile phones

Attended a presentation at the local Vodaphone HQ by Ruth Bruce from the Kinross Group and Craig Wilson from Black Coffee software this morning on their Applications message service (AMS) software. AMS provides the facility for forms to be filled in via mobile phone and sent back to a web based front end from which form templates can be formatted / edited and viewed. Data collected on the forms can be sent, received and shared.

Basically, forms are formatted on the AMS front end. Forms can be set out to allow strings, dates, gauge (slider to allow for input), choice (multiple choice with options for only one or many options), and text box (will allow 160 characters). Forms are disseminated to mobile phones via download by each individual phone with access via username and password. The form framework lives on the phone and data entered each time is sent back to the company that requires the form to be filled in. Photos collected on the phones can be attached to the forms but it means that the messages will be more expensive. More info on the AMS wiki which includes user guide and quicktime movies of how to use the AMS.

There is an annual cost to use AMS based on number of uses, a message cost and the usual mobile phone date usage charges. If you want forms to be formatted for you, there is a one off cost for each form but the AMS seems to be quite straight forward to use and CPIT has the infrastructure to deal with any forms that are sent back in CVS or XML format and should be able to do the conversions ourselves. This platform has many potential uses and since launch 4 months ago has users in the real estate, health and equipment servicing sectors.

From the point of view of our current mlearning project, the AMS is a substantial improvement on using etxt . We will be able to sent out formative and summative questions in a ready to fill in format instead of sending out questions just as unformatted text messages and then students having to text back question number and answer. The question types we use can be increased as entry of answers in simplified using the various form filling formats provided. I can also see a use in using a short standardised from to be attached to each portfolio item that apprentices send in for their assessments that provide details of the photo / video / text file that can then be later collated towards their final eportfolio.

For mlearning in general, short pieces of content which are interactive (ie allows students to enter answers to questions etc) could be downloaded by students for revision sessions, extra information, assignments, formative assessments etc. Forms could also be set up for students who need to fill in reports while they are out on field trips, completing internships / work placements or for gathering of assessment evidence. So AMS will be something worth investigating as part of our ongoing pilot.