More about forum participation in education

Another random set of notes about forums I’m afraid this week, but we’re close to finishing the paper that I’m co-authoring so normal service will be resumed shortly.

Initially, on-line forums were offered in addition to print-based correspondence courses, and were, alongside email and web-based articles, considered optional “so as not to reduce access to students without internet or computer facilities”  (Bates, 2008)

The 2002 paper by Wu and Hiltz sets out possible benefits of forum participation in education:

Online discussions that persist throughout the week should motivate students to be more engaged in their  course on a continuous basis […] Secondly, active participation in online discussions, which are student-dominated rather than instructor-dominated, should be enjoyable for the students. It should  make learning more active and “fun.”

To test these hypotheses, they surveyed 116 participants in three face-to-face courses (two undergraduate  and one graduate) for which active participation in forms was a requirement of the course. It’s important to note trial was observational, not a randomised, controlled trial, and the surveys tested perceptions of learning rather than testing learning itself. However, they concluded that students did find that the asynchronous discussion afforded by forums did make the course more enjoyable and increase motivation. They also discovered that the amount of previous experience with distance learning courses didn’t appear to affect how enjoyable or motivating student found the on-line discussions on the observed courses. The importance of the instructors’ involvement in setting topics for discussion, offering feedback and guiding discussion was highlighted by the students’ responses, with one saying that instructors should be “online for for two to three hours every day.”

Two years later, Biesenbach-Lucas (2004) put forward her interpretation of the benefits of forum participation (particularly in teacher training) :

Positive interdependence: Students organize themselves by assuming roles which facilitate their collaboration.

Promotive interaction: Students take responsibility for the group’s learning by sharing knowledge as well as questioning and challenging each other.

Individual accountability: Each student is held responsible for taking an active part in the group’s activities, completing his/her own designated tasks, and helping other students in their learning.

Social skills: Students use leadership skills, including making decisions, developing consensus, building trust, and managing conflicts.

Self-evaluation: Students assess individual and collective participation to ensure productive collaboration.

Her paper only expects the instructor to act as “Observer/evaluator, perhaps some participation” however, she admits that, over the course of her five semester experiment with forums, the instructor carried over more and more outputs from the forum into face-to-face sessions.

Vonderwell, Liang and Alderman (2007) explored asynchronous online discussions, assessment processes, and the meaning students derived from their experiences in five online graduate courses, and concluded:

Educators need to look more carefuly into the notions of “assessment for learning” as well as “assessment of learning.” online learning pedagogy can benefit from a notion of “assessment as inquiry” and “assessment of constructed knowledge” in asynchronous discussions.

Kearns (2012) offers a reasonable summary of the challenges including the sheer number of posts that might need the instructors’ attention:

One problem arising from the asynchronous nature of online discussion is the impact of late posting. For a discussion that runs from Monday to Sunday, for example, students in the discussion group may miss the opportunity to fully engage if some wait until Saturday to begin. On the other hand, even in classes where discussion is sometimes less than robust, students may face the challenge of having to keep up with voluminous postings across multiple groups and discussion forums. As one of the participants pointed out, “Sometimes it’s hundreds of entries.” […] A recurring theme among instructors who participated in this phase of the study was the amount of time and effort involved in providing effective feedback to online students. One source of demand was online discussion. Several instructors reported being overwhelmed with the amount of reading this required. As one instructor remarked, the discussion board became “cumbersome when done every week.” Another demand on an instructor’s time that was raised was having to enter comments on student papers using Microsoft Word rather than being able to handwrite in the margins. For one instructor, this was “time consuming” and “more tedious” than annotating the hard-copy assignment. One instructor mentioned needing a greater number of smaller assessments to oblige students to complete activities that might otherwise be completed during F2F class time. In her words, “If there is not a grade associated with an assignment, it is completely eliminated.” Finally, several instructors commented on having to answer the same questions more than once in the absence of a concurrent gathering of students.

Mentioning peer assessment as as useful strategy in coping with this challenge, she cites Yang and Tsai (2010) which relates an interesting experiment with peer assessment, finding peer reviewed marks reasonably comparable with an those of an external assessor, and measuring teh impacts of the students on perceptions and approaches to peer review. Though in the context of MOOCS, I’m not looking at this stage for a robust marking procedure, I am interested in peer review as a way of tagging posts so that they can be used to create procedural or semi-random narratives.

So there may be lots of analysis of that challenge, and some useful ideas in the packed paper by Meyer (2006). For example:

Many research studies do not use multiple raters to code the content in online discussions. This may be  owing to a number of factors, including the instructor’s preference for working alone or a lack of interested colleagues to help with coding. Researchers may not have the time to train other coders or the  money to pay them, or perhaps the aim is simply to collect data about the learning of a given set of  students rather than to produce reliable findings. In other words, there may be understandable reasons for not using multiple raters, despite the greater reliability that might result from their use.

Crown sourcing the rating, from other students, in peer review, may be an answer.

From Roman Portus to Medieval Bodiam – virtually

Today I had a meeting with brothers Joe and Ken Rigby. We met in a faux-medieval world, of the sort familiar to players of Skyrim, World of Warcraft and many (many) others. I’d arrived as a woman, so Joe helped me find a more masculine avatar, then a quick tutorial in walking, running, flying with a rocket-pack, and we were off exploring.

Joe is convinced there’s a market in building historic environments in the Unreal engine, and he and Ken have built a few proof-of-concept environments, including (and of particular interest to me) building five at Portus, and Bodiam Castle.

Once I was comfortable manipulating my avatar, Ken replaced the game-world we were in, and loaded Portus on the server. Joe and Ken got a model of building five from my colleagues at Southampton, and put it on a model of the Trajanic basin. Walking through it (or rather directing my avatar while we talked) I was immediately impressed by the sense of scale, if not by the somewhat oppressive sky texture they chose.  We talked about how, with enough server space, you could invite a lecture group to the model, and talk about the research and interpretation behind it while leading a group of avatars around it.

Here’s a video walkthrough Joe made previously:

Of course I was thinking about the Portus MOOC – but immediately I could think of challenges. For a start the environment sits on a sort of commercial virtual world server run by US telecoms company Avaya.  Joe explained they had a very reasonable price-plan, for smaller meetings. But even though in theory 2000 people could visit at once, Joe said the server fees would be prohibitively expensive. On top of that of course, MOOCs are inherently asynchronous, so without huge amounts of planning, many people would miss out, and possibly feel deprived. But regardless I asked whether the lecturer could change the appearance of the models as s/he  discussed the various theories behind them. After a bit of thought Joe said, although the models themselves couldn’t change on the fly, they could build a sort of “TARDIS” (yay, Doctor Who back tomorrow) that could transport the group between a number of models, or (obviously) through time to show different stages of Portus’ development.

Then we went to Bodiam, and arrived in the courtyard of a Bodiam Castle far less ruined than the one I know. Joe explained that they had a model of the Great (dining) Hall, created by a PhD student, and were thinking about how to build a simple model of the Castle around it, when the found exactly the model they were looking for on-line, available for £50. So that’s what we explored, but the only detailed interior was the Great Hall.

I must admit, though the smoothness of the experience was a lot more accessible than, say, Second Life (though maybe that’s because of the speed of my Broadband) I can’t think of a sustainable business model for environments like this. Build it and (maybe) they will come, but beyond these experiments, where is the reward for building it? Will visitors pay to visit a virtual Bodiam, or would they prefer to go to the real thing? Would my organisation (the National Trust) pay to have a virtual Bodiam accurately modelled? Who for?

Millions  of people (probably) have paid to tour a virtual medieval Florence in Assassins’ Creed, but they came mostly for the killing and the treasure – Florence itself was a pleasant extra.

I DO think virtual environments like this could benefit things like the Portus MOOC, but MOOCs ain’t cash cows… and Second Life lies in (relative) ruins, as do many other Virtual world platforms.

This one is free to visit though, and Ken and Joe have agreed to leave Portus on it for a while. Make sure Flash is up to date and click on this link to visit. You’ll need to download an Avaya extension, but its a painless process. If you see anyone there, wave by pressing the 1 key on your keyboard, and if you have a microphone attached, talk to them.

Creative Commons and Cultural Heritage

Creative Commons is something I’ve had in the back of my mind since joining the NT. This is a useful article…

Other Voices

I am very pleased to present a post and resource links on Creative Commons by my colleague Jason Baird Jackson.  More and more cultural heritage professionals and students are faced with questions about how to best present original documents for public access and the proper citation and use of internet files.  Jason provides a solid introduction and valuable links to Creative Commons licenses that are relevant today and will be increasingly important in the immediate future.

Creative Commons and Cultural Heritage

by Jason Baird Jackson

Do public archaeologists, heritage professionals, museum practitioners, and graduate students need to know about the Creative Commons? I think so. Robert Connolly does so as well, which is why he thought to ask me to contribute a short note to his blog. After you have learned a bit about it, I hope that you too will see the relevance of the tools provided by the…

View original post 636 more words

Reading about forum participation as a component of on-line learning

I’ve participated in two MOOCs so far, one through Coursera and one through FutureLearn. One difference between the two platforms is the use of Forums.

In the Coursera course on Statistics, the forum is presented as an add-on, a tool that was available to students who wished to interact with other students, discuss concepts raised, offer feedback on the course and, especially, seek help with the weekly assignments that were the main form of assessment during the course. But the forum didn’t feel part of the course, and there was no evidence that my participation on the forum was being, either formally or informally evaluated.

On the FutureLearn course, each learning element came with its own forum built in, and students were actively encouraged to submit work on the the forum, and to comment on each other’s submissions.

Neither course ended with any form of certification, so any evaluation of the student’s work on the FutureLearn forums was informal, but there was a more of a sense of the course team taking an active interest in how student participated in forum discussions than on the Coursera course.

With a growing number of courses delivered wholely or partly on-line, and in particular expansion of Massively Open On-line Courses (MOOCs), new models of student participation and evaluation have developed. One such model is the use of discussion forums. A development of the Bulletin Board Systems of the early internet, forums can be described as an asychronous form of conversation that uses type. Forums are often archived, at least temporarily, and the course of the whole conversation can be viewed at any time, which distinguishes discussion forms from other typed conversations such as Internet Relay Chat.Discussion forums are often a component of Vrtual Learning Evironments (VLEs) like Blackboard, etc.

Moodle is an open source virtual learning evironment first released in 2002 and used by a number of insititions worldwide (including, for example, the Open University) to deliver on-line education. It was originally developed by Martin Dougiamas who (for example in Dougiamas and Taylor, 2002) is a proponent of social constructionist pedagogy. Lewis (2002) is a much cited study that was one of the first to use a randomised trial to evaluate the effectivness of discussion forums as a learning tool. Although inconclusive on the main question, one new hypothesis raised was that “online group discussion activities must reach a certain level of intensity and engagement by the participants in order to result in effective learning.”

Indeed Hrastinski, S. (2008) is concerned that asynchonous online conversations can be difficult to get going if too few students participate. However, when they do succeed, Hrastinski offeres evidence that asynchonous conversations stay on-topic for longer, give students more time to reflect on complex issues, and allow students from different time-zones, and with different time commitments, to participate.

Given these advantages it’s no wonder that on-line course designers want to include discussion forums in the toolset that they offer to students. But if the forums are to be an effective learning tool, students must be incentivised to participate. One obvious incentive is to make participation in discussion forums part of the sudent’s assessment. Morrison (2012) offers an example rubric that makes clear to students how their participation could be assessed. In her example, the quality of the initial post is measured according to relevance, clarity and depth of understanding. Follow up posts are graded according to frequency and supportiveness. Word count and timelyness are also factors that affect grading.

This is just one example, but it demonstrates the effort required by instructors to properly assess each student’s work. An active and vibrant forum may have dozens or, especially likely with MOOCs, hundreds of posts. Automated tools, especially those that enable supportive peer review are required if the full learning potential of asynchronous discussion forms is to be realized.

Changing direction?

I’ve been doing a lot of thinking around my participation in the Portus MOOC a few weeks back. This post is an attempt to get my thoughts in order, so I apologise in advance for any disjointedness.

First of all, let me edit in some thoughts on locatative gaming, prompted by a Guardian article on social gaming I read today while I should have been bashing this post into shape. Describing the new game from Bungie, Destiny (which is of course a console game, not a location based one) she says

On a practical level, though, “social” is a business model. It means content engineered to be “liked” or shared. It means fundamentally we spend anxious time doing free labour for social infrastructures, providing our personal lives, disseminating links, making those platform-holders wealthy with our exhibitionism and interaction. When it comes to games, it’s increasingly on the player to create the meaning in their experience.

And passionate players provide unpaid labor to games development, too: games are being released in beta and updated in public, so that the end product will better meet their needs. Thus the eager front-line beta testers mitigate the expensive risk of developing a commercial tech product, just through the fuel of their social behavior.

[…]

It is social in that business sense: you must collaborate with and keep up with your friends, ensure that your statistics and equipment – your fitness for competition – are ever increasing. You participate excitedly in this capitalistic metaphor.

Having “played” Ingress for a couple of weeks now, I’m beginning to feel the same frustrations as the author. I simple don’t have the time and dedication to labour on behalf of Google and for the benefit of my fellow players. I know what I ought to do to have an enjoyable experience is recruit freinds and family into the game so that we can play as a team, or build relationships with other players to do the same. But its too much bother. Its not for my generation, I’ve concluded.

In the Guardian, Leigh Alexander concludes:

I believe in the potential for games to create incredible collaborative environments for play. But let’s think about what a “social” play experience would look like if it served us, the users, and not the platform, whose only real desire is to have us use it, to have us serve and propagate it, to lend hours of our time to its cold lunar ecosystem.

What would a locatative game look like, that served the users rather than the platform? We’ll have to wait and see.

Right now though, I’ve been thinking about how the Portus MOOC might better serve its users. I’ve been looking at all the comments that were posted by students on the MOOC, and though I’ve not yet done any proper text analysis, my impression is that the Portus team received great praise from participants, but there are two apparent challenges for web-based learning:

• Spatial and contextual awareness. Comments from participants consistently highlight the difficulty of understanding the spaces involved, their relationship to each other, and their scale. Efforts to understand spaces were further undermined by the struggle to understand the context as the topography and use of space changed during the 500 year period of occupation. Copious maps, plans, 360/spherical panorama and references to GoogleEarth and Bing Maps failed adequately to mitigate this challenge.
• A preference for didactic learning over investigation. Though many participants relished the more autodidactic optional activities, a considerable number expressed discomfort when faced with interpretation tasks where users generated their own content. Peer review was especially daunting.

My supervisor, Graeme Earl already addressed the first point in his post on the Portus MOOC blog. Therein he says:

Some of you have already used ingenious methods, such as pacing out the size of a canal on your driveway or finding household objects similar to those we find at Portus. This is fabulous and please keep sharing these ideas – it is really helpful for us and for other learners.

But what do we do if we want to immerse you in the site as it is today, and as it was in the past? I would like you to imagine the buildings towering above you, to feel as though you are walking the streets and avenues in the footsteps of the Roman sailors, warehouse workers, slaves and traders that walked there two thousand years ago. You did this in textual form fantastically already in theFirst Century discussion in week one and in the Summary of the Week in week five, and it would be great if you continued to produce image or audio versions and share them on the Flickr group pool.

We’ve been thinking about how, for the next run of the Portus MOOC, we might lift our model of Portus off the page, take it out of the tiny window of the average computer monitor.

Imagine this.

Armed with a smartphone (loaded with a simple app that we create), one of our MOOC participants takes a walk, where-ever they live, and finds a piece of ground of a reasonable size, a park perhaps, or a school playing field, or a parking lot even. As long as it’s reasonably clear of obstructions it should be fine. They walk around the field pacing out as large a rectangle as they can, using the smart-phone’s GPS function to define and log each of the four corners. The app (or maybe its an HTML5 webapp, so they (and we) don’t have to worry about app-stores) tells them how the area they’ve measured out compares to the area of the Portus site.

Then (and here is the clever bit) the app scales everything we know about the real Portus to the area they’ve described. Using the app, and maybe some physical markers of their own, they can locate the intersections of the streets, and the locations and sizes of buildings that we’ve excavated. The app would allow them to map the changes that took place over time too, so that could plan out Then they can walk those streets, and the app can help them visualise the building they are walking past, and how goods (and people) moved from one space to another on their journeys in and out of the Port.

When I say visualise, I bet you are thinking they hold their phone up and, looking through the screen, see 3D models that we’ve made of the buildings in AR. I guess it’s a possibility, but we’re beginning to push at the limits of the technology here: Smartphone GPS has been getting better, but most phones are likely to deliver something accurate only to between three meters and nine, and what with level changes on the site they are doing this, and at Portus, I fear that an AR presentation might end up with so many visual glitches that it becomes off-putting rather than insightful and inspiring.

So actually I’m thinking there is a better learning outcome by making them do it all in their heads. I like the idea of  learning that visualisation starts in the imagination, not at the 3D modelling interface. Grant Morrison, who wrote the challenging comic The Invisibles, coined the term Fictionsuit to describe a method by which an author interacts with the characters in his (or her) diegesis by becoming a character in the diegesis. In a way, this is what the MOOC asked students to do the “First century discussion” that Graeme referred to in his post. Some participants (according to the comments) were more comfortable than others with this exercise, but I’m convinced its a vital tool for interpreting archaeological evidence and learning about exploring a world that can, in a very literal sense, only be a creation of our collective imaginations.

In another way, game avatars are fictionsuits too. Whether they are created by authors of the game, like John Marston in Red Dead Redemption, customisable creations of the player as in Skyrim, or held entirely within the imagination of the player as in Dear Esther.

But there’s a dichotomy between the exercise of imagination, and the “truth” of an academic paper or computer model. And the evidence of the comments betrays, among participants on the MOOC, a preference for passive acceptance of an expert’s model over willingness to imagine a model of their own.

So I’m thinking about how we might use game mechanics to:

• Immerse participants in the geo-spatial relationships of different parts of the site. Exploring it by moving from place to place on a map (or even a scale recreation of that map in a real-word space) to access different content.
• Encourage the creation of fictionsuits to explore the possibilities of how the site might have worked
• Share (and even evaluate) interpretations of the site and the evidence

That last, the sharing and evaluation of interpretation is a particular challenge, the game mechanic solution to which might be in some work I looked at yesterday. Without wanting to reveal too much about the project of a colleague I only just met, I was introduced to a team which is working of using game mechanics to create Linked Data for an enormous  corpus, and also evaluate learning. It strikes me that this methodology could be incredibly useful for MOOCs. Yes, it uses game-play to source un-paid labour, just like the social games that Leigh Alexander was berating in today’s Guardian, but it does offer the intrinsic reward of actual, real learning.

I’m still trying to synthesize all this into a coherent project, but I do think I’m getting somewhere.

Please do comment if this all feels like nonsense though.

A great post about locatative technology at Walt Disney World, Florida

A forcibly disassembled MagicBand, revealing parts that include a coin-cell battery, RFID chip, coiled and copper antennae, microcontroller and integrated circuit, plastic battery and IC housing, and rubber wristband enclosure. Picture linked from Medium.com

I should be writing another post entirely, but I got sucked into reading Welcome to Dataland, which makes me admire Disney so much (again) and fear them, just a little…

I particularly like the citing of “what the science-fiction writer Bruce Sterling has called design fiction, a kind of design that “tells worlds rather than stories.””