 those that were unsourced or unrelated to the course content. We coded each on a simple three-point scale, determined by the TA for the course: 1 for good, thoughtful insight; 2 for relevant but not particularly insightful; and 3 for unrelated or completely trivial. We coded 64 percent (112 of 174) in category 1 and 87 percent (152 out of 174) as combined category 1 and 2. Again, we found these results heartening.
"...as a farmgirl, I find it interesting how knowledgeable they were, and also how some of the main ideas still carry through today"
"only mentioned gutenburg for printing press even though China had it centuries before (and they mentioned China for other things)"
"What role did the king/royalty have in the construction of cathedrals, especially ones who had divine right to govern?"
"the plague had nearly the same effect on Siena and London because the plague could not be purged no matter what government"
As we mentioned earlier, part of our model included using an aggregated tag cloud to identify themes. The idea was to use these tag clouds in a way similar to how clickers can be used to aggregate responses to multiple-choice questions.
Figure 3 represents the word cloud for the third movie, The Making of a City: Medieval London, 1066–1500, with identifying data blurred. Notice that the most tweeted word was "plague." The instructors considered issues concerning the plague important, but not really central, to the theme of the movie. This provided a moment of reflection for the class, where the instructors articulated clearly to the class the role that the plague played — and didn't play — in the formation of medieval London. It also guided the post-viewing discussion.
Beyond leveraging the word clouds to guide discussion, a simple attendance/participation grade was provided to learners who participated in a simple check (tweets not considered thoughtful by the instructors), check plus (tweets considered thoughtful by the instructors), check minus (did not tweet) system.
Only 13 of the 29 respondents (45 percent) agreed or strongly agreed that they paid closer attention because of the Twitter exercises; this means, of course, that the other 55 percent indicated that they disagreed or strongly disagreed with that statement.
Only six students (21 percent) agreed or strongly agreed that they worked harder in the course because of the Twitter exercise.
Twelve of the 29 respondents (41 percent) indicated that the Twitter exercise helped them learn, leaving nearly 60 percent that disagreed or strongly disagreed with this statement.
Eleven respondents (38 percent) suggested that the Twitter exercise enhanced interaction with the instructor, TAs, or other instructional staff.
Eleven students (38 percent) agreed or strongly agreed that they thought more about the course material because of the Twitter exercise than they would have if it had not been used.
"I like the idea of the Twitter exercise overall, but I feel like it distracted from the movie."
"Twitter was distracting and disorganized."
"I thought the Twitter exercise made me focus more on Twitter than on what I needed to learn."
"Twitter was fun because of the jumping point for discussion it provided after the film."
Certainly, the most charitable interpretation of these data would be that the results are mixed in terms of student perception of efficacy. A clear and consistent, if slight, majority of the students did not seem to see the learning or motivational benefits of the Twitter exercise that the instructional and evaluation teams observed. The student perceptions of efficacy certainly did not reflect our own sense that the project met with our learning goals and seemed to transform the culture of the course during the in-class films to engage students in thinking constructively about the movies shown. We'll discuss possible interpretations of these results in the Discussion section.
While the pilot of our pedagogical strategy yielded, to our minds, generally positive results, it was a deeply manual process that required far too much human resource support to scale. Too, we saw an opportunity to use this general approach to address other types of issues that more traditional clicker technologies had presented over time. We've heard many instructors complain over the years that loading an additional $30 or more onto already financially burdened students is a bridge too far, particularly when most students already have some form of computing device that they bring with them to class. Why can't students use their own devices, such as laptops, smartphones, or cell phones, to click in their responses? Clicker companies have begun offering cloud-based clicker tools that require students to purchase a license to participate, but of course this doesn't address the issue of loading additional costs onto students. We also believed that the commercial offerings have remained relatively uninspirnig, not fully realizing what we see as the potenitial of student response technology, including allowing more sophisticated forms of input and more interesting visualizations of students' collective work.
In discussions with several key faculty members about the pilot project, many expressed concerns about the "open" nature of Twitter and moving students' work to the cloud. One instructor, for instance, who wanted to (and has since begun) using the tool in a course in the Gender, Women, and Sexuality Studies department indicated that learners in her class might be asked to respond to questions about rape, gender, and sexuality that she would feel much better about if they were in a closed, authenticated, and still anonymous environment.
Not finding anything that specifically met these needs in the commercial or open-source market, we set our sights on developing a cloud-based tool that would power the same pedagogical strategies as our Twitter/Wordle pilot, but in a more tailored, user-friendly, scalable experience. ChimeIn was the result.
ChimeIn is a cloud-based student response system that (currently) allows instructors to ask students three types of question: true/false, multiple choice, and free text. Students "chime in" with their responses to the questions in a number of ways: via a web-based interface, via a smartphone interface, or by SMS/texting their responses. (As devices change, tweaks to the input interface and protocols should be trivial.)The system is relatively device-neutral — our goal was to make chiming as ubiquitously accessible as tweeting or updating a Facebook status. Responses from true/false and multiple choice questions appear as a bar graph and as a pie chart (see Figure 4). Responses to text-based questions appear as a word cloud (see Figure 5, which shows student answers to the question "What is the definition of the medication experience?" in a course in the College of Pharmacy). And, much like a Twitter feed, the chronological, dynamically updating archive of text-based chimes can be monitored (see Figure 6, which shows the chronological archive of chimes for Reyerson's history course on the Crusades with identifying data blurred). Questions may be anonymous or tied to a particular student for grading purposes. Instructors can also download the source data collected by ChimeIn in a CSV format, allowing them to perform other types of analysis or integrate responses with a learning management system (LMS) gradebook.
Included with permission of Drs. Rod Carter, Keri Naglosky, and Don Uden, course instructors.
ChimeIn took one full-time programmer less than two weeks to build, with the help of one undergraduate programmer and one undergraduate interface designer. It is housed on an existing server and requires very little maintenance. We estimate that the initial development costs were under $5,000. It is fully integrated with our PeopleSoft data and has an interface that allows instructors to create courses by the university course ID number that will auto-populate the students signed up in their course rosters. Students and additional instructors can also be added manually. Public ChimeIn "courses" can also be created to allow instructors or administrative units at the college to use the tool in noncurricular ways. Technically, the application was developed using a combination of PHP, jQuery, MySQL, jQTouch, and Google Voice.
All question types can be set to update dynamically as students are chiming. This allows the bar graph and pie chart or word cloud to update as new responses come in, changing relative size and shape dynamically to show the movement of learners' collective chiming.
In addition to dynamically generating the word cloud for text-based responses, individual words can be selected from the cloud, and the text-feed will auto-sort to include only those chimes with the selected word, which is highlighted. This feature facilitates identifying themes from chimes in real time.
Finally, we added a "hot swap" feature that allows learners to click a single button within the ChimeIn interface that logs them out and returns to the login screen, so devices can be easily shared among learners even in authenticated mode with a minimum of disruption.
For an example of the first two features, see the demo at http://z.umn.edu/dynamicchimes. Figure 7 shows a word cloud generated from student responses to a film in Reyerson's course on the Crusades. The word "blame" is selected in the word cloud (top) and sorted/highlighted in the free-text response feed (bottom), with identifying data blurred.
Note that ChimeIn was intended to power not only the specific pedagogical strategy from our pilot but also to provide a range of other features to instructors that would allow additional pedagogies to emerge. Some of those might simply be allowing students to use their personal mobile devices to answer multiple-choice clicker-style questions. Some might be free-text clicker-type questions. Others might be more dialogic, allowing a backchannel with a cloud-based representation of the most commonly chimed thoughts, similar to our initial pilot.
The system has 168 unique instructors who have created courses.
Over