The Pros and Cons of Using Personal Response Systems in an Interactive Scientific Debate

Aims: To assess the suitability of personal response system (PRS) clickers as an audience engagement tool to make conference debates a more interactive learning experience. Study Design: We studied the suitability of PRS to moderate and compile data on a scientific debate that was the centerpiece of an international conference. The debate explored the issues pertaining to the introduction of Genetically-Modified (GM) foods into Europe and involved a panel of some of the world's leading scientific authorities on this subject. Methodology: 173 delegates were issued with PRS handsets and audience demography was collected immediately prior to debate commencement. The audience was polled on their opinion on the debate motion during the event and PRS was used to instantly screen poll results and changes in inter-poll opinions to the audience. Results: (i) Pros: PRS effectively controlled debate timings, rendered the event interactive stimulating audience engagement, compiled the respondents’ opinions on GM foods and instantly Original Research Article Kavanagh and Jones; BJESBS, 19(2): 1-11, 2017; Article no.BJESBS.31348 2 screened polling results as a percent frequency histogram to the audience. The technology highlighted a clear shift in audience opinion from “Neutral” before the debate to a two-fold increase in the anti-GM group after the event which created a novel engaging experience for the delegates. (ii) Cons: Inefficient collection of the audience polling responses was observed exemplified by 36.1% of responses to poll questions not being recorded and only 10.4% of the audience submitted a full set of poll results (7 demographic multiple choice questions (MCQ) and 3 debate polls). Conclusion: To the best of the authors’ knowledge no studies have been published to date concerning the use of PRS to facilitate an interactive debate. The immediacy of the PRS enhanced the event by prompting discussion among delegates, which would not have happened had they not known which direction other delegates were leaning on the debate question. The PRS aided in controlling the debate timings and producing instant graphical feedback but was less efficient in the collection of the complete audience dataset.


INTRODUCTION
Debates can be defined as an old teachinglearning strategy [1] where in our example an audience (learners) listens to world-renowned experts in their field argue over complex or controversial scientific topics (e.g., genetically modified foods). This process is a type of passive learning similar to the traditional didactic lecture format whereas novel teaching-learning strategies promoting active learning through direct involvement in the process are constantly being sought by progressive educators [2]. In particular the debate audience at a scientific conference are professional adults and as such adults learn best by "doing" [3] i.e., being actively engaged in the process.
Personal Response Systems (PRS) are powerful pedagogical polling tools that encourage interactive learning and provide instant feedback to the student and teacher [4,5]. Students are anonymously engaged in an active learning process whilst instant feedback aids the instructor in remedial teaching and amalgamation of the class data for assessment purposes [6,7]. PRS, in a teaching context, involves the moderator (e.g., teacher) screening questions with PRS-embedded presentation software to which each respondent submits an answer via a hand-held credit card-sized wireless keypad transmitter with an alphanumerical keypad or "clicker". Respondent input is transmitted to a USB receiver connected to software stored on a personal computer where the responses are tallied and, upon a click of a button, feedback is immediately screened in illustrative participant response charts. At the end of a PRS session data can be instantly compiled into a database presenting it in written and graphical charts. The time of response can be controlled by the moderator ensuring the event proceeds at an ideal pace and it has been shown to reduce staff input time by up to 75% [6]. PRS possesses several attractive features that would promote audience participation making the debate process interactive thus supporting active learning by the audience and debaters alike. PRS polling can be used to track audience opinions almost instantly as the debaters speak and adds an element of fun to the process.
Recently the challenge was laid down to design, deliver and moderate an interactive debate as a centerpiece of an international scientific conference in Queen's University Belfast (QUB). With this in mind we investigated the practicality of using PRS to moderate the scientific debate and engage the audience in a novel interactive experience by polling opinions throughout. To the best of the authors' knowledge only one study has been published to date concerning the use of similar technology to facilitate an interactive debate [8] and their work focused on the information polled rather than the feasibility of the technology itself. An initial concept was to correlate any relationships in the respondent's views to basic demographics (sex, age, level of education, employment and location of residence). We highlight the advantages and disadvantages of using PRS in this context and suggest ways of improving the potential of this technology. In this fast moving world of technological innovation our clicker system has been superseded by superior PRS handsets and phone-based polling software however many of the lessons learned in this report should be taken into account when moderating an interactive debate with any PRS in the future.

Conference Debate
The debate was the centerpiece the 2 nd Food Integrity and Traceability conference (ASSET2014 http://www.qub.ac.uk/asset2014/GM_Debate.htm l) hosted by QUB and SafeFood (http://www.safefood.eu/Home.aspx) in Belfast, Northern Ireland April 2014. This three day conference was aimed at an international audience of scientists, policymakers and industry representatives working in the agri-food sector and provided a balance of science together with its practical application. The debate explored the controversial issues pertaining to the introduction of Genetically Modified (GM) feeds, crops and foods into Europe. The motion posed was "This House believes that GM crops are a safe and important means of improving food security in Europe" and involved some of the world's leading scientific authorities on the subject of GM crops. The debate was chaired by an independent agrifood journalist and featured four experts (eminent academics and industry leaders in the agri-food sector), two of whom spoke for, and two against, the motion. The debate was split into two segments where after the initial motion was presented one panelist from each side spoke, then there was a twenty minute interval following which the other two panelists presented their respective case. The audience was able to contribute their views on both GM food and the debate via Twitter (#assetgm) with selected messages posted to the main screen in the debate venue.

Personal Response Systems
The PRS handsets were purchased from Turning Technologies LLC (TurningPoint.co.uk) and utilized TurningPoint® 2008 software version which was embedded into Microsoft (MS) Office PowerPoint® Professional Plus 2013. A total of 350 PRS devices were provided by the university Information Services, from which, 2 were discovered to have flat batteries (Device ID 097569 and 0974EB) from our pre-poll checks and excluded from the study.

Debate Audience
The number of pre-registered delegates for the conference was 327 consisting of scientists, regulators and agri-food producers from 25 countries. Although each device has an individual 6 character ID linking the registered device user to their polled responses a simpler numerical system was applied where the clickers were labelled from 1-348 and each delegate was allocated a number according to their surname on an alphabetical basis into the following groups A-C (n=73), D-J (n=85), K-M (n=76) and N-S (n=52) and T-Z (n=41). This system facilitated the quick distribution of the handsets to the assigned delegate and recovery of the devices post-debate. To ensure anonymity of the delegates no record was made to link the 6 character ID individual device number logged by the PRS system (e.g., 94F780) to the simplified delegate numbering system (e.g., 1-348). On the day of the event 173 PRS handsets were distributed to the debate audience.

PRS Polling
Interactive PRS MCQs were presented to the audience using TurningPoint®-embedded MS PowerPoint® slides (one question per slide) and the debate polling was preceded by a demographic survey of the audience also using PRS. Polling multiple choice questions (MCQ) were prepared by the conference organizing committee and approved by a university ethics committee prior to the event. Upon screening, each PRS question was simultaneously read out loud by the moderator in case any audience members were visually impaired. Immediately after this point the poll was manually opened to the audience by the moderator. The length of voting was dependent on the maximum response number received via the PRS transmitter which was tallied on-screen. When a certain number of respondents (n=110, determined at the time of polling due to time constraints) was tallied onscreen the poll was shut. Immediately after poll closure, results were instantly displayed to the audience onscreen in the form of a PowerPoint percent frequency histogram.

Demographic survey
Before commencing with the demographic survey a "demographic survey purpose statement" was presented to the audience as follows; "In what follows, you will be asked some demographic questions followed by a question related to the motion for debate. This question will be asked before the debate and at specific intervals throughout the debate. There is no right or wrong answer, only your honest opinion which we value. Your answers will help inform us as to whether opinions can be altered by the arguments presented by the opposing sides and allow for a potential research paper on the subject. Please be reassured that the information provided by you will be kept confidential, data will be reported anonymously and used in an aggregated format. It will not be used for any other purpose. All collected data will be stored in a safe place and destroyed after the research has been completed. By using the personal response system (PRS) you are consenting to the collection of this data".
Immediately before the debate, the delegates were asked seven carefully chosen questions (each containing numbered answers) pertaining to their demography and they were asked to respond by pressing the corresponding number on their clickers that reflected their situation. A demographic assignment slide contains a question text box, an answer text box that allows for up to 10 answer choices and a chart to present results to the audience. For example: Q1. Please indicate your gender -1. "Male" or 2. "Female" and the audience was asked to press button 1 or 2 on their clicker to register their gender.

Debate polling
The debate motion was focused on the scientific merits of introducing GM foods into the EU and polling consisted of a three individual PRS polls with one before (poll 1= pre-debate), one in the middle following both "for" and "against" arguments (poll 2= interval) and another immediately after the second "for" and "against" arguments (poll 3= post-debate). Debate polling used a five-point Likert-scale [9] in which 1 indicated ''Strongly agree'', 2 indicated "Agree", 3 indicated ''Neutral,'', 4 indicated "Disagree" and 5 indicated ''Strongly disagree''. All responses were logged onto the TurningPoint® software embedded in MS PowerPoint® and MS Excel® data processing was done immediately after the debate and took less than 5 minutes to complete.

Debate Timings
The total time of the debate as logged by TurningPoint® software was 2 hours 3 minutes (13:21 to 15:24) and a complete breakdown of the event, as recorded by the technology, is presented in Table 1.

Demographic Survey
For the purposes of this study the demographic survey was considered valid only if the respondents answered all 7 MCQ. The number of delegates who received a PRS handset immediately before the event was 173 (i.e., the expected maximum number of participants). The most responses were recorded by TurningPoint® to the first question (n=122), but thereafter, the number of respondents decreased with 106 responses being received for the final question (data not shown). There was almost an equal gender split involved in polling (52% male: 48% female) and the majority of participants were between 25-34 years old ( Table  2). Approximately a third of the respondents were educated to PhD level and worked as academic researchers which would be expected at an international scientific research conference. Finally, there was a 61:39% split between urbanites and rural residing participants with most (71%) in relative close proximity to the Northern Ireland conference venue i.e., the UK and Ireland.

Debate Polling
PRS polling consisted of presenting the audience with the debate motion and the audience submitting responses numerically based on the Likert scale (1-5). Fig. 1a shows a representative example of how the results of a debate poll (poll 3= post-debate) were screened as a percentage frequency histogram to the audience. The graphical feedback was instantly generated in a PRS-embedded MS PowerPoint® slide upon closure of voting by the moderator. Immediately prior to the debate (poll 1) 44% of polled delegates agreed with the debate motion ("agree" or "strongly agree"), 19% disagreed ("disagree" or "strongly disagree") and 37% remained neutral whereas after the debate (poll 3) 41% agreed, 43% disagreed and only 17% remained neutral (Table 3). An interesting feature of PRS was the instant screening of a chart to the audience comparing voting patterns between two polls collected within the debate process. Fig. 1b provides a representative graphical example of one such chart screened immediately after the debate showing the shift in voting patterns after the debate described above. The audience was polled during a break in the debate (poll 2 = interval; i.e., between poll 1 and poll 3) to track any effect the first pro-and antimotion speaker had on public opinion (Table 3). Already the shift in public opinion that was observed in poll 3 was underway by the debate interval with the number of neutrals dropping (37% poll 1 to 24% poll 2) and a large vote accumulating against the motion ("Disagree": 19% poll 1 to 41% poll 2) as well as a 8% drop in those voting for the motion ("Agree": 44% poll 1 to 36% poll 2).

Poll Respondents
The USB PRS receiver connected to the PC logged responses from 173 delegate clickers (100% of expected maximum number of participants) during the debate which represents a 51.4% participation rate out of the total registered delegate list (n=348). As 173 respondents were surveyed using a total of 10 questions each (7 demographic and 3 polls) the maximum response rate would be expected to be 1730. However closer analysis of the PRS poll report showed a relatively poor audience response rate with 647 MCQ responses (36.1% of clicker audience) not received by the PRS technology. The highest rate of participation per question was found with MCQ 1 where 122 (71.5% of debate audience) clicker responses were received (Fig. 2) and the participation rate decreased marginally for each subsequent demographic question (mean of MCQ 2-7 participation rate = 63.1%). The debate polls received a similar response rate as MCQ 2-7 with poll 1 (pre-debate) receiving 63.1% participation rate, poll 2 (interval) and poll 3 (post-debate) both receiving responses from 62.3% of the audience. However the story is not as simple as the same delegates who voted in poll 1 also voting in poll 2 and poll 3 as closer analysis of the data represented in Fig. 2 showed that the voting pattern was more heterogeneous in nature with 29 delegates (16.7%) registering a vote in poll 1 but not poll 3 and conversely 27 (15.6%) voting in poll 3 that didn't register a vote in poll 1.
Further analysis of the total debate audience response pattern showed the low number of valid poll results tallied ( Table 4). The most complete set of poll results (7/7 demographic MCQ & poll 1+2+3 completed) was received from only 10.4% of participants and if the interval poll 2 was excluded (7/7 demographic MCQ & poll 1+3 completed) there was only a marginal increase in valid surveys to 12.7%. If demographic data is excluded from the analysis then 30.6% of responses were considered complete (poll 1+2+3) and further exclusion of poll 2 (poll 1+3) increased complete survey results to 45% of clicker audience. As all data for respondents was not collected it was deemed invalid to inter-relate the demographics directly to the debate motion.

DISCUSSION
PRS has a proven track record as a successful pedagogical tool [4], however to the best of the authors' knowledge, very little has been published on assessing its efficacy in polling and moderation of interactive events such as a debate. In an extensive literature search of online databases (SCOPUS & Google Scholar) using search terms such as "audience response systems" and "personal response systems" and "clickers" we only found one report described using PRS to collect data from a debate [8]. In this study, as in our case, the organizers of a gastrointestinal surgeons' conference used PRS to first collect audience demography before a panel of experts put forth their cases for various surgical interventions in a debate format. The audience (n=88) was polled before and after the debate and the collated data formed the basis of a collected elective decision-making process which helped optimize future surgical procedures in weight loss surgery. This report, though a clearly successful use of PRS to amalgamate audience opinion, only presented results as percentages and so any of the underlying problems with PRS we observed herein may have been overlooked. In our experience we felt that PRS possessed several attractive features that would make it ideal for moderating a debate namely (1) rapid collection of audience demography immediately prior to the debate (to give a more comprehensive understanding of the results), (2) polling the audience during the debate and compare polls to each other (to track if any one argument has shifted the public opinion as well as the overall final outcome), (3) actively engaging the audience by making the debate interactive, and, (4) instantly generating a written and illustrative report for comprehensive analysis of the debate outcomes. After using the PRS in the debate we found all these key features held true with some major caveats. The audience opinion on the debate motion was surveyed throughout and displayed instantly in graphical form onscreen (Fig. 1a) and the entire process (demographic polling and debate) proceeded smoothly and ended within the allocated time. On reflection, for question 1 of the demographic survey we should have accounted for delegates who identify as non-binary gender or do not want to state their gender identities. One of our major concerns in adopting this technology was losing handsets but our labeling system and a dedicated team of volunteers ensured 100% handset recovery. The PRS polling technology seemed to work flawlessly throughout and succeeded in creating a "buzz" in the audience which was particularly evident after a comparison slide (poll 1 vs. poll 3) reflecting any changes in public opinion was shown at the end of the debate (Fig. 1b). This graph created an audible gasp and much discussion long after the debate ended ensuring that PRS connected delegates in a social context thereby creating a powerful group experience. It is to the authors' regret that this positive reaction was not surveyed in an exit poll after the debate. Whereas the number of participants who agreed with the motion remained approximately the same between poll 1 and 3 the most notable trend was a clear movement of large numbers of "neutrals" to disagreeing with the use of GM crops in the EU with a two-fold rise in the "disagree" category and an almost three-fold rise in respondents who "strongly disagree" with the motion ( Table 3). The increase in poll 3 anti-GM voters mostly came from the "Neutral" pool as the pro-GM vote remained relatively static between polls (poll 1 = 44% vs. poll 3 = 40%) (data not shown). The PRS allowed further tracking of the origins of this change of public opinion to the first anti-motion speaker before the interval as a two-fold increase in anti-GM voters was recorded in the interval survey (poll 2) when compared to poll 1 (Table 3). but less efficient in the collection of the complete audience dataset. None of these non-responses were due to faulty clickers or delegates collecting a device, but not using it during the debate, as all 173 devices were used to transmit at least one MCQ during polling (Fig. 2). The fault mainly lies with (1) people selectively answering some questions and not others and (2) non-participant technophobes [10] or (3) the clicker keypads not efficiently registering responses. The issues raised in the first two points may be overcome by offering low cost incentives which have been shown to increase participation rate in online survey-based research [11]. We had expected the latter point to have been addressed prior to the event as the handsets contain a built in failsafe (a red light flashes on the clicker upon registering a response informing the user that they have answered the question). However this handset feature seems to have been insufficient to advise the delegate that their response was not received by the PRS technology and we would recommend testing the failsafe feature on any PRS technology prior to the event. As this is nearly impossible to ascertain due to time limits on the day, it is a clear disadvantage of this type of PRS. The technology uses superior radio frequency instead of infra-red and the clicker range (60 meters) was checked before the debate by sending a signal from all corners of the conference hall. Although PRS clickers are theoretically easy to use it would be wise to have practice questions to allow delegates to become familiar with the system before polling [6,12,13].
An easier route to overcome the handset issues would be to replace them altogether. The PRS technology used in this study was released in 2008 and since then newer improved versions of this technology have come on the market (e.g., ResponseWare® with TurningPoint 5 or TurningPoint Cloud software). These virtual response systems permit the integration of any wireless internet-enabled devices (e.g., laptop and tablet PCs, smartphones) that can interact with a network (e.g., Wi-Fi or Bluetooth) thus replacing the troublesome clicker handset completely. The respondent votes via short message service (SMS) messaging [14] or through a website/app using their device from whence they will be immediately informed if their vote was submitted [15][16][17]. Furthermore, newer PRS systems are not limited just to MCQ nowadays as exemplified by Qwizdom which has 8 question types plus a demographic option and a more sophisticated handset. Server-based cloud computing PRS technology [18] will greatly enhance the interactive experience by including the responses from stakeholders based globally who couldn't attend the conference.

CONCLUSION
Altogether the fast-paced improvements in information technology married with PRS should dramatically increase the polling experience for debates and other participant-based events and indeed in the wider field of technology-enhanced learning.