Agostino Gibaldi, The Physical Structure of Perception and Computation Group, University of Genova, Italy:
"The proposed experiment for evaluating the eye movement in a programming task was straightforward and understandable. The simplicity and clarity of the presented code allowed all the subjects, from native to experienced ones, to gain a comprehension of its functionality. Along the conduction of the experiment, we acquired data from 32 subjects, ranging from freshman students to post-doc researcher."
Bonita Sharif, Software Engineering Research and Empirical Studies Lab, Youngstown State University, USA:
"It was very easy to use the SMI tracker to collect eye tracking data. The setup was very simple and required very little time. I thoroughly enjoyed the experience of using the system."
Maria Bielikova and Jozef Tvarozek, Faculty of Informatics and Information Technologies, Slovak University of Technology, Slovakia:
"We collected eye gaze for both bachelor and master students during a week-long data collection. Students were glad to help with research on programming."
Ahmad Jbara, School of Mathematics and Computer Science of the Netanya Academic College, Netanya, Israel:
"The setup of the system was very easy and the run of the experiment went smoothly even for first year students who just passed one programming course. Overall, it is an impressive system."
"Collecting data from participants was extremely easy going and fun! Most of the participants expressed their interest to see the result of the processed data once it is ready and up on web. Some of them complained on why there is no BACK button on the slides. Apart from that, they seemed to enjoy the experiments too!"
Teemu Sirkiä, Department of Computer Science, Aalto University, Finland:
"18 Students from our CS1 course participated the study. We organized a room where students could participate and complete the study."
Fabian Fagerholm, Department of Computer Science, University of Helsinki, Finland:
"This was an interesting experiment and it was fun for both participants and researchers. The experiment system was self-contained and easy to set up and use. We did most of the experiment runs in our department's Software Factory laboratory (http://www.softwarefactory.cc/), because it has a quiet corner for just this kind of thing. Our participants were varied: we had students with different levels of experience, some senior lecturers who have been programming for decades, and a few professional programmers. Everyone thought the experiment was interesting and we had fun discussions afterwards. A positive experience!"
"First, Jan and I were amazed at huge difference in the effort required to collect eye movements for this experiment compared to our previous eye-tracking efforts 30 years ago. The collection process went quite smoothly. We did lose the eye-tracker a few times. However, we were able to restart the collection process by aborting the experiment, making certain the connections were in place, and restarting the IRED software. A few of the students didn’t stop after reading the second program so the results section may have a few extra files. When those events occurred the reading sequence might be out of order. When they were answering demographic questions, some of the non-native English speakers didn’t know the meaning of the term “mother tongue.” Also, a few students from the software engineering class mentioned that the answer for one of the programs depended on whether or not the input was a negative number as the noticed that the program didn’t check for that condition."
Kai Essig, Neuroinformatics, Bielefeld University, Germany
Ian van der Linde, Vision and Eye Research Unit, Anglia Ruskin University, Great Britain
James Paterson, Software Engineering, Glasgow Caledonian University, Great Britain