Assessment
Assessment Plan
Evaluating this product involved answering one major question: are they adopting and using a mechanistic mental model of programming? To answer this question, subjects will be given a crash course on using the mechanical programming language for the averaging problem and asked to solve it using a talk-aloud protocol. Then they will be given a crash course on programming in a text-based language (Python) and asked to solve the problem.
Tangible vs. Graphical
In addition to the tangible user interface for the mechanical programming language, I built a graphical version of the language, because another important question to answer with the evaluation is the impact of the use of a physical, tangible interface for the solution rather than a graphical version. Marshall showed that there was little empirical evidence to suggest that tangible interfaces enhanced learning (partially due to a dearth of comparison studies).
This is an important question to answer because tangible interfaces have many downsides compared to graphical interfaces: they are more expensive to develop, prototype, test and finally deploy compared to a graphical interface. To address these questions, I did a comparative study to compare the two, with half of my subjects starting with the tangible prototype and the other half starting with the graphical prototype.
Assessment Results
All results from the study are preliminary, but a few observations that stood out. There was a large difference in the amount of time it took subjects in the tangible versus graphical conditions. Subjects in the graphical condition managed to complete their task within 5–10 minutes of having the tasks explained in contrast to the 20–40 minutes in the tangible task.