CST's University Administrative Fellow, Bernat Guillén Pegueroles has conducted interactive workshops, dissecting the math behind patterns. Topics have included exploring their cultural origins and significance, creating new culturally inspired patterns and discussion on their significance. Bernat has blogged about the series here:
In the previous workshop we had a long discussion about patterns. What are they, why are they interesting, how do they relate to culture… In this session we looked for actual patterns in a dataset: The mouse movement of a user during two hours.
When we design an experiment, we need a question before even starting to gather data. In this way, we force ourselves to be aware of our potential biases and assumptions when we conduct the experiment. Sometimes, however, like when we simply observe something, there is no question and our brain just decides to infer a pattern. It is not wrong to see this pattern or study it (or create art based on it) but it can lead us to a wild goose chase very easily. In this case in particular, I was interested in seeing if I could identify periods of high stress looking at mouse and keyboard movement. In fact, I recently learned that Google will use our mouse patterns to detect if we are human or robot, which makes me think that the question isn’t that far out there. We discussed for a while whether data from a single user would be enough to see patterns. It is interesting to think of these patterns at different scales: Probably if we had data from many users we could start seeing things (for example, do people in the US move their mouse around like people in Spain or differently?) that we cannot see with a single user, but my hope was that even with only one user’s data we could start find different patterns in time.
We discussed how every decision we make when we analyse some data (be it mouse movements, paintings, textiles, food recipes, stories…) has imprinted in it our biases and assumptions, unless we are very careful. Subconsciously, we make decisions on what are the important parts of the data, the parts we think will help us answer our questions or understand the situation better. In doing this, and especially if we are studying patterns from another culture, we have the risk of misunderstanding and misrepresenting a pattern. Following the discussion from last session, the line between cultural pillaging and pedagogy is a thin one and we have to be careful and honest with our analytical process.
We started by analyzing the position of the mouse cursor in the data. The whole thing was a beautiful mess (see below, dots are mouse clicks and scrolls), and we quickly realized it probably won’t give information about stress. Maybe it could give information about website/app design? Why is the user clicking so much in the bottom of the screen?
We all agreed, intuitively, that the velocity of the movement should be factored in to try to detect stress. Indeed, there are calmer and more jittery periods:
Since the data is not annotated (that means, we don’t know when the user was or was not stressed), we cannot use any supervised learning technique. However, we can search for “commonly seen movements”. However, here comes another design choice: What is the “natural” span in which we should consider a movement? Do we consider a movement only between clicks? Between stops? Do we take fragments of 5 seconds? Or maybe variable fragments, depending on the velocity? In the end, partly because the workshop was coming to an end, we decided to look at variable fragments of 10 samples (the sampling rate of the program was variable and depended on the velocity of the mouse). However, we can see from the following plot that 10 samples is not enough: All the patterns we see are straight lines in 4 different directions (or the mouse just staying in 0).
We can take longer segments and we start seeing more interesting patterns
We would hope to be able to at some point have annotated data and be able to detect when this particular user is stressed or not. However, that doesn’t mean it would work for all users (but that is a much bigger and harder question).
In the last few minutes of the workshop we discussed a bit more about the tools. In particular, some neural networks (LSTMs) seem very well suited to this problem and that would be an interesting approach to try. However there would be little to no visualization possible of the results of the networks, which in itself warrants another discussion.
Finally, one question remains: First of all, if we suppose that these movements follow certain pattern, and even if they are influenced by culture, is analysing this the same as analyzing certain cultural objects like textiles, pottery, music, or food? What is different? Maybe the fact that this is not an intentional pattern and that we cannot ascribe any aesthetic sense to it makes it easier to analyze. However, someone more talented than me could maybe take this data and fuse it with a more intentional pattern (to put an example that hits close to home, what if a flamenco “bailaor” decided to dance on top of these patterns?). Would that make it “cultural”? Would that make it “art”? Would that be the creation of a pattern?