The more you measure them, the more they will game you
with Marta Stelmaszak (originally published at LSE Management with Impact)
Organizations are increasingly turning to ‘people analytics’ — using vast amounts of data about their employees — to gain insights into their workforce and to introduce more evidence-based decision-making. Our research into the use of big data analytics for performance measurement suggests that these efforts may easily backfire.
Today, companies hope to use measures such as clicks, views, transactions and many other online operations to accurately capture employee performance on top of traditional performance evaluation systems that are perceived as subjective or biased. However, employees become easily aware of data collection efforts and react by engaging in gaming strategies to influence data captured about them to their advantage. Smart organizations can respond to such reactivity of data collection in two ways. They can i) try to design measures that cannot be gamed or ii) embrace gaming that is often unavoidable.
Gaming: to manipulate (a situation) to one’s own advantage, especially in a way that is fraudulent or underhand; to rig, fix.
The promise of people analytics
Evaluating and measuring employee performance in organizations is traditionally a difficult and contentious process. Performance measurement is wrought with subjectivity, lack of precision and frustration with often cumbersome corporate systems. The emerging field of people analytics, that is the collection and analysis of vast amounts of data generated by employees in online systems, is often touted as a silver bullet. For example, Google is said to base its HR management on a complex people analytics system to uncover insights, solve problems and make decisions through analytical approaches instead of emotions, instincts and anecdotes.
The data-driven approach is understandably tempting — it promises objectivity, fairness and significant efficiencies vis-à-vis traditional ways of measuring employee performance. As a result, many companies may rush to implement increasingly scrupulous performance measurement systems, often based on clicks in online environments and applications, with the hope of capturing true worker performance. For instance, Amazon has patented wristbands that track warehouse employees’ hand movements and use haptic notifications to nudge them in certain directions to measure and increase order fulfilment rates. But does such close monitoring through data really work?
Gaming the analytics
To find out whether people analytics stands up to its promise, we conducted research in the context of higher education where teaching and administrative activities of academics and professional services staff were constantly logged real-time in a learning analytics system. Based on the theory of reactivity by sociologists Wendy Espeland and Michael Sauder, we wanted to know if and how people change their behaviors when they know they are being observed through data. This would not only impact their actions but also limit the usefulness of such monitoring in the first place.
What we found was that employees introduced various changes to their behavior as a reaction to an implemented analytics system. One of the most prominent and at the same time spurious effects was gaming: manipulating the numbers and data to improve the appearance of performance without necessarily making the actual work practices better. For example, in order to look busy and productive, users in the system could be clicking on different content items without actually studying them. Some revealed that instead of reading required documents, scrolling through them quickly would register as completing a given activity, so that their non-compliance would not be picked up. Others observed that in order to boost performance data, colleagues were posting very short but frequent forum updates (one of the performance metrics in use) as the length or intellectual depth of the content was not measured by the system.
It is clear that such gaming is detrimental to the goals of people analytics — it can partly undermine the very objectivity, fairness and efficiency of measurement that the new form of performance measurement promises. Instead of focusing on the actual work, people analytics can motivate employees to engage in sometimes elaborate schemes on how to make their work look better in analytics. This can lead to biased data and, consequently, poor decisions. With improvement being the ultimate goal of performance evaluation, there is a risk that the new analytics systems not only do little to improve performance but may even lower it as employees become distracted by playing the game of analytics.
What can organizations do?
In principle, organizations hoping to employ people analytics can set out to design analytics systems and metrics that cannot be gamed by their employees. Sometimes, there may be a way of observing and evaluating performance that will not trigger reactivity, and consequently will not lead to gaming. However, our findings suggest that this will often be difficult, if not impossible, given the fact that people are reflexive: employees constantly adjust their behaviors in response to how they perceive their own circumstances and what is beneficial to them individually. In other words, the more data organizations collect about their employees, the more reactive the employees will become. This will only intensify the likelihood of gaming.
Alternatively, a smart organization could try to embrace the reactivity of people analytics and use this to its advantage. Acknowledging and accepting the existence of gaming may by itself reduce incentives to game the system. More importantly, harnessing gaming for example through gamification could also be used to improve performance, if the individual gaming behaviour could be used to drive collectively desirable outcomes.
Ultimately, gaming performance measures is not a new phenomenon, but it will bear increasing influence on organizations as data-driven and algorithmic forms of managing work become prevalent.
