Moneyball for Business: Employee Engagement Meta-Analysis

Whether it's in baseball or business, team performance is difficult to predict.

Just ask baseball's sabermetricians, who spend their lives compiling player statistics. Their goal is to make "Moneyball" predictions about players who will succeed in the game by gathering detailed statistical data on their past performances. Combining these data should provide for robust predictions of team success -- or so the theory goes.

But if the winning equation were as simple as accumulating individual player performances to predict team success, baseball might be among the easiest sports to predict. After all, baseball has collected detailed performance metrics on everything from individual pitches to at-bats and fielding over a 162-game season for many, many years. As a dataset goes, this one is enormous.

These data are useful in predicting individual player tendencies. But when these data are combined across players on a team, the statistical algorithms approximate team wins for a given season with an average accuracy of plus or minus eight wins -- which is often the difference between a team making the playoffs or not. In some cases, predictions of a given team's wins are off by 15 or more games. In 2015, the sabermetricians missed the final regular season record of the Kansas City Royals -- the eventual World Series champion -- by an average of 16.75 games.

"The things we're so good at, they're not stat-able," said Royals Manager Ned Yost.

Yost might be referring to the "clubhouse culture," or the extent to which the collective players complement and have an impact on each other's performance. In other fields, studies that evaluate performance on a variety of tasks suggest a collective intelligence of teams that explains much more than the sum of the individuals' abilities.

Prediction Problems in Business

Businesses face similar prediction problems when anticipating how much performance or profit various teams and business units will produce. Many organizations have developed systems to reduce the variability in team characteristics to maximize profit, using everything from business unit size, location, marketing efforts and product availability, while providing extensive staff training to ensure quality performance. But even after accounting for these things, performance still varies substantially across teams.

Gallup has been collecting data on teams within businesses throughout the world for decades. This data collection includes measuring employees' perspectives on the crucial elements of workplace culture, which Gallup calls "employee engagement." Employee engagement includes factors such as role clarity, having an opportunity to do what you do best, opportunities to develop, opinions counting, strong coworker relationships, and a common mission or purpose.

Q01.	I know what is expected of me at work.
Q02.	I have the materials and equipment I need to do my work right.
Q03.	At work, I have the opportunity to do what I do best every day.
Q04.	In the last seven days, I have received recognition or praise for doing good work.
Q05.	My supervisor, or someone at work, seems to care about me as a person.
Q06.	There is someone at work who encourages my development.
Q07.	At work, my opinions seem to count.
Q08.	The mission or purpose of my company makes me feel my job is important.
Q09.	My associates or fellow employees are committed to doing quality work.
Q10.	I have a best friend at work.
Q11.	In the last six months, someone at work has talked to me about my progress.
Q12.	This last year, I have had opportunities at work to learn and grow.
These statements (Q01-Q12) are proprietary and copyrighted by Gallup, Inc. They cannot be reprinted or reproduced in any manner without the written consent of Gallup, Inc. Copyright © 1993-1998 Gallup, Inc. All rights reserved.
Gallup

 

Gallup also collects performance measures for those same teams, ranging from absenteeism, turnover rates, and customer perceptions of service to productivity and profit metrics. Combining employee engagement and performance data for teams across many organizations results in a meta-analysis, or a study of many studies. This method provides a more precise estimate of the influence of team engagement on performance than can be captured in any one study.

In the case of baseball team predictions, even though there are massive databases for individual performances, there are only 30 teams in any given year and 162 games for each team. Neither 30 nor 162 is a large sample size, which increases the error rate associated with evaluating team predictions for any given year and the resulting algorithms used to predict success in future years.

The same principle applies in business. Each business has a limited set of teams or business units that researchers can study at any given time. Any one study includes both imperfect measurement and limited sample size, which boosts the errors in predicting outcomes. In baseball, these imperfections affect predictions of team wins; in business, they affect estimates of future productivity, profit, sales, customer perceptions of service, retention rates and other outcomes. But by combining many studies through meta-analysis, researchers can approximate, with greater precision, the relationship between team engagement and, ultimately, performance.

Gallup has just completed its ninth meta-analysis of the relationship between team engagement and performance. This study includes more than 82,000 teams in 230 organizations -- including 1.8 million employees -- across 49 industries and in 73 different countries. Gallup assesses team engagement using 12 statements that measure critical workplace elements with proven linkages to performance outcomes. Teams often vary widely in how engaged they are -- even teams within the same organization -- much as they vary in performance.

One of the central findings of this meta-analytic study is that the relationship between team engagement and performance was consistent across the 12 elements, despite wide variation in industry and nationality, and across different economic times with massive changes in technology over the decades the studies were conducted.

When Gallup researchers compared teams in the top quartile to those in the bottom quartile on the measure of engagement, they found median differences in performance, reflected in the graphic below.

Combining these performance metrics into an overall composite performance metric, teams at the 99^th percentile had four times the odds of success (or above average performance) compared with those at the first percentile.

Though team performance will never be perfectly predictable, these results provide strong evidence that it is possible to measure the cultural elements of a team that predict how well that team will perform. In other words, team culture is, indeed, "stat-able." Businesses that measure and manage those elements can increase performance -- and improve their chances of success.