Mistakes in Employee Engagement Analysis: The Survey Score Is Not Your Outcome Measure

By Nicolas Saliba · 8 min read

An internal communication team is given an objective for the year: raise the communication item in the annual engagement survey by five points. The item is one Likert question. The survey runs once, in October. Between this October and the next, the organisation will complete a restructuring, hold a pay round, change two country managing directors and lose a product line. The team is now accountable for a number it cannot move on its own, measured on a cycle it cannot act within, and contaminated by events it does not control.

This is the central mistake in employee engagement analysis, and most of the others are consequences of it. The survey is a good instrument used for the wrong job. It describes a state of the workforce. It does not evaluate a communication programme.

Why an engagement score cannot be your outcome measure

Three properties disqualify it, and they are properties of the instrument rather than failures of the people running it.

It lags. Engagement is a durable state that Gallup’s research consistently ties to managers, role clarity and the daily experience of work^[1]. States like that move over quarters. A campaign that ran in March shows up, if at all, blended into an October reading alongside everything else that happened in between.

It is low-frequency. One or two readings a year gives you one or two data points. You cannot run a measurement discipline on two data points, and you certainly cannot use them to decide whether to change a channel mix in June.

It is confounded. Pay, workload, job security and line-management quality all press on the same score, and all of them press harder than communication does. When the score falls after a restructuring, attributing that to the internal communication team is not analysis. When it rises after a good year, claiming credit is the same error in the direction that feels better.

	Annual engagement survey	Communication measurement
Frequency	Once or twice a year	Per communication, continuous
What it describes	A workforce state	Whether a message reached and was read
Attribution	Heavily confounded	Direct for reach, weaker for behaviour
Acts on	Management practice, role design, reward	Channel mix, timing, targeting, format
Right use	Framing the problem	Measuring the intervention

Mistake two: reporting an average over a population that is not homogeneous

A single organisation-wide engagement figure is almost always two or three different results wearing one number. A 68 per cent favourable score composed of 81 per cent in corporate functions and 44 per cent in operations is not a moderately good result. It is a good result and a serious problem, and the average conceals which sites need attention.

The same applies to communication data. Reach of 34 per cent across a group is uninformative until you see that it is 61 per cent among desk-based staff and 9 per cent on the plant floor. Our piece on measuring frontline worker communications covers the population that averages most reliably hide, and audience segmentation covers how to structure the attributes so the split is available at all.

Mistake three: comparing groups with different response rates

Survey scores are compared across sites and functions as though the samples were equivalent. They rarely are. A function with an 84 per cent response rate and a site with 38 per cent are not two measurements of the same kind, because the people who did not respond are not a random subset. Disengaged people answer less often, which biases the low-response group upwards, which is the opposite of the direction most readers assume.

The practical rule is to publish the response rate next to every score, and to refuse comparisons below a threshold you set in advance. It makes the deck less tidy and stops you acting on an artefact.

Mistake four: treating movement smaller than the noise as a result

A two-point year-on-year change on a survey item is presented as improvement more often than it is presented as noise. Without a confidence interval, a stable response rate and a comparable population, a small movement carries no information. Deloitte’s Human Capital Trends work keeps pointing to how quickly the composition of workforces changes^[2], and in an organisation with 18 per cent turnover, a share of any year-on-year movement is simply a different set of people answering.

Say so. A finding of “no detectable change” delivered with confidence is more valuable to an executive audience than a two-point rise presented as a win, and it protects you the year the number moves the other way.

Mistake five: inferring from the channels that happen to report

Analysis gravitates to the channels with data. The intranet has viewer counts, the newsletter has opens, so conclusions get drawn about a communication programme from the two surfaces that keep records, while manager cascade, Teams channel posts, notice boards and shift briefings contribute nothing to the picture. Teams is a particular blind spot, which our piece on Teams as an internal communication channel works through.

The result is a systematic bias towards concluding that the audience is the desk-based population, because that is the population the measurable channels serve. A channel map that lists every channel, including the unmeasurable ones, and states plainly which are dark, is a more honest starting point than a dashboard covering three of eleven channels.

What to do instead: pair the slow measure with a fast one

The workable model is a pairing. The engagement survey frames the problem: it tells you which populations feel least informed and least connected to direction. Communication data measures the intervention: it tells you whether the campaign you ran in response actually reached those populations, at what rate, and how that compares with the last comparable campaign.

Concretely, if the October survey shows the lowest scores on “I understand the strategy” in three manufacturing sites, the objective for the next quarter is not to raise that item. It is to lift read rate on strategy communications in those three sites from 11 per cent to 30 per cent, verified monthly, with the survey used a year later as a slow confirmation. Our guide to defining success before you measure it covers how to write that objective so it can be evidenced, and the full argument for running both instruments is in engagement surveys against communications data.

Gallagher’s State of the Sector research keeps identifying measurement as the capability internal communicators most want to strengthen, and this pairing is the practical form of that capability^[3].

What Microsoft 365 gives you for the fast half

Native analytics is free, already switched on and sufficient for a single team on a single surface. SharePoint site and page analytics tell you honestly whether a page was read, and Viva Engage community insights do the same inside a community. If your fast measure is one page on one site, you may need nothing more.

The constraint is that native reporting has no view of your organisation, so it cannot express reach as a share of the three manufacturing sites in the example above. It also reports per channel, so a campaign that ran across the intranet, the newsletter and Teams produces three unlinked numbers. Those are the two capabilities the pairing model depends on. The four questions any number can answer are set out in our guide to internal communication analytics.

A Power BI build against the Microsoft Graph reporting API is a legitimate way to close that gap, with a known cost. The reporting API is not consistently reliable and produces days with missing data. Customers have observed KPI values in Power BI that do not reconcile with the native SharePoint reports for the same period. A dependable build needs a robust ingestion process and scheduled reconciliation against native figures, so budget for the pipeline rather than the dashboard.

Tryane provides the fast half directly. It reads SharePoint, Viva Engage, Teams and your newsletter platform together, joins the activity to your organisational structure through Entra ID or an HR file, and keeps history so this year’s campaign can be compared with the last one. Tryane is SOC 2 Type 2 certified and GDPR compliant by design, hosts in the EU by default with US data residency on request, and deploys in a couple of hours over single sign-on with Entra ID.

Frequently asked questions

Should internal communication own an engagement survey target at all?

Share it, do not own it. The engagement score is a shared organisational outcome influenced by reward, workload and line management as much as by communication. Owning a target you cannot move alone is how a function ends up defending a number rather than improving the work. Own the reach and readership measures and contribute to the engagement conversation.

What is a reasonable frequency for measuring communication?

Per communication for the important ones, and monthly in aggregate. That gives you enough observations to see a trend within a quarter, which is the cycle you can actually act on. Keep the survey as an annual or twice-yearly framing instrument rather than trying to increase its frequency until it becomes a burden.

Are pulse surveys a better outcome measure than the annual survey?

They are faster and they help, but they inherit the confounding problem and add response fatigue. A two-question pulse tied to a specific communication and a specific population is genuinely useful. A general monthly engagement pulse tends to produce a noisy line that nobody acts on.

How do we handle populations with no digital channel at all?

Name them explicitly in every report as unmeasured rather than allowing them to disappear into an average. Then measure a proxy you can defend, such as manager cascade completion or attendance at shift briefings, and state that it is a proxy. Silent exclusion of a population is the most common way an internal communication report misleads its own authors.

Does Tryane measure employee engagement?

Tryane measures communication: who was reached, in which populations, across which channels, and how that changes over time. It does not replace an engagement survey and does not try to. The two are complementary, and the pairing described above is how most of our customers use them together.

Sources

• Gallup, State of the Global Workplace

• Deloitte, Human Capital Trends

• Gallagher, State of the Sector