Can you trust the summary an app gives you about your own data?
Over about five weeks I ran into four measurements that worked. A survey instrument, a research programme, a web counter and a ring on my finger. Every one of them counted the thing it was built to count. Every one of them then printed a sentence that meant something else. Two of the four were about me.
This is not an essay about bad data. The data was fine. It is about the layer between the sensor and the sentence, which is where the information leaves.
An instrument that cannot record getting worse
Evidence The Posttraumatic Growth Inventory is the standard way researchers ask whether people change after something terrible. It asks you to “indicate for each of the statements below the degree to which this change occurred in your life as a result of your crisis.” The scale runs from 0, “I did not experience this change,” to 5, “to a very great degree.”
There is no negative end. A person who got worse in a domain records a zero, which is the same mark as a person who did not change at all. The authors say why in the 1996 paper. Negatively worded items would not have controlled for acquiescence. So an instrument used to ask whether people change after trauma has no way to record the direction a lot of them move.
Evidence You can see what that costs when somebody measures both ways at once. In 187 cancer patients tracked from before a stem-cell transplant, 5.6 percent reliably increased on the prospective measure and 40.8 percent reliably decreased. The retrospective instrument, filled in afterwards, reported growth.
A survey that offered three answers and published two
Evidence Nearly every public statistic about people who start businesses out of necessity rather than opportunity comes from one global survey. Its question had five answers, not two. One of them was “combination of both of the above.”
The programme’s own codebook shows what happened to those people. The derivation rule counts answer 1, answer 3 and answer 4 as opportunity. Answer 3 is the one that says both. Answer 4 is having a job already and wanting a better one. Only the single unmixed necessity answer counted as necessity.
Asked the same question, 38 percent of 1,547 German founders picked both. In a separate study of previously unemployed founders, 71.6 percent were pushed and pulled at once, which makes the mixed group the largest one and the two published categories the exceptions. The survey dropped the design in 2019, writing that the dichotomy “may not fully reflect the nuances in motivations for founding contemporary startups.” Every figure published before then still rests on the recode.
My website told me I had ten thousand readers
Observation In early October my host emailed me a congratulations card. Ten thousand pageviews in a month, a first for the site. I had published a lot that month, so it was not implausible, and it took opening the dashboard to find out.
The same company’s own analytics, running on the same site since August, had the number for the same window. Forty pageviews with automated traffic included. Thirty with it excluded.
Both numbers are correct. The card counts requests reaching the edge of the network, which includes every crawler, every scraper and every probe. The word on the card was “pageviews.” Nobody lied. A counter counted.
My ring called me royalty
Observation My sleep tracker sent a quarterly recap. It awards what it calls a crown for any day you score 85 or above on readiness, sleep or activity. I earned eleven crowns in ninety-two days. Readiness zero. Sleep three. Activity eight.
Readiness never once reached 85. The best single day in the quarter was 76. A zero sits on that screen, in its own column, under the word Readiness, and the sentence underneath reads “you’re definitely Oura royalty!”
Three screens earlier the same recap told me my bedtime was “completely fine (even desirable).” Three screens later it told me my sleep midpoint runs 58 minutes behind my own chronotype. Four different cards each name a different thing as the key to better sleep. None of them cites anything, and none acknowledges the others.
What the four have in common
In all four cases the measurement was competent and the caption was a separate artifact. Somebody chose a word. The word travelled and the measurement stayed behind.
The choices were not random, either. The growth instrument has no negative end because a positive-only scale was easier to score. The survey folded its mixed answers into the flattering category rather than the unflattering one. The host counted the bigger number. The ring congratulated rather than reported. Each caption is more pleasant than its own data, which is a weak pattern across four cases and worth saying out loud anyway.
The test I now use
Hypothesis Here is the question I have started asking, and I am labelling it a hypothesis because it is a habit rather than a finding. What result would have embarrassed this instrument?
If no answer pattern could have produced bad news, the output is a receipt for the activity of measuring. The growth inventory cannot return decline. The congratulations card has no version that says ten thousand requests and thirty readers. The quarterly recap has no version that leads with a zero. All three measured something real and none of them could have told me something I did not want to hear.
The honest summary
None of these took cleverness to catch. Each one took opening the thing underneath, which was minutes of work in every case. Two of the four were about me, and that is the part worth keeping. A bad caption on somebody else’s number is easy to notice. On your own it is harder, because the pleasant version arrives first and arrives addressed to you.
Where this stops. Four cases is not a sample. I did not go looking for them, which protects against one kind of bias and not others, and I cannot tell you how often captions are accurate because I only inspect the ones that surprise me.
What I am not saying. That the instruments are bad or that the people behind them acted in bad faith. Three of the four are measuring carefully. The growth inventory in particular is the product of real work and is fine as a record of what a person believes about themselves. The claim here is narrower. A number and the sentence printed beside it are two different objects, and only one of them was measured.