Sleep trackers: more of a data nightmare than a data dream?

I like my Android phone just fine… but I don’t want to marry it and have its babies.

Nor do I want to sleep with it perched on my mattress. Many people do, though: Sleep as Android, the most downloaded sleep tracking app in the Google Play store, had been downloaded nearly 250,000 times as of May 16.

The app functions the same way as many other popular sleep trackers, be they smartphone apps or wearable personal devices (there are more than 500 options from more than 15 brands now available). The apps track a sleeper’s movements with an accelerometer, listen in on whether they’re snoring, and chirps at them to make them roll over if they are. It also tracks how much sleepers toss and turn, and how much time they’re lightly or deeply sleeping. Unlike other sleep trackers, this one promises to track even REM, which is the phase that’s the most difficult to detect.

But here’s what a researcher at Brown University wanted to know: is the data being collected by these devices and apps worth anything?

To find out, Jina Yoon, an undergraduate researcher, teased out data from 10 popular sleep tracking devices, then spent nine days conducting a minute-by-minute comparison.

As Yoon tells it, she wanted to do the comparative study as a followup to one done three years ago by sleep specialist and physician Dr Christopher Winter, Personal Sleep Monitors: Do They Work?.

As both Yoon and Winter have noted, only true sleep studies can determine the quality of sleep. Trying to do so with accelerometers is like trying to gauge how well a car drives by looking at its odometer reading, as Winter says. The gold standard is a study that combines brain wave tracings, muscle-tone evaluation and eye-movement analysis, along with a live audio/video of sleep that allows researchers to see exactly what happens during slumber.

That’s what Winter himself underwent. Yoon did not; she notes that it’s too expensive and inconvenient to do a polysomnography to detect REM sleep with a worn device or app. A polysomnography records brain waves, blood oxygen levels, heart rate, breathing, and eye and leg movements. Such a test requires the placement of electrodes on the head and is usually performed in a clinic for one night.

Instead, Yoon strapped on 10 of some of the most popular sleep trackers (including one research-standard actigraph as a control). She wore them for 10 nights and then graphed the data against each other.

Those were some miserable nights:

I wore all five wristbands on the same arm (Fitbit Alta, Jawbone UP, Microsoft Band, Microsoft Band 2, AMI MotionLogger), three phone apps placed side-by-side on my bed (Sleep as Android, SleepCoacher iOS, SleepCoacher Android), one phone app placed on my nightstand (Sleep Cycle iOS), one device clipped to my pillow (Hello Sense Sleep Pill) and one product that required two sensors clipped onto my pillow and placed on my nightstand (Hello Sense). It was, as expected, not very comfortable – which might be why my sleep patterns seem quite restless over the 10 nights!

How do you even pull data out of those trackers? Yoon published a data extraction guide for all the devices here. One of them, Hello Sense, had to be hacked a bit. With the help of her advisor, Yoon says, she and her helpers were the first to ever unlock the thing.

As you can see in Winter’s results, back in 2014, the data from various devices appears to be all over the map. But Winter didn’t trash any of the tested devices; they all contributed something to sleep and fitness activity tracking, he said, and “anything that makes you think twice about staying up too late is a good thing,” he concluded.

Fast-forward to 2017 and Yoon’s study: how are the trackers doing in terms of accuracy?

Pretty much the same. Yoon notes that the data was “better than random,” with most devices recording similar findings if they’re based on the same type of sensor (accelerometer vs noise) and are the same device type (phone vs wristband vs other).

Ultimately, sleep is best tracked through polysomnography, Yoon says, but these trackers are good for those who can’t afford or don’t care to spend time in a clinic, strapped with electrodes.

But here’s the data privacy question: given that these things are not exactly the peak of scientific accuracy, then it’s reasonable to think about whether the trade-off – interesting but not particularly reliable stuff about your sleeping patterns vs. where the hell does all that data go – is worth it?

Yoon herself says that reports coming from consumer-level trackers should be “taken with a grain of salt”.

These devices offer some powerful data tracking tools, but it’s best to let the experts analyze your sleep if you have any chronic conditions.

Both Yoon and Winter before her still came to a similar conclusion: the researchers’ thinking goes that sleep trackers are great, given that anything that helps you to get a good night’s rest or reminds you to avoid staying up late is a positive thing.

But remember: personal data, once collected, is out there for good or bad. We’ve seen Fitbit data compromised, for example.

Is data compromise worth the risk, if it means your data gets pulled into, and may be exposed via, the gloriously security-addled Internet of Things (IoT)?

My pillow will remain bereft of gadgetry, though the lullabies of whalesong and the morning alarm of birds chirping offered by that Android tracker do appeal.

Do you use a sleep tracker? Please do let us know in the comments section below how you like it and what you think of being tracked while you’re sawing wood.