Sensors rarely fail outright — they drift, and drift triggers no alarm

A monitoring system gets installed and works well. Six months later someone opens it up: every device still reports as connected, data still arrives on schedule, the charts are smooth, and there has not been a single alert in half a year.

The only problem is that the numbers stopped being correct back in month two.

This is the most awkward failure mode any measurement system has, and it is awkward for exactly the reason that makes it hard to spot: the system never stops. It keeps running precisely as designed, keeps sending numbers on time, and those numbers simply no longer reflect what is being measured.

Two ways to fail, and the more tolerable one is the noisier one

Measuring equipment fails in two quite different ways.

The first is failing outright. The sensor stops sending, loses connectivity, or returns an obviously impossible value. This is annoying but it is actually the good kind: it reports itself. Monitoring catches it immediately, somebody gets an alert, somebody goes and replaces it.

The second is drifting. The sensor keeps working and keeps returning numbers that look entirely reasonable, but they sit slightly away from the true value — and further away each month. No threshold is crossed, no connection drops. There is nothing to alert on.

The second kind is far more dangerous, because throughout that period people keep making decisions on those numbers: ventilation still runs off the measured level, cold-room temperature still gets adjusted from the sensor, reports still go upstairs. Nobody uses the readings from a device that has died. Everybody uses the readings from a device that has drifted.

A very concrete example: a CO₂ sensor that re-zeroes itself wrongly every week

The clearest illustration lives in the most common type of CO₂ sensor on the market — the kind that shines infrared light through the air and measures how much of it gets absorbed.

This type has an inherent problem: the infrared source dims with age, and dust settles inside the measurement chamber. Both make readings drift. Manufacturers address it with a self-adjusting mechanism, usually abbreviated ABC — *automatic baseline correction*.

It works like this: the device remembers the lowest CO₂ level it has seen over roughly the last 180 hours, assumes that minimum was outdoor air, and pulls its own zero point to 400 ppm.

In an office, that assumption holds. Nobody works at night, doors get opened, and indoor air drifts back toward outdoor levels. The device catches that minimum and re-zeroes itself. The mechanism runs quietly, untouched for years.

Now install the same device in a three-shift factory, a greenhouse, or a sealed cold room. In those places the room is never close to empty. The lowest value in 180 hours is not outdoor air — it is merely the quietest hour, possibly still 700 or 900 ppm.

The device does exactly what it was programmed to do: treats that number as 400 ppm and pulls the whole scale down with it. After a few cycles the readings are off by hundreds of ppm — and they are off in the direction of always reading lower than reality, which is the dangerous direction for a number used to decide whether ventilation is needed.

The crux: the device is not broken. It is running exactly as designed. What is wrong is that the design's assumption does not hold where it was installed — and it has no way of knowing that in order to tell you.

And even where the assumption holds, that 400 is out of date

There is one more detail that makes this example stick.

The 400 ppm baseline embedded in many sensor firmwares traces back to a patent for a self-calibrating gas detector, filed in 2003 and published in 2005. At that time the global mean atmospheric CO₂ level was 378.98 ppm, so a round 400 was a sensible reference.

More than twenty years on, the atmosphere sits at roughly 429 ppm. The number 400 sits exactly where it was, in the firmware.

The consequence: even a device installed in an office, where the overnight assumption holds perfectly, will see a minimum of around 428 ppm, treat it as 400, and manufacture a systematic offset of about 28 ppm — with every component working flawlessly. The planet's baseline drifted away from a number hard-coded in software, and the software has no way to know.

At 28 ppm, for indoor air quality work, that error rarely changes any decision. But it illustrates the principle cleanly, and the principle is the part worth carrying elsewhere.

The principle: every self-calibration contains an assumption about the world

Self-adjustment is not magic. It is a rule somebody wrote, and that rule always rests on an assumption about where the device will be installed: that the room will empty out, that the pipe will sometimes carry no flow, that overnight temperature will return to something near a known level.

So the question to ask when choosing equipment is not "does this sensor self-calibrate" — nearly all of them do, and it always sounds like a feature. The right question is: what is it assuming, and does that assumption hold where I am putting it.

If it does not, there are usually two options: turn the self-adjustment off and move to scheduled manual calibration, or create the conditions that make the assumption true — for instance, taking the device outdoors periodically. Both require somebody to remember, which is also why this step tends to get skipped.

Why drift slips past every routine check

Worth listing out, because most monitoring systems have all of these layers and still miss it:

• Connectivity checks only answer whether the device is alive. A drifting sensor is very much alive.
• Data arrival checks only answer whether numbers are coming. A drifting sensor sends them punctually.
• Threshold alerts keep working normally — the thresholds simply now sit in the wrong place relative to reality, so they stay silent at exactly the moment they should fire.
• Plausibility checks do not help either, because drifted readings remain plausible. That is the definition of drift.

In other words, every routine layer asks "is there a number" and "is the number inside the allowed range". None of them asks "is this number correct" — because answering that requires something to compare against.

Four ways to catch it without a laboratory

• Install two identical devices in the same spot. This is the cheapest and most effective measure. A lone sensor has nothing to compare against; two adjacent ones always do.
• Alert on the gap between the two, not on the value. Two instruments measuring the same place whose readings diverge further each week is an unmistakable signal, and it shows up far earlier than waiting for someone to notice an implausible figure.
• Check whether readings ever reach a known baseline. For CO₂, the baseline is outdoor air. If a week goes by without readings ever approaching it, the self-calibration is running blind — and you learn this purely by looking at data you already have, with no extra hardware.
• Record every calibration and swap in the same data series. Each adjustment creates a step change. Without a record, six months later nobody can tell whether the step came from the equipment or from the thing being measured.

Do not confuse this with model drift

There is a similar-sounding concept that is an entirely different matter: model drift, where a prediction model degrades because reality has moved away from the data it was trained on. We have written separately about catching model drift before it is too late.

Telling the two apart matters because they call for opposite actions. Model drift is fixed by retraining. Sensor drift is not helped by retraining at all — it merely teaches the model to memorise the offset, after which everything looks more stable than before.

This is also why, in a system containing both sensors and models, you check the measurement layer first and the prediction layer second. Do it the other way round and it is easy to spend weeks tuning a model for a problem that lives in a dusty sensor out on the floor.

Several things speed this up in Vietnam

Three conditions make equipment drift faster than the datasheet suggests, since datasheet figures are typically measured in a laboratory:

• Year-round high humidity. Moisture condenses inside measurement chambers and on circuit boards, accelerating ageing in nearly every sensor type.
• Dust. For anything that measures using light — infrared CO₂, optical particulate sensors — dust on the optical path shifts readings directly, and in a direction that is hard to predict.
• Day-night and indoor-outdoor temperature swings. Repeated expansion and contraction degrade solder joints and seals faster.

Plus one factor unrelated to climate that shows up more often than all three: installed means finished. In most projects, periodic calibration is not in the contract, not in the operating budget, and has nobody's name attached. For the first six months nobody notices it is missing, because for the first six months the numbers are still right.

Conclusion

For a measurement system, the question "is the device still running" is easy to answer and usually answered automatically. The question "are its numbers still correct" is much harder, and in most systems nobody is asking it at all.

Self-calibration does not solve this, because it too is only an assumption about the world — and when that assumption does not hold where you installed the device, it does not raise an error. It just goes quietly wrong, with complete confidence.

What needs doing is not expensive: add a second device so there is something to compare against, alert on the distance between the two readings, and record every time somebody touches the equipment. All three cost far less than discovering, half a year in, that every recent decision rested on a number that had shifted.

If you operate a sensor system that has been running for a while and want to know how far its numbers can still be trusted, get in touch to review a specific cluster, or see KonexForge's IoT & Sensors capability.