New versions of the travel platform were usually deployed early on Tuesday mornings. Early enough to leave time for investigation, but not so early that the engineers and developers responsible for the system would be unavailable.
Then, often on Wednesday afternoon, a colleague responsible for traffic from search aggregators would appear at the application management department.
“You deployed again, didn’t you?”
He could see that sales and conversion had fallen. The team would check the platform, find it healthy and point out that the deployment had taken place the day before.
By the time he arrived, everything was indeed working normally.
After this happened several times, his reports became easy to dismiss. He gained a reputation for blaming deployments and asking for rollbacks when the current technical situation gave no reason to do so.
At first, I accepted that interpretation too.
He was reporting something real
Eventually, I asked him to sit down with me and explain exactly what he could see.
He could not identify a failing component, but he could give me an approximate time when the business results had started to deteriorate. I queried the transactional data to see whether his observation was reflected there.
It was.
The drop was substantial, and its starting point was remarkably clear.
I looked further back and found another unexplained conversion drop after the previous deployment. The delay was long enough that nobody had treated the two events as directly connected.
Then I found similar drops on Thursday afternoons.
There had been no new deployment between them.
A pattern hidden by the working day
I collected the timestamps and calculated the intervals between them.
The same interval kept returning: 1,740 minutes — exactly 29 hours.
Some occurrences had attracted attention because they happened during working hours. Others had taken place around midnight or early in the morning, when traffic was lower and nobody was waiting for the next report.
The deployment was not repeatedly causing a new failure. It was synchronising a sequence.
The application servers used a standard recycling interval of 29 hours. Because deployments restarted the servers within roughly the time it took people to click through them, their internal clocks began running almost in parallel.
Twenty-nine hours later, they recycled together.
The servers recovered. The business result did not.
A restart alone did not explain the lost conversion.
During startup, the application servers loaded a large amount of travel data into cache. Much of that data changed too often to be built permanently into the application, but slowly enough that repeatedly querying the database would have been inefficient.
When all servers rebuilt their caches at approximately the same time, the database came under heavy load. Queries slowed down, some timed out and retries added even more work. It could take about an hour for the platform to settle again.
For visitors using the company’s own websites, the problem was difficult to recognise. While the system searched for flight options, animations suggested that it was checking more airlines. A patient visitor might simply wait.
Search aggregators operated differently. They had a strict cutoff. Once that time had passed, a result was no longer useful, even if it arrived a second later.
The platform was still answering.
It was simply answering too late to be shown.
Breaking the synchronisation
I assigned different recycling intervals to the application servers so they would no longer restart together or gradually fall back into the same rhythm.
The recurring conversion drops disappeared.
No single component had provided a complete explanation. The colleague had seen the commercial impact. Application management had correctly observed a healthy platform after recovery. The application servers were behaving according to their configuration. The database eventually completed its work.
The problem only became clear when those observations were placed on the same timeline and the request was followed from deployment to customer result.
Sometimes the first useful question is not:
Which component is broken?
It is:
What did someone observe, and what happened before the system looked healthy again?