I had been working at the company for only a few days when the CIO called.
“There were no subscriptions yesterday. Can you check whether everything is okay?”
The company sold mobile entertainment subscriptions. On a normal day, thousands of people would begin or complete the subscription process. A day with none at all was not a small fluctuation.
I asked my new colleagues to check the monitoring.
Everything appeared healthy.
The databases were available. Replication was current. The web servers were running. Load, network traffic and connectivity between both data centres looked normal.
There was no obvious reason to worry.
But the distance between everything is green and there were no subscriptions was too large to dismiss.
Starting with the business result
I still knew very little about the environment, so I asked colleagues to explain the transactional database to me.
The data confirmed the CIO’s concern.
There had been no subscriptions. There had not even been any new leads — the earlier step where a visitor left a mobile number to begin the opt-in process.
The failure was therefore not limited to the final transaction. Customers were not entering the process at all.
I opened several landing pages.
They were blank.
The web servers still returned a successful HTTP response. From an availability perspective, the request had worked. From the visitor’s perspective, there was nothing to use.
I ran one of the PHP files directly and saw errors that had been hidden in the web response. A shared file could no longer be included correctly.
The immediate cause was small: a missing semicolon in logic that selected a tracking pixel for a newly added affiliate.
The impact was not small.
To make new affiliates quick to connect, all of that logic lived in one shared file included by every landing page. One typing error had therefore affected every page at once.
The fix itself took little time.
The more important question remained:
How could the entire commercial flow disappear and remain unnoticed for so long?
A delay built into the organisation
The error had been introduced when the new affiliate was added.
For part of that first day, the failed traffic disappeared inside the normal scale of the business. By the following day, the flow was completely down.
But the organisation did not receive its consolidated sales totals in real time. Business intelligence produced daily summaries. The absence of sales only triggered an investigation halfway through the next morning.
By then, the platform had been returning blank pages for at least a full day.
The infrastructure monitoring had done what it was designed to do. It showed that servers, databases and networks were available.
It did not show whether a visitor could move from a landing page to a lead and from a lead to a subscription.
Repairing more than the typo
We first removed the fragile affiliate logic from the shared code.
Instead of extending a growing set of conditional statements, affiliate-specific information was stored in the database. A mistake could then affect one configuration rather than every landing page.
Later, an interface was added so commercial teams could configure new affiliates without waiting for a technical work order and a manual database change.
But I was still not satisfied.
I began checking pages myself, regularly, to make sure the customer-facing flow still worked. That reduced the risk, but it did not scale and depended on someone remembering to look.
I considered adding a fixed marker to every page and checking whether it appeared. That would have detected another total white-page failure.
It would not have detected a gradual fall in leads or sales.
So I built a small tool for myself.
Watching the flow, not only the components
The first version followed three events: a page view, a lead and a completed subscription. It showed both the volumes and the conversion between them.
At first, the purpose was modest: reduce the amount of manual checking and make unusual changes visible sooner.
Then the complications appeared.
Normal conversion differed by market, product, mobile operator, affiliate network and time of day. A number that was alarming in one country could be ordinary in another. New products and affiliates had no long history against which to compare them.
The tool gradually learned those differences. It used recent behaviour to build expectations, created drill-downs and added new products, markets and affiliates to the monitoring automatically.
When the observed flow moved far enough outside its expected range, it generated an alert.
The small monitoring page became the Business Monitoring System.
From private tool to shared view
I had built it for myself.
Then application managers began using it. Sales and management followed. It became a real-time view of the business, often trusted more than the existing reports that depended on commercial tooling and delayed data-processing jobs.
That success created its own technical problem: a tool designed for one user now had many. I rebuilt parts of it and added caching so that the act of consulting the system would not overload the databases it was meant to observe.
From then on, business-impacting failures became visible.
In small or newly launched markets, low traffic could mean that an alarm needed more time to become reliable. Some interruptions might take minutes, and occasionally hours, before crossing the alert threshold.
But they no longer disappeared.
And when an application manager decided not to call a mobile operator in Brazil in the middle of the night, that became a conscious operational decision — not a failure nobody knew about.
What the green lights had not answered
The original monitoring had been correct.
The databases were up. The web servers were up. The network was up.
What it had never been asked was whether the customer journey still produced a result.
The CIO did not need to know that a semicolon was missing. “There were no subscriptions” was already a valid place to begin.
The immediate defect was repaired in minutes.
The lasting change was making sure that, from then on, someone could see whether the whole path from visitor to business result was still working.