Operating-system patching at the bank followed the usual sequence through development, test, acceptance and production.
Development and test had completed without incident.
Then the acceptance environment was patched.
Partway through the cycle, the load-balancer team called a stop. Application pools were gradually dropping their backends. By the time the pattern was recognised, many servers had already received the update and a growing number of pools were close to empty.
The immediate assumption was that the patched application servers had failed.
They had not.
Healthy when tested directly
I checked several of the servers that F5 had removed from their pools.
The network connection worked. The web server responded. After temporarily directing requests straight to individual application servers, I could open the websites normally.
The servers were available and the applications worked when approached directly.
So the relevant question was no longer:
Why are the servers down?
It was:
Why does the load balancer consider healthy servers unusable?
The network-level checks between F5 and the servers were successful. I then asked what application-level health check was being sent.
It was a minimal HTTP request without a hostname.
That request had worked for years.
The load-balancer configuration had not changed.
Apache had.
A hidden assumption stops working
The monthly patch set included an Apache update.
After the update, Apache no longer accepted the incomplete health-check request as it had before. Instead of returning the successful response F5 expected, it returned an error.
F5 behaved correctly. A failed health check meant that the backend should be removed from the pool.
Apache also behaved correctly. The request it received was incomplete.
The patching process had done what it was supposed to do.
The failure existed only in the way those working parts had been expected to collaborate.
Development and test had not exposed it because those environments were not behind F5. Acceptance was the first place where the full chain existed.
The ideal fix would take too long
The technically clean solution was clear: every health check should request a hostname that the relevant application actually served.
At this bank, that was not a small change.
There were roughly 300 affected pools in acceptance and another 600 in production. The load-balancer team did not always know which hostname belonged to which backend. Finding out would mean identifying hundreds of responsible teams, requesting the information, validating it and implementing each change through the normal process.
Meanwhile, delaying the patch would leave systems unnecessarily exposed.
We needed a response that was valid, safe and applicable immediately at scale.
A temporary bridge
Apache falls back to its default website when it receives a hostname that has not been configured as a virtual host.
So instead of sending an incomplete request, the health checks were changed to include an explicit placeholder hostname.
The placeholder was deliberately recognisable. It made the request valid and restored the expected response, while also marking the pools whose health checks should later be replaced with an application-specific configuration.
The backends returned to their pools.
Patching could continue.
The same change was prepared for the remaining acceptance pools and for production.
Making the solution fit the organisation
The technical ownership crossed several boundaries.
My team managed the operating system. The load-balancer team controlled F5. Another team formally owned Apache, although its main expertise was in Java and WebSphere rather than the web server itself.
The fix still had to enter the bank through the normal ownership and change process.
I prepared the technical explanation and implementation communication so the responsible team could propose and apply the change unless an application owner had a reason to object.
The immediate problem was solved without waiting for hundreds of perfect configurations.
Whether every placeholder was later replaced is another question.
Where the failure really was
Nothing in the incident required a dramatic component failure.
The servers were healthy.
The load balancer made the correct decision based on its health check.
Apache correctly stopped accepting a request that should never have been considered complete.
The outage appeared in the assumption between them.
Once that assumption was visible, the challenge was not only to find a technically correct answer. It was to find one that could restore service across hundreds of configurations, keep the security patch moving and still leave a path toward the better long-term design.