A power outage took out the on-premise server running an invoice matching service for one of our clients. The service reads vendor invoices, compares them line by line against purchase orders in P21, and posts vouchers for the clean matches. Losing it for a day is annoying and survivable: the invoices queue up in a mailbox and nothing is destroyed.
The interesting part was not the outage. It was the plan for what happens when the power comes back, and working through it turned up two things worth writing down, because we have since found both of them at other clients who were sure they had continuity covered.
Failover Is Easy to Buy. Failback Is Where It Breaks.
Standing up a standby copy of an application somewhere else takes an afternoon. That part is genuinely solved, and it is the part vendors sell you.
Now consider the moment the utility restores power. The containers on the original box are almost certainly configured to restart automatically, because that is the sane default and somebody set it years ago so the service would survive a reboot. So the old server comes back by itself, with no human involved, and starts doing its job. If the standby is still running, and if both of them point at the same database, you now have two independent writers processing the same queue.
Nobody chose that. It is the emergent result of one restart policy and one afternoon of good intentions.
Duplicate Postings Do Not Reverse
The reason this matters more for ERP automation than for a website is that the writes land in someone else's system of record. A voucher posted twice into the ERP is not an undo. It is a credit, a phone call to accounts payable, a correction that has to be explained at month end, and a permanent dent in the argument that the automation can be trusted. Two writers processing one invoice queue can produce a week of that in an hour.
Put in those terms, the risk math inverts. The cost of being down for a day is a day of delay. The cost of coming back wrong is a reconciliation project plus the political capital you needed to expand the automation next year. Given a choice between the two, staying down longer is usually correct, and almost no continuity plan says so.
Find Out Where Production Data Actually Lives, Now
The second finding was more embarrassing and more common. Everyone involved believed the application database was hosted, on a managed cluster, off the box. It was not. The hosted cluster was a free tier instance used for development, and production data was running on the same physical server as the application, alongside two other services.
Which means during the outage there was nothing to fail over to. The standby could have been started in minutes and it would have had no data.
This is not incompetence, it is drift. Someone set up the hosted cluster during development, someone else deployed to the on-premise box with a local database for speed, and the connection string that documents the truth is in an environment file nobody has read in two years. The fix costs an hour: for every service, write down which host holds the production data, which host holds the application, and which of those two facts you last verified by looking rather than remembering. Most teams do this exercise during the outage, at the worst possible time, with the client on the phone.
What a Workable Plan Looks Like
None of this requires expensive infrastructure. It requires four decisions made in advance.
- One writer, enforced by the software, not by a policy. The service should check for a lease or an active flag at startup and refuse to process anything if it does not hold it. A rule that lives in a runbook is a rule that fails at 6 AM on a holiday weekend when the person who read the runbook is not there.
- Come back paused, not running. The restart policy that survives a reboot is correct, but what comes back should be the application, not the ingestion. Resuming processing should take a deliberate human action, because that action is the only place a person gets to ask whether the other copy is still running.
- Make the queue the safety net. If intake is a monitored mailbox or a durable queue, a pause costs time and nothing else. Nothing is lost, work waits. Designing intake so that stopping is survivable is what buys you the freedom to stay down until you are sure.
- Name the person who declares failback. Not a team, a person, with a named alternate. The failback decision is the one that creates duplicates, so it needs an owner in the same way a production release does.
And then test the failback rather than the failover. Everybody rehearses the part where the standby comes up. Almost nobody rehearses powering the original server back on while the standby is live, which is the exact sequence that will happen for real, and the only one that produces the failure that costs money.
Uptimize Solutions builds and runs the automation that sits beside your ERP, which includes being honest about what happens to it during a bad week. If you are not sure which server holds your production data, see our system integration services or book a free workflow audit and we will map it with you.