Premature Server Decommission Knocked Out StorageCraft's Recovery Service for Weeks

Premature Server Decommission Knocked Out StorageCraft's Recovery Service for Weeks

A month-long disruption to StorageCraft's disaster recovery as a service (DRaaS) traced back to a botched maintenance procedure has renewed warnings from industry analysts that no single backup provider or location should be trusted as a sole safeguard. The outage, which began March 9 and wasn't fully resolved until April 8, left some Arcserve partners unable to replicate or fail over machines in affected data centers.

What went wrong

According to Arcserve CEO Brannon Lacey, a server array holding critical metadata was decommissioned prematurely during planned maintenance. That error corrupted metadata and permanently severed links between the storage environment and the StorageCraft DRaaS cloud, affecting regions in the United States, Ireland and Australia. Engineers identified the root cause by March 12, but partial outages in Ireland and Australia persisted until early April. Arcserve has not disclosed how many customers were affected or how much data was lost, though it says impacted partners were notified and given recovery options based on the type of machine they run.

Lacey said the company is "undertaking a complete internal review and audit" to prevent similar human errors going forward. Christophe Bertrand, practice director at Enterprise Strategy Group, noted that while cyberattacks and external events can cause outages, the bar for backup and DR vendors to control internal mistakes is especially high given what customers rely on them for.

Why redundancy still matters

Analysts say the incident underscores a long-standing best practice: don't concentrate all data protection in one service or location. Johnny Yu, research manager at IDC, pointed to the classic 3-2-1 strategy - three copies of data, on two different media types, with one copy off site - and added that an offline, air-gapped copy is increasingly essential given the prevalence of ransomware.

  • Keep a local backup alongside any cloud-based copy for operational recovery if the cloud source becomes unavailable.
  • Maintain an offline copy that ransomware cannot reach or encrypt.
  • Regularly test recovery processes, including full disaster recovery drills, to catch gaps before they cause downtime.

Krista Macomber, senior analyst at Evaluator Group, recommended routine documentation and testing of recovery procedures to confirm recovery point and recovery time objectives can actually be met. Bertrand added that a local backup doesn't need to be extensive - even a few days' worth of recent backups can be enough to keep operations running if a primary cloud service fails.

Context for a growing market

Use of backup as a service and DRaaS has climbed steadily: ESG's 2021 "Evolution of Data Protection Cloud Strategies" report found 69% of surveyed organizations used BaaS that year, up from 39% in 2016, while DRaaS adoption rose to 60% from 39% over the same period. StorageCraft, which merged with Arcserve in 2021, brought its OneXafe storage appliances and ShadowProtect software along with a managed service provider customer base that often lacks large in-house IT teams - making reliable, well-communicated service recovery especially important when outages occur.