UniSuper’s week-long systems outage has been traced to a “combination of rare issues at Google Cloud” that collectively caused a misconfiguration during resource provisioning and triggered a secondary software bug.
The industry superannuation fund last week said an “isolated” issue at Google Cloud was the reason for its online services being offline.
The cloud provider has now provided its own explanation for the outage – that a cascading series of “issues” led to problems with UniSuper’s primary and secondary systems.
“The disruption of UniSuper services was caused by a combination of rare issues at Google Cloud that resulted in an inadvertent misconfiguration during the provisioning of UniSuper’s private cloud, which triggered a previously unknown software bug that impacted UniSuper’s secondary systems,” a Google Cloud spokesperson told iTnews.
“This was an unprecedented occurrence, and measures have been taken to ensure this issue does not happen again.”
The exact combination of issues that led to the misconfiguration and its flow-on effects is still not clear, nor is the architecture of UniSuper’s environment.
The fund only moved many of its workloads into the Google Cloud Platform over the past year, having previously split them between Azure and two of its own data centres.
It is known to use the Google VMware Engine (GCVE) managed service on GCP; since it already ran workloads in VMware virtual machines, this made cloud migration effectively a re-hosting exercise.
UniSuper CEO Peter Chun said yesterday that restoration works would likely begin on Thursday this week.
“The progressive restoration of member services will begin Thursday May 9,” he said.
“Please note that some services will still be limited as we continue the restoration.
“Services that we expect to have online in some capacity will include the ability to login to online services, access mobile app, and see balances—initially as of Monday April 29 2024.
“As regular trading and investments have been continuing as normal, this will be reflected in [customers’] balances once our systems have been restored.”
In an updated FAQ, UniSuper suggested it was able to lean on other unspecified cloud systems to operate in some capacity, as well as for data records.
“Google Cloud is not the only cloud service provider UniSuper utilises, and this planning has ensured our ability to restore services and minimise data loss,” the fund said.
Google Cloud’s spokesperson said its teams continued “to work around the clock with UniSuper to fully remediate the situation, with the goal of progressively restoring services as soon as possible.”
