Disaster Recovery Strategies for Trading Firms
“If you fail to plan, you’re planning to fail,” so goes the famous quote by Benjamin Franklin that was later paraphrased by Winston Churchill. Far from just being a call to general preparedness, it’s also an appeal to protect the progress you’ve already made. To keep what you’ve built secure. The idea of insuring against unexpected events is now almost universal, but in our increasingly interconnected world, disaster recovery strategies for certain classes of businesses, particularly small-to medium financial services firms, is still sadly lacking.
So, what if you turn up for work tomorrow to find your headquarters a twisted pile of rubble? This is probably the worst case scenario (barring alien invasion or thermonuclear fallout), but it’s also where disaster recovery planning begins. Other possible calamities include outages, cyberattacks, internal system failures due to hardware or software failures, and even de-platforming from app stores for companies that are overly reliant on the Apple and Android ecosystems.
What is Disaster Recovery?
The idea is to have a disaster recovery plan with well defined protocols that allow your organisation to bounce back as quickly as possible, no matter how severe the hit. Every minute a company’s systems are down it haemorrhages money.
Every second clients can’t access their accounts, or fail to get meaningful answers and solutions, the broker suffers irreparable reputation damage.
In the above worst case, we assumed the destruction of property as well as equipment and records. Further down the list we had outages to power or network services, which can also cripple a company without affecting it physically. For all unforeseen events that lead to a business not being able to serve its clients online, a number of general recovery strategies exist that apply broadly to many types of business. They involve routine data back-up to a remote storage as well as the remote hosting of a duplicate website that can be switched to in the event of an emergency.
Disaster Recovery for Financial Firms
At Devexperts, our focus is specifically on financial services firms, which are a unique class of business where the stakes are naturally higher. Firms involved in financial markets not only have to contend with all the hosting and backup issues that affect other types of businesses with an online footprint, but they have a host of much stricter connectivity requirements to both clients and liquidity providers. Furthermore, the very nature of financial markets makes any downtime extremely problematic. Consider clients with open positions that can’t be closed as the market turns against them, pending orders that can’t be modified, or trading algorithms that can’t be shut down.
Avoiding Single Point of Failure:
Backup Site
To protect against the absolute worst case scenario where a company’s physical location comes under threat, a dedicated standby system running in parallel to the primary one can be deployed at a remote location. This standby system is also running broker’s applications that work in a read-only mode and do not process client requests. It becomes active in the event of an emergency. Requests are routed to this standby system, which is up-to-date with the current state of the primary system up until the point it ran into trouble.
This allows for business continuity in the event of a disaster affecting the physical location of the primary system, or for the primary system to be taken offline in the event of maintenance, or a hardware or software failure for troubleshooting and repair. The systems are kept up to date in a bi-directional manner, so when the primary system is active the standby is updated, and when the standby system is active, the primary one is updated with the new up-to-date state of the standby system.
Redundancy of Providers
Many financial services firms, especially smaller ones, could not afford complete system redundancy in the manner outlined above. This leaves them vulnerable to severe disruptions such as those caused by natural disasters or extreme weather conditions. However, the cost of implementing and maintaining such systems can be prohibitive as it essentially involves completely duplicating the trading infrastructure at a second location.
In order to mitigate the risk of an LP not accepting the trades, or the incoming feed serving stale quotes, brokers can add specific solutions to their setup. They can work with multiple liquidity providers via an aggregator, or use professional data vendors such as dxFeed, allowing them to continue serving price data even in the event of a problem with one or more of their feeds. This, of course, only addresses issues affecting LPs and fails to address internal hardware and software issues, network problems, DDOS attacks or other cybersecurity threats. The above should be regarded as the bare minimum in terms of disaster readiness.
Fallback Trading Apps
An underestimated but highly-effective middle ground between full off-site redundancy and no fail-safes at all, is the creation of a basic platform to be used only in the case of emergencies. The solution is not intended to replace existing brokerage systems in the event of failure, but rather to provide a temporary means for clients to access their accounts so that they can manage positions, or to provide staff members with the ability to do so on their behalf.
It prevents brokerage businesses from grinding to a halt when a failure occurs. More importantly, it allows clients to be able to do something while they await the resumption of normal service. Anyone who’s had the experience of their trading venue going down will tell you what a stressful time this can be, particularly when the failure occurs at moments of high volatility. Traders will punish brokers that let them down in this manner by taking their custom elsewhere and reward reliable ones or those with proven fail-safes.
At Devexperts we have developed several such fallback systems in the past, and they have provided notable advantages to the companies that have deployed them. The briefs have usually called for the creation of stripped-down trading interfaces that allow clients to login with their existing credentials. The software integrates with the most recent backup of the production database and allows clients to view all their trading activity such as open positions, pending orders or social trading preferences.
These fallback platforms are not usually intended for full trading, as such, but just as a means for clients to close open trades or pending orders until the main platform comes back online. During 2020 alone, household name brokers suffered long lasting outages that took their systems down for hours..
Solutions such as this can also be used as backups in the case of de-platforming, i.e. suspending broker’s app from the stores, as they allow clients to access an alternate web-based platform, until the decision to have the primary application removed is appealed.
Think Outside of the Box
Our Devexa smart assistant allowed brokers to open a further line of communication with their clients that’s both efficient and scalable. The same benefits it confers to busy customer support teams inundated with calls, emails and live chats, can also be a leverage in a disaster recovery scenario. For any disruption that does not directly affect a company’s trading servers, Devexa can be used as an alternative method for sending trading instructions. Much like the practice of calling your trades in directly via phone, Devexa allows positions and pending orders to be placed, modified or cancelled simply by texting or voice messaging.
Removing human support from this process brings scalability. The difference with trading over the phone is that Devexa is plugged directly into the trading servers, so she can handle far more than one instruction at a time. This means no human-related bottlenecks in cancelling trades and pending orders in the event that web terminals, web servers, or data centres are down or under attack. The same goes for issues with desktop platforms, or in the event that a mobile app removed from iOS or Android app stores. The key here is to have workarounds for every conceivable type of disaster that inspire confidence in your clients.
Final Thoughts
Striving to stay competitive, brokers are trying to move fast but break things: marketing can come before product or business development, budgets can be overstretched in the interests of causing a splash, and adequate headroom is rarely in place even when it comes to peak server loads, let alone disaster recovery.
As the space grows evermore competitive, the cost of not having reliable fallbacks in place and disaster recovery protocols is becoming too great to bear. Especially as cybercrime becomes a greater part of our daily lives and even the physical world around us doesn’t seem as predictable as it once was.