Migrating Market Data to a Cloud Environment
Physical storage is finite. Market data is not. Every day, financial markets generate terabytes of new data to add to the bulk volume of historical market data firms need to make critical decisions. Helping to take that burden off physical infrastructure is cloud migration—firms can deliver historical market data to public, private, or hybrid cloud environments, for use cases that include application development and strategy backtesting.
Market participants, like others in the financial services industry, have been slow to incorporate functions into the cloud, often because of worries about security. The landscape has shifted in recent years, with firms becoming comfortable with the security of public cloud providers such as Amazon Web Services (AWS), Google Cloud Platform, and Microsoft Azure.
In fact, a survey of C-level information security professionals found that most considered cloud environments to generally be as safe or safer from data breach than on-premises data storage.
2020’s work-from-home environment brought even more attention to the advantages of the cloud for financial services firms. Teams sitting in their homes across town—or across the globe—can work with datasets easily. As a result, cloud migration is increasingly on the agenda of market participants of all sizes.
Regardless of how your firm plans to use historical data and in what cloud environment you are operating, obtaining the data you need in the format that works doesn’t have to be complex and error-prone. Exegy has been delivering data to the cloud for years—the most visible example of this being an arrangement with FINRA to provide mission-critical market data for the Consolidated Audit Trail via AWS.
The process of migrating market data to the cloud does come with some decisions and caveats, but it doesn’t have to be all or nothing. Many firms negotiate a gradual transition, doing more in the cloud as they grow more comfortable with it. Schedule a free consultation with Exegy’s experts to talk about your firm’s path.
Carrying Out Mission-Critical Functions in the Cloud
All the major use cases for historical market data can be performed in a cloud environment—and often at an advantage over traditional on-premises functionality:
- Application development and testing—Developers can use historical data as they write trading applications, running their applications against the data before and after code changes. The goal is to pull data into the environment and test it quickly, paying for compute time by the second.
- Regulatory compliance—Regulations in the US and abroad require ever-larger volumes of market data, to bolster record-keeping and show how risk management is carried out. Depending on a firm’s approach, cloud-based solutions can amass data more economically or provide easier access to only the information needed.
- Archiving and data lake creation—Cloud storage is the only practicable solution to managing large data sets in varied formats. Firms needing to keep data in the same format that it was transmitted by the exchanges can benefit, as well as firms seeking out alternative data.
- Tick database and analytics tools—Major database providers, including Kx Systems, OneTick, and TickSmith can be deployed in the cloud and work with firms’ data.
- Strategy backtesting—Many data analytics tools use normalized market data in comma-separated values (CSV) file format to test strategies, a process that can be carried out efficiently via the cloud.
AI and machine learning applications are particularly suited to cloud development; they require vast amounts of data that can be more economically processed in the cloud. Computing power can be expanded when needed and pulled back when conditions no longer require it. This is of particular advantage in volatile markets, where a strategy may be needed in a hurry, but of finite usefulness.
A firm using Exegy’s Capture Replay appliance to record and replay historical data can easily move that market data to the cloud for storage. Once imported into a firm’s cloud environment, the same data can be used for all these functions, ensuring consistency.
An Evolving Cloud Footprint
Firm’s cloud configurations can vary significantly, including private cloud infrastructure maintained by the company, public cloud environments, and hybrid models made up of some combination of the two.
Many firms typically start out moving older, less-used data to cloud storage, to free up on-premises infrastructure for more mission-critical uses. But as market data volume balloons, savvy firms have begun to see the benefits of cloud migration for more active data.
North American equities and commodities feeds now generate 2.5 terabytes of data every day—more than double the volume of only a few years ago. Meanwhile, the massive Options Price Reporting Authority’s SIP feed can generate 10 or more terabytes a day, up from about 6 TB in 2019.
As a firm’s plans evolve and the amount of stored data expands, it’s easy to expand the cloud footprint to match it.
There are some considerations firms must work out with their cloud providers and internally before migrating market data to the cloud:
- Provider billing—Major cloud providers all have their advantages and disadvantages; firms will have to choose the one that best suits their needs. But they create accounts and bill for their services in different ways, and a firm will need to understand upfront how this process works, to avoid unpleasant surprises down the road.
- Data retention—Like billing, cloud providers store data in different ways, so firms must have a plan for how to retain it in a way that makes it easy to access when needed.
- Moving data across regions—Most cloud providers charge more to move data from one region of the world to another (from APAC to North America, for example). For large market data consumers with a global reach, those costs can add up. As a result, plans should allow for some functions (e.g., pre-trade risk analysis or transaction cost analysis) to be carried out in the cloud environment closest to where the data is generated. Then only the necessary data needs to be transmitted to another region.
Exegy can Support Your Cloud Migration Plans
Every firm is different, and cloud computing needs can vary—from storage to simple computing capability to accommodating networks of containerized applications such as Kubernetes clusters. Whatever the scenario, Exegy has seen it before.
Exegy can work with your firm to store and use historical data in the cloud in all the formats you need: Replay via the Exegy Client API (XCAPI), PCAP or raw data files, and CSV files that are used with tick database providers.
We can migrate data to an existing private or public cloud, providing all the features of our Exegy Capture Replay (XCR) appliance within the scalability of a cloud environment. This configuration ensures that a capture device won’t run out of storage space and allows you to spin up as many instances of needed services as your various teams require—around the globe, if necessary.
The XCR can also smooth out delivery of bursty market data to the cloud, helping to optimize available bandwidth. We’ll discuss this capability in more detail in a future blog post.
Exegy’s services are cloud provider-agnostic, so we can work with any provider to take you through your first steps. We can help you collaborate with other organizations and their own cloud environments. Wherever you are on your cloud migration plan, Exegy can ensure you remain connected to the data you need. Request a free consultation with our team to see how to get started.
