Big Data Ingestion
Most organisations have an initiative to capture and store unstructured data . Much of the focus is on the HDFS deployment , yet the majority of work involved is in the plumbing to get the data into HDFS. Magtia have built an ingestion framework that takes cartridges built specifically for financial gateways and protociols e.g. FIX , FPML .
Magtias “adaptors” eavesdrop on the existing gateways and replicate the data payload for ingestion. The framework originated in an exercise to capture web clicks from a search engine and insert into HFDS in near real-time . The framework is horizontally scalable and ingests data in real-time as opposed to End of Data batch. The framework has an operational console that allows for the monitoring of the gateways and the centralised management of resources e.g. in the event a gateway adaptor reports a back log is building, the operator can deploy and run more processes from the console.
Hence the framework accelerates projects and removes many of the operational headaches of managing the ingestion of data
- Integrated approach to deployment , monitoring and management of ingestion gateways
- Library of financial gateway taps and adaptors
- Adaptors and taps can be retro fitted to existing infrastructure and gateways
- Near real-time ingestion data rate to HDFS
- Centralised management and deployment of gateway cartridges
- Monitoring dashboard built in to the solution
- Accelerate projects deployment times
- Future proof against increasing data volumes
- Rapid ROI
- Complete with monitoring system of end solution
- 3rd party management of protocol updates and maintenance of multiple versions