The Data & Reporting team's main focus is gathering of flight, price and availability data from third party websites and selected partners. Every day is different, with unique challenges ranging from implementing data collection from new websites, fixing and maintaining our links to existing partners, or setting up tracking essential to our business model.
The code-base for the web scraping library is written in Python, and XML/Regular expressions are used to parse out relevant data from third party websites and APIs. The Data & Reporting team also manage and maintain our relationships with the Global Distribution Systems (GDS) that provide pricing information for many of our affiliated airlines.
The reporting element of the team focuses on the storage of data in our internal logging system. The logging system uses a combination of Python and RabbitMQ to validate, correlate and insert the data into the high volume OLTP system based on a SQL Server platform.
This data is then exploited to provide insights and information across the company, including monitoring operational environments, dealing with commercial queries, and undertaking research projects when required. Other important responsibilities include the maintenance of the business intelligence infrastructure and governance of all data that enters the logging system.
