ExxonMobil’s journey to unleash time-series data with open source technology
Download as PPTX, PDF1 like746 views
The document outlines ExxonMobil's use of open source technology for managing time-series data, detailing their global data collection strategies and various technologies including Apache NiFi, Spark, HBase, and Hive. Key challenges include data contextualization and normalization due to variations in metadata and sensor information. The document emphasizes the importance of effective data storage, aggregation, and consumption for enhancing data utility across the company.
ExxonMobil’s journey to unleash time-series data with open source technology
- 1. ExxonMobil’s journey to unleash time-series data with open source technology June, 2018 Kevin Brown Big Data Platform Engineer
- 2. ExxonMobil’s journey to unleash time-series data with open source technology June, 2018 Kevin Brown Big Data Platform Engineer
- 3. Introduction 3 Kevin Brown Big Data Platform Engineer ExxonMobil
- 4. Introduction to time-series data at ExxonMobil Global data collection and ingestion with Apache NiFi™ Apache Spark™: normalization, validation, aggregations, and interpolation Apache HBase® & Apache Hive™ partitioning, performance Consumption APIs Today’s Objectives 4
- 5. A series of data points indexed in time order. What is time-series data? 5
- 6. • Global refineries and chemical plants • Millions of sensors/tags • Decades worth time-series data Time-series data at ExxonMobil 6
- 7. Collection and Ingestion with Apache NiFi™
- 8. • Interoperability • Ease of use • Fine control of flow Why Apache NiFi™
- 10. Simple Regional Design - NiFi 10
- 12. • Repository sizing • Run Schedule • Back Pressure • Monitoring • NiFi Expression Language Apache NiFi - Flow Design 12
- 14. “Your metadata is key, but it’s probably not consistent.” Data Contextualization • Global data challenges • Language and abbreviation variation • Diversity of vendors and tools. • Naming standard • Variance in functionality (confidence levels, frequency, resolution) • Synchronization and mutable data • Calculated tags, faulty sensors • Delayed lab test results Getting your data ready 14
- 16. • Archival and compression of data • Interpolation • Aggregation • Partitioning “Sequence Matters” Storage, interpolation, aggregation … 16
- 17. Consumption 17
- 18. • Who are your users? • Apache HBase® & Apache Hive™ ? • Off cluster? • APIs • Serialization Consumption 18
- 19. Questions 19
- 20. Apache HBase ® , Apache Hive™ and Apache NiFi™ are trademarks of the Apache Software Foundation.
Editor's Notes
- #4: Personal Introduction: Education 2012 – BYU – BS in Information Technology ExxonMobil 6 years @ ExxonMobil, Linux Systems Administrations Pioneered our initial journey into Hadoop/Big Data Responsible for architecting/maintaining/supporting our current Big Data platform
- #11: From Left to right: Data originates at site Regional NiFi Instance pulls from sites. Data is sent securely to a clustered central NiFi.
- #17: Archival and compression of data Interpolation Rely on both Step and Linear depending usually on data type. (float with no manual input = linear) otherwise step (integer, or float with manual input) Aggregation Partitioning