The energy industry faces unprecedented challenges, and opportunities, in working with the vast amount of data gathered in extraction processes. However just gathering terabytes upon terabytes of data from thousands of sensors across oil, gas, geothermal, wind, solar, and more isn’t enough – the real challenge is then storing, transforming, integrating, and finally analyzing that data for useful and actionable insights. The potential value is granular insight into maximum resource collection at minimal impact and cost.

The Challenge

Manipulating and building flexible, scalable, and powerful data flows at rapid speed is critical to getting accurate insights and visualization. Having a system that is able to scale up and down instantly, work seamlessly with every data system in the end-to-end pipeline, and handle structured or unstructured data is key. Data has to be integrated from different API protocols like REST, SOAP, ETP, etc. without breaking across different protocol versions and updates. Finally, on top of a system with these capabilities, enabling technical and non-technical analysts and data scientists to work collaboratively is required to reduce back-and-forth, while simplifying the process.

The Solution

Nexla’s no/low-code solution for rapidly building powerful data pipelines powered by data products is perfect to address this particular data challenge. Several features unique to Nexla’s offering apply directly to energy data use cases. Integrating metadata intelligence to Nexla’s automatically generated data products enables scalable, customizable, pipelines that feed always reliable insights and visualizations. One of them is Nexla’s out-of-the-box WITSML parsers that make data useful from well log formats that work for any version of WITSML. Nexla is also API protocol agnostic and version agnostic – which means it will always work with ETP, SOAP, REST, and other APIs with any past, present, or future version. 

Let’s take a look.

Flexible Connectors and Technology Mix

The end-to-end data pipelines required to perform analysis and get insights require different types of data and data systems. Nexla seamlessly integrates with many of them out of the box, while being able to connect to any new data system without writing code. This supports data at any speed from batch to real-time streaming on Kafka, Amazon Kinesis, or other streaming platforms. In addition, Nexla’s universal connectors work just as easily with a custom REST API built by a developer as your Snowflake warehouse, Tableau dashboard, and much more. Choosing wherever the data is coming from or needs to be is as easy as selecting the connector and your credentials.

Support for File Protocols and Formats for WITSML and Other Energy System Formats

The Wellsite Information Transfer Standard Markup Language (WITSML) standard from Energistics for consistent XML documents of wellsite data plays perfectly in Nexla – as well as JSON, CSV, Parquet, and any other format data might come in. Due to Nexla’s ability to generate standardized Data Products from any data, formats and protocols are easily abstracted away to reduce complexity. Receive and Send data in any format, velocity, and system, including WITSML. Then use Nexla’s transform builder to parse nested information in the log into a usable and workable table format.

Introducing Nexsets – Automatically Generated Data Products

Nexsets are automatically generated data products from any source. Nexsets abstract the velocity, schema, protocol, version, and underlying data format.  The moment you connect Nexla to a WITSXML server, Nexla automatically detects the nexsets and presents them in an easy to use interface. Users can then filter, transform, combine multiple of these nexsets together to create more ready to use nexsets.

Example Flow From a Real Oil Well Built in Nexla in Minutes

Putting all of these features together, we can build flows like the one below in minutes in Nexla. In this flow, WITSML format data is ingested – in this case the Rig Data logs – then packaged into a data product, transformed based on a schema template, and sent to an S3 bucket, a Snowflake Database, and a Tableau dashboard for real-time visualization. 

The WITSML data we receive is in a messy nested XML format, which isn’t easy to analyze. It needs to be converted in a format that renders itself to easy analysis and consumption.

Using the transform builder in Nexla, we can apply no/low code transformations to parse the information, apply mathematical, statistical, etc. transformations to attributes, and then send the cleaned table to any destination. The cleaned table format data is sent to multiple destinations:

1. Data Scientist – Parquet format on S3, so that they can be loaded directly into Data science libraries 
2. Operational user – Tableau, operational user can load the Nexset on Tableau to analyze the data.

We can send the same data to multiple other destinations in Nexla  – 

1. Message bus like Kafka
2. API’s to power internal/third party apps.
3. Data warehousing tools like Snowflake, Redshift, Bigquery, Synapse, etc.
4. Data lakes like S3, GCS, Azure blob storage, etc.

Building this flow was possible with no code at all, in minutes in the Nexla UI. With that capability, data can be made ready to use and analyze in minutes rather than days to week. While is a simple example of a flow, more complex operations are extensible with custom transformations, other connectors, and the use of Nexla’s CLI and SDK.

Wrap-Up

The energy industry has tons of data from thousands of sensors. The challenge today is providing the tools and platforms for users to build the necessary data pipelines on the fly to derive meaningful insights quickly. With Nexla, building pipelines to and from any data system in the pipeline is made easy, with full support for the systems, formats, and protocols unique to the energy industry.

To learn more, reach out for a 1:1 demo tailored to your use case or to request a free trial.