What Is Data Exchange? How to Buy and Sell Third-party Data

Let’s say you’re emailing your colleagues an important document.

Real simple, right? You attach the document to your email and hit the send button. Your co-workers almost instantly receive the data they need. 

Now, think of this information exchange on a bigger scale. 

Your company deals with suppliers, manufacturers, customers, and vendors. They share all sorts of information like inventory provisioning data, product maintenance information, construction drawings, simulation models, quality planning data, contracts, commercial documents, program source codes – the list goes on. And, all this data comes in different formats. 

How do you normalize this huge volume of data without changing its meaning? That’s where data exchange comes in. Data exchange software offers data-as-a-service  (DaaS) capabilities to help providers and consumers share and procure information effortlessly. As a result, companies can gather market intelligence and fuel data-driven decisions with minimal lift

Data exchange ensures smooth data transfer between data suppliers and consumers. Suppliers, data syndicators, and brokers share or sell data. Consumers collect or purchase data from data suppliers.

A data exchange platform allows suppliers and consumers to exchange, commercialize, source, and distribute data. These platforms help suppliers and consumers meet legal, security, technical, and compliance requirements.

Importance of data exchange

Companies produce, collect, and acquire huge volumes of data from daily operations. However, this first-party data is barely sufficient to make business decisions based on fresh perspectives. That’s when companies become data consumers. They use verifiable second-party and third-party data points to fill information gaps, analyze data, and meet intelligence needs.

On the other side of things, data distributors selling data don’t always have as much information as they need. They use online data exchange to monetize informational assets and acquire data from other sources. When data isn't useful, businesses monetize it. Most companies use data and sell it to other firms.

Companies value data exchange because it ensures data quality – something that the traditional data buying and selling process often overlooks.

When data consumers purchased data sets in the past, they found heaps of duplicate records. Sometimes, the data lacked regularity and normalization. Other times, the data contained missing records, nulls, invalid numbers, and illegible labels. Data exchange software solutions eliminate these problems by letting buyers preview data and address quality issues before purchasing. 

Data exchange also solves data discovery issues. Previously, organizations had to browse through countless websites before acquiring data. Add to that the great ordeal of price negotiations, contract signing, data cleaning, and integration. Not a great equation for good business.

Data exchange systems make the entire process effortless for consumers and suppliers. Data consumers can use multi-criteria, filtered searches, sampling tools, and data visualization to find what they’re looking for. 

History of data exchange

The humble beginning of data exchange as we know it today started in the 1960s when IBM and General Electronics (GE) invented databases. Data transfer between databases wasn’t necessary until the 1970s, when databases had finally gathered enough data.

CSV

The need to transfer data led the IBM Fortran compiler to support the comma separated values (CSV) format in 1972. Businesses used CSV to collect data in tables and import it into another database. 

CSV continues to be the most common data distribution method even today. Large corporations, government bodies, and academic institutions use CSV to distribute data on the internet.

XML and JSON

Soon, businesses realized that they weren’t exchanging the entire information table. Instead, they rendered limited records to end-users. This need to provide access to a handful of records led them to use application programming interfaces (APIs) that connected lightweight applications. 

APIs facilitated data exchange with small, hierarchical information collections. The process of shipping data with APIs required two API calls: one for the base object and another for the tags list from a relational database. This problem led to the invention of extensible markup language (XML) in 1998 and Javascript serialized object notation (JSON) in 2001. 

Businesses quickly shifted away from XML because it created tags larger than the data payload. JSON could represent key-value pairs and arrays only. As a result, APIs started using JSON to connect apps. 

Today, businesses use API management tools to monitor APIs and facilitate data exchange. 

Source Code Control System

 Computer scientist Marc Rochkind invented a version control system called Source Code Control System (SCCS) while working at Bell Labs in 1972. Multiple code authors used SCCS and found they could collaborate efficiently using version control features such as diffs, merges, and branches.

Before SCSS, companies relied on manual compilation and integration of everyone’s work into code. Collaboration on the same code became effortless with SCSS. 

CVS

Organizations used proprietary version control systems until computer scientist and lecturer Dick Grune released Concurrent Versions Systems (CVS) in 1986. Most open source projects used CVS to share code using free, open formats. 

In 2005, Finnish software engineer Linus Torvalds moved his open source project to Git, and product companies followed.

Git and Github

Using distributed formats, Git made source code collaboration easy. The platform stored all code versions locally, and companies just needed to sync remote server changes. The easy version control enabled companies to handle huge volumes of diff, branch, and merge operations orders more quickly. 

Unlike other version control systems, Git used the Merkle-directed acyclic graph (DAG) structure to enable branches to be pointers to commits. With virtually unlimited branches, Git made it easier for people to collaborate and work on the same code. 

The launch of Github in 2008 further improved source code collaboration, resulting in many open source projects. 

Data exchange features

Data exchange systems offer the following features to help businesses obtain data and derive insights.

Data normalization

Data normalization organizes similar data across records to generate clean data. The normalization process ensures logical data storage, minimizes data modification errors, simplifies querying, and eliminates redundancy and unstructured data. This feature allows businesses to standardize different information entries, including phone numbers, street addresses, and contact names. 

Normalization uses normal forms to maintain database integrity and check dependencies between attributes and relations. 

Most relational databases don’t usually require more than 3NF to normalize data. However, businesses use fourth normal form (4NF), fifth normal form (5NF), and sixth normal form (6NF) to handle complex datasets.

DaaS

Data exchange solutions use the data-as-a-service (DaaS) model to store data, process it, and deliver analytics services. Businesses turn to cloud service delivery to enhance agility, improve functionality, set up quickly, automate maintenance, and save cost. 

DaaS works similarly to SaaS but didn’t see widespread adoption until recently. Originally, cloud computing services handled application hosting and data storage instead of data integration, analytics, and processing.  Today, low-cost cloud storage makes it easier for cloud platforms to manage and process data at scale. 

Data management

Data management, the process of collecting, organizing, transforming, storing, and protecting data, starts with data acquisition in the control center. Once you acquire data, you continue with the subsequent processes such as data preparation, conversion, cataloging, and modeling. These steps help data meet data analysis goals. 

Efficient data management optimizes data usage across teams and organizations. Plus, it’s crucial for meeting policy and regulation requirements. 

Dynamic data exchange

Dynamic data exchange (DDE) transfers data with a messaging protocol. DDE shares data between applications using various data formats. Remote data exchange platforms using dynamic data exchange help you update applications based on new data availability. 

DDE uses client and server models along with shared memory to exchange information. In this model, the client requests and the application offers information. You can use DDE more than once to exchange data.

Data exchange automation

Data exchange automation helps businesses save time, simplify data processing, and execute data lifecycle tasks faster. Data exchange software systems featuring automation emulate manual actions to make processes more efficient.

Types of data exchange

Below are the four types of data exchange, depending on the data transfer relationships between data consumers and suppliers.

1. Peer-to-peer data exchange is the direct data exchange between two different companies or two divisions within the same company. For example, a large corporation with multiple data warehouses can use peer-to-peer data exchange to share data subsets among departments. 

2. Private data exchange happens when two companies share data using a secure channel. Common examples include industry-specific data sharing among users. Likewise, when a company shares data with suppliers who share it with customers, it's known as private data exchange. 

This kind of data exchange uses representational state transfer (REST) API, simple object access protocol (SOAP) web service, message queue, file transfer protocol (FTP), electronic data interchange (EDI), or business-to-business (B2B) gateway technology. 

3. Electronic data exchange operates via the cloud. This data exchange type protects data with passwords and can make data available for download. 

4. Data marketplace is a public data exchange open to companies willing to consume or supply data. For example, Amazon Web Services (AWS) is a global data marketplace catering to various industries and functions. You’ll also come across niche data marketplaces offering financial data exchange or healthcare data exchange services to consumers and suppliers. 

Data catalog vs. data exchange vs. data marketplace

A data catalog creates and maintains data asset inventory in an enterprise setting. Business analysts, data engineers, and scientists use data catalogs to extract business value from relevant datasets.

To automate data cataloging, machine learning data catalog tools use natural language querying and data masking solutions, enabling secure and efficient metadata discovery, ingestion, enrichment, and translation. 

Data exchange platforms connect data suppliers and buyers through a digital data interface that simplifies how businesses find, use, and manage relevant data. Data exchange interactions can be transactional or collaborative. 

A data marketplace facilitates external data exchange via financial transactions. Data marketplaces allow businesses to discover, publish, license, and distribute data. All data marketplaces are data exchanges, but marketplaces don’t support non-financial use cases.

How does data exchange work?

Data exchange software solutions bring together sellers and buyers. This collaboration happens in the following steps.

• Partner agreements: Once buyers know what data they want, they sign agreements or contracts with sellers. These agreements define data exchange protocols, usage guidelines, and other collaboration principles.
• Node client setup: Depending on consumers’ needs, suppliers set up nodes to share data over the network. These node clients enable consumers to request and receive data over a secure channel. Some companies only use nodes to automate data request monitoring.
• Data standardization: Suppliers standardize and enrich data using agreed upon data formats.
• Information exchange: Data suppliers share data using node clients and buyers receive the data.

Data exchange patterns

Data exchange patterns combine data format, communication protocols, and architectural patterns to facilitate data sharing. Let’s break down some of the most common data exchange patterns.

API

APIs use hypertext transfer protocol (HTTP) and web services to communicate between applications. Web services like the ones below standardize interoperability provision between applications. 

• The SOAP standardized protocol uses HTTP and a simple mail transfer protocol (SMTP) to send messages. The World Wide Web Consortium (W3C) develops and maintains the SOAP standard specifications.
• REST architectural style offers RESTful web service with a set of guidelines.
• GraphQL or similar API design architecture tools feature query and manipulation language along with associated runtime.

ETL

To read and write data, applications transferring data need to connect to other databases. Extract, transform, and load (ETL) tools enhance database connections with data batching, transformation, and scheduling. 

ETL solutions help businesses gather data from multiple databases into a single repository for formatting and data analysis preparation. This unified data repository is key to simplifying analysis and data processing. 

File transfer

The file transfer process uses a network or internet connection to store and move data from one device to another. Data exchange solutions use file transfer to share, transmit, or transfer logical data objects between local and remote users. JSON, XML, and CSV are common file formats used in the data exchange process.

Remote procedure call

Distributed computing uses a remote procedure call (RPC) to translate and send messages between client-server-based applications. RPC facilitates point-to-point communications during data exchange. 

An RPC protocol asks a remote server to execute specific procedures based on the client’s parameters. Once the remote server responds, RPC transfers results to the calling environment. 

Event-based brokered messaging

Event-based brokered messaging uses middleware software to deliver data messages. In this process, different technical components manage queuing and caching. It relies on a business rules engine to manage publication and subscription services. 

Data streaming

Data streaming is the process of receiving continuous data flow or feed from different sources. Data exchange tools use data streaming to receive data sequences and update metrics for each arriving data point. This data exchange pattern is suitable for real-time monitoring and response activities. 

Consider your local and enterprise needs before choosing a data exchange pattern. 

The use of universal data exchange standards enables seamless data access and integration across all levels of healthcare.

Health facilities use healthcare integration engines to ensure electronic health records (EHR) accessibility, reduce disparate data silos, and achieve better compatibility and compliance.

Data exchange framework

A data exchange framework facilitates data transfer between systems. It defines the logic needed to read data from source files, transform data in compatible formats, and share transformed data with the destination system. To make this process easier, developers generally connect third-party and destination systems with the framework.

Data exchange frameworks feature the following functionalities to help data consumers and suppliers interact.

• Searchable catalog simplifies data asset search by using data set description, including the number of records, file type, pricing, profile statistics, and ratings. Data consumers search these catalogs to find suitable data sets and assess the sample data quality.
• Asset management lets you upload, manage, and publish data assets. Data suppliers use this functionality to specify data licensing, access rights, and manage inventory.
• Access control helps data suppliers define data asset access rules. For example, a supplier can restrict data set access until payment completion or agreement. Some data exchange layers also offer encryption key exchange for file delivery.
• Data transfer is the process suppliers use to share data with consumers. Common data transfer methods include file transfer, multi-tenant data sharing, and APIs. Cloud-based transfer holds files and simplifies data access with object storage. On the other hand, multi-tenant data sharing needs suppliers and consumers to use the same data management platforms (DMPs) for transparency.
• Subscription management streamlines data asset subscriptions offerings for data suppliers. Some data exchanges also offer a “bring your own subscription” (BYOS) feature that connects different subscriptions via tokens.
• Transaction management offers payment transaction and payment processing via credit cards, bank transfers, and account billing. Data consumers track purchases and subscriptions, stay updated about renewal terms, and modify subscriptions using transaction management modules.
• Account management gathers details related to users, buyers, sellers, as well as payment mechanisms, billing information, and account activity.
• Administration and data exchange operators monitor user activities and troubleshoot issues.
• Collaboration offers a secure area for suppliers and consumers to work together on data sets.
• Data enrichment improves quality with data standardization, address verification, deduplication, file merging, validation, and data cleansing.
• Selective sharing allows data set configuration for select consumers.
• Data mapping recommends supplementary data for further enrichment.
• Multi-tenant data sharing eliminates traditional data sharing headaches of replacing FTP, and copying and moving data.
• Connector software development kit (SDK) creates custom connectors for data exchange suppliers to access other data platforms.
• Derived aggregate data lets consumers run user-defined functions (UDFs) and receive aggregated output. Suppliers generally offer this functionality when they don’t want consumers to have access to sensitive raw data.
• Enhanced onboarding simplifies supplier onboarding with supplier data compliance evaluation.
• Alerts notify consumers when a new data publication matches what they’re looking for.
• Pipeline management combines and integrates before delivering third-party data to end-users.
• Enhanced reporting shows data exchange sales performance and helps suppliers find insights for targeting the right buyers.
• Custom data products blend, segment, and engineer data to create suitable data products for consumers.
• Change of custody prohibition prevents license term violation with sensitive data preview and trials.

Most data exchange solutions combine the above features to create easy and compliant transactions between data buyers and sellers.

Data exchange benefits

Whether your company wants to break down data silos, govern data access or securely share data with customers, data exchange software has plenty of benefits for you.

• Simplifies data buying and selling. Finding credible third-party data has been painstaking for data consumers. And consider the challenges of price negotiation, data evaluation, and integration. Data exchange systems make it effortless for data suppliers to sell data and buyers to purchase it.
• Makes data sourcing for insights easier. Data exchanges provide faster data access to companies looking to make crucial data-driven decisions. This ease of access helps businesses boost revenue and enhance forecasts with machine learning models.
• Streamlines data monetization opportunities. Businesses selling data traditionally relied on a go-between to find suitable buyers. Data exchanges mean sellers sell data on their own terms with an easily accessible platform.
• Facilitates data commercialization. Data exchange helps data originators and acquirers build an ecosystem that benefits both parties. Data exchanges help data buyers use newly found insights to make strategic moves while giving sellers opportunities to create new revenue streams.
• Improves data quality and minimizes wasted expenses. Data exchanges help you access reliable data and eliminate data bots so you don’t spend time on false leads. Plus, data exchange software offers accurate data for correct segmentation, leading to more successful business outcomes.

Data exchange challenges

Data exchange solves some problems and creates a few. Below are some of the common issues businesses face with data exchange.

• Requires robust data compliance policy. Without it, you can barely synchronize data management systems. Compliance rules help companies to define data management frameworks for tracking what data they share and with whom. These frameworks ease data access control applications for data engineering teams.
• Needs sufficient providers and consumers. Data exchange platforms without enough consumers find it difficult to reach their full potential. Suppliers remain skeptical to list their companies on these platforms. Perhaps this is why many cloud solutions with data exchange capabilities help systems gain buyers and sellers.
• Relies on data integration and validation. Data consumers can’t find insights unless they integrate the data with internal data management tools. This integration requires data exchange software to be able to validate, clean, and format raw data into a readable format.
• Needs some technical expertise. Businesses can’t navigate data exchange solutions without knowing how to package, filter, or validate data.
• Limits data filtering ability. Data exchanges don’t let buyers pick exactly what they need. Data acquirers can’t create or purchase precise data sets according to their preferences.

Data exchange approach considerations

There isn’t a one-size-fits-all data exchange approach that every business can use. Each method has its pros and cons, but keep these points in mind while choosing the data exchange approach for your enterprise.

• Data complexity tells you whether you need direct database access or not. For example, if you don’t have access to specific data entity components, you’ll be better off with direct access. On the other hand, REST APIs require multiple calls and coding to build relationships among data elements. You can also use JSON and XML for more complex data models.
• Data update frequency reveals if you have to replace datasets regularly. APIs and messaging system methods ensure better resynchronization in case of large data updates.
• Data set size determines whether you’ll need a direct database connection or file transfer for performance optimization. You can also look for ways to improve performance while sending data via REST or other APIs.
• Data versions or schemas also help you choose between API or other data exchange protocols. For example, APIs aren’t ideal for representing different data formats. If your applications need data in various versions, you’re better off with another data exchange protocol.
• Data security controls guide you to the best data exchange approach. For example, you may need to design APIs to require keys, configure web servers, or set up database management system (DBMS) security controls to protect data.
• Data transformation difficulty tells what you need to move data. You need a direct database connection and ETL tools for an extensive transformation with complex rules. Also, evaluate transformation complexity to see if API management platforms can be of any use.
• Connection type is another decision to make before you pick an approach. Short-lived protocols suit a specific action or series of actions, whereas long-lived protocols keep connections open indefinitely. Consider end-user requirements while choosing connection persistence.

Successful organizations also look at broader organizational goals before making decisions about requirements for specific projects and applications. They collaborate and coordinate approaches to avoid data conflicts and inconsistencies across teams.

Data exchange design best practices

A well-implemented data exchange requires correct data configuration and synchronization. Rely on the following best practices to design data exchange processes accurately and validate data across the implementation cycle.

• Check the XML schema registry before creating a new schema.
• Follow exchange network design rules and schema design standards.
• Divide logical data groups into separate schema files.
• Use schema constraints effectively to ensure compatibility with your target database.
• Minimize required fields and use them only when necessary.
• Use count, list, or detailed result sets for easy data synchronization.
• Disallow large data transactions during non-business hours.
• Leverage asynchronous methods for large datasets.
• Pre-process requests to assess node impact.
• Simplify relational data to XML conversion with data preparation.
• Choose a flexible schema design to optimize returning data options.
• Limit query parameter choices to avoid large datasets.
• Compress files to limit data transmission size.
• Use data differencing to identify changes since the last data transmission.
• Choose a simple and flexible data service naming convention.
• Document data service parameters before data exchange.

Data exchange software

Data exchange software is used to share and transmit data without changing its meaning. 

A data exchange solution must do the following to meet the requirements for inclusion in the data exchange category:

• Share data without altering its meaning
• Normalize data for ease of consumption
• Offer market data-as-a-service acquisition service
• Integrate with other data solutions for ease of sharing and analysis

*Below are the five leading data exchange platforms based on G2 data collected on July 18, 2022. Some reviews may be edited for clarity.

1. PartnerLinQ

PartnerLinQ is a supply chain visibility platform that streamlines data visibility and connectivity. This platform features electronic data interchange (EDI), non-EDI, and API integration capabilities for connecting multiple supply networks, marketplaces, real-time analytics, and core systems. 

What users like:

“This platform remains to be one of the best data mapping platforms.The setup is so ideal,  enhancing great management on supply chain issues.The interface design is so marginal,  improving high performance. The support offered to users is just on point.”

– PartnerLinQ Review, Chris J.

What users dislike:

“Price is expensive, I think. And slight development of analytics would be more useful.”

– PartnerLinQ Review, Rashad G.

2. Crunchbase

Crunchbase is a leading prospecting and research solution provider. Companies, sales teams, and investors use this platform to find new business opportunities.

What users like:

“The most helpful thing about Crunchbase is the powerful filters you can use to create super-targeted lists of businesses you want to contact for future collaboration.”

– Crunchbase Review, Aaron H.

What users dislike:

“The one problem I encountered was that if you use the website's query feature instead of the API, you will get relatively unclean data that you will need to clean up before processing properly! This problem can be circumvented if you use the API but you will need basic knowledge of JSON.”

– Crunchbase Review, Kasra B.

3. Flatfile

Flatfile is a data onboarding platform that enables companies to import clean, ready-to-use data faster. This platform automates column matching recommendations and lets you set target data models for data validation. 

What users like:

“Flatfile is a powerful import tool that just works. It has all the features you'd expect from an importer plus ones you wouldn't initially consider. For developers, their API is well documented and their support was always available to discuss approaches. We've made Flatfile a critical part of our onboarding process and it's worked out great!”

– Flatfile Review, Ryan F.

What users dislike:

“The only minor issue is that the client-side version is not quite as fully-featured as the version that sends data to the Flatfile backend, meaning that the column matching is not quite as smart. But this is really minor - I'd strongly recommend Flatfile.”

– Flatfile Review, Rob C.

4. AWS Data Exchange

AWS Data Exchange simplifies how companies use the cloud to find third-party data. 

What users like:

“It's impressive to find hundreds of commercial data products from category-leading data providers across drives such as retail, financial services, healthcare, and more.”

– AWS Data Exchange Review, Ahmed I.

What users dislike:

“Subscription prices are costly and it becomes very hard to manage the budget.”

– AWS Data Exchange Review, Mohammad S.

5. Explorium

Explorium is a data science platform that connects thousands of external data sources using automatic data discovery and feature engineering. Businesses use this platform to acquire data and make predictive insights that drive business decisions. 

What users like:

“The richness and breadth of data is incredible. I really like instant access to the most useful and reliable external data. It helps us provide better customer service because it is the data we need to make faster and better decisions. The platform is very easy to use and extremely versatile.”

– Explorium Review, Ishi N.

What users dislike:

“I wish they had more point-in-time data sources.”

– Explorium Review, Noa L. 

Harmonize master data governance across business domains

When you’re ready to synchronize company-wide tools, processes, and applications with a single source of truth (SSOT), let data exchange platforms free you from data silos. Decompartmentalize your insights and make better data-driven decisions. 

Leverage master data management (MDM) to create a trusted view of data and achieve operational efficiency.