 |
Data is at the center of many processes and products, whether it’s a large-scale dataset used to train machine learning models, a relational database, or an API-based integration. AWS Data Exchange lets you discover, subscribe to, and use hundreds of file-based datasets via Amazon Simple Storage Service (Amazon S3) offered by third parties such as Reuters, Foursquare, Change Healthcare, Vortexa, IMDb, and many more. Additionally, AWS Data Exchange for Amazon Redshift makes it even easier to ingest third-party data in your Amazon Redshift data warehouse, without any manual processing or transformation.
However, in many cases your data projects require more than static datasets because you need frequent and synchronous retrieval of small amounts of information – for example, you might need to fetch a stock price every hour. Data APIs let you answer specific questions quickly and without having to build ad-hoc data pipelines to ingest, process, and analyze bulk datasets. But each API provider has its own ease of use, SDK, documentation, and authentication mechanisms, which makes this harder than it needs to be.
Today, I’m happy to announce the general availability of AWS Data Exchange for APIs, a new capability that lets you find, subscribe to, and use third-party APIs with a consistent access using AWS SDKs, as well as consistent AWS-native authentication and governance. This simplifies the lives of developers and IT administrators who have to integrate and secure the access to multiple third-party APIs.
Now you can make RESTful or GraphQL API calls directly to AWS Data Exchange and receive synchronous responses that contain the information you need, using the AWS SDK in the programming language of your choice. We take care of integrating with the API provider, implementing proper authentication, managing the API subscription, and ensuring charges appear on your AWS bill. You can manage API access centrally with AWS Identity and Access Management (IAM).
As a data provider, you make your API discoverable by millions of AWS customers by listing it in the AWS Data Exchange catalog using an OpenAPI specification and fronting it with an Amazon API Gateway endpoint.
AWS Data Exchange for APIs in Action
First, I look for an API product in the AWS Data Exchange catalog, review its subscription terms, support information, and auto-renewal. Each API product might include multiple public or private subscription offers and periods.
I select Subscribe and a couple of minutes later I’m successfully subscribed.
Within the API product, I select an entitled data set and its latest revision.
Each API revision contains one or more API assets that correspond to a specific API endpoint and a unique Asset ARN.
AWS Data Exchange takes care of invoking API endpoints with the correct authentication.
All I need to do is check the Integration notes, which include instructions and code snippets based on the AWS Command Line Interface (CLI).
Of course, I could implement the very same API call with my favorite programming language using one of the AWS SDKs.
For example, here’s how I’d implement a simple wrapper function in Python:
import json import urllib import boto3 adx = boto3.client(‘dataexchange’) def get_api_response(path, method=”GET”, querystring={}, headers={}, body={}): return adx.send_api_asset( DataSetId=”4b3fbabc31171662851531b8576a3411″, RevisionId=”e8e78e921af12c76499edc40f92e3082″, AssetId=”557d858c317efdfb5b6c9a2860ec4a03″, Method=method, Path=path, QueryStringParameters=urllib.urlencode(querystring), RequestHeaders=urllib.urlencode(headers), Body=json.dumps(body), )
Please note that there are no hard-coded credentials in the code above because all the authorization happens via AWS Identity and Access Management (IAM).
And that’s how you make your first API call via AWS Data Exchange for APIs.
Available Today
AWS Data Exchange for APIs is generally available in all AWS Regions where AWS Data Exchange is available. We’re looking forward to helping you simplify and centralize the management and governance of third-party APIs while we take care of the undifferentiated heavy lifting for you.
Today you can start integrating third-party APIs such as Infutor, Variety Business Intelligence, IMDb, PeopleDataLabs, Neustar, Experian, Foursquare, PredictHQ, WeatherTrends International, and many more.
If you’re a developer, check out the new AWS Data Exchange for APIs documentation to learn more about subscribing and using APIs. If you’re an API provider, check out the new publishing documentation to learn more about publishing new APIs on the AWS Data Exchange catalog.
— Alex
