LogoLogo
Back to OsmosDeveloper DocsOsmos BlogWhat's New
  • Welcome to Osmos
  • 👋Getting Started with Osmos
    • Terminology
  • 🎉What's New
  • 🧩Osmos API Reference
  • ⌨️Osmos Chat
  • 👩‍💻Developer Docs
    • Manage API Keys
    • Embedding an Osmos Uploader
    • Embedding Uploader Jobs Table
    • Turning on Advanced Mode Uploader
    • Customizing Uploader Styling
    • Passing Parameterized Fields
    • Configuring Uploader's "Recall" functionality
    • Optional Uploader Settings
    • Uploader Submission Callback
    • Configuring AutoClean for your Uploader
    • Uploader Client-Side Validation
      • Data Validators
      • Checking for Duplicate values in a field
      • Creating Dropdown-Controlled Fields
      • Dynamic Dropdown Options
      • Dropdown Interaction with Validation Functions
    • Validation and Transformation Webhooks
      • OpenAPI Validation Webhook Testing
    • Parser Webhook for file based connectors
  • 🔠Datasets
    • Osmos Datasets
      • Uploading Data to your Table
      • Creating Primary and Foreign keys
      • Osmos Dataset Destination Connector
      • Osmos Dataset Source Connector
      • Dataset Edits
    • Datasets Query Builder
      • Query Builder Metadata
    • Performing Look Ups
      • Performing Joins
        • Types of Joins
  • ⏏️Uploader
    • Creating an Osmos Uploader
      • Testing your Osmos Uploader
    • Uploader Validation Summary
    • Advanced Mode
      • Overview
      • Process
    • Standard Mode
      • Overview
      • AutoClean
      • Process
    • AI AutoMapping
    • Uploaders Page
    • Uploader Details Page
  • 🔀Pipelines
    • Step 1. Select the Source
    • Step 2. Select a Destination
    • Step 3. Map & Transform Data
    • Step 4. Schedule the Pipeline
    • Step 5. Review & Confirm
    • Pipelines Page
    • Pipeline Details Page
  • ⏩Data Transformations
    • AutoMap
    • Column Mapping & Data Cleanup Panel
    • QuickFixes
    • AI Value Mapping
    • AI AutoClean
    • Lookups
      • Performing Lookups
    • SmartFill
    • Formulas
      • Date & Time Formulas
        • DateTime Format Specifiers
        • Timezone specifiers
      • Math Formulas and Operators
      • Logical Formulas & Operators
        • True & False Casting
      • Text Formulas
      • Other Formulas
    • Deduplication
  • ↘️Source Connectors
    • Amazon S3
    • Azure Blob Storage
    • BigQuery
    • Email
    • FTP
    • Google Cloud Storage (GCS)
    • Google Drive
    • Google Sheets
    • HTTP API (Call an Osmos API)
    • HTTP API (Osmos Calls Your API)
    • Osmos Dataset
    • Snowflake
    • Accessing Sources behind firewall
  • ↖️Destination Connectors
    • Amazon S3
    • BigQuery
    • FTP
    • Google Cloud Storage (GCS)
    • Google Drive
    • Google Sheets
    • HTTP API (Call an Osmos API)
    • HTTP API (Osmos Calls Your API)
      • Passing Dynamic Tokens in the API Header
    • MySQL
    • Osmos Dataset
    • PostgreSQL
    • Snowflake
    • Accessing Destinations behind firewall
  • 🗂️Projects
  • ⚙️Administration
    • Email Notifications
  • 🔒Security
  • 📞Support
  • Back to Osmos.io
Powered by GitBook
On this page
  • Overview
  • Types of APIs Supported
  • Prerequisites
  • Creating an HTTP Endpoint Source Connector
  • Detecting the Schema
  • Step 5: API URL
  • Step 6: HTTP Method
  • Step 7: API Request Type
  • Step 8: HTTP Headers
  • Step 9: Deduplication Mode
  • Advanced Options
  • Connector Options

Was this helpful?

  1. Source Connectors

HTTP API (Osmos Calls Your API)

PreviousHTTP API (Call an Osmos API)NextOsmos Dataset

Last updated 1 year ago

Was this helpful?

Overview

There are two modes of operation for this connector. Either Osmos will call your API, or Osmos can generate an API for you to call. Put another way, either you have an API endpoint created, or you want Osmos to create an API endpoint for you. These two modes will be referred to as Osmos Calls Your API and Call an Osmos API respectively. This page is for the former. To see documentation on the latter, see .

This Connector sends an HTTP request to a specified HTTP API and treats the returned data from that API as source data. This source data is then sent through an Osmos Pipeline to a Destination Connector.

Types of APIs Supported

This is not a Postman or curl replacement -- the scope of HTTP requests that Osmos will create are limited. Generalized authentication handshakes, forwarding data from a previous API request to the next one, and non-HTTP (TCP, FTP, Telnet) are not supported by this connector.

Example Use Cases

For example, let's say there's an API out there, www.example.com/weather?city=seattle. Let's say that issuing a GET request to this endpoint returns an array of temperature measurements -- one measurement for every hour of the day. Something like this:

[75, 75, 75, 74, 72, 72, 
 70, 69, 70, 71, 75, 74,
 72, 72, 70, 69, 68, 67,
 67, 66, 65, 65, 66, 65]

Issuing an HTTP GET request from an Osmos source would result in 24 records, one for each of the above numbers, being fed into the pipeline. You may also choose to have these 24 numbers treated as one record with 24 fields in it.

Prerequisites

Information needed:

  • An HTTP API that you would like to retrieve data from and send through an Osmos Pipeline.

Creating an HTTP Endpoint Source Connector

Step 1: After selecting + New Connector, under the System prompt, click HTTP API

Step 2: Enter a Connector Name.

Step 3: Select Source Connector.

Step 4: Select Osmos Calls Your API

Detecting the Schema

We will automatically detect the schema of the data returned by the specified HTTP API.

Step 5: API URL

Specify the URL, including the port number if any, of the API that Osmos should be sending the request to. For example, this could be http://example.com/api/v2:8824

Step 6: HTTP Method

Step 7: API Request Type

Select whether the API should return one record at a time, or an array of records. This setting turns on/off unpacking a top level array into a series of records.

Step 8: HTTP Headers

Specify the headers, if any, that Osmos should be using to request data from this API. Headers are often used for authentication, authorization, or declarations of what type of content you're sending. Headers have both a Name and a Value. For example, an authorization header might look like:

Name: Authorization

Value: qaX1SrppzSe1Y1qOOc08

Step 9: Deduplication Mode

Osmos supports record level deduplication when pulling data from an API. In this mode, Osmos will store hashes of each record pulled from the source API, and will only forward new records.

Advanced Options

Header Normalization

The source file may have characters at the start or end that includes spaces, tabs, carriage returns and line endings. You can choose to keep all characters from the source or remove all whitespace. Select one of the options:

  1. Don't normalize headers. Use headers exactly as they appear in the source: If this option is selected, we will retain all characters from the source file.

  2. Remove extra whitespace and other common untypable characters from headers: If this option is selected, we remove all whitespace (spaces, tabs, carriage returns, line endings) at start/end.

Connector Options

The connector can be deleted, edited and duplicated.

Duplication

To save time, the connector can be duplicated. This new connector needs to be named and can be edited, as needed.

Select the that Osmos should use when sending the request for data to this API.

↘️
HTTP API (Call an Osmos API)
HTTP Method