LogoLogo
Back to OsmosDeveloper DocsOsmos BlogWhat's New
  • Welcome to Osmos
  • 👋Getting Started with Osmos
    • Terminology
  • 🎉What's New
  • 🧩Osmos API Reference
  • ⌨️Osmos Chat
  • 👩‍💻Developer Docs
    • Manage API Keys
    • Embedding an Osmos Uploader
    • Embedding Uploader Jobs Table
    • Turning on Advanced Mode Uploader
    • Customizing Uploader Styling
    • Passing Parameterized Fields
    • Configuring Uploader's "Recall" functionality
    • Optional Uploader Settings
    • Uploader Submission Callback
    • Configuring AutoClean for your Uploader
    • Uploader Client-Side Validation
      • Data Validators
      • Checking for Duplicate values in a field
      • Creating Dropdown-Controlled Fields
      • Dynamic Dropdown Options
      • Dropdown Interaction with Validation Functions
    • Validation and Transformation Webhooks
      • OpenAPI Validation Webhook Testing
    • Parser Webhook for file based connectors
  • 🔠Datasets
    • Osmos Datasets
      • Uploading Data to your Table
      • Creating Primary and Foreign keys
      • Osmos Dataset Destination Connector
      • Osmos Dataset Source Connector
      • Dataset Edits
    • Datasets Query Builder
      • Query Builder Metadata
    • Performing Look Ups
      • Performing Joins
        • Types of Joins
  • ⏏️Uploader
    • Creating an Osmos Uploader
      • Testing your Osmos Uploader
    • Uploader Validation Summary
    • Advanced Mode
      • Overview
      • Process
    • Standard Mode
      • Overview
      • AutoClean
      • Process
    • AI AutoMapping
    • Uploaders Page
    • Uploader Details Page
  • 🔀Pipelines
    • Step 1. Select the Source
    • Step 2. Select a Destination
    • Step 3. Map & Transform Data
    • Step 4. Schedule the Pipeline
    • Step 5. Review & Confirm
    • Pipelines Page
    • Pipeline Details Page
  • ⏩Data Transformations
    • AutoMap
    • Column Mapping & Data Cleanup Panel
    • QuickFixes
    • AI Value Mapping
    • AI AutoClean
    • Lookups
      • Performing Lookups
    • SmartFill
    • Formulas
      • Date & Time Formulas
        • DateTime Format Specifiers
        • Timezone specifiers
      • Math Formulas and Operators
      • Logical Formulas & Operators
        • True & False Casting
      • Text Formulas
      • Other Formulas
    • Deduplication
  • ↘️Source Connectors
    • Amazon S3
    • Azure Blob Storage
    • BigQuery
    • Email
    • FTP
    • Google Cloud Storage (GCS)
    • Google Drive
    • Google Sheets
    • HTTP API (Call an Osmos API)
    • HTTP API (Osmos Calls Your API)
    • Osmos Dataset
    • Snowflake
    • Accessing Sources behind firewall
  • ↖️Destination Connectors
    • Amazon S3
    • BigQuery
    • FTP
    • Google Cloud Storage (GCS)
    • Google Drive
    • Google Sheets
    • HTTP API (Call an Osmos API)
    • HTTP API (Osmos Calls Your API)
      • Passing Dynamic Tokens in the API Header
    • MySQL
    • Osmos Dataset
    • PostgreSQL
    • Snowflake
    • Accessing Destinations behind firewall
  • 🗂️Projects
  • ⚙️Administration
    • Email Notifications
  • 🔒Security
  • 📞Support
  • Back to Osmos.io
Powered by GitBook
On this page
  • What are Osmos Pipelines?
  • Connect to a Source
  • Connect to a Destination
  • Transform & Map Your Data
  • Schedule the Osmos Pipeline

Was this helpful?

Pipelines

In this section you will learn the basics of how to setup Osmos Pipelines.

PreviousUploader Details PageNextStep 1. Select the Source

Last updated 11 months ago

Was this helpful?

What are Osmos Pipelines?

Osmos Pipelines allow you to send data from a Source Connector to a Destination Connector. You can also clean up and restructure the data before it is sent to the Destination Connector using easy-to-use transformations like column mapping, SmartFill, QuickFixes, and formulas. Osmos Pipelines can be set up on a recurring schedule or manually triggered to run at any time.

Connect to a Source

The first step in creating an Osmos Pipeline is selecting the Source Connector. This connector provides access to the source data.

Check out the Source Connector step for more information.

Connect to a Destination

The second step when creating an Osmos Pipeline is selecting the Destination Connector. This Connector provides access to the system to where you want to send the source data after it's transformed.

Check out the Destination Connector step for more information.

Transform & Map Your Data

After selecting the Source and Destination Connectors for your Osmos Pipeline, the next step is to map and transform the source data into a format that aligns with the Destination Connector's schema. You can quickly clean up and restructure the data using QuickFixes, SmartFill, and formulas.

Check out the transformation step for more information.

Schedule the Osmos Pipeline

After transforming the data, you will be taken to the next step where you can set a schedule for the Osmos Pipeline.

Check out the schedule step for more information.

Continue to the next page to get started with creating Osmos Pipelines!

🔀