LogoLogo
Back to OsmosDeveloper DocsOsmos BlogWhat's New
  • Welcome to Osmos
  • 👋Getting Started with Osmos
    • Terminology
  • 🎉What's New
  • 🧩Osmos API Reference
  • ⌨️Osmos Chat
  • 👩‍💻Developer Docs
    • Manage API Keys
    • Embedding an Osmos Uploader
    • Embedding Uploader Jobs Table
    • Turning on Advanced Mode Uploader
    • Customizing Uploader Styling
    • Passing Parameterized Fields
    • Configuring Uploader's "Recall" functionality
    • Optional Uploader Settings
    • Uploader Submission Callback
    • Configuring AutoClean for your Uploader
    • Uploader Client-Side Validation
      • Data Validators
      • Checking for Duplicate values in a field
      • Creating Dropdown-Controlled Fields
      • Dynamic Dropdown Options
      • Dropdown Interaction with Validation Functions
    • Validation and Transformation Webhooks
      • OpenAPI Validation Webhook Testing
    • Parser Webhook for file based connectors
  • 🔠Datasets
    • Osmos Datasets
      • Uploading Data to your Table
      • Creating Primary and Foreign keys
      • Osmos Dataset Destination Connector
      • Osmos Dataset Source Connector
      • Dataset Edits
    • Datasets Query Builder
      • Query Builder Metadata
    • Performing Look Ups
      • Performing Joins
        • Types of Joins
  • ⏏️Uploader
    • Creating an Osmos Uploader
      • Testing your Osmos Uploader
    • Uploader Validation Summary
    • Advanced Mode
      • Overview
      • Process
    • Standard Mode
      • Overview
      • AutoClean
      • Process
    • AI AutoMapping
    • Uploaders Page
    • Uploader Details Page
  • 🔀Pipelines
    • Step 1. Select the Source
    • Step 2. Select a Destination
    • Step 3. Map & Transform Data
    • Step 4. Schedule the Pipeline
    • Step 5. Review & Confirm
    • Pipelines Page
    • Pipeline Details Page
  • ⏩Data Transformations
    • AutoMap
    • Column Mapping & Data Cleanup Panel
    • QuickFixes
    • AI Value Mapping
    • AI AutoClean
    • Lookups
      • Performing Lookups
    • SmartFill
    • Formulas
      • Date & Time Formulas
        • DateTime Format Specifiers
        • Timezone specifiers
      • Math Formulas and Operators
      • Logical Formulas & Operators
        • True & False Casting
      • Text Formulas
      • Other Formulas
    • Deduplication
  • ↘️Source Connectors
    • Amazon S3
    • Azure Blob Storage
    • BigQuery
    • Email
    • FTP
    • Google Cloud Storage (GCS)
    • Google Drive
    • Google Sheets
    • HTTP API (Call an Osmos API)
    • HTTP API (Osmos Calls Your API)
    • Osmos Dataset
    • Snowflake
    • Accessing Sources behind firewall
  • ↖️Destination Connectors
    • Amazon S3
    • BigQuery
    • FTP
    • Google Cloud Storage (GCS)
    • Google Drive
    • Google Sheets
    • HTTP API (Call an Osmos API)
    • HTTP API (Osmos Calls Your API)
      • Passing Dynamic Tokens in the API Header
    • MySQL
    • Osmos Dataset
    • PostgreSQL
    • Snowflake
    • Accessing Destinations behind firewall
  • 🗂️Projects
  • ⚙️Administration
    • Email Notifications
  • 🔒Security
  • 📞Support
  • Back to Osmos.io
Powered by GitBook
On this page
  • What are Osmos Datasets?
  • Why use Osmos Datasets?
  • A few features of Osmos Datasets

Was this helpful?

Datasets

In this section you will learn the basics about Osmos Datasets and Dataset Tables.

PreviousParser Webhook for file based connectorsNextOsmos Datasets

Last updated 10 months ago

Was this helpful?

What are Osmos Datasets?

Datasets keep information organized, housing tables and their data. Datasets offers a clean logical entity to build and manage your data models. A Dataset serves as the container for all your tables where the actual data resides. Each of these tables can have Primary keys and reference other tables using Foreign keys, thereby helping you organize the elements of data and standardize how they relate to one another by enforcing referential integrity between your data elements.

You can create new datasets, duplicate from another project, and manage existing datasets.

Why use Osmos Datasets?

Datasets gives you the ability to achieve high fidelity Data Management in one place, without the need for SQL skills.

Let's say you are working on a migration project. As a part of the migration project, you need to import 10 types of data elements (tables) from your customer. These data elements are interconnected and reference one another using foreign keys. When migrating the data, you need to ensure that you are keeping the data integrity intact and perform Lookups and Joins at the time of data migration.

The cleanest way to accomplish such a project would be using Datasets. You would end up creating a migration Dataset, housing all the tables with their Foreign Key relationships. You would then use Uploaders or Pipelines to import data into these tables, using Lookups to ensure referential integrity. Once the data is loaded in Datasets, you would then use Pipelines to extract the clean data from Dataset into your Database and APIs.

A few features of Osmos Datasets

  • There are no limitations to the number of datasets or the tables you can create inside a dataset.

  • There is no cost to storing data inside Datasets. You only pay for the in-transit data processing costs to and from Datasets, per the service tier you are in.

  • You can write to and read from Datasets. You can use Uploaders or Pipelines to perform read and write operations on Datasets.

  • Depending upon your Osmos plan, you can leverage datasets to do lookups, joins, and aggregations.

🔠
Demo Osmos Uploader
Osmos Uploader Sandbox