LogoLogo
Back to OsmosDeveloper DocsOsmos BlogWhat's New
  • Welcome to Osmos
  • 👋Getting Started with Osmos
    • Terminology
  • 🎉What's New
  • 🧩Osmos API Reference
  • ⌨️Osmos Chat
  • 👩‍💻Developer Docs
    • Manage API Keys
    • Embedding an Osmos Uploader
    • Embedding Uploader Jobs Table
    • Turning on Advanced Mode Uploader
    • Customizing Uploader Styling
    • Passing Parameterized Fields
    • Configuring Uploader's "Recall" functionality
    • Optional Uploader Settings
    • Uploader Submission Callback
    • Configuring AutoClean for your Uploader
    • Uploader Client-Side Validation
      • Data Validators
      • Checking for Duplicate values in a field
      • Creating Dropdown-Controlled Fields
      • Dynamic Dropdown Options
      • Dropdown Interaction with Validation Functions
    • Validation and Transformation Webhooks
      • OpenAPI Validation Webhook Testing
    • Parser Webhook for file based connectors
  • 🔠Datasets
    • Osmos Datasets
      • Uploading Data to your Table
      • Creating Primary and Foreign keys
      • Osmos Dataset Destination Connector
      • Osmos Dataset Source Connector
      • Dataset Edits
    • Datasets Query Builder
      • Query Builder Metadata
    • Performing Look Ups
      • Performing Joins
        • Types of Joins
  • ⏏️Uploader
    • Creating an Osmos Uploader
      • Testing your Osmos Uploader
    • Uploader Validation Summary
    • Advanced Mode
      • Overview
      • Process
    • Standard Mode
      • Overview
      • AutoClean
      • Process
    • AI AutoMapping
    • Uploaders Page
    • Uploader Details Page
  • 🔀Pipelines
    • Step 1. Select the Source
    • Step 2. Select a Destination
    • Step 3. Map & Transform Data
    • Step 4. Schedule the Pipeline
    • Step 5. Review & Confirm
    • Pipelines Page
    • Pipeline Details Page
  • ⏩Data Transformations
    • AutoMap
    • Column Mapping & Data Cleanup Panel
    • QuickFixes
    • AI Value Mapping
    • AI AutoClean
    • Lookups
      • Performing Lookups
    • SmartFill
    • Formulas
      • Date & Time Formulas
        • DateTime Format Specifiers
        • Timezone specifiers
      • Math Formulas and Operators
      • Logical Formulas & Operators
        • True & False Casting
      • Text Formulas
      • Other Formulas
    • Deduplication
  • ↘️Source Connectors
    • Amazon S3
    • Azure Blob Storage
    • BigQuery
    • Email
    • FTP
    • Google Cloud Storage (GCS)
    • Google Drive
    • Google Sheets
    • HTTP API (Call an Osmos API)
    • HTTP API (Osmos Calls Your API)
    • Osmos Dataset
    • Snowflake
    • Accessing Sources behind firewall
  • ↖️Destination Connectors
    • Amazon S3
    • BigQuery
    • FTP
    • Google Cloud Storage (GCS)
    • Google Drive
    • Google Sheets
    • HTTP API (Call an Osmos API)
    • HTTP API (Osmos Calls Your API)
      • Passing Dynamic Tokens in the API Header
    • MySQL
    • Osmos Dataset
    • PostgreSQL
    • Snowflake
    • Accessing Destinations behind firewall
  • 🗂️Projects
  • ⚙️Administration
    • Email Notifications
  • 🔒Security
  • 📞Support
  • Back to Osmos.io
Powered by GitBook
On this page
  • Contents
  • What is AI Value Mapping?
  • Getting Started
  • Type 1 Scenario
  • Features
  • One-Time Setup
  • Creating the Valid Options List & Destination Uploader
  • Performing the AI Value Mapping
  • Uploading Data
  • How to map unmapped fields (or adjust existing mapped fields).

Was this helpful?

  1. Data Transformations

AI Value Mapping

This feature is currently in Beta.

PreviousQuickFixesNextAI AutoClean

Last updated 7 months ago

Was this helpful?

Contents

  1. What is AI Value Mapping?

  2. Getting Started

What is AI Value Mapping?

AI Value Mapping allows you to map end-user data into categories with your organization’s internal representation (commonly known as enumerations “enums”). Leveraging advanced Large Language Models (LLMs), this QuickFix automatically maps source data to its nearest semantic match in the destination schema. This saves your data operations teams hours of manual work while dynamically adapting to evolving source data without requiring updates to formulas or webhooks.

Value Mapping can be used throughout Osmos in both Uploaders and Pipelines. It is most commonly used with an Uploader where end users map enumerations ("enums"). Enums are integrated as part of the Uploader Validation. Users map a group of constants to a specific valid option. The list of valid options is configured in your validation and the mapping occurs in the Value Mapping QuickFix.

Getting Started

There are two primary types of configurations.

Type 1: Create a Dataset Table to hold your list of valid options. Then, set up a column in your destination Dataset Table. This configuration can be used for Pipelines and Uploaders. .

Type 2: With an Embedded Uploader, create a dropdown-controlled field that contains your list of valid options in the code snippet. Here is a link to this .

Type 1 Scenario

One-time setup items:

  1. Create a valid options list in a Dataset table

  2. Create a standard Uploader (with Destination Datasets) with a Foreign Key field to look up the valid option.

Customer ABC is ingesting employee data. The employee data includes employee type, which needs to be narrowed down to Customer ABC's list of valid options. In this scenario, see how to use Value Mapping to group a field to a specific list of options.

Features

  • Datasets

  • Pipelines and/or Uploader

One-Time Setup

Creating the Valid Options List & Destination Uploader

Note: This scenario requires the use of primary and foreign keys. It requires the use of dataset tables. The first table is used as the repository for the valid options and the second is the destination for your uploaded data (think of it as your standard uploader).

  1. Create the Dataset to hold the valid options.

Best Practice is to use one Dataset to hold all of your lookup tables.

  1. Within your new Dataset, create the first table called Employee_Category. This table will have one field called 'Employee Category', set to required field, and selected as a Primary key field. Additional fields may be included but will not be featured in this scenario. Save your table.

  1. Either create a new Dataset or within the existing Dataset, create a second table called 'Employees'. This table will be the destination for your ingested data and will have all of our destination data fields.

    1. AI Value Mapping requires a foreign key relationship on the destination field where the valid option will reside. To set up the relationship, select Add Foreign Key from the table creation page.

      1. Enter local field name: 'Employee_Type',

      2. Select the project where the valid options Dataset resides

      3. Select the Dataset where the valid options reside

      4. Select the Table

      5. Select the Reference Field for the corresponding valid option.

      6. Hit Add Foreign Key

Notes:

  1. If this destination will be an uploader, don't forget to create the Uploader from the Dataset Table.

  2. Make sure the Foreign Key field is not in the schema multiple times (one as a foreign key and one as the expected type). This often occurs when uploading the schema.

At this point, we have established a relationship between the Employee_Category table and the Employees table. This relationship tells the system that only values that have been submitted to the 'Employee Category' field of the Employee_Categories table are valid options for the 'Employee_Type' field in the Employees table.

Performing the AI Value Mapping

Uploading Data

Step 1: Go to your newly created Uploader and upload your data. Once the AutoMapping is complete, select the field you wish to initiate the AI Value Mapping. In our scenario it is Employee_Type.

Step 2: However on the Column Mapping > Select Transform on the left-hand side column mapping field. In this scenario, it will be the destination Employee_Type field.

Step 3: Select the AI Value Mapping QuickFix.

AI Value Mapping will automatically run and map. You can then make adjustments and/or map any fields left unmapped.

Step 4. Hit Save when you are done to save the mapping logic.

How to map unmapped fields (or adjust existing mapped fields).

Step 1: In the AI Value Mapping Window, select the box next to an unmapped field

Step 2: Scroll to the bottom and select the valid option to map the field to

Step 3: Hit Apply

Step 4: Hit Save Mapping

⏩
configuration
See the scenario below
foreign key