AI Value Mapping

This feature is currently in Beta.

What is AI Value Mapping?
Getting Started

What is AI Value Mapping?

AI Value Mapping allows you to map end-user data into categories with your organization’s internal representation (commonly known as enumerations “enums”). Leveraging advanced Large Language Models (LLMs), this QuickFix automatically maps source data to its nearest semantic match in the destination schema. This saves your data operations teams hours of manual work while dynamically adapting to evolving source data without requiring updates to formulas or webhooks.

Value Mapping can be used throughout Osmos in both Uploaders and Pipelines. It is most commonly used with an Uploader where end users map enumerations ("enums"). Enums are integrated as part of the Uploader Validation. Users map a group of constants to a specific valid option. The list of valid options is configured in your validation and the mapping occurs in the Value Mapping QuickFix.

Getting Started

There are two primary types of configurations.

Type 1: Create a Dataset Table to hold your list of valid options. Then, set up a foreign key column in your destination Dataset Table. This configuration can be used for Pipelines and Uploaders. See the scenario below.

Type 2: With an Embedded Uploader, create a dropdown-controlled field that contains your list of valid options in the code snippet. Here is a link to this configuration.

Type 1 Scenario

One-time setup items:

Create a valid options list in a Dataset table
Create a standard Uploader (with Destination Datasets) with a Foreign Key field to look up the valid option.

Customer ABC is ingesting employee data. The employee data includes employee type, which needs to be narrowed down to Customer ABC's list of valid options. In this scenario, see how to use Value Mapping to group a field to a specific list of options.

Features

Datasets
Pipelines and/or Uploader

One-Time Setup

Creating the Valid Options List & Destination Uploader

Note: This scenario requires the use of primary and foreign keys. It requires the use of dataset tables. The first table is used as the repository for the valid options and the second is the destination for your uploaded data (think of it as your standard uploader).

Create the Dataset to hold the valid options.

Best Practice is to use one Dataset to hold all of your lookup tables.

Within your new Dataset, create the first table called Employee_Category. This table will have one field called 'Employee Category', set to required field, and selected as a Primary key field. Additional fields may be included but will not be featured in this scenario. Save your table.

Either create a new Dataset or within the existing Dataset, create a second table called 'Employees'. This table will be the destination for your ingested data and will have all of our destination data fields.
1. AI Value Mapping requires a foreign key relationship on the destination field where the valid option will reside. To set up the relationship, select Add Foreign Key from the table creation page.
  1. Enter local field name: 'Employee_Type',
  2. Select the project where the valid options Dataset resides
  3. Select the Dataset where the valid options reside
  4. Select the Table
  5. Select the Reference Field for the corresponding valid option.
  6. Hit Add Foreign Key

Notes:

If this destination will be an uploader, don't forget to create the Uploader from the Dataset Table.
Make sure the Foreign Key field is not in the schema multiple times (one as a foreign key and one as the expected type). This often occurs when uploading the schema.

At this point, we have established a relationship between the Employee_Category table and the Employees table. This relationship tells the system that only values that have been submitted to the 'Employee Category' field of the Employee_Categories table are valid options for the 'Employee_Type' field in the Employees table.

Performing the AI Value Mapping

Uploading Data

Step 1: Go to your newly created Uploader and upload your data. Once the AutoMapping is complete, select the field you wish to initiate the AI Value Mapping. In our scenario it is Employee_Type.

Step 2: However on the Column Mapping > Select Transform on the left-hand side column mapping field. In this scenario, it will be the destination Employee_Type field.

Step 3: Select the AI Value Mapping QuickFix.

AI Value Mapping will automatically run and map. You can then make adjustments and/or map any fields left unmapped.

Step 4. Hit Save when you are done to save the mapping logic.