# AI Value Mapping

## Contents

1. What is AI Value Mapping?&#x20;
2. Getting Started

## What is AI Value Mapping?

AI Value Mapping allows you to map end-user data into categories with your organization’s internal representation (commonly known as enumerations “enums”).   Leveraging advanced Large Language Models (LLMs), this QuickFix automatically maps source data to its nearest semantic match in the destination schema. This saves your data operations teams hours of manual work while dynamically adapting to evolving source data without requiring updates to formulas or webhooks.

Value Mapping can be used throughout Osmos in both Uploaders and Pipelines.  It is most commonly used with an Uploader where end users map enumerations ("enums").  Enums are integrated as part of the Uploader Validation.  Users map a group of constants to a specific valid option.  The list of valid options is configured in your validation and the mapping occurs in the Value Mapping QuickFix.

## Getting Started

There are two primary types of configurations.

Type 1: Create a Dataset Table to hold your list of valid options.  Then, set up a [foreign key](https://docs.osmos.io/datasets/getting-started/creating-primary-and-foreign-keys#foreign-keys) column in your destination Dataset Table.    This configuration can be used for Pipelines and Uploaders.   [See the scenario below](#scenario).

Type 2: With an Embedded Uploader, create a dropdown-controlled field that contains your list of valid options in the code snippet.  Here is a link to this [configuration](https://docs.osmos.io/developer-docs/uploader-client-side-validation/dynamic-dropdown-options).&#x20;

## Type 1 Scenario

One-time setup items:

1. Create a valid options list in a Dataset table
2. Create a standard Uploader (with Destination Datasets) with a Foreign Key field to look up the valid option.

Customer ABC is ingesting employee data.  The employee data includes employee type, which needs to be narrowed down to Customer ABC's list of valid options.  In this scenario, see how to use Value Mapping to group a field to a specific list of options. &#x20;

<figure><img src="https://353417064-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYrsDW6vGBTygB1qqSE%2Fuploads%2FreKf2ro2CtuXpr7jjSZR%2Fimage.png?alt=media&#x26;token=82678cd6-625a-4f08-b0ae-43138dd5170c" alt=""><figcaption></figcaption></figure>

## Features

* Datasets
* Pipelines and/or Uploader

## One-Time Setup &#x20;

## Creating the Valid Options List & Destination Uploader

{% hint style="info" %}
Note: This scenario requires the use of primary and foreign keys. It requires the use of dataset tables. The first table is used as the repository for the valid options and the second is the destination for your uploaded data (think of it as your standard uploader).
{% endhint %}

<figure><img src="https://353417064-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYrsDW6vGBTygB1qqSE%2Fuploads%2FFZWTkZs9zopvg2XdyOSE%2FCleanShot%202024-07-17%20at%2012.47.12%402x.png?alt=media&#x26;token=85831477-62a9-4494-9964-f001f8177b75" alt=""><figcaption></figcaption></figure>

1. Create the Dataset to hold the valid options.&#x20;

{% hint style="success" %}
Best Practice is to use one Dataset to hold all of your lookup tables.
{% endhint %}

1. Within your new Dataset, create the first table called *Employee\_Category*.  This table will have one field called '*Employee Category*', set to ***required*** field, and selected as a **Primary key** field.  Additional fields may be included but will not be featured in this scenario. Save your table.

<figure><img src="https://353417064-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYrsDW6vGBTygB1qqSE%2Fuploads%2FP7HWLmG7tj6cXtxDqdWM%2Fimage.png?alt=media&#x26;token=12eabe7f-cee7-4886-b369-e9f86d5ffe92" alt="" width="563"><figcaption></figcaption></figure>

3. Either create a new Dataset or within the existing Dataset, create a second table called '*Employees*'.  This table will be the destination for your ingested data and will have all of our destination data fields. &#x20;
   1. AI Value Mapping requires a foreign key relationship on the destination field where the valid option will reside.  To set up the relationship,  select **Add Foreign Key** from the table creation page.&#x20;
      1. Enter local field name:  '*Employee\_Type*',&#x20;
      2. Select the project where the valid options Dataset resides
      3. Select the Dataset where the valid options reside
      4. Select the Table
      5. Select the Reference Field for the corresponding valid option.
      6. Hit Add Foreign Key

<figure><img src="https://353417064-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYrsDW6vGBTygB1qqSE%2Fuploads%2Fa1C2OoFm8CETfz7ZU8KU%2Fimage.png?alt=media&#x26;token=6aca74b2-50c5-4ae7-9ab0-ec1e6175901c" alt="" width="357"><figcaption></figcaption></figure>

<figure><img src="https://353417064-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYrsDW6vGBTygB1qqSE%2Fuploads%2FRlHBQ3jqiF6I8BGmSZ05%2Fimage.png?alt=media&#x26;token=5f2bd58b-c35b-4f1c-92a6-0a55a17270b2" alt="" width="563"><figcaption></figcaption></figure>

{% hint style="info" %}
Notes:&#x20;

1. If this destination will be an uploader, don't forget to create the Uploader from the Dataset Table.
2. Make sure the Foreign Key field is not in the schema multiple times (one as a foreign key and one as the expected type).  This often occurs when uploading the schema.
   {% endhint %}

At this point, we have established a relationship between the *Employee\_Category* table and the *Employees* table. This relationship tells the system that only values that have been submitted to the '*Employee Category*' field of the *Employee\_Categories* table are valid options for the '*Employee\_Type*' field in the *Employees* table.

## Performing the AI Value Mapping

### Uploading Data

Step 1:  Go to your newly created Uploader and upload your data.   Once the AutoMapping is complete, select the field you wish to initiate the AI Value Mapping.  In our scenario it is Employee\_Type.&#x20;

Step 2: However on the Column Mapping > Select Transform on the left-hand side column mapping field.  In this scenario, it will be the destination Employee\_Type field.&#x20;

<figure><img src="https://353417064-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYrsDW6vGBTygB1qqSE%2Fuploads%2F867ZjApt0LvSAqT8rZwY%2Fimage.png?alt=media&#x26;token=45df4577-1cb9-4745-8020-61dc118eee85" alt=""><figcaption></figcaption></figure>

<figure><img src="https://353417064-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYrsDW6vGBTygB1qqSE%2Fuploads%2FaQJlk9iciRuTY9sjy9GS%2Fimage.png?alt=media&#x26;token=5cc4ac21-b220-4201-9475-da8f94a62e44" alt=""><figcaption></figcaption></figure>

Step 3: Select the AI Value Mapping QuickFix. &#x20;

AI Value Mapping will automatically run and map.  You can then make adjustments and/or map any fields left unmapped. &#x20;

Step 4. Hit Save when you are done to save the mapping logic.&#x20;

<figure><img src="https://353417064-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYrsDW6vGBTygB1qqSE%2Fuploads%2F6YHO0mZLiJGJ08nCksfp%2Fimage.png?alt=media&#x26;token=70995ff2-d544-4530-a715-c9dd1ca33cf0" alt=""><figcaption></figcaption></figure>

### How to map unmapped fields (or adjust existing mapped fields).

Step 1:  In the AI Value Mapping Window, select the box next to an unmapped field

Step 2:  Scroll to the bottom and select the valid option to map the field to

Step 3:  Hit Apply

Step 4:  Hit Save Mapping

<figure><img src="https://353417064-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYrsDW6vGBTygB1qqSE%2Fuploads%2FbPuz4zSoOFkPcKwTW1dO%2FCleanShot%202024-10-11%20at%2015.29.08.gif?alt=media&#x26;token=53780821-4e54-424d-af36-d455f29d7a5e" alt="" width="375"><figcaption></figcaption></figure>
