Links

AutoClean

Function

AutoClean is a capability which allows end users clean data with one click. Your Osmos Uploader comes with prebuilt AutoClean capability for certain scenarios detailed below, but also empowers developers with the ability to transform data on the fly by creating their own AutoClean functions.
For more detail on how to clean messy data in Advanced Mode, please see our Advanced Mode: Map & Transform Data page
In step 2 of an upload, Map and Transform Data, AutoClean is displayed as a toggle above the "our Field" pane.

AutoClean in Action

AutoClean Configuration Example

On your schema fields, you can add an attribute called autoCleanMode that will let you configure whether or not AutoClean is on or off by default, or disabled entirely, for a given field. The options are:
  • 'auto': Toggle is off by default, can be turned on. This is the default setting if you don't specify a mode for a field.
  • 'onForced': AutoClean is on by default and cannot be turned off.
  • 'onButAllowDisable': AutoClean is on by default and can be turned off.
  • 'disabled': AutoClean is off and cannot be turned on for this field.
Osmos.configure({
schema: {
fields: [
{
name: 'date',
displayName: 'date',
description: '<Your field description here>',
// Disabled for this field, cannot be turned on
autoCleanMode: 'disabled',
},
{
name: 'datetime',
displayName: 'datetime',
description: '<Your field description here>',
// autoCleanMode defaults to 'auto' if you don't set it
},
{
name: 'bool',
displayName: 'bool',
description: '<Your field description here>',
// Toggle starts off, and you can turn it on
autoCleanMode: 'auto',
},
{
name: 'int',
displayName: 'int',
description: '<Your field description here>',
// Toggle starts on, and you cannot turn it off
autoCleanMode: 'onForced',
},
{
name: 'double',
displayName: 'double',
description: '<Your field description here>',
// Toggle starts on, and you can turn it off
autoCleanMode: 'onButAllowDisable',
},
],
},
token: 'your_token',
uploadDescription:
'<Include a description of your uploader upload here which will be shown to users that click the uploader>',
hideUploadSchema: true,
hideUploadDescription: true,
disableAdvancedMode: true,
});

Osmos AutoClean

The table below describes what cleanup operations will be performed by AutoClean, depending on the data type of the destination field, and whether or not the field is required.
Destination Field Type
Nullable
Required
Integer
  1. 1.
    Strip non-numeric symbols
    • Example: $8 -> 8
  2. 2.
    Round to the nearest whole number
    • Example: 8.34 -> 8
    • Example: $8.79 -> 9
  1. 1.
    Strip non-numeric symbols
    • Example: $8 -> 8
  2. 2.
    Round to the nearest whole number
    • Example: 8.34 -> 8
    • Example: $8.79 -> 9
  3. 3.
    If no value is in the source data, enter 0
  4. 4.
    If source data is not parse-able as a number, it will remain unaltered and show an error
Float
  1. 1.
    Strip non-numeric symbols
    • Example: $8.79 -> 8.79
  1. 1.
    Strip non-numeric symbols
    • Example: $8.79 -> 8.79
  2. 2.
    If no value is in the source data, set to 0.0
  3. 3.
    If source data is not parse-able as a float, it will remain unaltered and show an error
Date
If data is not parse-able as a date, it will be set to null
If data is not parse-able as a date, it will remain unaltered and show an error
Datetime
If data is not parse-able as a date and time, it will be set to null
If data is not parse-able, it will remain unaltered and show an error
Boolean
  • 0, F, False, N, No (case insensitive) will map to false
  • 1, T, True, Y, Yes (case insensitive) will map to true
If data is none of the above, the output will be set to null
  • 0, F, False, N, No (case insensitive) will map to false
  • 1, T, True, Y, Yes (case insensitive) will map to true
If data is none of the above, it will remain unaltered and show an error
Text
Text will remain unaltered by Osmos AutoClean
Text will remain unaltered by Osmos AutoClean

Custom AutoClean

You can also set up your own instances of AutoClean to perform custom cleanup operations on input data by setting up Server Side Validation Webhooks for the destination connector of your uploader.
By default, Osmos AutoClean will run, then send the resulting data to the webhook. The webhook can then return final writeback values, errors, or warnings to the Osmos data mapping page.
If you don't want Osmos AutoClean to be performed before data is sent to the webhook for a given field you can set that field's autoCleanMode to 'disabled' as mentioned in the AutoClean Configuration Example section above. In which case the input data will be sent to your webhook unaltered.
Custom Autoclean in action