AutoClean
AutoClean is a capability which allows end users clean data with one click. Your Osmos Uploader comes with prebuilt AutoClean capability for certain scenarios detailed below, but also empowers developers with the ability to transform data on the fly by creating their own AutoClean functions.
For more detail on how to clean messy data in Advanced Mode, please see our Advanced Mode: Map & Transform Data page
In step 2 of an upload, Map and Transform Data, AutoClean is displayed as a toggle above the "our Field" pane.
On your schema fields, you can add an attribute called
autoCleanMode that will let you configure whether or not AutoClean is on or off by default, or disabled entirely, for a given field. The options are:'auto': Toggle is off by default, can be turned on. This is the default setting if you don't specify a mode for a field.'onForced': AutoClean is on by default and cannot be turned off.'onButAllowDisable': AutoClean is on by default and can be turned off.'disabled': AutoClean is off and cannot be turned on for this field.
Osmos.configure({
schema: {
fields: [
{
name: 'date',
displayName: 'date',
description: '<Your field description here>',
// Disabled for this field, cannot be turned on
autoCleanMode: 'disabled',
},
{
name: 'datetime',
displayName: 'datetime',
description: '<Your field description here>',
// autoCleanMode defaults to 'auto' if you don't set it
},
{
name: 'bool',
displayName: 'bool',
description: '<Your field description here>',
// Toggle starts off, and you can turn it on
autoCleanMode: 'auto',
},
{
name: 'int',
displayName: 'int',
description: '<Your field description here>',
// Toggle starts on, and you cannot turn it off
autoCleanMode: 'onForced',
},
{
name: 'double',
displayName: 'double',
description: '<Your field description here>',
// Toggle starts on, and you can turn it off
autoCleanMode: 'onButAllowDisable',
},
],
},
token: 'your_token',
uploadDescription:
'<Include a description of your uploader upload here which will be shown to users that click the uploader>',
hideUploadSchema: true,
hideUploadDescription: true,
disableAdvancedMode: true,
});
The table below describes what cleanup operations will be performed by AutoClean, depending on the data type of the destination field, and whether or not the field is required.
Destination Field Type | Nullable | Required |
|---|---|---|
Integer |
|
|
Float |
|
|
Date | If data is not parse-able as a date, it will be set to null | If data is not parse-able as a date, it will remain unaltered and show an error |
Datetime | If data is not parse-able as a date and time, it will be set to null | If data is not parse-able, it will remain unaltered and show an error |
Boolean |
If data is none of the above, the output will be set to null |
If data is none of the above, it will remain unaltered and show an error |
Text | Text will remain unaltered by Osmos AutoClean | Text will remain unaltered by Osmos AutoClean |
You can also set up your own instances of AutoClean to perform custom cleanup operations on input data by setting up Server Side Validation Webhooks for the destination connector of your uploader.
By default, Osmos AutoClean will run, then send the resulting data to the webhook. The webhook can then return final writeback values, errors, or warnings to the Osmos data mapping page.
If you don't want Osmos AutoClean to be performed before data is sent to the webhook for a given field you can set that field's
autoCleanMode to 'disabled' as mentioned in the AutoClean Configuration Example section above. In which case the input data will be sent to your webhook unaltered.Last modified 3mo ago