Links
🎉

What's New

Each month we update this page of all the latest features to hit the Osmos platform.

August 2023

Osmos Pipelines

Pipelines Detail Page Updates

The Pipelines Detail page now includes the ability to view skipped records at the pipeline run (job) level. Under the Status tab, you can view the skip count for your job.

Osmos Uploader

Save an Uploader with Errors

You now have an option to save an upload with errors.

Uploader Detail Page Updates

The Uploader Detail page has a number of exciting updates including taking viewing skips and taking action on errors.

Osmos Datasets

Schema Import

Dataset schema now can be imported via uploading the file or via drag & drop.

Query Builder

Leverage the Query Builder to join tables, select columns, add filters and group your data.

Connectivity

New Source Connectors

Transformations

Upload Destination Schema

For the following destination connectors, you now have the ability to import the file schema: S3, FTP, Drive, and HTTP API (Osmos Calls Your API).

Support for Enumerations (Enums)

Enum mapping is now available to map a group of constants to a specific valid option in the Uploader. Check out Enums for Advanced Mode vs Standard Mode Uploader.

Header Normalization

For Source Connectors you now have the ability to remove all whitespace (spaces, tabs, carriage returns, line endings) at start/end of each header. You can find this option under Advanced Options > Header Normalization.

File Name Available in the Column Mapping for Uploader

The Source originating File Name field is now available to add to a destination column.

New Formulas

Summer 2023 🏖️

Osmos Datasets

The power of database organization for frontline teams!!!! Datasets keep information organized, housing tables and their data. Datasets offers a clean logical entity to build and manage your data models.

June 2023

Transformations

The Data Cleanup panel has been moved to the left side so all column mapping and clean up activity is organized in one area.

May 2023

Infrastructure

German Region

We now have a German specific region available.

Osmos Uploader

Enabling/Disability Recall + Auto-Mapping

You can now disable Auto-Map and/or Auto Recall

Support for XLSM

You now have the option upload XLSM file types.

Performance

Webhook Performance Updates

Added support for setting a maximum request size for individual validation webhook requests in the new setting called Validation Webhook Max Request Size in Bytes. The default is set to unlimited.
For the Validation Webhook Batch Size, If maxRequestSizeBytes is set, then the request will either include the batchSize number of records or be maxRequestSizeBytes bytes long, whichever limit is encountered first.

Transformations

File Name Available in the Column Mapping for Pipelines

The Source originating File Name field is now available to add to a destination column.

Formula Updates

Every pipeline or uploader job sent through Osmos is assigned a unique JobID. This UUID is now available to leverage in a formula for destination connectors.

April 2023

Osmos Pipelines

Pipelines Detail Page Updates

The Pipelines Detail page has a number of exciting updates!
Managing errors can now be executed at the pipeline run (job) level. Under the Status tab, you can view and take action for the individual pipeline runs.
The new actions include running a download error report, fixing the errors only at the pipeline run level as well as the ability to ignore the pipeline specific errors

Download Error Report

This new error report is included under the new Action icon for an individual pipeline.

Fixing Errors

We have added new functionality to fix errors that are unique to the specific pipeline run but do not require updates to the entire pipeline.

Ignore Errors

You now have the ability to ignore errors for a specific pipeline job.

Transformations

Formula Updates

We've added new names to some of our existing Formulas that will be more familiar to savvy Excel users. These formulas have the same or additional functionality under two different names.

March 2023

Osmos Uploader

Show/Hide Preview Pane when Uploading CSV Files

You now have an option to show or hide the Preview Pane when the user is uploading a CSV File.

Callback Methods

You now have two Methods to Trigger the Callback- at the time when the Uploader is submitted and when the Uploader is closed before submit.

JobID Added to Callback

A unique JobID is now a component of the Osmos Uploader Callback.

Performance

Organizing File Structure

You can now include datetime values in your file output for FTP, GCS, Google Drive, and S3.

February 2023

Transformations

Disable Deduplication

We have added an option to disable de-duplication for either files or records. Allowing all rows of all files in a source location to be processed whenever a job is triggered. You can find this option under Advanced Options > Deduplication Methods for GCS, S3, Drive, FTP, and Email file based source connectors.

Latin-1 Encoding Support

We now auto detect encoding and support latin-1 encoded files for GCS, S3, Drive, FTP and Email file based connectors.

January 2023

Performance

HTTP Performance Improvements

We have made huge performance improvements as you can now send 250K requests per second or more at a time with our automated batching feature. We have also added a useful view in the status page as we now display batches of requests as jobs instead of one job per request.

Webhook Performance Updates

  • The Validation Webhook default batch size has been updated to default to 10,000 rows for faster response.
  • The Validation webhook default max parallel requests have been updated to 1,000 requests at a time faster response.

Guidance

Webhook Guidance

  • A Parser Webhook tutorial is now available. You will learn how to implement a webhook server that receives the bytes of a file and parses it to return the data inside the file in tabular form.
  • We have added OpenAPI Webhook definition and testing guidance. You can take the YAML file, modify it and test it out in Postman or similar tool.

Transformations

New Formulas

Formula Updates

We've added new names to some of our existing Formulas that will be more familiar to savvy Excel users. These formulas have the same or additional functionality under two different names.

December 2022

Usability

Spreadsheet View for Osmos Pipelines

After you’ve cleaned your data in Osmos Pipelines, you’re able to edit single-cells just like you would in a spreadsheet. You can now review your data in a familiar spreadsheet format before submitting for a little extra peace of mind when uploading data.

Transformations

New Formulas

We've also added new names to some of our existing Formulas that will be more familiar to saavy Excel users. These formulas have the same functionality under two different names.

November 2022

Usability

Better Skip Handling

We made it easier to see which columns caused rows to skip during uploads. While cleaning your data, you can now filter rows that were skipped, or had any errors or warnings. Plus in the column mapping panel, there’s now a message noting if any rows were skipped in a particular column giving you better visibility while transforming your data.

Connectivity

New Source Connectors

New Destination Connectors

Transformations

Mapping and transformation recall is now available for nested JSON Connectors. Regardless of how complex your schemas are, Osmos remembers and brings back Formulas that had been previously used to transform data to a given Destination saving you time each upload.

October 2022

Performance

Ingest Large Files

Osmos Pipelines now supports unlimited file sizes for STFP, GCS, and S3 files.

Organizing File Structure

This is an additional configuration for GCS, Google Drive, and S3 that allows users to have Osmos Uploaders or Osmos Pipelines dynamically name files (GCS and Google Drive) and create file paths (S3).

Validation Webhook Updates

Validation Webhook Batch Size and Validation Webhook Max Parallel Requests are new configurations which allow you to do the following:
  • Batch webhook requests to specific sizes (row counts) to support webhook performance
  • Run webhook requests in parallel meaning multiple batches of data will be sent to the webhook simultaneously

Connectivity

New Source Connectors

  • Shopee (only available on the Enterprise tier)
  • Coinbase
  • GCS

New Destination Connectors

  • Coinbase
  • GCS

Transformations

Improved support for nested data

When connecting to sources that have nested data, Osmos now automatically flattens the data, making it easier to understand and to map it to your destination.

New Formulas

September 2022

Updates to Osmos Uploader

We simplified the Osmos Uploader experience with a redesigned UI that is easier to navigate for nontechnical users.
Plus, we're introducing single-click data cleanup with the addition of AutoClean.
By turning on Advanced Mode in the Uploader configuration snippet, you can regain access to the advanced data transformations like QuickFixes, SmartFill, and Formulas.
These updates will be available on all pricing tiers.

Performance

Ingest Large Files

We’re focusing on scaling performance to meet your needs. The work's not done, but today Osmos can now support up to 10GB files in Osmos Uploader, and Osmos Pipelines can run massive 100GB+ files. Free trials are limited to 1 million rows.

Webhooks Write Back

You now have the ability to programmatically write back to individual fields from your validation endpoint. You can use this feature to do automatic data cleanup, lookups, and replace for users. This feature is exclusive to server side webhooks. You can find more information about this here. Webhooks Write Back is available on all pricing tiers.

Connectivity

New Source Connectors

New Destination Connectors

Transformation

New Formulas