🎉What's New
Stop by regularly to catch all of the latest features in Osmos!
October 2024
Embedded Uploaders
The Uploader Description text now displays in column mapping to provide users instructions as they review mappings.
September 2024
Pipelines
Increase the visibility of your recurring pipelines by showing only jobs with records.
Dataset Updates
Datasets are now more intuitive as you can review table schema, view linked connectors, and delete and clone tables in one place.
Projects
To increase your working space and to align with other actions, Projects has been moved to the bottom of the collapsible sidebar on the Osmos homepage.
Summer 2024 🍦
AI Value Mapping
We are stoked to release our next iteration of the popular value matching (“enum”) QuickFix. AI Value Mapping is a groundbreaking feature that transforms data mapping with the power of AI. Leveraging advanced Large Language Models (LLMs), this innovation automatically maps source data to its nearest semantic match in the destination schema. Enhance productivity and streamline complex processes with Osmos AI Value Mapping, the future of data transformation.
AI AutoClean
Osmos AutoClean eliminates mundane manual cleanup tasks and the need to use complex spreadsheet techniques, by giving business users the power to use natural language. Simply tell Osmos how data needs to be cleaned, and get clean data, fast!
New APIs
Osmos released fourteen new APIs making it even easier to interact with your Uploader and Pipeline detail. These API endpoints include:
Get Uploader
List Uploader Jobs
List Pipelines
Get Pipeline
List Pipeline Jobs
Update Pipeline
Create (Pipeline) Schedule
Update (Pipeline) Schedule
Get Pipeline Transformer
List Job Errors
List Connectors
Get Connector
Update Connector
Create Connector
Spring 2024 🌸
Homepage Chat
Introducing the all-new Osmos Chat – your gateway to seamless navigation and effortless interaction within the Osmos universe! Leveraging simple natural language queries, we've revolutionized the way you engage with Osmos, making it easier than ever to access vital information and perform tasks with just a few keystrokes
March 2024
Homepage
Collapsable Sidebar
Increase your working space by collapsing the sidebar on the Osmos homepage.
February 2024
Uploader
Embedded Jobs Table
Users can now view the job-level details of their uploads from within the embedded uploader.
Winter 2024 🏂🏽
AutoMap AI
With enhancements to AutoMap, our new AutoMap uses generative AI to significantly reduce the need for manual column mapping that is required during data ingestion. AutoMap AI takes that task and automatically maps your columns quickly and more accurately. Immediately map things like “MSRP” to “Price” and “Energy” to “Calories”. No additional setup up required.
January 2024
Authentication
New Authentication Methods
You can now sign into Osmos using Microsoft Authentication or with a passkey.
Osmos Uploader
Editing Uploader Name
The Uploader Name can now be modified by selecting the pencil icon to edit.
December 2023
Osmos Datasets
Clone Dataset
Datasets can now be duplicated within or across projects.
Osmos Uploader
Customizing Uploader Styling
Additional styling options are now supported for font, headers, buttons, and spinners. For the font body and headers, any css property can be used to customize them such as fontFamily, marginTop, marginbottom, padding, lineHeight, fontSize, and fontWeight.
Connectivity
Clone Connectors
Both Source and Destination Connectors can now be duplicated.
Data Transformations
New Formulas
November 2023
Data Transformations
Column Mapping Filters
Streamline your column mapping with filters.
Osmos Pipelines
Pipelines Page Updates
The Pipelines page now includes the ability to filter pipelines based on a variety of criteria.
Osmos Uploader
Uploader Page Updates
The Uploaders page now includes the ability to filter Uploaders based on a variety of criteria.
Fall 2023 🍁
Osmos API
Our all new Osmos API not only provides simple request-response semantics to read and write data, but it also provides rich data history, lineage, and auditability. Full high fidelity API definitions to create, read, update, and delete Datasets and Tables with a REST API.
October 2023
Osmos Datasets
Save a Dataset Query
Create and Save custom queries in Datasets.
Dataset Query Builder
We have added advanced query abilities by providing AND OR scenario handling.
Osmos Uploader
Optional Settings
For embedded Uploader's, Osmos stores all data transformations submitted for recall. You now can disable this functionality, at any point in time, and prevent the transforms for uploads to be saved for recall.
For internal Uploader's, there is an Admin level Feature Flag which allows Admin Users within an organization to set the baseline recall behavior of an uploader.
September 2023
Osmos Datasets
Clone Table
Dataset tables can now be duplicated into existing Datasets or into a different Dataset entirely. You can edit the schema before the completion of the cloning.
View Data
Datasets now has a number of new ways to view data
Osmos Query Builder
Metadata
Query Builder now includes a set of metadata fields that are available to add as columns to the any dataset table.
Osmos Dataset Source Connector
Only New Rows
A new connector configuration option has been released that will only pull rows changed since last pipeline was run.
Osmos Uploader
View Data
You now have an option view Datasets data directly from the Uploader
August 2023
Osmos Pipelines
Pipelines Detail Page Updates
The Pipelines Detail page now includes the ability to view skipped records at the pipeline run (job) level. Under the Status tab, you can view the skip count for your job.
Osmos Uploader
Save an Uploader with Errors
You now have an option to save an upload with errors.
Uploader Detail Page Updates
The Uploader Detail page has a number of exciting updates including taking viewing skips and taking action on errors.
Osmos Datasets
Schema Import
Dataset schema now can be imported via uploading the file or via drag & drop.
Query Builder
Leverage the Query Builder to join tables, select columns, add filters and group your data.
Connectivity
New Source Connectors
Transformations
Upload Destination Schema
For the following destination connectors, you now have the ability to import the file schema: S3, FTP, Drive, and HTTP API (Osmos Calls Your API).
Support for Enumerations (Enums)
Enum mapping is now available to map a group of constants to a specific valid option in the Uploader. Check out Enums for Advanced Mode vs Standard Mode Uploader.
Header Normalization
For Source Connectors you now have the ability to remove all whitespace (spaces, tabs, carriage returns, line endings) at start/end of each header. You can find this option under Advanced Options > Header Normalization.
File Name Available in the Column Mapping for Uploader
The Source originating File Name field is now available to add to a destination column.
New Formulas
Summer 2023 🏖️
Osmos Datasets
Osmos is proud to announce Datasets, a significant step forward in no-code data ingestion technology. Built for everyone, not just engineers, it places the power of database organization in the hands of a specific user who hasn’t had access to it before. Datasets keep information organized, housing tables and their data. Datasets offers a clean logical entity to build and manage your data models.
June 2023
Transformations
The Data Cleanup panel has been moved to the left side so all column mapping and clean up activity is organized in one area.
May 2023
Infrastructure
German Region
We now have a German specific region available.
Osmos Uploader
Enabling/Disability Recall + Auto-Mapping
You can now disable Auto-Map and/or Auto Recall
Support for XLSM
You now have the option upload XLSM file types.
Performance
Webhook Performance Updates
Added support for setting a maximum request size for individual validation webhook requests in the new setting called Validation Webhook Max Request Size in Bytes. The default is set to unlimited.
For the Validation Webhook Batch Size, If maxRequestSizeBytes is set, then the request will either include the batchSize number of records or be maxRequestSizeBytes bytes long, whichever limit is encountered first.
Transformations
File Name Available in the Column Mapping for Pipelines
The Source originating File Name field is now available to add to a destination column.
Formula Updates
Every pipeline or uploader job sent through Osmos is assigned a unique JobID. This UUID is now available to leverage in a formula for destination connectors.
April 2023
Osmos Pipelines
Pipelines Detail Page Updates
The Pipelines Detail page has a number of exciting updates!
Managing errors can now be executed at the pipeline run (job) level. Under the Status tab, you can view and take action for the individual pipeline runs.
The new actions include running a download error report, fixing the errors only at the pipeline run level as well as the ability to ignore the pipeline specific errors
Download Error Report
This new error report is included under the new Action icon for an individual pipeline.
Fixing Errors
We have added new functionality to fix errors that are unique to the specific pipeline run but do not require updates to the entire pipeline.
Ignore Errors
You now have the ability to ignore errors for a specific pipeline job.
Transformations
Formula Updates
We've added new names to some of our existing Formulas that will be more familiar to savvy Excel users. These formulas have the same or additional functionality under two different names.
March 2023
Osmos Uploader
Show/Hide Preview Pane when Uploading CSV Files
You now have an option to show or hide the Preview Pane when the user is uploading a CSV File.
Callback Methods
You now have two Methods to Trigger the Callback- at the time when the Uploader is submitted and when the Uploader is closed before submit.
JobID Added to Callback
A unique JobID is now a component of the Osmos Uploader Callback.
Performance
Organizing File Structure
You can now include datetime values in your file output for FTP, GCS, Google Drive, and S3.
February 2023
Transformations
Disable Deduplication
We have added an option to disable de-duplication for either files or records. Allowing all rows of all files in a source location to be processed whenever a job is triggered. You can find this option under Advanced Options > Deduplication Methods for GCS, S3, Drive, FTP, and Email file based source connectors.
Latin-1 Encoding Support
We now auto detect encoding and support latin-1 encoded files for GCS, S3, Drive, FTP and Email file based connectors.
January 2023
Performance
HTTP Performance Improvements
We have made huge performance improvements as you can now send 250K requests per second or more at a time with our automated batching feature. We have also added a useful view in the status page as we now display batches of requests as jobs instead of one job per request.
Webhook Performance Updates
The Validation Webhook default batch size has been updated to default to 10,000 rows for faster response.
The Validation webhook default max parallel requests have been updated to 1,000 requests at a time faster response.
Guidance
Webhook Guidance
A Parser Webhook tutorial is now available. You will learn how to implement a webhook server that receives the bytes of a file and parses it to return the data inside the file in tabular form.
We have added OpenAPI Webhook definition and testing guidance. You can take the YAML file, modify it and test it out in Postman or similar tool.
Transformations
New Formulas
Formula Updates
We've added new names to some of our existing Formulas that will be more familiar to savvy Excel users. These formulas have the same or additional functionality under two different names.
December 2022
Usability
Spreadsheet View for Osmos Pipelines
After you’ve cleaned your data in Osmos Pipelines, you’re able to edit single-cells just like you would in a spreadsheet. You can now review your data in a familiar spreadsheet format before submitting for a little extra peace of mind when uploading data.
Transformations
New Formulas
We've also added new names to some of our existing Formulas that will be more familiar to saavy Excel users. These formulas have the same functionality under two different names.
November 2022
Usability
Better Skip Handling
We made it easier to see which columns caused rows to skip during uploads. While cleaning your data, you can now filter rows that were skipped, or had any errors or warnings. Plus in the column mapping panel, there’s now a message noting if any rows were skipped in a particular column giving you better visibility while transforming your data.
Connectivity
New Source Connectors
New Destination Connectors
Transformations
Mapping and transformation recall is now available for nested JSON Connectors. Regardless of how complex your schemas are, Osmos remembers and brings back Formulas that had been previously used to transform data to a given Destination saving you time with each upload.
October 2022
Performance
Ingest Large Files
Osmos Pipelines now supports unlimited file sizes for STFP, GCS, and S3 files.
Organizing File Structure
This is an additional configuration for GCS, Google Drive, and S3 that allows users to have Osmos Uploaders or Osmos Pipelines dynamically name files (GCS and Google Drive) and create file paths (S3).
Validation Webhook Updates
Validation Webhook Batch Size and Validation Webhook Max Parallel Requests are new configurations that allow you to do the following:
Batch webhook requests to specific sizes (row counts) to support webhook performance
Run webhook requests in parallel meaning multiple batches of data will be sent to the webhook simultaneously
Connectivity
New Source Connectors
Shopee (only available on the Enterprise tier)
Coinbase
New Destination Connectors
Coinbase
Transformations
Improved support for nested data
When connecting to sources that have nested data, Osmos now automatically flattens the data, making it easier to understand and to map it to your destination.
New Formulas
September 2022
Updates to Osmos Uploader
We simplified the Osmos Uploader experience with a redesigned UI that is easier to navigate for non-technical users.
Plus, we're introducing single-click data cleanup with the addition of AutoClean.
By turning on Advanced Mode in the Uploader configuration snippet, you can regain access to advanced data transformations like QuickFixes, SmartFill, and Formulas.
These updates will be available on all pricing tiers.
Performance
Ingest Large Files
We’re focusing on scaling performance to meet your needs. The work's not done, but today Osmos can now support up to 10GB files in Osmos Uploader, and Osmos Pipelines can run massive 100GB+ files. Free trials are limited to 1 million rows.
Webhooks Write Back
You now have the ability to programmatically write back to individual fields from your validation endpoint. You can use this feature to do automatic data cleanup, lookups, and replace for users. This feature is exclusive to server-side webhooks. You can find more information about this here. Webhooks Write Back is available on all pricing tiers.
Connectivity
New Source Connectors
New Destination Connectors
Transformation
New Formulas
Last updated
