🎉
What's New
Each month we update this page of all the latest features to hit the Osmos platform.
We have added an option to disable de-duplication for either files or records. Allowing all rows of all files in a source location to be processed whenever a job is triggered. You can find this option under Advanced Options > Deduplication Methods for GCS, S3, Drive, FTP, and Email file based source connectors.
We now auto detect encoding and support latin-1 encoded files for GCS, S3, Drive, FTP and Email file based connectors.
We have made huge performance improvements as you can now send 250K requests per second or more at a time with our automated batching feature. We have also added a useful view in the status page as we now display batches of requests as jobs instead of one job per request.
- The Validation Webhook default batch size has been updated to default to 10,000 rows for faster response.
- The Validation webhook default max parallel requests have been updated to 1,000 requests at a time faster response.
- A Parser Webhook tutorial is now available. You will learn how to implement a webhook server that receives the bytes of a file and parses it to return the data inside the file in tabular form.
- We have added OpenAPI Webhook definition and testing guidance. You can take the YAML file, modify it and test it out in Postman or similar tool.
We've added new names to some of our existing Formulas that will be more familiar to savvy Excel users. These formulas have the same or additional functionality under two different names.
After you’ve cleaned your data in Osmos Pipelines, you’re able to edit single-cells just like you would in a spreadsheet. You can now review your data in a familiar spreadsheet format before submitting for a little extra peace of mind when uploading data.
We've also added new names to some of our existing Formulas that will be more familiar to saavy Excel users. These formulas have the same functionality under two different names.
We made it easier to see which columns caused rows to skip during uploads. While cleaning your data, you can now filter rows that were skipped, or had any errors or warnings. Plus in the column mapping panel, there’s now a message noting if any rows were skipped in a particular column giving you better visibility while transforming your data.
Mapping and transformation recall is now available for nested JSON Connectors. Regardless of how complex your schemas are, Osmos remembers and brings back Formulas that had been previously used to transform data to a given Destination saving you time each upload.
This is an additional configuration for GCS, Google Drive, and S3 that allows users to have Osmos Uploaders or Osmos Pipelines dynamically name files (GCS and Google Drive) and create file paths (S3).
Validation Webhook Batch Size and Validation Webhook Max Parallel Requests are new configurations which allow you to do the following:
- Batch webhook requests to specific sizes (row counts) to support webhook performance
- Run webhook requests in parallel meaning multiple batches of data will be sent to the webhook simultaneously
- Coinbase
When connecting to sources that have nested data, Osmos now automatically flattens the data, making it easier to understand and to map it to your destination.
We simplified the Osmos Uploader experience with a redesigned UI that is easier to navigate for nontechnical users.
By turning on Advanced Mode in the Uploader configuration snippet, you can regain access to the advanced data transformations like QuickFixes, SmartFill, and Formulas.
These updates will be available on all pricing tiers.
We’re focusing on scaling performance to meet your needs. The work's not done, but today Osmos can now support up to 10GB files in Osmos Uploader, and Osmos Pipelines can run massive 100GB+ files. Free trials are limited to 1 million rows.
You now have the ability to programmatically write back to individual fields from your validation endpoint. You can use this feature to do automatic data cleanup, lookups, and replace for users. This feature is exclusive to server side webhooks. You can find more information about this here. Webhooks Write Back is available on all pricing tiers.
.png?alt=media&token=faf2e6db-2b5a-4d5f-a97c-3011cb69b91e)
Last modified 21d ago