FTP

Overview
You can create an SFTP Destination Connector to write CSV files (by default) or JSONL files to an SFTP folder using an SSH Private Key or your SFTP password. 
Note:  We only support SFTP Destination Connector and not FTP.
Supported file formats: CSV and JSONL
Prerequisites
Required information:
SFTP Host
Port Number
Username
Password
Directory
Creating a SFTP Destination Connector
Step 1: After selecting + New Connector, under the System prompt, click SFTP.
Step 2: Provide a Connector Name.
Step 3: Select Destination Connector.
Step 4: Select the SFTP protocol used by the server.
Step 5: Provide the SFTP host. 
Step 6: Provide the port number.
Step 7: Provide your SFTP username.
Step 8: Provide the directory path for the SFTP folder you want to connect to. (Note: the path is case sensitive)
Step 9: Authentication
You have the option to authenticate with an SSH Private key or with a Username and Password. 
SSH Private Key
Step 10a: If you choose to authenticate with an SSH Private Key, provide the SSH private key with the BEGIN prefix and END suffix, as shown below. 
Step 11: Some private keys have a password as well. If you have one, provide the password in the in the SSH Private Key Password field. If you do not have one, you can leave this field blank. 
Step 10b: Password
If you select this option, provide your SFTP password.
Step 11: Building the Schema for the Destination Connector
Use the schema designer to build the output schema for this Destination Connector. 
Parameter
Description
Field Name
Provide a field name for the output fields. These names will be used as the column headers or field names in the output file you are writing to.
Type
Define the type of each field. The field types will be used to enforce rules when you send data to this Connector.
Nullable
Check this box if the field is nullable. If the field is not nullable, you will be required to provide values for this field when sending data to this Connector.
Delete
Deletes the field.
Add Field
Adds another field to the schema.
1. Click Add Field for each additional field required in the schema

2. Select Create Schema once you have built the schema. 
Advanced Options
Output File Format
By default, this Destination Connector writes CSV files, and each Osmos Pipeline run produces a new file. If preferred, you can choose to change the output to a JSONL file instead of a CSV file. 
File Prefix Format String
We support the designation of file prefixes in order to more easily manage the output of this connector. The contents of this field will be written into the filename of the data this Connector writes. If a prefix is specified, a UUID will be appended to it to prevent filename conflicts. 
For additional configuration go to Additional Configuration for File Prefix Format.​
Limit Records Per File
By default, we do not set a limit on the number of records to be written to a single destination file by a single job (i.e. a single run of a Pipeline or Uploader).  If this box is checked, the data written to the destination will be "chunked" into separate files which contain at-most the number of records designated here. These "chunked" files will be suffixed with it's position in the sequence i.e. filename_part_1.csv, filename_part_2.csv, etc.
Validation Webhook
We support the use of Validation Webhooks to prevent bad data from being written to your systems, adding another layer of protection to the built-in validations that Osmos provides. The Webhook URL can be posted here.
For more information on Validation Webhook configuration, see Server Side Validation Webhooks​
Overwrite Output Column with Raw Input Data
Enter the name of the destination column where you'd like to store the entire raw source record data. The raw source record data will be stored as a JSON string in the provided destination column.
Additional Configuration for File Prefix Format
File Output
We support the designation of file prefixes in order to more easily manage the output of this connector. The contents of this field will be written into the filename of the data this Connector writes. If a prefix is specified, a UUID will be appended to it to prevent filename conflicts.  Osmos leverages magic string identifiers in order to include additional information in your file prefix.
DateTime: You can include datetime values in your file output using String from time (strftime) format specifiers. The time values created here correspond to Osmos internal system time at the moment the job was started. See example 3
Output Scenarios: 
1.No file prefix 
Output: <user base path>/chunk-<chunk num>-<UUID>.<file extension>
2.File includes description in the prefix
Sample prefix: my_osmos_output_
Output: <user base path>/my_osmos_output_chunk-<chunk num>-<UUID>.<file extension>
3.File includes datetime specifiers in the prefix
Sample prefix: %F_%T_
Output: <user base path>/<YYYY-MM-DD>_<HH:mm:ss>_chunk-<chunk num>-<UUID>.<file extension>

CosmosDB

Google Cloud Storage (GCS)

Last modified 1mo ago

Parameter	Description
Field Name	Provide a field name for the output fields. These names will be used as the column headers or field names in the output file you are writing to.
Type	Define the type of each field. The field types will be used to enforce rules when you send data to this Connector.
Nullable	Check this box if the field is nullable. If the field is not nullable, you will be required to provide values for this field when sending data to this Connector.
Delete	Deletes the field.
Add Field	Adds another field to the schema.