Go Back
SFTP File Management
Publisher
: Lingk
Run In Lingk
Description
This recipe reads a file from SFTP processes the data and then overwrites the file with new data
Browse the knowledge base
Twitter
E-Mail
# _____ _ _____ __ # | __ \ (_) |_ _| / _| # | |__) |___ ___ _ _ __ ___ | | _ __ | |_ ___ # | _ // _ \/ __| | '_ \ / _ \ | | | '_ \| _/ _ \ # | | \ \ __/ (__| | |_) | __/ _| |_| | | | || (_) | # |_| \_\___|\___|_| .__/ \___| |_____|_| |_|_| \___/ # | | # |_| # # Project Name - SFTP FILE MANAGEMENT # Recipe URL - https://app.lingk.io/a/10932/tf/17559 # Description - # This recipe reads a file from SFTP processes the data and then overwrites the file with new data # Industry - Higher Ed # Business Process - Graduate Reporting # Systems - SFTP # Connectors - SFTP # Data Flows - Single Direction # Connection Type - SFTP to SFTP # Data Models #1. # Add Recipe notes / Change log information here! # _____ _ # / ____| | | # | | ___ _ __ _ __ ___ ___| |_ ___ _ __ ___ # | | / _ \| '_ \| '_ \ / _ \/ __| __/ _ \| '__/ __| # | |___| (_) | | | | | | | __/ (__| || (_) | | \__ \ # \_____\___/|_| |_|_| |_|\___|\___|\__\___/|_| |___/ # # CONNECTORS specify what data will be pulled into the in-memory database during processing connectors: # Configure SFTP credentials in your Environment before running this recipe # SFTP Setup - https://help.lingk.io/en/articles/296-sftp-setup-guide ###### Start - SFTP Reader connectors ###### ## SFTP connectors - name: lastRun type: sftpReader format: csv properties: path: /automation/lastrun.csv ###### End - SFTP Reader connectors ###### ###### Start - SFTP Writer connectors ###### - name: runStatus type: sftpWriter format: csv properties: path: /automation/lastrun.csv ###### End - SFTP Writer connectors ###### # ______ _ # | ____| | | # | |__ ___ _ __ _ __ ___ __ _| |_ ___ # | __/ _ \| '__| '_ ` _ \ / _` | __/ __| # | | | (_) | | | | | | | | (_| | |_\__ \ # |_| \___/|_| |_| |_| |_|\__,_|\__|___/ # Configures the format of the file written to SFTP ## CSV data formats for SFTP Connector readFormats: - name: csv type: delimited properties: quoteAllFields: true delimiter: ',' header: true writeFormats: - name: csv type: delimited properties: quoteAllFields: true delimiter: ',' header: true # _____ _ _ _ # / ____| | | | | | # | (___ | |_ __ _| |_ ___ _ __ ___ ___ _ __ | |_ ___ # \___ \| __/ _` | __/ _ \ '_ ` _ \ / _ \ '_ \| __/ __| # ____) | || (_| | || __/ | | | | | __/ | | | |_\__ \ # |_____/ \__\__,_|\__\___|_| |_| |_|\___|_| |_|\__|___/ # STATEMENTS specify how the data should be processed while in memory statements: #******************************************************************** D I S C L A I M E R *********************************************************************************************** # * # Note that in an effort to keep recipes optimized for DPH (Data Processing Hours), print statements should be commented out after development has concluded for a recipe. * # For more information on DPH optimization, please visit the following help article - https://help.lingk.io/en/articles/212-minimizing-data-processing-hours-on-the-lingk-platform * # * #******************************************************************** D I S C L A I M E R *********************************************************************************************** ### GET LAST RUN DATA - statement: | (currentTimestamp) => SELECT current_timestamp #- statement: print currentTimestamp - statement: | (allData) => SELECT type, date, status FROM lastRun #- statement: print allData - statement: | (lastRun) => SELECT type, date, status FROM lastRun where type ="last" and status="complete" #- statement: print lastRun # Through "date_format" function it provides the format needed to the date retrieved - statement: | (test) => SELECT date_format(from_utc_timestamp(lastRun.date, "America/New_York") , "MM/dd/yyyy hh:mm:ss"), date_format(from_utc_timestamp(current_timestamp(), "America/New_York") , "MM/dd/yyyy hh:mm:ss") FROM lastRun WHERE type = "last" and status= "complete" #- statement: print test # create two rows of data - statement: | (currentRun) => SELECT type, date, status FROM lastRun where type ="last" and status="complete" UNION SELECT "current" type, current_timestamp date, "started" status #- statement: print currentRun # Write file to SFTP - statement: INSERT currentRun INTO runStatus # add some time between writes if you want # - statement: SLEEP 10000 # create two rows of data - statement: | (currentRun) => SELECT "last" type, current_timestamp date, "complete" status UNION SELECT "current" type, current_timestamp date, "complete" status #- statement: print currentRun # Write file to SFTP - statement: INSERT currentRun INTO runStatus # Add more statements to convert, join, aggregrate, transform, and integrate your data
Join data between two Google Spreadsheets
Salesforce Data Cleanup