Go Back
Transforming and Writing Bulk Data in Batches
Publisher
: Lingk
Run In Lingk
Description
This recipe batches and transforms data for enables batched output to API or file-based data writers. To use this recipe, click Run!
Browse the knowledge base
Twitter
E-Mail
# _____ _ _____ __ # | __ \ (_) |_ _| / _| # | |__) |___ ___ _ _ __ ___ | | _ __ | |_ ___ # | _ // _ \/ __| | '_ \ / _ \ | | | '_ \| _/ _ \ # | | \ \ __/ (__| | |_) | __/ _| |_| | | | || (_) | # |_| \_\___|\___|_| .__/ \___| |_____|_| |_|_| \___/ # | | # |_| # Project Name - TRANSFORMING AND WRITING BULK DATA IN BATCHES # Recipe URL - https://app.lingk.io/a/10932/tf/17910 # Description - # This recipe batches and transforms data for enables batched output to API or file-based data writers. # To use this recipe, click Run! # Industry - Higher Ed # Business Process - Graduate Reporting # Systems - # Connectors - JSON # Data Flows - Single Direction # Connection Type - JSON # Add Recipe notes / Change log information here! # _____ _ # / ____| | | # | | ___ _ __ _ __ ___ ___| |_ ___ _ __ ___ # | | / _ \| '_ \| '_ \ / _ \/ __| __/ _ \| '__/ __| # | |___| (_) | | | | | | | __/ (__| || (_) | | \__ \ # \_____\___/|_| |_|_| |_|\___|\___|\__\___/|_| |___/ # # CONNECTORS specify what data will be pulled into the in-memory database during processing connectors: # JSON Setup - https://help.lingk.io/en/articles/74-json-connector-reference ###### Start: JSON Connectors ####### - name: sourceLeadData type: json properties: jsonObject: > [ { "leadId": 1, "firstName": "John", "lastName": "Thomas", "leadScore": 88 }, { "leadId": 2, "firstName": "Anne", "lastName": "Jacobs", "leadScore": 32 }, { "leadId": 3, "firstName": "Anne", "lastName": "Jacobs", "leadScore": 32 }, { "leadId": 4, "firstName": "Anne", "lastName": "Jacobs", "leadScore": 32 }, { "leadId": 5, "firstName": "Anne", "lastName": "Jacobs", "leadScore": 32 }, { "leadId": 6, "firstName": "Anne", "lastName": "Jacobs", "leadScore": 32 }, { "leadId": 7, "firstName": "Anne", "lastName": "Jacobs", "leadScore": 32 }, { "leadId": 8, "firstName": "Anne", "lastName": "Jacobs", "leadScore": 32 }, { "leadId": 9, "firstName": "Anne", "lastName": "Jacobs", "leadScore": 32 }, { "leadId": 10, "firstName": "Anne", "lastName": "Jacobs", "leadScore": 32 } ] ###### End: JSON Connectors ####### # _______ _ # |__ __| | | # | | __ _ ___| | _____ # | |/ _` / __| |/ / __| # | | (_| \__ \ <\__ \ # |_|\__,_|___/_|\_\___/ tasks: - name: batchBigJson type: dataOperation function: batch parameters: inputBatchGroupBy: leadId: desc leadScore: desc inputBatchSize: 3 # inputBatchSize inputBatchFields: "*" # comma delimited for other fields #inputBatchFields batchedColumnAlias: output # outputColumnName - optional with a default of "output" # _____ _ _ _ # / ____| | | | | | # | (___ | |_ __ _| |_ ___ _ __ ___ ___ _ __ | |_ ___ # \___ \| __/ _` | __/ _ \ '_ ` _ \ / _ \ '_ \| __/ __| # ____) | || (_| | || __/ | | | | | __/ | | | |_\__ \ # |_____/ \__\__,_|\__\___|_| |_| |_|\___|_| |_|\__|___/ # STATEMENTS specify how the data should be processed while in memory statements: #******************************************************************** D I S C L A I M E R *********************************************************************************************** # * # Note that in an effort to keep recipes optimized for DPH (Data Processing Hours), print statements should be commented out after development has concluded for a recipe. * # For more information on DPH optimization, please visit the following help article - https://help.lingk.io/en/articles/212-minimizing-data-processing-hours-on-the-lingk-platform * # * #******************************************************************** D I S C L A I M E R *********************************************************************************************** - statement: (sourceLeadDataRes) => select * from sourceLeadData #- statement: print sourceLeadDataRes - statement: | execute task --name batchBigJson --inputTable sourceLeadDataRes --outputBatchTable outputByBatch --result myResult #- statement: print myResult #- statement: print outputByBatch # Add more statements to convert, join, aggregrate, transform, and integrate your data
Retrieve Leads from the Marketo REST API
Amazon RDS (PostgreSQL) to Google Spreadsheet Writer