Data Pipeline
Package
Description
Read and write to Amazon-S3.
Read and write Apache Avro files.
Read Bloomberg Message data.
Organize incoming data into groups of sequential records without aggregation or transformation.
Core DataPipeline classes for reading, writing, and processing data.
Read and write comma separated values (CSV) and other delimited data.
Troubleshoot DataPipeline's setup and environment.
Read emails.
In-memory, publish-subscribe hub using simple Java interfaces or loosely-coupled readers and writers.
Read and write Microsoft Excel documents.
Parse and execute DataPipeline's expression language.
Read and write records using DataPipeline's native binary format.
Read and write to Dropbox.
Read and write files to an FTP Server.
Read and write files to Hadoop Distributed File System.
Read and write files to an SFTP Server.
Read and write data by treating Zips as a file system.
Filter or exclude data by criteria.
Read and write fixed-width / fixed-length records.
Model and execute source-target data mapping using expressions and functions.
Decision table rule engine.
Decision tree rule engine.
Common classes for comparing objects to detect elements that were added, removed, renamed, and changed.
Database metadata model and code generation.
Detect number patterns in strings.
Declarative pipeline modeling and execution.
Pluggable algorithms to detect record breaks and fields in a
Tree, including collapsing down
(merging) tree-node-paths where a naming/wildcard pattern can be derived.Schema models for validation, mapping, and code generation.
Compare schema models to detect elements that were added, removed, renamed, and changed.
Detect date-time patterns in strings.
DataPipeline Foundation tools for generating code and models.
Read Google Analytics Report.
Read Google Calendar Events.
Read Google Contacts.
Read Gmail messages.
Read and write files to Google Drive.
Summarize batch and streaming data similar to "group by" in SQL.
Search and read Instagram media and user data.
Read Java Beans as a job's data source.
Database reading and writing via JDBC.
Insert strategies for
JdbcWriter and JdbcMultiWriter.Common SQL generation classes.
SQL generation classes for DELETE.
SQL generation classes for INSERT.
SQL generation classes for MERGE.
SQL generation classes for SELECT.
SQL generation classes for UPDATE.
Upsert strategies for
JdbcUpsertWriter and JdbcMultiUpsertWriter.Read Jira project, sprint and issue data.
RESTful client and Java wrapper for Jira.
Read and write Java Message Service data.
Running, tracking, and managing data transfer pipelines.
JSON reading and writing.
Programmatic JSON creation.
Apply JSONata transformations.
Read and write Apache Kafka distributed messaging system data.
Extract data lineage from records and fields.
Read MailChimp lists.
Reading and writing data to and from in-memory buffers.
Measures the rate (bytes/second or records/second) at which data is read or written.
Writes records to a Microsoft Word (OOXML .docx) document stream.
Read and write data in MongoDB.
Strategies to convert a single source
DataReader into many downstream readers.Read OFX / QFX (Quicken) / QBO (QuickBooks) files.
Read and write Apache ORC columnar files.
Read and Write Apache Parquet columnar files.
Write data to a PDF file.
Classes to retry and continue reading or writing data on failure.
Writes records to an RTF document stream.
Common task scheduling.
Encrypt and decrypt data.
Read data from Shopify.
REST client to read and write Shopify store data.
Code generation classes.
Generate SQL code for the H2 database with default mode.
Generate SQL code for the MySQL database.
Generate SQL code for the PostgreSQL database.
Write data to a text stream using FreeMarker template.
Write records as an easy-to-read, plain-text table with padded, aligned columns.
Limits the rate (bytes/second or records/second) at which data is read or written.
Restructure data using a variety of built-in and customizable operations.
Enrich data by with data from another source.
Read and write data to a Trello board and cards.
Search and read Twitter users and tweets.
Search and read Twitter users and tweets using Twitter v2 Api.
Validate data while reading.
Read web server logs.
XML reading and writing.
Programmatic XML creation.