Data Pipeline User Guide
Contents
- Introduction
- Getting Started
- 1. Add Maven or Gradle Dependencies
- 2. Download a license
- 3. Install License
- 4. Start Coding
- Integrations
- Configure Gradle/Maven for Compressed Parquet Support
- Additional Resources
- Running Diagnostics
- How Data Pipeline Works
- Records
- Field Names
- Field Values
- Field Types
- Data Readers and Writers
- Stream Operators
- Java Streams and Iterators
- Running Jobs
- Synchronous Jobs
- Asynchronous Jobs
- RunAsync
- Thread
- ExecutorService
- Waiting for Jobs to Start and Finish
- Managing Running Jobs
- Cancel
- Pause and Resume
- Monitoring Jobs
- Directly Access Job Properties
- Job Callback Hooks
- Job Lifecycle Listener
- Jobs Logging
- JMX Monitoring of Jobs
- Data Formats
- Excel
- Excel Documents and Providers
- Reading Excel Files
- Writing Excel Files
- Streaming Large Workbooks
- Formatting Styles and Hyperlinks
- Excel Examples
- LaTeX
- Add the LaTeX Dependency
- Writing a LaTeX Table
- Escaping and Schema Rules
- LaTeX Pipeline Output
- Filtering Data
- Data Validation
- Transforming Data
- Conditional Transformations
- Lookup and Joins
- Data Conversions
- Data Aggregation
- Simple Aggregation
- Aggregate Operators
- Sliding Window Aggregation
- Batch/Adjacent Sliding Windows
- Overlapping Sliding Windows
- Sparse/Sample Sliding Windows
- Time-Based Sliding Windows
- Built-in Window Strategies
- Custom Window Strategies
- Custom Aggregate Operators
- Detail and Summarize in a Single Pipeline
- Expression Language
- JSONata
- Add the JSONata Dependency
- Compile and Evaluate Expressions
- Query Any Data Format
- Working with Results
- Variables and Custom Functions
- JSONata Examples
- SQL Generators
- How the Builders Work
- SELECT Queries
- Joins
- WHERE and HAVING Criteria
- Nested Queries and Unions
- INSERT UPDATE and DELETE Statements
- MERGE and ON CONFLICT
- Running the Generated SQL
- Database-Specific Generators
- SQL Generator Examples
- Multi-threaded Processing
- Asynchronous Reading
- Asynchronous Writing
- Reading Many-to-One Asynchronously
- Reading One-to-Many Asynchronously
- Writing One-to-Many Asynchronously
- Process Incoming Data on Multiple Threads
- Event Bus
- How the Event Bus Works
- EventListener Interface
- Starting and Stopping an Event Bus
- Using The System Event Bus
- Publish and Subscribe to Events on the Bus
- Accessing the Current Event
- Connecting Pipelines to an Event Bus
- EventBusWriter
- EventBusReader
- Event Bus Pipelines
- Monitoring Event Buses
- Directly Access Event Bus Properties
- Event Bus Lifecycle Listener
- Watching All Records on an Event Bus
- Watching All Events on an Event Bus
- Monitoring Event Buses with JMX
- Handling Exceptions in the Event Bus
- Meter and Throttle DataPipeline Jobs
- Debugging DataPipeline Jobs
- Error Handling in DataPipeline Jobs
- Credentials Resolver
- Credentials
- Built-in Resolvers
- Connecting Resolvers to File Systems
- Credential Rotation
- Caching and the Resolver Registry
- Writing Your Own Resolver
- Credentials Resolver Examples
- Creating Custom DataPipeline Components
