Read from Amazon S3 Using a Rotating Credentials File

This example shows how to read a CSV file from Amazon S3 with credentials kept in a Java properties file. A FileCredentialsResolver reads the file each time the file system opens and re-reads it whenever its last-modified time changes, so a secret that an orchestrator rotates in place (a Kubernetes secret volume or a Vault Agent sidecar, for example) is used on the next connection without restarting the job.

Credentials file

Each property in the file becomes one credential entry, so the property names are the key names defined on Credentials: accessKey, secretKey and, optionally, sessionToken.

accessKey=YOUR ACCESS KEY
secretKey=YOUR SECRET KEY

Java Code Listing

package com.northconcepts.datapipeline.examples.security;

import java.io.InputStreamReader;

import com.northconcepts.datapipeline.amazons3.AmazonS3FileSystem;
import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.core.StreamWriter;
import com.northconcepts.datapipeline.csv.CSVReader;
import com.northconcepts.datapipeline.job.Job;
import com.northconcepts.datapipeline.security.FileCredentialsResolver;

public class ReadFromAmazonS3UsingRotatingCredentialsFile {

    private static final String CREDENTIALS_FILE = "example/data/input/s3-credentials.properties";
    private static final String BUCKET = "YOUR BUCKET";
    private static final String KEY = "output/trades.csv";

    public static void main(String[] args) throws Throwable {
        AmazonS3FileSystem s3 = new AmazonS3FileSystem()
                .setCredentialsResolver(new FileCredentialsResolver(CREDENTIALS_FILE));
        s3.open();
        try {
            DataReader reader = new CSVReader(new InputStreamReader(s3.readFile(BUCKET, KEY)))
                    .setFieldNamesInFirstRow(true);
            DataWriter writer = StreamWriter.newSystemOutWriter();

            Job.run(reader, writer);
        } finally {
            s3.close();
        }
    }

}

Code Walkthrough

1. CREDENTIALS_FILE points at the properties file; BUCKET and KEY name the object to read.
2. An AmazonS3FileSystem is created with a FileCredentialsResolver as its credentials resolver.
3. s3.open() resolves the credentials and connects. The resolver checks the file's modification time on every resolve, so after the file is rewritten the next open() uses the new keys.
4. s3.readFile(BUCKET, KEY) returns an InputStream for the object, wrapped in a CSVReader with field names in the first row.
5. Job.run() transfers the records to a StreamWriter that prints them to the console.
6. s3.close() in the finally block disconnects and clears the credentials snapshot held by the file system.

Console Output

Each record in trades.csv is printed to the console, followed by the record count.