Read from Amazon S3 Using an AWS Profile

This example shows how to read a CSV file from Amazon S3 using a named profile from the AWS credentials file instead of keys in code. useProfileCredentialsProvider() on AmazonS3FileSystem tells the AWS SDK to load the given profile from ~/.aws/credentials (or the file named by AWS_SHARED_CREDENTIALS_FILE), the same file the AWS CLI uses, and setRegion() names the region explicitly. A CredentialsResolver, if one is set, takes precedence over the profile.

AWS credentials file

The profile name in the code must match a section in the credentials file:

[YOUR AWS PROFILE]
aws_access_key_id = AKIA...
aws_secret_access_key = ...

Java Code Listing

package com.northconcepts.datapipeline.examples.amazons3;

import java.io.InputStreamReader;

import com.northconcepts.datapipeline.amazons3.AmazonS3FileSystem;
import com.northconcepts.datapipeline.core.DataReader;
import com.northconcepts.datapipeline.core.DataWriter;
import com.northconcepts.datapipeline.core.StreamWriter;
import com.northconcepts.datapipeline.csv.CSVReader;
import com.northconcepts.datapipeline.job.Job;

public class ReadFromAmazonS3UsingAnAwsProfile {

    private static final String PROFILE = "YOUR AWS PROFILE";
    private static final String REGION = "us-east-1";
    private static final String BUCKET = "YOUR BUCKET";
    private static final String KEY = "output/trades.csv";

    public static void main(String[] args) throws Throwable {
        AmazonS3FileSystem s3 = new AmazonS3FileSystem()
                .useProfileCredentialsProvider(PROFILE)
                .setRegion(REGION);
        s3.open();
        try {
            DataReader reader = new CSVReader(new InputStreamReader(s3.readFile(BUCKET, KEY)))
                    .setFieldNamesInFirstRow(true);
            DataWriter writer = StreamWriter.newSystemOutWriter();

            Job.run(reader, writer);
        } finally {
            s3.close();
        }
    }

}

Code Walkthrough

1. PROFILE names the section of the credentials file to use, REGION the AWS region, and BUCKET and KEY the object to read.
2. An AmazonS3FileSystem is created; useProfileCredentialsProvider(PROFILE) selects the profile and setRegion(REGION) sets the region.
3. s3.open() loads the profile through the AWS SDK and connects.
4. s3.readFile(BUCKET, KEY) returns an InputStream for the object, wrapped in a CSVReader with field names in the first row.
5. Job.run() transfers the records to a StreamWriter that prints them to the console.
6. s3.close() in the finally block disconnects.

Console Output

Each record in trades.csv is printed to the console, followed by the record count.