S3

The S3 Source captures an ObjectCreated event from an AWS S3 bucket. DCNiOS creates S3 bucket event redirections to the SQS queue. Then, Apache NiFi captures the event and introduces it to the dataflow. The whole pipeline is created using DCNiOS. The SQS queue is deleted with DCNiOS, but the Event Notification in the S3 section needs to be removed manually.

The S3 Source requires:

• An identifier name of the process. It must be unique. Required.
• The region where the SQS is going to be deployed. AWS_DEFAULT_REGION. Required.
• The bucket name. The queue name is not necessary. DCNiOS creates an SQS with this pattern name <bucket-name>-events. Required.

Here is an example of the configuration file. Check the documentation of AWS credentials to define the Access Key and Secret Key.

S3 Source consists of the following component:

• GetSQS

GetS3:
 - name: S3
   AWS_DEFAULT_REGION: us-east-1
   AWS_S3_BUCKET: <bucket-name>