Intercepting ECS CloudWatch Logs

The ECS Task can be started from the other machine and can be waited till completed or failed. The logs are streamed directly to CloudWatch. The waiting process can poll and show them while waiting for the task.

Idea

The problem occurs if I run 20 or more tasks in my binary pipeline. I know something is working and even progressing, but I do not want to see just ECS task started and complete messages. I don’t want to check logs manually in the AWS CloudWatch neither. But I want to see all the logs happening in ECS container and aggregate them back in my master process.

Solution

I changed the ECS task in pipeline to not just wait, but to wait and pull available logs and write it back to the logger as local log entry. The change was simple and turned out to be working as expected.

As always the code is available here.