This solution was designed for ETL workflow. AWS StepFunction service was used to orchestrate the workflow. It gets triggered by a time based CloudWatch event rule. Once triggered, a docker container is launched to execute scripts which further pushes the data to S3 and metadata to SQS. A Lambda function pulls data from SQS. Next Lambda in workflow launches EMR cluster. Once Cluster is ready, EMR Steps are submitted. Appropriate status checks are implemented with "Wait" tasks in Step Functions.
This solution was implemented to automated the DR process. Daily EBS snapshots being taken everyday using DLM service. All EBS volumes are tagged with mount point and instance id (by DLM service). In the event of failure, a new instance is launched, all the snapshots for original instance are pulled based on DescribeSnapshots API call, volumes are created and attached with the new instance based on the mount point tags.
DLM, Lambda, Step Functions
Disaster Recovery - II
This solution was implemented to automated the DR process. Daily EBS snapshots being taken everyday using AWS Backup service. All EBS volumes are tagged with mount point and instance id. In the event of failure, a new instance is launched, all the snapshots for original instance are pulled from Backup service, volumes are created and attached with the new instance based on the mount point tags.
This implementation allows auto creation of infrastructure and then deploy the artifacts. The pipeline is triggered by external service which pushes artifacts to S3 bucket. This is highly useful for quickly creating the infra, deploy the app, test and then tear down the infrastructure.
This solution consolidates the build agents required for Azure DevOps Build and Release pipelines. Agents run inside EKS with docker installed in order to communicate with ECR. Build Agents running in EKS have docker container running as a side-car in same pod so that instead of mounting EKS host volume for docker client, it is actually referred to side container over TCP.
Amazon EKS, Docker, Azure DevOps
CFN Drift Remediation
This solution provisions IAM roles and policies using CloudFormation templates. In addition to that, if anyone manually edits the role, the drift will be detected and changes will be removed automatically.
CloudWatch Events, Lambda
Detect & Remove CloudFormation Drift
AWS HPC Design
This implementation process high workload on cluster on EC2 instances and managed using CfnCluster