A recurring business problem is achieving the ability to capture data in near-real time to act upon any significant event close to the moment it happens. For example, you may want to tap into a data stream and monitor any anomalies that need to be addressed immediately rather than during a nightly batch. Not only that, but maintaining, patching, and upgrading these clusters takes valuable time and effort away from business-impacting goals. Our client is looking for a stellar individual to work with the team in the following:

• Support in several projects with real-time data integration
• Through the POS system, goes to a vendor cloud and receive once a day to obtain real-time information to be analyzed
• Utilize AWS tech stack experience in building real-time pipeline
• Utilize AWS Tech Stack to handle incoming data streams and to manage the infrastructure
• Experience working on different file types like json,xml,parquet etc
• Proficient working in glue,emr,lambda etc different AWS services
• Know Athena and can load data in Redshift/Snowflake
• Know Airflow and CloudFormation is plus.

Responsibilities

• Create data pipelines in AWS cloud
• Experience with building Datalake in S3
• Understand the current application infrastructure and suggest changes to it.
• Define and document best practices and strategies regarding application deployment and infrastructure maintenance.
• Migrate our infrastructure with zero downtime to a highly available, scalable one.
• Set up a monitoring stack.
• Define service capacity planning strategies.
• Implement the application’s CI/CD pipeline using the AWS CI/CD stack.
• Write infrastructure as code using CloudFormation or similar.
• Utilize AWS Data Pipeline to reliably process and move data between different AWS compute and storage services, as well as on-premises data sources, at specified intervals.
• Efficiently transfer the data/results to AWS services such as Amazon S3, Amazon RDS, Amazon DynamoDB, and Amazon EMR.