Data Engineering with AWS: Acquire the skills to design and build AWS-based data transformation pipelines like a pro 2nd ed. Edition

 

Looking to revolutionize your data transformation game with AWS? Look no further! From strong foundations to hands-on building of data engineering pipelines, our expert-led manual has got you covered.

Key Features

Delve into robust AWS tools for ingesting, transforming, and consuming data, and for orchestrating pipelines

Stay up to date with a comprehensive revised chapter on Data Governance

Build modern data platforms with a new section covering transactional data lakes and data mesh

Book Description

This book, authored by a seasoned Senior Data Architect with 25 years of experience, aims to help you achieve proficiency in using the AWS ecosystem for data engineering. This revised edition provides updates in every chapter to cover the latest AWS services and features, takes a refreshed look at data governance, and includes a brand-new section on building modern data platforms which covers; implementing a data mesh approach, open-table formats (such as Apache Iceberg), and using DataOps for automation and observability.

You'll begin by reviewing the key concepts and essential AWS tools in a data engineer's toolkit and getting acquainted with modern data management approaches. You'll then architect a data pipeline, review raw data sources, transform the data, and learn how that transformed data is used by various data consumers. You’ll learn how to ensure strong data governance, and about populating data marts and data warehouses along with how a data lakehouse fits into the picture. After that, you'll be introduced to AWS tools for analyzing data, including those for ad-hoc SQL queries and creating visualizations. Then, you'll explore how the power of machine learning and artificial intelligence can be used to draw new insights from data. In the final chapters, you'll discover transactional data lakes, data meshes, and how to build a cutting-edge data platform on AWS.

By the end of this AWS book, you'll be able to execute data engineering tasks and implement a data pipeline on AWS like a pro!

What you will learn

Seamlessly ingest streaming data with Amazon Kinesis Data Firehose

Optimize, denormalize, and join datasets with AWS Glue Studio

Use Amazon S3 events to trigger a Lambda process to transform a file

Load data into a Redshift data warehouse and run queries with ease

Visualize and explore data using Amazon QuickSight

Extract sentiment data from a dataset using Amazon Comprehend

Build transactional data lakes using Apache Iceberg with Amazon Athena

Learn how a data mesh approach can be implemented on AWS

Who this book is for

This book is for data engineers, data analysts, and data architects who are new to AWS and looking to extend their skills to the AWS cloud. Anyone new to data engineering who wants to learn about the foundational concepts, while gaining practical experience with common data engineering services on AWS, will also find this book useful. A basic understanding of big data-related topics and Python coding will help you get the most out of this book, but it’s not a prerequisite. Familiarity with the AWS console and core services will also help you follow along.

Table of Contents

An Introduction to Data Engineering

Data Management Architectures for Analytics

The AWS Data Engineer’s Toolkit

Data Governance, Security, and Cataloging

Architecting Data Engineering Pipelines

Ingesting Batch and Streaming Data

Transforming Data to Optimize for Analytics

Identifying and Enabling Data Consumers

A Deeper Dive into Data Marts and Amazon Redshift

Orchestrating the Data Pipeline

Hard Copy: Data Engineering with AWS: Acquire the skills to design and build AWS-based data transformation pipelines like a pro 2nd ed. Edition

Read More

DevOps on AWS: Release and Deploy

 

Build your subject-matter expertise

This course is part of the DevOps on AWS Specialization

When you enroll in this course, you'll also be enrolled in this Specialization.

Learn new concepts from industry experts

Gain a foundational understanding of a subject or tool

Develop job-relevant skills with hands-on projects

Earn a shareable career certificate

Join Free: DevOps on AWS: Release and Deploy

There are 2 modules in this course

AWS provides a set of flexible services designed to enable companies to more rapidly and reliably build and deliver products using AWS and DevOps practices. These services simplify provisioning and managing infrastructure, deploying application code, automating software release processes, and monitoring your application and infrastructure performance. 

The third course in the series explains how to improve the deployment process with DevOps methodology, and also some tools that might make deployments easier, such as Infrastructure as Code, or IaC, and AWS CodeDeploy.

The course begins with reviewing topics covered in the first course of the DevOps on AWS series. You will learn about the differences between continuous integration, continuous delivery, and continuous deployment. In Exercises 1 and 2, you will set up AWS CodeDeploy and make revisions that will then be deployed. If you use AWS Lambda, you will explore ways to address additional considerations when you deploy updates to your Lambda functions.

Next, you will explore how infrastructure as code (IaC) helps organizations achieve automation, and which AWS solutions provide a DevOps-focused way of creating and maintaining infrastructure. In Exercise 3, you will be provided with an AWS CloudFormation template that will set up backend services, such as AWS CodePipeline, AWS CodeCommit, AWS CodeDeploy, and AWS CodeBuild. You will then upload new revisions to the pipeline.

Read More

DevOps on AWS: Operate and Monitor

 

Build your subject-matter expertise

This course is part of the DevOps on AWS Specialization

When you enroll in this course, you'll also be enrolled in this Specialization.

Learn new concepts from industry experts

Gain a foundational understanding of a subject or tool

Develop job-relevant skills with hands-on projects

Earn a shareable career certificate

Join Free: DevOps on AWS: Operate and Monitor

There are 2 modules in this course

The third and the final course in the DevOps series will teach how to use AWS Services to control the architecture in order to reach a better operational state. Monitoring and Operation are key aspects for both the release pipeline and production environments, because they provide instruments that help discover what's happening, as well as do modifications and enhancements on infrastructure that is currently running. 

This course teaches how to use Amazon CloudWatch for monitoring, as well as Amazon EventBridge and AWS Config for continuous compliance. It also covers Amazon CloudTrail and a little bit of Machine Learning for Monitoring operations!

Read More

Exam Prep: AWS Certified Cloud Practitioner Foundations

 

What you'll learn

The four domains - Cloud Concepts, Security and Compliance, Technology and Billing and Pricing - for the AWS Certified Cloud Practitioner exam

Certification exam-level practice questions written by experts from AWS

Simulations designed to solidify understanding of cloud concepts you need to know for the exam

Join Free: Exam Prep: AWS Certified Cloud Practitioner Foundations

There are 4 modules in this course

This new foundational-level course from Amazon Web Services (AWS), is designed to help you to assess your preparedness for the AWS Certified Cloud Practitioner certification exam.  You will learn how to prepare for the exam by exploring the exam’s topic areas and how they map to both AWS Cloud practitioner roles and to specific areas of study. You will review sample certification questions in each domain, practice skills with hands-on exercises, test your knowledge with practice question sets, and learn strategies for identifying incorrect responses by interpreting the concepts that are being tested in the exam. At the end of this course you will have all the knowledge and tools to help you identity your strengths and weaknesses in each certification domain areas that are being tested on the certification exam. 

The AWS Certified Cloud Foundations Certification the AWS Certified Cloud Practitioner (CLF-C01) exam is intended for individuals who can effectively demonstrate an overall knowledge of the AWS Cloud independent of a specific job role. The exam validates a candidate’s ability to complete the following tasks: Explain the value of the AWS Cloud, Understand and explain the AWS shared responsibility model, understand security best practices, Understand AWS Cloud costs, economics, and billing practices, Describe and position the core AWS services, including compute, network, databases, and storage and identify AWS services for common use cases

Read More

AWS Cloud Practitioner Essentials

 

What you'll learn

Understand the working definition of the AWS Cloud

Differentiate between on-premises, hybrid-cloud, and all-in cloud

Describe the basic global infrastructure of the AWS Cloud

Explain the benefits of the AWS Cloud

Join Free: AWS Cloud Practitioner Essentials

There are 7 modules in this course

Welcome to AWS Cloud Practitioner Essentials. If you’re new to the cloud, whether you’re in a technical or non-technical role such as finance, legal, sales, marketing, this course will provide you with an understanding of fundamental AWS Cloud concepts to help you gain confidence to contribute to your organization’s cloud initiatives. This course is also the starting point to prepare for your AWS Certified Cloud Practitioner certification whenever it’s convenient for you.

After you complete the course, you’ll understand the benefits of the AWS Cloud and the basics of its global infrastructure. You’ll be able to describe and provide an example of the core AWS services, including compute, network, databases, and storage. For the finance-minded, you’ll be able to articulate the financial benefits of the AWS Cloud, define core billing and pricing models, and learn how to use pricing tools to make cost-effective choices for AWS services.

Read More

Migrating to the AWS Cloud

 

Build your subject-matter expertise

This course is part of the AWS Fundamentals Specialization

When you enroll in this course, you'll also be enrolled in this Specialization.

Learn new concepts from industry experts

Gain a foundational understanding of a subject or tool

Develop job-relevant skills with hands-on projects

Earn a shareable career certificate

Join Free: Migrating to the AWS Cloud

There are 4 modules in this course

This introductory course is for anyone who wants a deeper dive into AWS migration. Whether you want to understand what services are helpful, need to plan a migration for your organization, or are helping other groups with their own migration, you will find valuable information throughout this course. The course sessions structure cloud migration through the three-phase migration process from AWS: assess, mobilize, and migrate and modernize. This process is designed to help your organization approach and implement a migration of tens, hundreds, or thousands of applications. By learning about this three-phase structure—and the various AWS tools, features, and services that can help you during each phase—you will complete this course with a better understanding of how to design and implement migrations to AWS.

Read More

Architecting Solutions on AWS

 

Build your subject-matter expertise

This course is available as part of 

When you enroll in this course, you'll also be asked to select a specific program.

Learn new concepts from industry experts

Gain a foundational understanding of a subject or tool

Develop job-relevant skills with hands-on projects

Earn a shareable career certificate

Join Free: Architecting Solutions on AWS

There are 4 modules in this course

Are you looking to get more technical? Are you looking to begin working in the cloud, but don’t know where to go next? Are you looking to up your game by prepping for the AWS Solutions Architect Associate Exam? Do you see yourself as a cloud consultant, but can’t quite envision how your days would be? Are you puzzled how to match a customer’s requirements with the right AWS services/solutions? If so, you are in the right place!! You’ll learn how to plan, think, and act like a Solution Architect in a real-life customer scenario.

In this course, you’ll get prepared to begin your career architecting solutions on AWS. Through a series of use case scenarios and practical learning, you’ll learn to identify services and features to build resilient, secure, and highly available IT solutions in the AWS Cloud. Each week, a fictional customer will present a different need. We will then review the options, choose the best one for the use case and walk you through the architecture design on a whiteboard. You’ll learn about event-driven architectures with a focus on performance efficiency and cost. You’ll then gain knowledge on how to architect a solution using many purpose-built AWS services. With this understanding, you’ll get a sense of hybrid architectures with a refined focus on reliability and operational efficiency. Finally, you’ll wrap up your learning by understanding a multi-account strategy centered on security and cost.

Read More

AWS Cloud Solutions Architect Professional Certificate

 

What you'll learn

Make informed decisions about when and how to apply key AWS Services for compute, storage, database, networking, monitoring, and security.

Design architectural solutions, whether designing for cost, performance, and/or operational excellence, to address common business challenges.

Create and operate a data lake in a secure and scalable way, ingest and organize data into the data lake, and optimize performance and costs.

Prepare for the certification exam, identify your strengths and gaps for each domain area, and build strategies for identifying incorrect responses.

Join Free: AWS Cloud Solutions Architect Professional Certificate

Professional Certificate - 4 course series

This professional certificate provides the knowledge and skills you need to start building your career in cloud architecture and helps you prepare for the AWS Certified Solutions Architect - Associate exam. You will start by learning key AWS Services for compute, storage, database, networking, monitoring, and security, then dive into how to design architectural solutions, how to create and operate a data lake, and how to prepare for the certification exam.

The AWS Certified Solutions Architect – Associate certification showcases knowledge and skills in AWS technology across a wide range of AWS services. The certification focuses on the design of cost and performance optimized solutions and demonstrating a strong understanding of the AWS Well-Architected Framework. This AWS Certification is one of the top-paying IT certifications, per the 

 SkillSoft IT Skills and Salary report

. Per

 Enterprise Strategy Group

, surveyed AWS Certification holders credited their certification for their higher earnings (74%), increased confidence (87%), and increased influence among coworkers (79%).

To prepare for your AWS Certification exam, we recommend that — in addition to attaining this professional certificate — candidates review the free exam guide, sample questions, and AWS technical documentation (e.g. white papers and product FAQs) on the

 AWS Certified Solutions Architect - Associate exam page

 to understand what content and services are covered by the exam.

Applied Learning Project

Through 15 hands-on labs, you’ll use the AWS Management Console to apply skills learned in the videos. 

For example: 

In Architecting Solutions on AWS, you’ll use Amazon API Gateway, AWS Lambda, Amazon SQS, Amazon DynamoDB, and Amazon SNS to build a serverless web backend.

In Introduction to Designing Data Lakes, you’ll use Amazon S3, Amazon OpenSearch Service, AWS Lambda and Amazon API Gateway to create an Amazon OpenSearch Service Cluster. You’ll also use Amazon S3, Amazon EC2, Amazon Kinesis Data Firehose, Amazon Kinesis Data Analytics, Amazon Elasticsearch Service to create a data ingestion pipeline with the use of high-scale AWS Managed services. 

In Cloud Technical Essentials, you’ll design a 3-tier architecture using services like Amazon VPC, Amazon EC2, Amazon RDS with high availability and Elastic Load Balancing following AWS best practices. You’ll upload an architecture diagram laying out your design including the networking layer.

Read More

AWS Cloud Technical Essentials

 

What you'll learn

Describe terminology and concepts related to AWS services     

Articulate key concepts of AWS security measures and AWS Identity and Access Management (IAM)    

You will learn to distinguish among several AWS compute services, including Amazon EC2, AWS Lambda, and Amazon ECS.  

Understand AWS database and storage offerings, including Amazon Relational Database Service (Amazon RDS), Amazon DynamoDB, and Amazon S3.

Join Free: AWS Cloud Technical Essentials

There are 4 modules in this course

Are you in a technical role and want to learn the fundamentals of AWS? Do you aspire to have a job or career as a cloud developer, architect, or in an operations role? If so, AWS Cloud Technical Essentials is an ideal way to start. This course was designed for those at the beginning of their cloud-learning journey - no prior knowledge of cloud computing or AWS products and services required!

Throughout the course, students will build highly available, scalable, and cost effective application step-by-step. Upon course completion, you will be able to make an informed decision about when and how to apply core AWS services for compute, storage, and database to different use cases. You’ll also learn about cloud security with a review of AWS' shared responsibility model and an introduction to AWS Identity and Access Management (IAM). And, you’ll know how AWS services can be used to monitor and optimize infrastructure in the cloud.

AWS Cloud Technical Essentials is a fundamental-level course and will build your competence, confidence, and credibility with practical cloud skills that help you innovate and advance your professional future. Enroll in AWS Cloud Technical Essentials and start learning the technical fundamentals of AWS today!

Note: This course was designed for learners with a technical background. If you are new to the cloud or come from a business background, we recommend completing AWS Cloud Practitioner Essentials (https://www.coursera.org/learn/aws-cloud-practitioner-essentials) before enrolling in this course.

Read More

Serverless Architectures on AWS

 

Build your subject-matter expertise

This course is part of the Developing Applications on AWS Specialization

When you enroll in this course, you'll also be enrolled in this Specialization.

Learn new concepts from industry experts

Gain a foundational understanding of a subject or tool

Develop job-relevant skills with hands-on projects

Earn a shareable career certificate

Join Free: Serverless Architectures on AWS

There are 2 modules in this course

A modern software engineer knows how to use the benefits of managed services from Amazon Web Services to reduce the coding needed to get a project across the line. There’s a lot of code you really don’t need to write when you can use a managed service for your applications. Less code means less tests, less bugs, and quicker delivery. 

In this course, we get hands on with automation tools and serverless managed services. Get your projects completed faster without needing to maintain the underlying servers hosting the managed services. Treat your infrastructure as code using AWS CloudFormation and AWS Serverless Application Model as an automated way to build the resources hosting your applications. We use AWS Amplify to rapidly add front-end hosting and AWS Cognito to add authentication to our application. With Cognito in place, we upgrade the application API to require authentication. Next, we learn to use AWS Step Functions to move a lot of the workflow coordination code out of your applications. Using serverless services, we contrast some options for building event driven architectures with Amazon SNS, Amazon SQS and Amazon EventBridge. Join our expert instructors as we dive deep on real-world use cases for each of the featured services in the course. 

This course will provide a combination of video-based lectures, demonstrations and hands-on lab exercises that will get you working with automation tools, Cognito authentication, Step Function workflows and event-driven architectures.

Read More

AWS Fundamentals Specialization

 

Advance your subject-matter expertise

Learn in-demand skills from university and industry experts

Master a subject or tool with hands-on projects

Develop a deep understanding of key concepts

Earn a career certificate from Amazon Web Services

Join Free: AWS Fundamentals Specialization

Specialization - 3 course series

This specialization gives current or aspiring IT professionals an overview of the features, benefits, and capabilities of Amazon Web Services (AWS). As you proceed through these four interconnected courses, you will gain a more vivid understanding of core AWS services, key AWS security concepts, strategies for migrating from on-premises to AWS, and basics of building serverless applications with AWS. Additionally, you will have opportunities to practice what you have learned by completing labs and exercises developed by AWS technical instructors.

Applied Learning Project

This specialization gives current or aspiring IT professionals an overview of the features, benefits, and capabilities of Amazon Web Services (AWS). As you proceed through these four interconnected courses, you will gain a more vivid understanding of core AWS services, key AWS security concepts, strategies for migrating from on-premises to AWS, and basics of building serverless applications with AWS. Additionally, you will have opportunities to practice what you have learned by completing labs and exercises developed by AWS technical instructors.

Read More

Introduction to Machine Learning on AWS

 

What you'll learn

Differentiate between artificial intelligence (AI), machine learning, and deep learning. 

Select the appropriate AWS machine learning service for a given use case.

Discover how to build, train, and deploy machine learning models.

Join Free: Introduction to Machine Learning on AWS

There are 2 modules in this course

In this course, we start with some services where the training model and raw inference is handled for you by Amazon. We'll cover services which do the heavy lifting of computer vision, data extraction and analysis, language processing, speech recognition, translation, ML model training and virtual agents. You'll think of your current solutions and see where you can improve these solutions using AI, ML or Deep Learning. All of these solutions can work with your current applications to make some improvements in your user experience or the business needs of your application.

Read More

DevOps on AWS: Code, Build, and Test

 

What you'll learn

Understand the DevOps philosophies and its lifecycle

Implement and manage continuous delivery systems and methodologies on AWS

How to use the right tools to measure code quality by identifying workflow steps

Join Free: DevOps on AWS: Code, Build, and Test

There are 2 modules in this course

DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity: evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes. This speed enables organizations to better serve their customers and compete more effectively in the market.

DevOps process can be visualized as an infinite loop, comprising these steps: plan, code, build, test, release, deploy, operate, monitor. Throughout each phase, teams collaborate and communicate to maintain alignment, velocity, and quality. This course in the DevOps on AWS specialization focuses on code, build and test parts of the workflow. We will discuss topics such as source control, best practices for Continuous Integration, and how to use the right tools to measure code quality, by identifying workflow steps that could be automated.

Read More

Hands-on Machine Learning with AWS and NVIDIA

 

There are 4 modules in this course

Machine learning (ML) projects can be complex, tedious, and time consuming. AWS and NVIDIA solve this challenge with fast, effective, and easy-to-use capabilities for your ML project.

Join Free: Hands-on Machine Learning with AWS and NVIDIA

This course is designed for ML practitioners, including data scientists and developers, who have a working knowledge of machine learning workflows. In this course, you will gain hands-on experience on building, training, and deploying scalable machine learning models with Amazon SageMaker and Amazon EC2 instances powered by NVIDIA GPUs. Amazon SageMaker helps data scientists and developers prepare, build, train, and deploy high-quality ML models quickly by bringing together a broad set of capabilities purpose-built for ML. Amazon EC2 instances powered by NVIDIA GPUs along with NVIDIA software offer high performance GPU-optimized instances in the cloud for efficient model training and cost effective model inference hosting.

In this course, you will first get an overview of Amazon SageMaker and NVIDIA GPUs. Then, you will get hands-on, by running a GPU powered Amazon SageMaker notebook instance. You will then learn how to prepare a dataset for model training, build a model, execute model training, and deploy and optimize the ML model. You will also learn, hands-on, how to apply this workflow for computer vision (CV) and natural language processing (NLP) use cases. After completing this course, you will be able to build, train, deploy, and optimize ML workflows with GPU acceleration in Amazon SageMaker and understand the key Amazon SageMaker services applicable to computer vision and NLP ML tasks.

Read More

Introduction to Designing Data Lakes on AWS

 

What you'll learn

Where to start with a Data Lake?

How to build a secure and scalable Data Lake?

What are the common components of a Data Lake?

Why do you need a Data Lake and what it's value?

Join Free: Introduction to Designing Data Lakes on AWS

There are 4 modules in this course

In this class, Introduction to Designing Data Lakes on AWS, we will help you understand how to create and operate a data lake in a secure and scalable way, without previous knowledge of data science! Starting with the "WHY" you may want a data lake, we will look at the Data-Lake value proposition, characteristics and components.

Designing a data lake is challenging because of the scale and growth of data. Developers need to understand best practices to avoid common mistakes that could be hard to rectify. In this course we will cover the foundations of what a Data Lake is, how to ingest and organize data into the Data Lake, and dive into the data processing that can be done to optimize performance and costs when consuming the data at scale. This course is for professionals (Architects, System Administrators and DevOps) who need to design and build an architecture for secure and scalable Data Lake components. Students will learn about the use cases for a Data Lake and, contrast that with a traditional infrastructure of servers and storage.

Read More

Getting Started with Data Analytics on AWS

 

What you'll learn

Explain different types of data analyses – descriptive, diagnostic, predictive, prescriptive

Understand how to perform descriptive data analytics in the cloud with typical data sets

How to build simple visualizations in AWS QuickSight to do descriptive analytics (using S3, Cloudtrail, Athena)

Join Free: Getting Started with Data Analytics on AWS

There is 1 module in this course

Learn how to go from raw data to meaningful insights using AWS with this one-week course. Throughout the course, you’ll learn about the fundamentals of Data Analytics from AWS experts.

Start off with an overview of different types of data analytics techniques - descriptive, diagnostic, predictive, and prescriptive before diving deeper into the descriptive data analytics. Then, apply your knowledge with a guided project that makes use of a simple, but powerful dataset available by default in every AWS account: the logs from AWS CloudTrail. The CloudTrail service enables governance, compliance, operational auditing, and risk auditing of your AWS account. Through the project you’ll also get an introduction to Amazon Athena and Amazon QuickSight. And, you’ll learn how to build a basic security dashboard as a simple but practical method of applying your newfound data analytics knowledge.

Read More

DevOps on AWS Specialization

 

What you'll learn

Implement DevOps culture and practices in the AWS Cloud

Adopt and enforce Continuous Integration and Continuous

Delivery best practices on AWS

Explore deployment strategies for serverless applications

Join Free: DevOps on AWS Specialization

Specialization - 4 course series

DevOps on AWS specialization teaches you how to use the combination of DevOps philosophies, practices and tools to develop, deploy, and maintain applications in the AWS Cloud. Benefits of adopting DevOps include: rapid delivery, reliability, scalability, security and improved collaboration.

The first course introduces you to essential AWS products, services, and common solutions. The course covers the fundamental concepts of compute, database, storage, networking, monitoring and security that learners and professionals will need to know when working with AWS.

The second course in the specialization discusses topics such as source control, best practices for Continuous Integration, and how to use the right tools to measure code quality, by identifying workflow steps that could be automated.

The third course explains how to improve the deployment process with DevOps methodology, and also some tools that might make deployments easier, such as Infrastructure as Code, or IaC, and AWS CodeDeploy.

Finally, the last course teaches how to use Amazon CloudWatch for monitoring, as well as Amazon EventBridge and AWS Config for continuous compliance. It also covers Amazon CloudTrail and a little bit of Machine Learning for Monitoring operations.

Applied Learning Project

AWS provides a set of flexible services designed to enable companies to more rapidly and reliably build and deliver products using AWS and DevOps practices. These services simplify provisioning and managing infrastructure, deploying application code, automating software release processes, and monitoring your application and infrastructure performance. This specialization has a significant hands-on component involving the AWS Free Tier in which you will explore AWS services and concepts using AWS SDKs, AWS APIs, and the AWS Console.

Read More