I'd come across this term quite often, but I wanted to understand how it actually works by building something myself. So, I decided to build my first CI/CD pipeline from scratch rather than just learning the concepts.

But before getting into the project, let's understand what CI/CD means.

CI/CD stands for Continuous Integration and Continuous Delivery/Deployment. In simple terms, it automates the process of testing, building, and delivering an application whenever its code changes.

Without CI/CD, we'd need to test our code, build the application, and deploy updates manually. This can become repetitive and time-consuming as projects grow.

Here's how the two parts work:

• Continuous Integration (CI): Automatically runs checks and tests when code changes are pushed or a pull request is opened.

• Continuous Delivery/Deployment (CD): Automates preparing an application for release or deploying it to a server.

The key difference between Continuous Delivery and Continuous Deployment is that Continuous Delivery may still require manual approval before release, while Continuous Deployment automatically releases changes that pass the required checks.

The idea is simple: automate repetitive steps and make the journey from code changes to a working application more reliable.

And that's exactly what I wanted to explore through this project: how different tools fit together to take an application from my local machine to a cloud server.

The Technologies Behind My Pipeline

Before getting into the implementation, let's look at the technologies I used and the role each plays in my pipeline.

• Node.js and Express.js: Used to build a simple task-management REST API.

• Docker: Packages the application and its dependencies into an image that can run on another machine.

• GitHub Actions: Automates testing, building and publishing the Docker image, and triggering deployments.

• GitHub Container Registry (GHCR): Stores the Docker images so EC2 can download them during deployment.

• AWS EC2: Provides the cloud server where my Docker container runs.

• AWS IAM and OIDC: Handle secure authentication and permissions between GitHub Actions and AWS without storing permanent AWS access keys in GitHub Secrets.

• AWS Systems Manager (SSM): Lets GitHub Actions send deployment commands to EC2 without relying on SSH.

Each tool has its own role, and the goal of this project was to connect them into one working pipeline.

Let's start with the application itself!

#1) Building the Express REST API

Before building the pipeline, I needed an application to deploy. Since my main goal was to learn CI/CD, I kept the application simple and built a task-management REST API using Node.js and Express.js.

The API supports basic CRUD operations: creating, fetching, updating, and deleting tasks. I also added input validation, a /health endpoint, and automated tests.

I skipped the database for now and stored tasks in memory, keeping the setup simple so I could focus on the deployment workflow.

With the API working locally and the tests passing, the next step was to package it using Docker so it could run on another machine without depending on my local setup.

#2) Dockerizing the Application

I wanted to package everything my application needed to run into a Docker image instead of relying on the Node.js setup and dependencies available on my laptop.

A Docker image is a packaged template, while a container is a running instance of that image.

To build the image, I created a Dockerfile containing the necessary instructions. I also added a .dockerignore file to exclude unnecessary files, such as local dependencies and Git history, from the build context.

Once the image was built, I ran a container and tested the API to make sure everything worked as expected before moving to the cloud.

With the application running successfully inside Docker, the next step was to automate the testing process using GitHub Actions.

#3) Automating Tests with GitHub Actions

With the application running in Docker, I moved on to automating its tests using GitHub Actions.

GitHub Actions lets us automate tasks through workflows that run in response to events such as code pushes and pull requests. I configured mine to run whenever code is pushed to main or a pull request targets that branch. The workflow sets up Node.js, installs dependencies, and runs my automated tests.

I also made the image-building job depend on the test job, so if the tests fail, the pipeline stops before publishing the Docker image or proceeding with deployment.

With testing automated, the next step was to publish the Docker image somewhere my EC2 server could access it.

#4) Publishing the Docker Image to GHCR

After automating the tests, I needed a place to store the Docker image so my EC2 server could download it during deployment. For this, I used GitHub Container Registry (GHCR), which lets us store and distribute container images.

I updated my GitHub Actions workflow to build and publish the image only after the tests passed and the code was pushed to main.

I used two image tags: latest and the Git commit SHA. While latest identifies the most recently published version, the commit SHA lets me identify and deploy the image built from a specific commit.

For authentication, I used GitHub Actions' built-in GITHUB_TOKEN with the required package permissions, avoiding the need for a separate token to publish images.

With this in place, successful pushes to main could automatically produce and publish tested Docker images to GHCR.

#5) Setting Up AWS EC2 for My Application

With my Docker image available on GHCR, I needed a server to run it. For this, I used Amazon EC2 (Elastic Compute Cloud), an AWS service that lets us rent virtual servers in the cloud.

Think of an EC2 instance as a computer running in AWS. I can choose an operating system, install software, and run applications on it without depending on my laptop being switched on.

I launched an EC2 instance running Ubuntu Server and installed Docker on it. This gave me a Linux server where I could pull my Docker image from GHCR and run the container.

Configuring the Security Group

I also needed to control which network connections could reach my server. That's where security groups come in.

A security group acts like a virtual firewall for an EC2 instance, using rules to control incoming and outgoing network traffic.

Initially, I configured these inbound rules:

• HTTP (port 80): Allows users to send HTTP requests to my API.

• SSH (port 22): Allowed me to connect to the server from my laptop for initial setup.

After setting up AWS Systems Manager Session Manager, I removed the inbound SSH rule, so I no longer needed to expose port 22 for routine server access.

With my cloud server ready, I still needed a way for GitHub Actions to trigger deployments whenever a new version of my application was ready. That's where IAM, OIDC, and AWS Systems Manager came in.

#6) Connecting GitHub Actions to AWS Securely

I needed a secure way for GitHub Actions to communicate with AWS and trigger deployments on my EC2 instance. To set this up, I used IAM and OpenID Connect (OIDC).

Understanding IAM and IAM Roles

IAM (Identity and Access Management) controls who can access AWS resources and what actions they can perform.

For example, GitHub Actions needed permission to request deployments on my EC2 instance, but it didn't need unrestricted access to my AWS account. I used IAM policies to define the actions it was allowed to perform.

An IAM role is an identity that a trusted entity can temporarily assume to obtain AWS permissions. A role has two important parts:

• Trust policy: Defines who is allowed to assume the role.

• Permission policy: Defines what actions the role can perform after assuming it.

Using OIDC Instead of Permanent AWS Access Keys

I could have stored permanent AWS access keys in GitHub repository secrets, but I wanted to avoid managing long-lived credentials for this pipeline. Instead, I used OpenID Connect (OIDC).

OIDC allows GitHub Actions to prove its identity to AWS using a verifiable identity token. AWS checks the token against the role's trust policy before deciding whether to allow the workflow to assume the role.

Here's how I configured it:

1. I configured AWS to trust GitHub's OIDC identity provider.

2. I created an IAM role whose trust policy restricts access to my specific repository and the main branch.

3. I attached a permission policy allowing the role to request deployment commands through AWS Systems Manager and check their execution results.

When an eligible workflow runs, GitHub Actions requests an identity token. If AWS validates it and the trust conditions match, the workflow receives temporary AWS credentials and can perform only the actions allowed by its role.

This way, I didn't need to store permanent AWS access keys in GitHub Secrets.

GitHub Actions wasn't the only component that needed AWS permissions, though. My EC2 instance also needed its own IAM role to communicate with Systems Manager.

#7) Using AWS Systems Manager to Deploy Without SSH

With GitHub Actions authenticated with AWS, I still needed a way to send commands to my EC2 instance. For this, I used AWS Systems Manager, which helps manage and operate servers remotely.

Its Run Command feature lets us execute commands on a configured EC2 instance. This allowed GitHub Actions to request the commands needed to deploy my application without opening an SSH connection.

What Does the SSM Agent Do?

The SSM Agent is software that runs on EC2 and communicates with AWS Systems Manager. It executes the commands sent through Run Command and reports their status and results.

To enable this communication, I attached a separate IAM role to my EC2 instance. This gave it the permissions needed to communicate with Systems Manager.

I now had two IAM roles with different responsibilities:

• GitHub Actions role: Allows the workflow to request deployment commands and check their results.

• EC2 instance role: Gives the instance the permissions needed to communicate with Systems Manager.

How GitHub Actions Reaches My EC2 Instance

Here's how the process works:

1. GitHub Actions authenticates with AWS through OIDC and assumes its IAM role.

2. The workflow asks Systems Manager to run deployment commands on EC2.

3. The SSM Agent executes the commands and reports their status through Systems Manager.

4. GitHub Actions checks the execution results to determine whether the commands succeeded.

With this setup, I could automate deployments without relying on SSH. Next, I connected these pieces to my GitHub Actions deployment job to pull and run the new Docker image.

#8) Automating the Deployment with GitHub Actions

With the AWS setup ready, I connected everything to my GitHub Actions workflow to automate deployments.

I added a separate deployment job that runs after the Docker image is successfully published to GHCR. I configured it to deploy only when changes are pushed to main, rather than on every pull request.

Authenticating and Triggering the Deployment

I used the aws-actions/configure-aws-credentials action to authenticate with AWS using the IAM role and OIDC configuration from the previous sections. I also added the id-token: write permission so GitHub Actions could request an OIDC identity token and obtain temporary AWS credentials.

Next, I used the AWS CLI's aws ssm send-command command with the AWS-RunShellScript document to send deployment instructions to EC2 through Systems Manager, without needing an SSH connection.

Giving EC2 Access to GHCR

My GHCR package was private, so my EC2 instance needed to authenticate before it could pull the Docker image. The GITHUB_TOKEN I used in GitHub Actions to publish the image wasn't automatically available on EC2.

To handle this, I created a GitHub Personal Access Token (PAT) with the read:packages permission and used it to authenticate Docker with GHCR on my EC2 instance.

Since I ran Docker commands as root, Docker stored the registry credentials in /root/.docker/config.json on the EC2 instance. This allowed Docker to use the saved credentials when pulling the image during deployment, without requiring me to provide the token with every pull command.

One important thing to keep in mind is that Docker's default configuration doesn't encrypt these credentials. So, access to this file should be restricted, and the token should never be included in the Docker image, deployment logs, or source code. For production setups, using a secure credential store or secrets manager is a better approach.

Updating the Running Container

The deployment commands sent to EC2 handle three main steps:

1. Pull the Docker image tagged with the current Git commit SHA from GHCR.

2. Stop and remove the existing application container.

3. Start a new container using the updated image, with the appropriate port mapping to make the API accessible.

Using the commit SHA lets me identify the exact version being deployed instead of relying only on the latest tag.

Finally, the workflow checks the command's execution status through Systems Manager. Successfully submitting a command doesn't guarantee that it completed successfully, so checking its result is an important part of the deployment process.

#9) Making Sure My Deployment Actually Works

At this point, GitHub Actions could trigger deployments on my EC2 instance. But there was still a problem: how could I be sure that the new version was actually working?

A Docker container starting successfully doesn't necessarily mean the application is working correctly. To verify each deployment, I added a health check using the /health endpoint in my Express API.

Checking Whether the Application Is Healthy

After starting the new container, the deployment process sends a request to /health and checks the response. If the API responds as expected, the deployment is considered successful. Otherwise, the workflow should report a failure.

What If the New Version Fails?

This is where rollback comes in. It means returning to a previously working version when a new deployment fails.

Since I tagged my Docker images with their Git commit SHA, I could identify the exact version to restore. If the new version fails its health check, the deployment logic attempts to stop the failed container and start the previous version again, while reporting the deployment as failed.

Rollback doesn't fix the underlying bug, but it helps restore the previous version while I investigate the issue.

With health checks and rollback, my pipeline could verify deployments and attempt to recover when something went wrong.

The Complete CI/CD Pipeline

We've covered GitHub Actions, Docker, GHCR, AWS EC2, IAM, OIDC, and Systems Manager. Here's how everything comes together when I push a change to my application.

From a Git Push to a Running Application

1. I push code to GitHub: Pushing changes to main triggers the workflow.

2. GitHub Actions runs the tests: If the tests fail, the pipeline stops before publishing the image or proceeding with deployment.

3. Docker builds and publishes the image: Once the tests pass, GitHub Actions builds the Docker image and pushes it to GHCR, tagging it with the Git commit SHA.

4. GitHub Actions authenticates with AWS: The deployment job uses OIDC to obtain temporary AWS credentials by assuming the IAM role I configured, without storing permanent AWS access keys in GitHub Secrets.

5. Systems Manager deploys the image to EC2: GitHub Actions sends deployment commands through Systems Manager, and the SSM Agent executes them on my EC2 instance. These commands pull the new image and replace the running container.

6. The pipeline verifies the deployment: The deployment process checks the application's /health endpoint. If the check fails, the rollback logic attempts to restore the previous version.

After the deployment succeeded, I verified that the application was accessible through my EC2 instance's public IP. The /health endpoint returned HTTP 200 OK with a {"status":"ok"} response, confirming that the API was reachable from outside AWS.

With that, my first end-to-end CI/CD pipeline was working! I no longer needed to build images manually, transfer them to a server, or SSH into EC2 for every deployment. The journey from a Git push to a running application was now automated.

What I Learned While Building My CI/CD Pipeline

My goal with this project wasn't to build a complex application, but to understand how code makes its way from GitHub to a running application on a server.

Building the pipeline helped me connect technologies I'd been learning separately, including GitHub Actions, Docker, AWS EC2, IAM, OIDC, and Systems Manager.

I learned that CI/CD isn't just about automating commands. It's also about managing permissions securely, checking whether deployments succeed, and recovering when something goes wrong.

There's still room to improve this setup, especially with zero-downtime deployments, HTTPS, and monitoring. But building the pipeline from scratch helped me understand how these pieces fit together in practice.

For me, the biggest takeaway wasn't just getting the application running on a server, but understanding what happens Under the Hood to get it there.

Let's say you create a file called app.py and save it in your Projects folder. You can open it, edit it, rename it, or delete it whenever you want. But how does Linux organize this file? How does it connect the files and directories you see to the actual storage inside your computer?

There's more going on than you might think!

Linux uses a few different concepts to manage storage and organize data. Once you understand how they work together, things like disks, partitions, filesystems, and mount points will make a lot more sense.

In this blog, we'll break these concepts down step by step, using simple explanations and diagrams to understand what's happening behind the scenes. The goal is to build a clear mental picture of how Linux storage is organized, from the physical disk all the way to the files you use every day.

We'll focus on understanding the core concepts here. In future blogs, we'll get more practical and explore these topics using Linux commands and hands-on examples.

Let's start with the basics!

#1) What Is Storage?

Storage is where your data is kept, even when you shut down your computer.

Your laptop might have an SSD or a hard disk drive (HDD) to store the operating system, applications, documents, projects, and other files.

For example, your laptop might have a 500 GB SSD:

The SSD is the physical storage device inside your computer. It provides the space where your data is kept.

But how does Linux identify and access this device?

# How does Linux represent storage devices?

Linux represents storage devices through device files under the /dev directory.

For example, an NVMe SSD might appear as:

/dev/nvme0n1

A SATA or SCSI-style disk might appear as:

/dev/sda

These names are how Linux refers to the corresponding storage devices. The exact names depend on the type of device and your system's configuration.

For now, you don't need to memorize these names. Just remember that /dev contains device files through which Linux can access hardware and other devices.

Now that we know what a disk is and how Linux identifies it, let's look at how a single disk can be divided into smaller sections called partitions.

#2) Disk vs Partition

Now that we know what a disk is, let's understand partitions.

Imagine your laptop has a 500 GB SSD. You could use the entire disk as one storage area, but you can also divide it into smaller sections called partitions.

For example, here's a 500 GB disk:

You could divide this disk into three partitions:

Each partition represents a defined region of the disk.

So, the relationship looks like this:

Think of a disk as a large storage space and partitions as separate sections within it.

# Why do we need partitions?

Partitions let us divide a disk into separate areas for different purposes.

For example, a Linux installation might use partitions like these:

• EFI System Partition: Stores boot-related files used by UEFI firmware to start the operating system.

• Root partition (/): Stores the Linux system, installed software, configuration files, and other data within the root filesystem.

• Home partition (/home): Can be used to store users' personal files and settings separately from the root filesystem.

• Swap partition: Can provide disk space for Linux swap.

These are common uses, not a list of partitions that every Linux system must have. For example, /home can simply be a directory inside the root filesystem instead of having its own partition. Swap can also be configured as a file rather than a partition.

One more thing worth remembering: a partition is not the same as a filesystem. A partition defines a region of storage, while a filesystem organizes data within that region. We'll look at filesystems next!

#3) What Is a Filesystem?

So far, we've learned that a disk can be divided into partitions. But here's a question: how does Linux organize data inside a partition?

Imagine you have a partition with 100 GB of storage. It provides space to store data, but we still need a way to organize that data into files and directories.

For example, how would Linux keep track of a file named project.py, its contents, who owns it, and which directory it belongs to?

That's where a filesystem comes in.

# What does a filesystem do?

A filesystem is a method of organizing and managing data on a storage device. It keeps track of things like:

• Files and directories: How files are organized and how they relate to each other.

• Metadata: Information about files, such as their size, timestamps, ownership, and permissions.

• Data locations: Where the contents of a file are stored within the filesystem.

Without a filesystem, Linux wouldn't have the familiar structure of files and directories that we use every day.

Think of a filesystem as the system that organizes storage and makes it possible to work with data as files and directories rather than having to manage raw storage blocks yourself.

# Common Linux filesystems

Linux supports several filesystem types, each with its own features. Here are some common examples:

• ext4: A widely used filesystem on Linux desktops and servers.

• XFS: A filesystem commonly used on servers, including systems that work with large files and large amounts of data.

• Btrfs: A filesystem that supports features such as snapshots, checksums, and compression.

• ZFS: A storage and filesystem system known for features such as data integrity checks, snapshots, and storage pooling.

• tmpfs: A temporary filesystem that stores data primarily in memory, rather than on a regular disk-based filesystem.

# How does a filesystem fit into our storage setup?

Let's return to our earlier example. Suppose you have a partition named /dev/nvme0n1p3.

You can create an ext4 filesystem on that partition. Once the filesystem is created, it can organize data into files and directories.

For example:

Creating a filesystem is commonly called formatting a partition. Be careful, though, creating a new filesystem can destroy data that was previously stored there.

One important thing to remember is that a filesystem doesn't have to be created on a partition specifically. It can also be created on other suitable storage devices, such as a logical volume. We'll keep our examples focused on partitions for now.

Now that we understand disks, partitions, and filesystems individually, let's put them together and see how they fit into the same storage hierarchy.

#4) Disk → Partition → Filesystem

Let's put everything together.

Imagine your SSD is divided into partitions, and one of those partitions has an ext4 filesystem.

The relationship looks like this:

Each layer has a different purpose:

• Disk: Provides physical storage.

• Partition: Defines a region of the disk.

• Filesystem: Organizes data into files and directories.

Remember, a partition is not the same as a filesystem. A partition defines a region of storage, while a filesystem organizes data within it. This is a common setup, though filesystems can also be created on other suitable storage devices.

Now, how do we access these files through the Linux directory structure? Let's understand mounting.

#5) What Is Mounting?

Unlike Windows, which commonly uses drive letters like C: and D:, Linux makes filesystems accessible through a single directory tree starting at / (the root directory).

Mounting means attaching a filesystem to a directory in this tree so you can access its files. This directory is called a mount point.

For example, suppose /dev/nvme0n1p3 contains an ext4 filesystem. You could mount it at /mnt/data.

A filesystem can be mounted at different locations, such as /mnt/data, /home, or /opt/storage. This is possible because Linux uses a single directory tree instead of assigning a separate drive letter to each filesystem.

One thing to remember: mounting a filesystem over a directory hides that directory's existing contents until the filesystem is unmounted. The files aren't deleted.

#6) Disk vs Partition vs Filesystem vs Mount Point

These four terms are closely related, but each has a different purpose.

One thing to remember is that this is a common storage setup, but not the only possibility. A filesystem can also be created directly on a whole disk or another suitable block device without creating a partition first.

#7) UUID and /etc/fstab

When you mount a filesystem manually using the mount command, it usually stays mounted until you unmount it or restart your computer. So, how does Linux know which filesystems to mount automatically when the system starts?

That's where /etc/fstab comes in.

# What is /etc/fstab?

/etc/fstab is a configuration file that tells Linux which filesystems to mount, where to mount them, and which options to use.

For example, an entry might look like this:

UUID=xxxx-xxxx  /mnt/data  ext4  defaults  0  2

Here's what the fields mean:

• UUID=xxxx-xxxx: Identifies the filesystem.

• /mnt/data: The mount point.

• ext4: The filesystem type.

• defaults: The mount options to use.

• 0 2: Fields related to the legacy dump utility and filesystem-check order.

An entry in /etc/fstab allows Linux to mount the filesystem automatically at boot, instead of requiring you to mount it manually each time.

# What is a UUID?

A UUID (Universally Unique Identifier) is an identifier assigned to a filesystem.

Instead of referring to a device by a name like /dev/sdb1, an /etc/fstab entry can use its UUID. This is useful because device names can change depending on how storage devices are detected.

In simple terms, a UUID identifies the filesystem, while /etc/fstab tells Linux where and how to mount it.

#8) The Complete Linux Storage Hierarchy

We've covered quite a few concepts, so let's put everything together and see how they fit into a typical Linux setup.

In this example, an ext4 filesystem is created on the partition /dev/nvme0n1p2 and mounted at /, the root of the Linux directory tree.

Directories such as /etc, /home, and /var are accessible through this tree. They belong to the root filesystem unless separate filesystems are mounted at those locations.

So, when you access /home/tanmay/Projects/app.py, you're accessing a file through the Linux directory tree, backed by the filesystem mounted at / in this example.

#9 Exploring storage on your Linux system

Now that we understand the concepts, let's look at a few commands that can help us explore storage on our own Linux systems.

• lsblk - Displays block devices in a tree-like structure, showing disks and their partitions.

• lsblk -f - Shows filesystem types, UUIDs, labels, and mount points.

• findmnt - Shows which filesystems are currently mounted and where they are mounted.

• df -hT - Shows filesystem types, total space, used space, and available space in a human-readable format.

• du -sh ~/Projects - Shows the disk space used by your Projects directory. Replace the path with a directory on your own system.

• cat /etc/fstab - Displays the configuration for filesystems and other storage resources that Linux can mount or activate, including entries configured for automatic mounting at boot.

You don't need to memorize all these commands right away. The goal is to understand what information each one provides and how it connects to the concepts we've covered.

# And That's the Big Picture!

Linux storage might seem complicated at first, but its basic concepts become much easier to understand when we break them down step by step.

A disk provides the physical storage, partitions divide it into regions, and filesystems organize data into files and directories. Mount points connect those filesystems to Linux's directory tree, while UUIDs and /etc/fstab help identify and configure filesystems for automatic mounting.

Now that we've covered these fundamentals, we have a solid starting point for exploring more advanced Linux storage topics.

In future blogs, we'll explore these concepts further with practical examples and Linux commands, including Logical Volume Management (LVM) and how it makes storage management more flexible.

This is just the beginning.

There's a lot more to explore under the hood of Linux!