GitOps deep dive

What is GitOps?

GitOps is a collection of best practices that are applied throughout the entire development workflow, from start to finish. These practices extend the use of Git, which is already used by developers for source code management, to include the management of an application’s configuration, infrastructure, and operational procedures. Any changes made to applications or infrastructure are recorded in a source control system and automatically synchronized with the live environment.

It’s important to understand that GitOps is not a replacement or an upgrade to DevOps. Rather, it is a specific set of deployment practices, while DevOps is a broader paradigm or mindset. The common principles between the two make it easier for teams to incorporate GitOps workflows into their existing DevOps techniques.

The 4 Commandments of GitOps

Here are the four things GitOps practitioners swear by. You absolutely need them to do GitOps right.

1. Declarative Everything By leveraging a declarative format, you can describe how your infrastructure works as well as the applications that are running on top of it. Doing this enables traceability of changes made to any environment and enables rollback, recoverability, and self-healing attributes using any source control system. Move away for imperative adhoc scripts to declarative configuration in all levels (app and infra).

2. V for Versioning Declarative descriptions stored within the repository supports immutability, versioning, and versioning history. For example, using Git for the declarations mentioned above, allows you a single place from which everything for your application is derived and driven. This allows you to easily pinpoint any changes made at any time. Don’t try to understand what is different between two environments by hand. Just look at all changes found in version control history, having the guarantee that the platform always matches what is described there.

3. Auto-Pull Using GitOps means that you employ software agents that are always running in the cluster, automatically pulling Git state at regular intervals and checking it against the live cluster state. This way you always know if the version in Git is the same as the live state or not.

4. Loop the Loop Operating in a close loop ensuring the desired state of a system matches the declared state. This is one of the most important features because it provides feedback allowing you and your team to better control your operations and workflow.

When used with your repository, GitOps software agents can perform various functions, ensuring self-healing. Agents automate fixes in case there’s an outage, perform QA processes for your workflow, and protect against human error or a manual intervention.

Getting Hooked on GitOps

It’s easy to get started with GitOps, because is a model developed on pre-existing practices:

Git workflows – GitOps uses Git repositories as the change mechanism for application and infrastructure updates. Any change to an application or environment is represented as a change in a Git repository – providing a full audit trail and the ability to roll back to a previous version at any time. CD pipelines – with GitOps you have a clear separation between CI and CD. Your CI pipelines stay as is and their end result is still an artifact and/or container that is stored in a binary repository. Once the artifact is created, GitOps takes over and applies it to the target cluster. Instead of triggering the CI/CD with a commit in the source code, with GitOps the trigger is a commit to the configuration repository. Infrastructure as Code (IaC) – GitOps uses a Git repository as the single source of truth for your environments. Infrastructure as code (IaC) means storing all infrastructure configuration as code in a version control system. IaC systems use declarative configuration, meaning you only state what you need deployed, not how. Tracking and observability – observability, a property of a system which allows it to be easily observed, is an important GitOps concept. It lets you ensure that the desired state described in declarative configuration, and actual current state, are the same. Immutable servers (containerized infrastructure) – GitOps is often used to operate Kubernetes application development. It lets you build development pipelines, continuously deploy applications, provision clusters, and deploy on Kubernetes or container registries. However, GitOps can also be used in other infrastructure environments. This means you can adopt GitOps today with your existing tools while still following the DevOps mindset.

Ouch...5 Things that can Make GitOps Hard

While GitOps has compelling benefits, it’s not always smooth sailing. Understanding the common challenges can help you make your journey smoother. Here are the main drawbacks of GitOps and what you can do about them.

1. Automating Git Commits May Create Conflicts

It is important to update applications in the CD automatically to accommodate frequent updates. The CI process must create pull requests when it requests to release an update to the testing environment. However, Git supports manual conflict resolution and editing, not automatic updates, so multiple CI processes can cause conflicts when writing to the same repository.

These conflicts don’t result from specific files but from the lack of synchronization between different processes cloning the same repo. You can mitigate this issue by using multiple repositories – for example, one for each namespace.

2. Too Many Git Repositories

The number of GitOps repositories typically increases with each new environment or application, increasing the repo management burden. Every repository must have the same access permissions and connect to a sync agent on various clusters. Provisioning these repositories can take up a large chunk of the development time.

To mitigate this issue, use fewer Git repositories (e.g., one per cluster). However, this approach means each repository must do more regarding pull-request and access control management.

3. Limited Visibility

GitOps provides visibility into everything within an environment, with all intended state data stored in Git in plain text. However, this only works with simpler setups (i.e., fewer repositories containing manageable numbers of config files). Enterprise environments often have many GitOps repositories and config files, making it impractical to comb through plain text files.

4. Doesn’t Centrally Manage Secrets

A complex enterprise environment requires a solution to manage secrets outside the standard CI/CD process. It is important to audit secrets (i.e., keys, passwords), so storing them in a secure, centralized data store makes sense. While GitOps won’t prevent you from managing secrets centrally, it doesn’t facilitate this either.

A Git repository is not suitable for storing secrets requiring encryption and decryption. Git history will remember the secrets forever.

5. Auditing May Be Insufficient

Git repositories are useful for auditing processes because they retain a full history of all state changes. However, it is harder to answer other questions with a Git repo, such as the number of times an application was deployed. These tasks require walking through the Git history or searching through text files.

GitOps vs. DevOps

DevOps (development + operations) is a collaborative approach to performing tasks by development and operations teams. While DevOps is a general philosophy of how teams should work (i.e., best practices applicable across an organization’s processes), GitOps is a way to effectively implement DevOps. It is a method that depends on a specific tool for source control (Git) and is more goal-oriented.

GitOps uses Git to verify and deploy infrastructure changes automatically. It handles operations with a declarative approach – DevOps involves prescriptive and declarative approaches. DevOps can apply to any infrastructure, not just containers.

DevOps focuses on operations and deployment, which is useful for organizations making slow application changes and handling frequent infrastructure issues and changes. It is suitable for software development and deployment and can help manage infrastructure as code (IaC).

GitOps is becoming a common workflow for managing containerized applications and cloud infrastructure because it helps implement CI/CD pipelines on Kubernetes. It uses Git (GitLab or GitHub) to redefine CI/CD and serve as an immutable, central state declaration.