CircleCI - Enhance your CI/CD code with private orbs
This post describes how to create private Orbs using the Orb Development Kit and the benefit of utilising it in our codebase.
Challenges without using Orbs:
We created a small Mocha test repository with few tests and config.yml to run the tests on CircleCI to explain the orb creation process.The purpose of this repository is to run both sanity and functional tests as needed.
The config.yaml file is created without the use of an Orb. Let us find some of the existing problems.
- When closely looking into the job of sanity-dev and functional-dev the steps like “cache-repo, dependency-cache, Create artefacts directory, notify-google-chat” are repeatedly used. Since these are commonly used commands we may be using it in other projects as well
- Non Encapsulated way of writing config
Doesn’t it indicate an anti-pattern method of writing configuration? Yes, below are some of the violations
- Duplicate config code: Having duplicate code will make our code smelly and increases technical debt associated with it
- Large config file: As the pipeline evolves, the configuration file grows in size and becomes difficult to manage
How do we address the challenges listed above in CircleCI? This is where the Orbs come into picture.
About CircleCI orbs:
CircleCI Orbs are a parameterised and sharable YAML configuration (orb.yml) that allows developers to easily create and customise CircleCI configurations.Executors, jobs, and commands are examples of reusable configuration items.These orbs allow us to reduce the time it takes to create a configuration and easily integrate it across all projects.
Types of Orbs:
In circleCI, there are two ways to publish an Orb to its registry.
“Public Orbs” As the name implies, the configuration we publish in the registry is open to the public and anyone can use it in their project. CircleCI categorises them into three, namely Certified, Partner, and Community. We could have already used them in our config.yaml.
“Private Orbs” The Orb we publish will only be accessible within a Github organisation and not visible to the outside world.
We now know that both public and private Orbs are available, but which one should we choose with respect to the problem statement we stated above? It is not recommended to create public orbs because there may be various sensitive configurations that are particular to internal projects . Private Orbs can be a good option as it does not exposes the code to outside world
Types of Orb Authoring process :
There are two ways to create an orb with circleCI
“Manual Orb author” is a process of writing a reusable configuration as a single yaml file and publishing it to the registry using circleCI CLI commands.
circleci orb publish ./orb.yml <namespace>/<orb-name>@dev:firstorb
circleci orb publish promote <namespace>/<orb-name>@dev:first patch
“Orb Development kit author” is a process for creating a orb with a set of components that simplifies orb creation, with automated testing and deployment
Create a orb with orb development kit:
We might think why we choose the orb development kit over manually creating Orbs. The solution is straightforward. Continuous testing and deployment is used in today’s development process to deploy even a small piece of code or a single line of code. Similarly, we must ensure that any CICD config code we write or modify is not causing any breaking changes. ODK will help us in achieving this.
Another benefit of using the orb development kit is that, rather than writing all of the commands, jobs, or parameterised executors in a single yaml file (Manual Orb Author), we can break and write them into separate chunks, making it more readable and maintainable.
To do this, let’s first understand the list of components inside the kit
“Orb Template” is a repository project that provides a template for a complete CI/CD pipeline flow that packs, tests, and publishes our orb automatically.
“CircleCI CLI” is a command line tool that provides two commands related to the authoring process.
- circleci orb init: A CLI command to initialise a new orb project, and in the case of private, an extra — private attribute must be added
- circleci orb pack: A CLI command that combines all YAML files for the entire project and packages them as a single YAML file at run time
“Orb-Tools Orb” is an orb for creating orbs and jobs inside this will make publishing process more easier
Stages involved in creating Orb with ODK
The following are the three main stages involved in the creation of Orb.
Create or initialise orbs : Its the first stage where we create a repository and initiate orb creation process.
- Create a new repository in GitHub, for our use case we have created a repo with name “e2e-test-orbs”
- Create a directory in local machine from where we want to initialise the orb creation process. Its recommended to have a directory with same name which we used for creating repository
- Open a terminal from the path where we want to create orbs and invoke below command
circleci orb init . --private
When we run the above command, we’ll be asked to select our preferred way of project setup. “Yes, walk me through the process,” will download the Orb template based on inputs.
(Namespace) is a unique identifier claimed by a user or organisation to group a set of orbs by author. Each user or organisation can claim one unique and immutable namespace. Once a namespace is reserved cannot be modified.
4. Now we should see a template downloaded automatically based on
Test the Orbs: Its the second stage which help us to test the orb commands and job we created
We now know that Orbs are a critical component of our cicd pipeline, responsible for tasks such as code checkout, notification, and artefact creation, and others. Its important to create test and verify the behaviour and protect from any breaking changes
In the orb directory we have 2 config files
config.yaml : This file is responsible for publishing a dev version of Orb. At this stage in the pipeline, we can verify the orbs with below tests
- orb-tools/lint: It’s an orb job from the orb tool that lint the orb’s YAML and syntax
- orb-tools/pack: Packs and validates the orb source
- orb-tools/review: This will run a suite of tests against our orb designed to find opportunities to implement best practices and improve the quality of the orb
- shellcheck/check: Similar to YAML linting this will validate the shell scripts
- orb-tools/publish: Once the above Job are passed at this stage, publish will create a development version of a job
- orb-tools/continue: Once development version of orb is created, this job will invoke the next workflow which is available in test-deploy.yaml
Additionally, if the orb we’re building has more complex internal bash scripts, we may use the bats-core framework to develop unit tests for improved code quality.We don’t have any complex scripts in this demo project, so we can skip this step.
We can directly test our orb using integration testing in this config file by running an orb command and verifying it.
- command-tests: Its a step where we run the integration step and verify the behaviour by asserting it
- orb-tools/pack: As we saw in the introduction orbs are sharable config which is defined in orb.yaml. This job will Pack the orb source into a single “orb.yml” file and validates for orb config errors
- orb-tools/publish: This will now create a production version of a orb
Publish the Orbs: Its the third and final stage on orb creation process
So far, we’ve finished authoring an Orb and published a dev version; now it’s time to publish the production version in semantic format.
Open a pull request from the Orb repository’s main branch and squash merge it.The lint-pack and test-deploy workflows will be triggered against the main branch after it has been merged.
Once test is completed, from the main branch create a new release with tag added in semantic version(ex: v1.0.0). Again a new workflow with int-pack and test-deploy workflow will be triggered. Once tests are passed orb-tools/publish will publish the production version of Orb
Our orb is now ready to be consumed throughout our codebase, where it is required. We have created a new repository “simple-mocha-test-with-orbs” to find the difference on creating the config.
The above-mentioned issues are now handled with this orb way of creating a CICD code, such as duplication being removed, cicd code creation taking less time, being more encapsulated, and being more readable.
Finally, let us compare two repositories that contain CICD code with and without orbs.
Happy Orbing !!
— Kowcik R