To foster rapid product development and deliver superior offerings to market, Continuous Integration and Continuous Deployment (CI-CD) practices have gained remarkable traction within product teams. At Druva, we strive to swiftly, efficiently, and reproducibly transform code within repositories into deployable components through these CI-CD pipelines. CI-CD heavily relies on the foundational infrastructure machines responsible for executing resource-intensive operations.
In this article, we will delve into existing CI-CD challenges, and shed light on the need for a service capable of scaling existing infrastructure in real-time to accommodate dynamic job queues within CI-CD systems. Let us explore the barriers to CI-CD's optimization and discuss Druva’s path to transform in this space.
About CI-CD systems
As a result of virtualization technology, CI-CD infrastructure machines are predominantly comprised of virtual machines (VMs). These VMs may be situated in an on-premise data center, a public cloud, or a private cloud. Containerization technology has revolutionized the process of encapsulating dependencies and the build itself, bringing remarkable portability. This has helped build teams leverage these technological innovations to configure an array of VM infrastructure resources and author scripts, and seamlessly integrate them into CI-CD platforms to help automate workflows.
Existing issues with CI-CD systems
The seamless realization of the CI-CD system's full potential faces several obstacles. The following issues present significant hurdles within CI-CD systems.
Variable resource demands: Distinct job loads necessitate varying resource allocations. However, limited configured resources coupled with heavy workloads bring added difficulty with job queues and increased waiting periods for users.
Dilemmas with legacy code bases: Containerizing legacy product components poses considerable difficulties when components exhibit dependencies on the machine environment.
Cumbersome dependency management: The gradual accumulation of dependencies installed on machines over time brings increased challenges. This complicates the segregation and replication of these dependencies during runtime.
Weighty containerized Windows images: Containerized Microsoft Windows systems tend to possess substantial size, rendering them less versatile and operating system-agnostic compared to their Linux counterparts.
Arduous infrastructure scaling: Scaling infrastructure resources requires the manual cloning of machines and the configuration of settings — tedious and time-consuming processes.
Integration overhead: The addition of new machines to CI-CD systems necessitates meticulous integration procedures, such as configuring the GitLab-runner for GitLab CI-CD, and nodes for Jenkins.
Overcoming these challenges requires a service that can dynamically scale the infrastructure and accommodate the ebb and flow of workload demands within the CI-CD systems.
Our Autoscaler Service Parmanu
We built Parmanu — an autoscaler service inspired by a Hindi Superhero comic book. Parmanu serves as a dynamic backend service, empowering Druva to effortlessly scale its infrastructure resources in response to workload demands. The service holds YAML file-based configuration for the following entities:
Job emitter platform credentials
Infrastructure platform credentials
Job emitter probing frequency
Executor machine configurations
Minimum/maximum requirement for a job
Rules for auto-scaling