POET Pipeline Framework
Submitted By Jenkins User Ravi Sharma
T-Mobile’s system reliability engineering unit took on the task of simplifying their CI/CD pipelines. In doing so, they’ve dramatically reduced developer downtime, enabling them to focus on developing and testing applications rather than operations.
Organization: T-Mobile Inc USA | https://www.t-mobile.com
Programming Language: Groovy
Version Control System: Bitbucket
Build Tools: ANT, Maven, Gradle, NodeJ
Project website: https://github.com/tmobile/POET-pipeline-library
Community Support: Relied on Jenkins.io websites & blogs and spoke with colleagues and peers
Dramatically reducing developer downtime by simplifying CI/CD pipelines.
Background: T-Mobile’s system reliability engineering (SRE) unit worked with a plugin-heavy, single-master Jenkins environment that hogged CPU cycles, caused all kinds of configuration headaches, and was constantly going down. On any given week, we had to restart Jenkins two or three times. Sometimes, our builds put such a strain on our environment that we had to restart it overnight and reset everything when our teams weren’t working. The team took on the task of simplifying their CI/CD pipelines.
Goals: Our team’s goal was to build a continuous delivery pipeline to support different technologies and align them cohesively with a solid release plan, including iOS, Android, Server, H5, React, etc. The platform needed flexibility as well as high availability to meet our requirements. Our main goal was to dramatically reduce developer downtime.
Solution & Results: We started by building a streamlined container-based pipeline infrastructure that is centrally managed and easily adaptable to development methodologies. We then reduced the number of Jenkins plugins we use in our master from 200 to four. Next, we moved from a single master powering all our Jenkins agents to multiple masters, and now have 30 pipeline engines powering roughly ten teams each. This setup has reduced CPU loads and other bottlenecks while allowing T-Mobile’s DevOps teams to continue enjoying the benefits of horizontal scaling.
As a result, we can now spin up a Jenkins master from a Docker image in roughly two minutes, test it, and roll it out to our production environment. Individual teams can then customize their CI/CD pipelines to meet the needs of specific projects. This streamlined and centralized approach to deploying our pipeline has been successfully rolled out to 30+ teams. Phase-1 of POET implementation was a huge success and the plan is to scale stable, reliable CI/CD pipelines to our 350 developer teams and 5,000 active users.
As developer teams take ownership of the pipelines, they no longer have to worry about the underlying technology. Simplified CI/CD pipelines have enabled us to:
- free developer teams to focus on developing and testing applications rather than maintaining the Jenkins environment
- reduce developer downtime
- eliminate the need for a pipeline specialist on every development team
- realize substantial labor and cost savings
- redirect resources towards building revenue-generating products that better serve T-Mobile customers
Like what you see? Share your Jenkins user story today.