Jenkins is the way to save thousands of hours and millions of dollars

POET Pipeline Framework

Submitted By Jenkins User Ravi Sharma

 T-Mobile’s system reliability engineering unit took on the task of simplifying their CI/CD pipelines. In doing so, they’ve dramatically reduced developer downtime, enabling them to focus on developing and testing applications rather than operations. 

Organization: T-Mobile Inc USA | https://www.t-mobile.com

Programming Language: Groovy 

Platform: Docker/Kubernetes/AWS 

Version Control System: Bitbucket 

Build Tools: ANT, Maven, Gradle, NodeJ

Project website: https://github.com/tmobile/POET-pipeline-library

Community Support:  Relied on Jenkins.io websites & blogs and spoke with colleagues and peers

Dramatically reducing developer downtime by simplifying CI/CD pipelines.

Background: T-Mobile’s system reliability engineering (SRE) unit worked with a plugin-heavy, single-master Jenkins environment that hogged CPU cycles, caused all kinds of configuration headaches, and was constantly going down. On any given week, we had to restart Jenkins two or three times. Sometimes, our builds put such a strain on our environment that we had to restart it overnight and reset everything when our teams weren’t working. The team took on the task of simplifying their CI/CD pipelines. 

Goals: Our team’s goal was to build a continuous delivery pipeline to support different technologies and align them cohesively with a solid release plan, including iOS, Android, Server, H5, React, etc. The platform needed flexibility as well as high availability to meet our requirements. Our main goal was to dramatically reduce developer downtime.

"Jenkins has helped our developers be much more innovative as they are no longer needed to also be infrastructure engineers."
Ravi Sharma, Software Development Manager, T-Mobile

Solution & Results: We started by building a streamlined container-based pipeline infrastructure that is centrally managed and easily adaptable to development methodologies. We then reduced the number of Jenkins plugins we use in our master from 200 to four. Next, we moved from a single master powering all our Jenkins agents to multiple masters, and now have 30 pipeline engines powering roughly ten teams each. This setup has reduced CPU loads and other bottlenecks while allowing T-Mobile’s DevOps teams to continue enjoying the benefits of horizontal scaling. 

As a result, we can now spin up a Jenkins master from a Docker image in roughly two minutes, test it, and roll it out to our production environment. Individual teams can then customize their CI/CD pipelines to meet the needs of specific projects. This streamlined and centralized approach to deploying our pipeline has been successfully rolled out to 30+ teams. Phase-1 of POET implementation was a huge success and the plan is to scale stable, reliable CI/CD pipelines to our 350 developer teams and 5,000 active users.  

As developer teams take ownership of the pipelines, they no longer have to worry about the underlying technology. Simplified CI/CD pipelines have enabled us to:

  • free developer teams to focus on developing and testing applications rather than maintaining the Jenkins environment
  • reduce developer downtime
  • eliminate the need for a pipeline specialist on every development team
  • realize substantial labor and cost savings
  • redirect resources towards building revenue-generating products that better serve T-Mobile customers

Click here to read the full case study.

Like what you see? Share your Jenkins user story today.

Share on facebook
Facebook
Share on twitter
Twitter
Share on linkedin
LinkedIn

Jenkins® is a CD Foundation project and a registered trademark of Software in the Public Interest, Inc. Copyright Jenkins 2020