Orchestrating Complex Builds for the gCube Framework
Submitted By Jenkins User Manuele Simi
D4Science has created, maintains, and continually enhances an advanced test-bed that allows scientific communities to share knowledge and collaborate in a secure Virtual Research Environment. When they needed a cost-effective way to build and release their software framework (gCube), they chose Jenkins in order to support multi-project releases at scale.
Organization: D4Science in support of the Institute of Information Science and Technologies of National Research Council, (ISTI-CNR), Italy
Team: All members worked for ISTI-CNR including: Manuele Simi, Software Engineer; Roberto Cirillo, Release Manager; Andrea Dell’Amico, Availability and Capacity Manager; Pasquale Pagano, Technical Director
Programming Language: Java Platform: Linux/MacOS Version Control System: Git Build Tool: Maven
Project website: https://www.d4science.org/
Community Support: Relied on the Jenkins Users Google Group
Coordinating multi-project releases with a single pipeline.
Background: gCube, D4Science’s software framework, has been under active development since 2004. For many years, we have been using SVN as a version control system with a single repository hosting all source code and an ad-hoc tool to build the software. Once we migrated to Git and got hundreds of repositories to release, our build system could no longer meet the needs of gCube. It was clear that we had to transition to a new Continuous Integration and Delivery pipeline for this demanding scenario.
Goals: We had very clear expectations from this transition:
- shorten the integration time
- shorten the release time
- have incremental releases
- minimize the effort required at each release
Solution & Results: We created a single pipeline that:
- orchestrates the submission of hundreds of build jobs;
- creates dynamic stages according to their dependencies. Each stage submits build jobs for parallel executions to our agent nodes;
- switches among different modalities (snapshot, staging, release and dry-run);
- generates a build report that represents the fingerprint of the release.
Other pipelines support our pre- and post-release activities.
These Jenkins Declarative Pipelines give us full control of the orchestration activities of our complex releases. We were able to coordinate all the build jobs in a configurable and dynamic fashion. With Jenkins, D4Science now has the capabilities to:
- build and release software components from 200+ Git repositories within one day by launching a single pipeline job;
- reuse the same build jobs across different build stages;
- require zero effort to the development team at release time;
- get reliable, fast Continuous Integration feedback