One of our clients faced significant challenges in managing their codebase and ensuring consistent, high-quality deployments. The primary issue arose from frequently merging code directly into the master/main branch, which led to a loss of stable code and jeopardized the reliability of software deployments. This practice hindered the ability to deliver a consistently high-quality product.

Client is one of the largest US-based fintech companies, providing innovative financial solutions to institutions, financial professionals, and individuals worldwide. They use continuous integration (CI) and continuous deployment (CD) practices. They use GitHub for version control and Jenkins for automating testing and deployment.

1. Code Stability: Direct merge into the master branch led to instability and unpredictable releases, resulting in a lack of confidence during deployment.
2. Lack of Regression Testing: As the repository grew and new test cases were added daily, manual monitoring of tests became impractical. Ensuring that previously created test cases still worked correctly as the application evolved was a major challenge.
3. Efficient Test Prioritization and Execution: With a growing test suite, determining the priority of test cases and automating regression testing for critical paths became crucial.
4. Pipeline Efficiency: Setting up pipelines for automated test execution in Jenkins required careful management to avoid bottlenecks and improve overall testing efficiency.

To address the challenges, we proposed the following solutions:

1. Git Branching Strategy: The key objective was to implement a streamlined branching strategy to ensure stability and improve deployment confidence.

• No Direct Merges into the Master Branch: The core principle of the new strategy was to prevent direct merges into the master/main branch. This would ensure that only stable and tested code makes it to production.
• Beta/Integration Branch: All updates should flow from a Beta/Integration branch, where rigorous testing takes place. Only once the code passes comprehensive testing should it be merged into the master branch for deployment.

Benefits:

• Isolation of Environments: Separating development and production code allowed the team to maintain an isolated environment for testing and integration, minimizing disruptions to production.
• Risk Management: By ensuring that code was thoroughly tested in the Beta branch before merging to production, the team could identify potential issues early and mitigate risks.
• Continuous Integration and Deployment (CI/CD): This strategy helped streamline CI/CD workflows by ensuring code stability and reducing the likelihood of broken builds.

2. Regression Testing Automation: Regression testing was essential to verify that changes didn’t inadvertently break existing features. With an expanding codebase, automating this process became crucial.

• Need for Automated Regression Testing: As the repository grew, manually tracking the status of test cases became too cumbersome. To ensure reliability, automated regression testing was implemented using Jenkins, an open-source tool already deployed in the organization.

Prioritization of Test Cases:

• P1 - Highest Priority: Critical test cases that must pass every time (e.g., happy paths).
• P2 - Business Validation: Test cases that validate core business functionality.
• P3 - Lower Priority: Edge cases and input validation (e.g., alphanumeric, numeric, null/empty checks).
• Automated Tests: Some tests were automated but not prioritized for immediate regression testing.
• WIP (Work in Progress): Test cases that were under development and hadn’t yet been fully automated.
• Manual Tests: Cases that couldn't be automated, such as those requiring specific UI interactions.
• Postproduction Verification (PPV): Regression testing also supported PPV, ensuring that any post-production changes did not break critical functionality.

3. Jenkins Pipeline Creation and Execution: Jenkins pipelines were set up for two main environments—beta and production—to automate the execution of regression tests.

Beta Pipeline:

• The first pipeline focused on the P1 test cases, ensuring critical functionalities were tested first.
• The second pipeline runs P2 and P3 tests, focusing on business logic.

Production Pipeline:

• The first pipeline focused on the P1 test cases, ensuring critical functionalities were tested first.
• The second pipeline runs P2 and P3 tests, focusing on business logic and lower priority test cases.

• Scheduling and Notifications: Pipelines were set to run on a weekly schedule to allow time for test fixes. In case of failure, automated notifications were sent to the QA team, prompting them to address issues.
• Ticket Creation on Failure: As one test case failed in the pipeline, it automatically created ticket for it on the Azure/JIRA board.

4. Bug Fixing and Repository Maintenance: Bugs identified during test execution were addressed and fixed in the beta branch first. After thorough testing, stable fixes were merged into the master branch. This ensured that only validated changes were pushed to production.

• Master Branch Updates: The master branch should only be updated from the beta branch once a release goes live. For bug fixes, the master could be updated from beta at any time, ensuring continuous stability.
• Best Practices: The team emphasized that no code should be merged directly into the master branch, reinforcing the stability of production code.

The implementation of the branching strategy and automated regression testing led to significant improvements in both deployment reliability and testing efficiency:

1. Improved Code Stability: The strategy of maintaining separate development and production branches reduced the risk of instability in the master branch, ensuring that only thoroughly tested code reached production.
2. Efficient Test Execution: Automated regression testing in Jenkins reduced the manual effort required for test execution and enabled faster identification of issues.
3. Clear Prioritization: Test case prioritization ensured that critical tests were executed first, optimizing both the testing process and resource allocation.
4. Faster Bug Fixing: The streamlined approach to bug fixing, using the beta branch for validation, helped the team address issues faster and with more confidence.

By adopting a structured Git branching strategy and automating regression testing via Jenkins pipelines, the client was able to achieve higher code stability, faster test execution, and more reliable deployments. This approach also enhanced the team's ability to manage a growing codebase and evolve the product with confidence, ensuring that new changes didn’t disrupt the stability of existing functionality.