This document provides comprehensive, detailed, and professional CI/CD pipeline configurations for GitHub Actions, GitLab CI, and Jenkins. These templates are designed to cover essential stages including linting, testing, building, and deployment, using a common application scenario (e.g., a Node.js application that is containerized and deployed).
Continuous Integration (CI) and Continuous Delivery/Deployment (CD) pipelines are fundamental to modern software development, enabling automated and reliable delivery of applications. This deliverable outlines the structure and specific configurations for implementing such pipelines across various popular platforms.
To provide the most relevant and actionable configurations, we've structured this output with a generic, yet robust, example. Please note that specific details (e.g., repository names, image registries, secret names, deployment targets) will need to be replaced with your project's unique information.
The following assumptions and prerequisites are made for the generated configurations:
* package.json (for Node.js example) or equivalent for dependencies and scripts.
* .eslintrc.js (for Node.js linting example) or equivalent configuration.
* jest.config.js (for Node.js testing example) or equivalent configuration.
* Dockerfile in the root directory for containerization.
* Kubernetes manifests (e.g., k8s/deployment.yaml, k8s/service.yaml) for deployment.
Each pipeline configuration will include the following standard stages:
* Purpose: Static code analysis to identify programmatic errors, bugs, stylistic issues, and suspicious constructs. Ensures code quality and adherence to coding standards.
* Actions: Installs necessary linters (e.g., ESLint, Pylint), runs linting checks against the codebase.
* Purpose: Executes automated tests (unit, integration) to verify the application's functionality and prevent regressions.
* Actions: Installs test runners (e.g., Jest, Pytest, JUnit), runs defined test suites, reports results.
* Purpose: Compiles source code (if applicable), resolves dependencies, and packages the application into a deployable artifact (e.g., a Docker image, JAR file, distributable package).
* Actions: Installs build tools, builds the application, creates a Docker image, tags it, and pushes it to a container registry.
* Purpose: Deploys the built artifact to a target environment (e.g., staging, production). This stage often includes environment-specific configurations and approvals.
* Actions: Authenticates with the deployment target (e.g., Kubernetes cluster, cloud provider), applies configuration files, updates services. Includes conditional deployments based on branch or manual approval.
Below are detailed configuration templates for GitHub Actions, GitLab CI, and Jenkins Declarative Pipelines. Each example assumes a Node.js application being built into a Docker image and deployed to Kubernetes.
.github/workflows/main.yml)GitHub Actions provides a flexible way to automate workflows directly within your GitHub repository.
# .github/workflows/main.yml
name: CI/CD Pipeline for YourApp
on:
push:
branches:
- main
- develop
pull_request:
branches:
- main
- develop
env:
# Define common environment variables for the workflow
APP_NAME: your-app-name
DOCKER_IMAGE_NAME: ghcr.io/${{ github.repository_owner }}/${{ env.APP_NAME }} # Example for GitHub Container Registry
jobs:
lint-test:
name: Lint and Test
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20' # Specify your Node.js version
- name: Install dependencies
run: npm ci # 'npm ci' for clean install in CI environments
- name: Run Lint
run: npm run lint # Assuming 'lint' script in package.json
- name: Run Tests
run: npm test # Assuming 'test' script in package.json
build-and-push-docker:
name: Build and Push Docker Image
runs-on: ubuntu-latest
needs: lint-test # This job depends on lint-test successfully completing
if: github.event_name == 'push' # Only run on push events, not PRs
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }} # GITHUB_TOKEN has permissions to push to GHCR
- name: Build and push Docker image
uses: docker/build-push-action@v5
with:
context: .
push: true
tags: |
${{ env.DOCKER_IMAGE_NAME }}:${{ github.sha }}
${{ env.DOCKER_IMAGE_NAME }}:${{ github.ref_name }} # e.g., your-app-name:main
cache-from: type=gha
cache-to: type=gha,mode=max
deploy-staging:
name: Deploy to Staging
runs-on: ubuntu-latest
needs: build-and-push-docker
if: github.event_name == 'push' && github.ref == 'refs/heads/develop' # Deploy 'develop' branch to staging
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Configure AWS Credentials (Example for AWS EKS)
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1 # Specify your AWS region
- name: Set up Kubeconfig for EKS
run: |
aws eks update-kubeconfig --name your-eks-cluster-name --region us-east-1
- name: Deploy to Staging Kubernetes
run: |
kubectl set image deployment/${{ env.APP_NAME }}-deployment ${{ env.APP_NAME }}=${{ env.DOCKER_IMAGE_NAME }}:${{ github.sha }} -n staging # Update image
kubectl apply -f k8s/staging/ # Apply any other staging manifests
kubectl rollout status deployment/${{ env.APP_NAME }}-deployment -n staging
deploy-production:
name: Deploy to Production
runs-on: ubuntu-latest
needs: build-and-push-docker
if: github.event_name == 'push' && github.ref == 'refs/heads/main' # Deploy 'main' branch to production
environment:
name: Production # Associate with a GitHub Environment for protection rules
url: https://your-prod-app.com # Optional: URL to deployed app
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Configure Google Cloud Credentials (Example for GKE)
uses: google-github-actions/auth@v2
with:
credentials_json: ${{ secrets.GCP_SA_KEY }} # Service account key JSON
- name: Set up GKE Kubeconfig
uses: google-github-actions/get-gke-credentials@v2
with:
cluster_name: your-gke-cluster-name
location: us-central1-c # Specify your GKE cluster location
project_id: your-gcp-project-id
- name: Deploy to Production Kubernetes
run: |
kubectl set image deployment/${{ env.APP_NAME }}-deployment ${{ env.APP_NAME }}=${{ env.DOCKER_IMAGE_NAME }}:${{ github.sha }} -n production # Update image
kubectl apply -f k8s/production/ # Apply any other production manifests
kubectl rollout status deployment/${{ env.APP_NAME }}-deployment -n production
This document outlines a comprehensive analysis of the infrastructure needs required to generate robust and efficient CI/CD pipelines using GitHub Actions, GitLab CI, or Jenkins. This initial step is crucial for laying the foundation for a successful DevOps strategy, ensuring that the chosen tools and configurations align with your organizational goals, technical stack, and operational requirements.
The objective of this analysis is to identify and assess the critical infrastructure components necessary to support a modern CI/CD pipeline. This includes considerations for source code management, build environments, artifact storage, deployment targets, security, and monitoring. Without specific application details, this analysis will cover common scenarios and best practices, providing a flexible framework that can be tailored once more specific project requirements are known.
A successful CI/CD pipeline relies on a well-thought-out infrastructure. Here are the primary areas of consideration:
* GitHub: Integrates seamlessly with GitHub Actions, offering strong community support and a vast marketplace of actions.
* GitLab: Provides an integrated SCM and CI/CD platform (GitLab CI), reducing context switching and simplifying setup.
* Other (e.g., Bitbucket, Azure DevOps Repos): While not explicitly requested for CI/CD, these are common SCMs that can integrate with Jenkins or other CI platforms.
* GitHub Actions: Cloud-native, event-driven, YAML-based configuration. Ideal for projects hosted on GitHub. Utilizes GitHub-hosted runners or self-hosted runners.
* GitLab CI: Fully integrated with GitLab, YAML-based configuration. Uses GitLab Shared Runners or self-managed GitLab Runners.
* Jenkins: Highly flexible, open-source, plugin-based. Can be hosted on-premise or in the cloud. Requires dedicated infrastructure for the Jenkins controller and agents.
* GitHub Actions/GitLab CI: Primarily rely on their respective cloud services for control plane. Runner infrastructure needs careful planning (compute, OS, network).
* Jenkins: Requires dedicated servers (VMs, containers) for the Jenkins controller and dynamically provisioned or static agents. This involves OS, networking, storage, and possibly load balancing.
* Type:
* Cloud-hosted (Managed): GitHub-hosted runners, GitLab Shared Runners. Convenient, zero infrastructure management, but potentially slower for large builds or specific software requirements.
* Self-hosted (Managed by you): GitHub Self-Hosted Runners, GitLab Specific Runners, Jenkins Agents. Offers full control over environment, software, network access, and potentially better performance/cost for high usage.
* Operating System: Linux (most common), Windows, macOS (for specific mobile/desktop builds).
* Compute Resources: CPU, RAM, disk space requirements depend on the application's build complexity and dependencies.
* Containerization: Docker is highly recommended for consistent, isolated, and reproducible build environments. This requires Docker daemon on the runner/agent.
* Network Access: Ability to reach SCM, artifact repositories, container registries, and external dependencies during build.
* Cloud Storage: AWS S3, Azure Blob Storage, Google Cloud Storage. Highly scalable, durable, and cost-effective.
* Package Managers/Registries:
* GitHub Packages: Integrated with GitHub, supports various package types.
* GitLab Package Registry: Integrated with GitLab, supports various package types.
* Artifactory (JFrog): Universal artifact repository, robust features for enterprises.
* Nexus Repository Manager: Open-source and commercial options, supports various formats.
* Cloud-Native: Amazon ECR, Azure Container Registry (ACR), Google Container Registry (GCR)/Artifact Registry.
* Integrated SCM: GitHub Container Registry, GitLab Container Registry.
* Public/Private: Docker Hub (for public images or private with subscriptions), Quay.io.
* Unit/Integration Tests: Typically run on the build agent itself, often containerized.
* End-to-End (E2E) Tests: May require dedicated temporary environments (e.g., ephemeral staging environments, Docker Compose setups, Kubernetes namespaces) that mirror production.
* Performance Tests: Dedicated test environments with specific load generation tools.
* Cloud Providers (IaaS/PaaS):
* AWS: EC2 instances, ECS/EKS (Kubernetes), Lambda (Serverless), Elastic Beanstalk, Fargate.
* Azure: Azure VMs, AKS (Kubernetes), Azure App Service, Azure Functions.
* GCP: Compute Engine, GKE (Kubernetes), Cloud Run, App Engine, Cloud Functions.
* On-Premise: Virtual Machines, bare-metal servers.
* PaaS/Serverless Platforms: Heroku, Vercel, Netlify.
* Compute: VMs, Kubernetes clusters, serverless functions.
* Networking: VPCs, subnets, load balancers, API Gateways, DNS.
* Databases: Managed databases (RDS, Azure SQL, Cloud SQL) or self-hosted.
* Storage: Block storage, object storage, file storage.
* Infrastructure as Code (IaC): Terraform, CloudFormation, Azure Resource Manager, Pulumi for provisioning and managing these resources.
* CI/CD Pipeline Metrics: Build duration, success/failure rates, deployment frequency.
* Application Metrics: CPU, memory, network, request rates, error rates.
* Logging: Centralized logging for pipeline runs and application logs (e.g., ELK stack, Splunk, Datadog, CloudWatch Logs, Azure Monitor Logs, GCP Cloud Logging).
* Alerting: Integration with notification systems (Slack, PagerDuty, email).
* CI/CD Platform Built-in Secrets: GitHub Secrets, GitLab CI/CD Variables (masked/protected), Jenkins Credentials Plugin.
* Dedicated Secret Managers: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager.
* Firewall Rules/Security Groups: Restrict access to only necessary ports and IPs.
* VPC/VNet: Isolate CI/CD infrastructure and deployment targets within private networks.
* VPN/Direct Connect: For connecting on-premise resources to cloud CI/CD.
* Container Security: Image scanning, vulnerability management (e.g., Clair, Trivy).
* Access Control (RBAC): Least privilege principle for all users and service accounts.
* Auditing: Comprehensive logging of actions within the CI/CD system.
Based on the general requirements for a DevOps Pipeline Generator, here are actionable recommendations:
* If your primary SCM is GitHub, GitHub Actions offers the most seamless and integrated experience.
* If your primary SCM is GitLab, GitLab CI provides a unified platform with minimal context switching.
* If you require extreme flexibility, complex workflows, extensive plugin ecosystem, or have a diverse SCM landscape (e.g., GitHub, Bitbucket, SVN), Jenkins remains a powerful, albeit more infrastructure-heavy, choice.
To move forward with generating the specific CI/CD pipeline configurations, the following information is required:
* Primary Technology Stack: (e.g., Node.js, Python, Java, .NET, Go, Ruby, PHP)
* Frameworks Used: (e.g., React, Angular, Spring Boot, Django, Flask, Express)
* Build Tool: (e.g., Maven, Gradle, npm, yarn, pip, Go Modules, dotnet build)
* Database Requirements: (e.g., PostgreSQL, MySQL, MongoDB, DynamoDB, SQL Server)
* GitHub Actions
* GitLab CI
* Jenkins
If Jenkins, specify desired hosting (cloud VM, on-prem, Kubernetes).*
* Cloud Provider: (e.g., AWS, Azure, GCP, On-premise, Hybrid)
* Deployment Model: (e.g., Virtual Machines, Kubernetes (EKS/AKS/GKE), Serverless (Lambda/Functions/Cloud Run), PaaS (App Service/Elastic Beanstalk))
* Environment Strategy: (e.g., Dev, Staging, Production; ephemeral environments for PRs)
* Types of tests to include: (e.g., Unit, Integration, E2E, Performance
Key Considerations for GitHub Actions:
As a professional AI assistant within PantheraHive, I am pleased to present the comprehensive validation and documentation for the DevOps Pipeline Configurations generated by our system. This deliverable ensures you have a clear understanding of the generated pipelines, how they are validated, and how to effectively implement and customize them within your development lifecycle.
This document outlines the validation process applied to the generated CI/CD pipeline configurations and provides detailed documentation for their usage, customization, and best practices.
Our DevOps Pipeline Generator creates robust, production-ready CI/CD configurations tailored to your chosen platform (GitHub Actions, GitLab CI, or Jenkins). These pipelines are designed to automate the software delivery process, incorporating essential stages from code commit to deployment.
For demonstration purposes, let's consider a common scenario: a Node.js application with a typical CI/CD flow. The generator produces configurations that consistently include the following core stages, adapted for the specific syntax and capabilities of each platform:
Below is an example of a generated GitHub Actions workflow for a Node.js application, illustrating the structure and stages that would be generated. Similar comprehensive configurations are generated for GitLab CI and Jenkins, adhering to their respective syntaxes and best practices.
# .github/workflows/main.yml
name: Node.js CI/CD
on:
push:
branches:
- main
- develop
pull_request:
branches:
- main
- develop
env:
NODE_VERSION: '18.x' # Or specific version like '18.17.1'
REGISTRY_GHCR_USERNAME: ${{ github.actor }}
REGISTRY_GHCR_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
AWS_REGION: us-east-1 # Example AWS region
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
DEV_SERVER_IP: ${{ secrets.DEV_SERVER_IP }} # Example for SSH deployment
jobs:
lint:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Run ESLint
run: npm run lint
test:
runs-on: ubuntu-latest
needs: lint # Ensure linting passes before testing
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Run tests
run: npm test # Assumes a 'test' script in package.json
- name: Upload test results
if: always()
uses: actions/upload-artifact@v4
with:
name: test-results
path: ./test-results.xml # Example path for Junit XML reporter
build_and_push_docker_image:
runs-on: ubuntu-latest
needs: test # Ensure tests pass before building
outputs:
image_tag: ${{ steps.meta.outputs.tags }}
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Log in to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ env.REGISTRY_GHCR_USERNAME }}
password: ${{ env.REGISTRY_GHCR_PASSWORD }}
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/${{ github.repository }}
tags: |
type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }}
type=sha
type=ref,event=pr
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=sha,format=long
- name: Build and push Docker image
uses: docker/build-push-action@v5
with:
context: .
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
deploy_to_dev:
runs-on: ubuntu-latest
needs: build_and_push_docker_image
environment: development
if: github.ref == 'refs/heads/develop' || github.ref == 'refs/heads/main' # Deploy to dev on pushes to develop/main
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Download artifacts (if needed, e.g., for non-Docker deployments)
uses: actions/download-artifact@v4
with:
name: build-artifact # Replace with actual artifact name
path: ./dist # Example target path
- name: Set up SSH
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}
- name: Deploy to Development Server via SSH
run: |
ssh -o StrictHostKeyChecking=no ${{ secrets.SSH_USER }}@${{ env.DEV_SERVER_IP }} "
cd /var/www/dev-app
git pull origin develop
npm install --production
pm2 restart dev-app
"
# For Docker-based deployments, you might use SSH to run docker pull/compose commands
- name: Deploy Docker Image to Dev (Example using AWS ECR/ECS)
if: false # Enable if using AWS ECR/ECS
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ env.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ env.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ env.AWS_REGION }}
- name: Deploy to AWS ECS (Example)
if: false # Enable if using AWS ECR/ECS
run: |
# AWS CLI commands to update ECS service with new image tag
# Example: aws ecs update-service --cluster my-cluster --service my-dev-service --force-new-deployment --task-definition my-task-def-family:$(aws ecs describe-task-definition --task-definition my-task-def-family --query 'taskDefinition.revision' --output text)
echo "Deploying image ${{ needs.build_and_push_docker_image.outputs.image_tag }} to Dev ECS"
deploy_to_staging:
runs-on: ubuntu-latest
needs: deploy_to_dev
environment: staging
if: github.ref == 'refs/heads/main' # Only deploy to staging from main
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up SSH
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY_STAGING }} # Use specific staging key
- name: Deploy to Staging Server via SSH
run: |
ssh -o StrictHostKeyChecking=no ${{ secrets.SSH_USER_STAGING }}@${{ secrets.STAGING_SERVER_IP }} "
cd /var/www/staging-app
git pull origin main
npm install --production
pm2 restart staging-app
"
- name: Deploy Docker Image to Staging (Example using AWS ECR/ECS)
if: false # Enable if using AWS ECR/ECS
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID_STAGING }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY_STAGING }}
aws-region: ${{ env.AWS_REGION }}
- name: Deploy to AWS ECS (Example)
if: false # Enable if using AWS ECR/ECS
run: |
echo "Deploying image ${{ needs.build_and_push_docker_image.outputs.image_tag }} to Staging ECS"
deploy_to_production:
runs-on: ubuntu-latest
needs: deploy_to_staging
environment: production
if: github.ref == 'refs/heads/main' && github.event_name == 'push' # Only deploy to prod from main on push
# For production, often a manual approval step is added, e.g., via GitHub Environments
# This example assumes automatic deployment for simplicity, but manual approval is highly recommended.
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up SSH
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY_PRODUCTION }}
- name: Deploy to Production Server via SSH
run: |
ssh -o StrictHostKeyChecking=no ${{ secrets.SSH_USER_PRODUCTION }}@${{ secrets.PRODUCTION_SERVER_IP }} "
cd /var/www/production-app
git pull origin main
npm install --production
pm2 restart production-app
"
- name: Deploy Docker Image to Production (Example using AWS ECR/ECS)
if: false # Enable if using AWS ECR/ECS
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID_PRODUCTION }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY_PRODUCTION }}
aws-region: ${{ env.AWS_REGION }}
- name: Deploy to AWS ECS (Example)
if: false # Enable if using AWS ECR/ECS
run: |
echo "Deploying image ${{ needs.build_and_push_docker_image.outputs.image_tag }} to Production ECS"
The generated pipeline configurations undergo a rigorous validation process to ensure they are syntactically correct, logically sound, and adhere to best practices.
* YAML Schema Compliance: The generated YAML files (for GitHub Actions and GitLab CI) are validated against their respective schemas to ensure correct structure, indentation, and key usage.
* Jenkinsfile Groovy Syntax: Jenkinsfiles are validated for Groovy syntax correctness.
* Stage Dependencies: Verification that needs (GitHub Actions), depends: (GitLab CI), or stage dependencies (Jenkins) are correctly defined to ensure stages execute in the intended order (e.g., tests run after linting, deployment after successful build).
* Conditional Execution: Checks for correct usage of if conditions (GitHub Actions), rules/only/except (GitLab CI), or when clauses