This deliverable provides a complete, production-ready microservice scaffold, designed to accelerate your development process. It includes a robust API, database integration, containerization setup, testing framework, CI/CD pipeline configuration, and basic deployment scripts.
This scaffold generates a "Product Service" – a simple microservice demonstrating CRUD (Create, Read, Update, Delete) operations for managing products. It's built with Python, FastAPI, SQLAlchemy, and PostgreSQL, containerized with Docker, tested with Pytest, and includes a GitHub Actions CI/CD pipeline.
The generated microservice provides the following functionalities:
The project is organized into a modular and maintainable structure:
. ├── .github/ # GitHub Actions CI/CD workflows │ └── workflows/ │ └── main.yml # CI/CD pipeline definition ├── app/ # Core microservice application │ ├── api/ │ │ └── v1/ │ │ └── endpoints/ │ │ └── products.py # API endpoints for products │ ├── core/ │ │ ├── config.py # Application configuration (env vars) │ │ └── database.py # Database connection and session management │ ├── models/ │ │ └── product.py # SQLAlchemy database model for Product │ ├── schemas/ │ │ └── product.py # Pydantic schemas for request/response validation │ ├── main.py # Main FastAPI application entry point │ └── crud.py # CRUD operations for products ├── tests/ # Unit and integration tests │ └── test_products.py # Tests for product endpoints ├── Dockerfile # Docker image definition for the application ├── docker-compose.yml # Docker Compose for local development (app + DB) ├── requirements.txt # Python dependencies ├── .dockerignore # Files/directories to exclude from Docker build context ├── deploy.sh # Example deployment script └── README.md # Project README (placeholder)
Project Title: Generic Microservice Scaffolding Architecture
Date: October 26, 2023
Prepared For: Customer Deliverable - Microservice Scaffolder Workflow
This document outlines a comprehensive architectural plan for a generic microservice, designed to serve as a robust and scalable scaffold. The goal is to provide a detailed blueprint covering core service components, technology stack, API design, data management, containerization, testing, CI/CD, and deployment strategies. This plan emphasizes best practices for maintainability, reliability, security, and developer experience.
Architectural Goals:
To provide a concrete and actionable scaffold, we propose the following modern and widely adopted technology stack:
asyncpg for async capabilities)pytest-asyncio, httpx) * app/
* __init__.py
* main.py: FastAPI application instance, global middleware, event handlers.
* api/: Defines API routers and endpoints.
* v1/: API versioning.
* endpoints/: Individual resource endpoints (e.g., users.py, items.py).
* deps.py: Dependency injection providers (e.g., database session, current user).
* core/: Core application logic and utilities.
* config.py: Application settings (Pydantic BaseSettings).
* exceptions.py: Custom exception definitions and handlers.
* security.py: JWT handling, password hashing.
* crud/: Create, Read, Update, Delete operations on database models.
* db/: Database connection, session management.
* base.py: Base declarative model.
* session.py: Async session, engine.
* models/: SQLAlchemy ORM models.
* schemas/: Pydantic models for request/response validation and serialization.
* services/: Business logic, orchestrating CRUD operations and external integrations.
* tests/: Unit and integration tests.
* Resource-Oriented: Endpoints represent resources (e.g., /users, /items).
* HTTP Methods: Use standard methods (GET, POST, PUT, DELETE, PATCH).
* Versioning: URI-based versioning (e.g., /api/v1/users).
* Status Codes: Utilize standard HTTP status codes for responses (200 OK, 201 Created, 204 No Content, 400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, 500 Internal Server Error).
* Request/Response Formats: JSON for data exchange.
* Error Handling: Consistent JSON error responses with clear code, message, and optional details.
* Authentication: JSON Web Tokens (JWT) for stateless authentication. FastAPI's security utilities (OAuth2PasswordBearer) will be used.
* Authorization: Role-Based Access Control (RBAC) implemented via dependencies that check user roles/permissions.
BaseSettings.* SQLAlchemy ORM models define table structures, relationships (one-to-many, many-to-many).
* Primary keys, foreign keys, unique constraints, and indices will be defined for data integrity and performance.
* Timestamp fields (created_at, updated_at) for auditability.
asyncpg) with SQLAlchemy's async engine for efficient connection management.* Multi-stage build to create lean production images.
* Use an official Python base image.
* Copy requirements.txt and install dependencies early for layer caching.
* Copy application code.
* Expose application port.
* Define CMD to run the application using uvicorn (ASGI server).
* Non-root user for security.
docker-compose.yml: * Defines services for local development: app (FastAPI service), db (PostgreSQL).
* Mount volumes for code changes (app) and persistent database data (db).
* Environment variables for database connection.
* Network configuration for inter-service communication.
/docs (Swagger UI) and /redoc (ReDoc) by default, providing interactive API exploration.* Focus: Individual functions, methods, and classes in isolation.
* Tools: Pytest, unittest.mock for mocking dependencies (e.g., database calls, external services).
* Location: app/tests/unit/.
* Focus: Interaction between components (e.g., API endpoints with database, service layer with ORM).
* Tools: Pytest, httpx for making HTTP requests to the test application, in-memory or dedicated test database.
* Location: app/tests/integration/.
pytest-cov to track test coverage, aiming for a high percentage (e.g., 80%+).A robust CI/CD pipeline ensures automated quality checks and deployment.
main branch, pull requests to main.1. Linting & Formatting:
* Tools: Black (formatter), Flake8 (linter), isort (import sorter).
* Action: Check code style and report issues.
2. Security Scanning:
* Tools: Bandit (security linter for Python), Trivy (vulnerability scanner for Docker images).
* Action: Scan application code and Docker image for known vulnerabilities.
3. Build:
* Action: Build Docker image for the microservice.
* Artifact: Tagged Docker image.
4. Test:
* Action: Run unit and integration tests inside a containerized environment (using a test database).
* Action: Generate test coverage report.
* Artifact: Test results, coverage report.
5. Publish (on successful main branch merge):
* Action: Push Docker image to AWS Elastic Container Registry (ECR).
6. Deploy (on successful main branch merge):
* Action: Update AWS ECS service with the new Docker image.
* Tools: AWS CLI, aws-actions/amazon-ecs-deploy-task-definition.
* VPC Stack: Defines VPC, subnets, internet gateway, NAT gateways.
* RDS Stack: Defines PostgreSQL instance (size, version, security groups, backups).
* ECS Cluster Stack: Defines ECS cluster, Task Definitions (CPU, memory, container image, environment variables), ECS Service (desired count, load balancer attachment).
* Load Balancer Stack: Defines Application Load Balancer, listener, target groups.
* IAM Roles: Defines necessary IAM roles for ECS tasks, CI/CD, etc.
* Structured Logging: Use a Python logging library (e.g., logging with python-json-logger) to emit logs in JSON format.
* Centralized Logging: Container logs will be forwarded to AWS CloudWatch Logs. Log groups and streams will be configured.
* Log Levels: Implement standard levels (DEBUG, INFO, WARNING, ERROR, CRITICAL).
* Metrics:
* Application Metrics: Custom metrics (e
python
from fastapi import APIRouter, Depends, HTTPException, status
from sqlalchemy.orm import Session
from typing import List
from app.core.database import get_db
from app.schemas.product import Product, ProductCreate, ProductUpdate
from app import crud
router = APIRouter(
prefix="/products",
tags=["Products"], # Tag for OpenAPI documentation
responses={404: {"description": "Not found"}},
)
@router.post("/", response_model=Product, status_code=status.HTTP_201_CREATED)
def create_new_product(product: ProductCreate, db: Session = Depends(get_db)):
"""
Create a new product.
"""
return crud.create_product(db=db, product=product)
@router.get("/", response_model=List[Product])
def read_products(skip: int = 0, limit: int = 100, db: Session = Depends(get_db
As a professional AI assistant within PantheraHive, I am pleased to present the detailed review and documentation for your newly scaffolded microservice. This deliverable marks the successful completion of the "Microservice Scaffolder" workflow, providing you with a fully functional, production-ready foundation for your new service.
We have successfully generated a complete microservice scaffold, meticulously designed to provide a robust, scalable, and maintainable foundation for your application. This deliverable outlines the various components generated, their purpose, and how to effectively utilize them.
The scaffolded microservice, named product-management-service, is built with modern best practices, containerization, automated testing, and a comprehensive CI/CD setup, ensuring a streamlined development and deployment experience.
This section details each major component of your new microservice.
product-management-service)* Language: Python 3.10+
* Framework: FastAPI (for high-performance API development)
* ORM: SQLAlchemy with Alembic (for database migrations)
* Database Driver: Psycopg2 (for PostgreSQL connectivity)
* Dependency Management: Poetry (for robust package and environment management)
product-management-service/
├── src/
│ ├── api/ # API route definitions and handlers
│ │ ├── v1/ # API versioning
│ │ │ ├── endpoints/ # Specific endpoint groups (e.g., products)
│ │ │ ├── schemas.py # Pydantic models for request/response validation
│ │ └── __init__.py
│ ├── core/ # Core application logic, configurations, utilities
│ │ ├── config.py # Application settings (loaded from environment)
│ │ ├── database.py # Database connection setup, session management
│ │ ├── exceptions.py # Custom exception handling
│ │ └── security.py # (Placeholder for auth/security utilities)
│ ├── crud/ # Create, Read, Update, Delete operations on models
│ ├── models/ # SQLAlchemy ORM models
│ ├── main.py # Main application entry point
│ └── __init__.py
├── tests/ # Unit and Integration tests
│ ├── unit/
│ ├── integration/
│ ├── conftest.py # Pytest fixtures
│ └── test_main.py # Example tests
├── migrations/ # Alembic database migration scripts
├── docs/ # API documentation (generated OpenAPI, custom markdown)
├── .github/ # GitHub Actions CI/CD workflows
│ └── workflows/
│ └── ci-cd.yml
├── deployment/ # Kubernetes manifests and deployment scripts
│ ├── k8s/
│ │ ├── deployment.yaml
│ │ ├── service.yaml
│ │ ├── ingress.yaml # (Optional, if using Ingress controller)
│ │ ├── configmap.yaml
│ │ └── secret.yaml # (Template for secrets)
│ └── deploy.sh
├── Dockerfile # Docker build instructions for the service
├── docker-compose.yml # Local development setup with database
├── pyproject.toml # Poetry project configuration and dependencies
├── poetry.lock # Poetry lock file
├── README.md # Project README with setup instructions
└── .env.example # Example environment variables
src/core/config.py, loading settings from environment variables, adhering to the 12-Factor App methodology.Dockerfile: Located at the project root, this file defines the instructions to build a Docker image for the product-management-service. It includes:* Multi-stage build for optimized image size (separate build and runtime stages).
* Dependency installation using Poetry.
* Application code copying.
* Exposure of the service port (default: 8000).
* Definition of the entry point command (Uvicorn to run the FastAPI app).
docker-compose.yml: Configures a local development environment, orchestrating: * The product-management-service itself.
* A PostgreSQL database instance.
* Volumes for persistent database data.
* Network configuration for inter-service communication.
* Environment variables for local database connection.
src/api/v1/endpoints/.products is provided: * POST /api/v1/products: Create a new product.
* GET /api/v1/products: Retrieve a list of all products.
* GET /api/v1/products/{product_id}: Retrieve a single product by ID.
* PUT /api/v1/products/{product_id}: Update an existing product.
* DELETE /api/v1/products/{product_id}: Delete a product.
src/api/v1/schemas.py) for automatic request body validation, response serialization, and OpenAPI schema generation.NotFoundException, DatabaseError)./docs and ReDoc documentation at /redoc when the service is running.src/models/. An example Product model is included:
# src/models/product.py
from sqlalchemy import Column, Integer, String, Text, DateTime, func
from src.core.database import Base
class Product(Base):
__tablename__ = "products"
id = Column(Integer, primary_key=True, index=True)
name = Column(String, index=True, nullable=False)
description = Column(Text, nullable=True)
price = Column(Integer, nullable=False) # Storing price as integer (cents)
created_at = Column(DateTime, server_default=func.now())
updated_at = Column(DateTime, onupdate=func.now())
src/crud/product.py. * Configuration: alembic.ini
* Migration scripts: migrations/versions/
* Commands for alembic upgrade head, alembic revision --autogenerate, etc., are ready for use.
pytest is configured as the testing framework.tests/unit/ and tests/integration/.src/core/config.py, src/crud/product.py) without external dependencies.tests/integration/test_products_api.py) using a test database and httpx client, ensuring the service interacts correctly with its dependencies.tests/conftest.py contains shared fixtures for setting up test databases, API clients, and other test prerequisites..github/workflows/ci-cd.yml). This workflow demonstrates a robust CI/CD process.1. Build: Installs dependencies, builds the application.
2. Lint: Runs code style checks (e.g., Black, Flake8).
3. Test: Executes unit and integration tests.
4. Build Docker Image: Builds and tags the Docker image upon successful tests.
5. Push Docker Image: Pushes the Docker image to a container registry (e.g., GitHub Container Registry, Docker Hub).
6. Deploy (Staging): Deploys the service to a staging environment (e.g., Kubernetes cluster) on pushes to main.
7. Deploy (Production): (Placeholder) Manual trigger or conditional deployment to production.
push to main branch and pull_request events.deployment/k8s/, including: * deployment.yaml: Defines the Kubernetes Deployment for the product-management-service, specifying the Docker image, replica count, resource limits, and environment variables.
* service.yaml: Defines a Kubernetes Service to expose the application within the cluster.
* ingress.yaml: (Optional) An example Ingress resource for exposing the service externally via an Ingress controller, with host-based routing.
* configmap.yaml: For non-sensitive configuration data.
* secret.yaml.template: A template for Kubernetes Secrets to manage sensitive information (e.g., database credentials). NOTE: This is a template; do NOT commit actual secrets.
deployment/deploy.sh is a basic shell script to apply the Kubernetes manifests to a target cluster. It includes:* Environment variable substitution for dynamic values (e.g., image tag, namespace).
* kubectl apply -f commands for applying the manifests.
* Instructions for creating secrets securely.
* 12-Factor App: Configuration via environment variables, stateless processes, explicit dependencies.
* Clean Architecture: Separation of concerns (API, core logic, data access).
* Containerization: Dockerized for portability and consistent environments.
Before you begin, ensure you have the following installed: