Workflow Step: gemini → plan_architecture
Date: October 26, 2023
Prepared For: Customer
This document outlines a comprehensive architecture plan for a robust and scalable Error Handling System. The proposed system aims to provide a centralized, efficient, and proactive mechanism for capturing, processing, storing, notifying, and analyzing errors across various applications and services within the organization. By implementing this architecture, we will significantly improve system reliability, reduce mean time to resolution (MTTR) for incidents, and enhance overall operational visibility.
The core objectives are to ensure that all critical errors are captured, enriched with relevant context, persisted reliably, and trigger appropriate notifications, enabling rapid diagnosis and remediation. The architecture emphasizes modularity, extensibility, and the use of industry-standard tools and practices.
The primary objectives for the Error Handling System are:
graph TD
A[Applications/Services] --> B(Error Capture Layer);
B --> C(Message Queue / Event Bus);
C --> D[Error Processing & Enrichment Layer];
D --> E{Error Storage Layer};
E --> F[Monitoring & Reporting Layer];
D --> G[Notification & Alerting Layer];
F --> H[Dashboards / UI];
G --> I[Notification Channels];
H --> J[Developers/Operations];
I --> J;
G --> K[Ticketing Systems];
Description:
This layer is responsible for intercepting errors as close to their origin as possible.
* Application-level Exception Handlers: Global handlers (e.g., AppDomain.CurrentDomain.UnhandledException in .NET, process.on('uncaughtException') in Node.js, set_exception_handler() in PHP).
* Framework-specific Middleware: Interceptors in web frameworks (e.g., ASP.NET Core middleware, Express.js error handling middleware, Spring Boot @ControllerAdvice).
* Logging Framework Integration: Utilize standard logging libraries (e.g., SLF4J/Logback/Log4j2 for Java, Serilog/NLog for .NET, Winston/Bunyan for Node.js, Python's logging module) to capture and format error details.
* Aspect-Oriented Programming (AOP): For cross-cutting concerns like error logging without modifying core business logic.
* timestamp: UTC timestamp of the error occurrence.
* serviceName / applicationName: Identifier of the service/application.
* environment: (e.g., development, staging, production).
* hostname / instanceId: Specific machine or container where the error occurred.
* correlationId / traceId: Unique identifier for the request/transaction across services.
* errorType: (e.g., System.NullReferenceException, DatabaseConnectionError).
* errorMessage: The exception message.
* stackTrace: Full stack trace of the error.
* severity: (e.g., DEBUG, INFO, WARN, ERROR, CRITICAL).
* sourceFile / lineNumber: Location in code.
* userId / sessionId: Identifier of the user involved.
* requestUrl / httpMethod: For web requests.
* requestHeaders / requestBody: (Sanitized to remove sensitive info).
* customTags / metadata: Any additional relevant key-value pairs.
This acts as a buffer and transport layer for error messages.
* Decoupling: Separates error capture from error processing.
* Resilience: Prevents data loss if downstream processing components are temporarily unavailable.
* Scalability: Allows asynchronous processing and horizontal scaling of consumers.
* Load Balancing: Distributes error processing across multiple instances.
* Apache Kafka: High-throughput, fault-tolerant, durable messaging system suitable for large-scale event streaming.
* RabbitMQ: Robust and mature message broker, good for smaller to medium-sized deployments and complex routing.
* AWS SQS / Azure Service Bus / Google Cloud Pub/Sub: Managed cloud-native message queuing services, ideal for serverless or cloud-based architectures.
This layer consumes raw error messages, refines them, and prepares them for storage and alerting.
* Ingestion: Consumes error messages from the Message Queue.
* Normalization: Standardizes error data schema across different application types.
* Enrichment:
* Contextual Data Lookup: Retrieve additional data (e.g., user details from a user service, service topology information).
* Geo-location: Based on IP address (if captured).
* Code Version: Automatically add Git commit hash or build version.
* Deduplication: Identify and group identical errors within a configurable time window to prevent alert storms. Increment a counter for repeated errors.
* Filtering: Discard known transient or ignorable errors (e.g., specific HTTP 4xx errors, expected third-party service timeouts).
* Severity Adjustment: Dynamically adjust severity based on frequency or impact patterns.
* Correlation: Link related errors (e.g., multiple errors stemming from a single user request across microservices) using correlationId or traceId.
* Data Masking/Sanitization: Remove or mask sensitive information (PII, secrets) before storage.
This layer provides durable storage for all processed error data.
* Scalability: Must handle growing volumes of error data.
* Searchability: Efficient indexing and querying capabilities.
* Retention: Configurable data retention policies.
* Reliability: High availability and data durability.
* Elasticsearch: Highly recommended for its full-text search capabilities, scalability, and integration with Kibana for visualization. Ideal for time-series log data.
* MongoDB / PostgreSQL: Suitable for storing structured error records, especially if complex relationships or specific reporting queries are needed beyond simple log searching.
* Cloud Object Storage (S3, Azure Blob Storage): For archiving raw, unprocessed logs or very long-term, low-cost storage of processed data.
This layer is responsible for triggering alerts based on predefined rules.
* Rule Engine: Define rules based on error type, severity, frequency, service, environment, specific keywords, or custom metrics.
* Thresholds: Configure thresholds (e.g., "more than 5 critical errors in 1 minute," "error rate above 1%").
* Notification Channels:
* Email: For less urgent or summary reports.
* SMS/Voice Calls: For critical, immediate alerts (via services like Twilio, PagerDuty).
* Chat Platforms: Slack, Microsoft Teams, Google Chat integrations.
* Paging Systems: PagerDuty, Opsgenie, VictorOps for on-call rotation management and escalation.
* Webhooks: For integration with custom systems or other services.
* Escalation Policies: Define a sequence of notifications if an alert is not acknowledged within a certain timeframe.
* Suppression: Temporarily mute alerts for known issues or during maintenance windows.
* Dedicated Alerting Platforms: PagerDuty, Opsgenie, VictorOps (best for on-call management).
* Monitoring Tools with Alerting: Grafana Alerting (with Prometheus), Kibana Alerting (with Elasticsearch), Datadog, New Relic.
* Custom Microservice: For highly specific or complex alerting logic, integrating with various notification APIs.
This layer provides visibility into error trends, statistics, and individual error details.
* Dashboards: Customizable dashboards showing:
* Total error count over time.
* Error rate per service/application.
* Top N errors by type, message, or affected service.
* Errors by severity.
* Distribution of errors by environment.
* Mean Time To Acknowledge (MTTA) and Mean Time To Resolve (MTTR) metrics.
* Search & Filtering: Powerful interface to search, filter, and drill down into individual error occurrences.
* Reporting: Generate scheduled or ad-hoc reports on error trends, compliance, and system health.
*
This document outlines the comprehensive, detailed, and professional output for the "Error Handling System" as part of the "gemini → generate_code" step. This deliverable provides production-ready code, complete with explanations and best practices, designed to establish a robust and maintainable error handling framework for your application.
This deliverable provides the foundational code for a robust and centralized error handling system. Effective error handling is crucial for application stability, user experience, and ease of debugging. This system focuses on:
The proposed error handling system comprises the following key components:
APIError class, allowing for semantic categorization of errors (e.g., NotFoundError, BadRequestError).logging module to capture error details, including stack traces, for internal monitoring and debugging.The following code provides a production-ready implementation example using Python and Flask. Each component is separated into logical files for better organization and maintainability.
.
├── app.py
├── config.py
├── errors/
│ ├── __init__.py
│ └── exceptions.py
├── handlers/
│ ├── __init__.py
│ └── error_handlers.py
└── utils/
├── __init__.py
└── logging_config.py
config.py - ConfigurationThis file holds general application settings, including logging levels.
# config.py
import os
class Config:
"""Base configuration class."""
DEBUG = False
TESTING = False
LOG_LEVEL = os.environ.get('LOG_LEVEL', 'INFO').upper()
LOG_FILE_PATH = os.environ.get('LOG_FILE_PATH', 'app.log')
class DevelopmentConfig(Config):
"""Development specific configuration."""
DEBUG = True
LOG_LEVEL = os.environ.get('LOG_LEVEL', 'DEBUG').upper()
class ProductionConfig(Config):
"""Production specific configuration."""
LOG_LEVEL = os.environ.get('LOG_LEVEL', 'WARNING').upper()
# Map environment names to configuration classes
config_map = {
'development': DevelopmentConfig,
'production': ProductionConfig,
'default': DevelopmentConfig,
}
def get_config(env_name=None):
"""
Returns the appropriate configuration class based on the environment name.
Defaults to 'development' if no env_name is provided or recognized.
"""
if env_name is None:
env_name = os.environ.get('FLASK_ENV', 'default')
return config_map.get(env_name, config_map['default'])
errors/exceptions.py - Custom Exception ClassesThese custom exceptions allow for a more semantic way of raising and handling application-specific errors.
# errors/exceptions.py
from http import HTTPStatus
class APIError(Exception):
"""
Base class for custom API exceptions.
All custom API errors should inherit from this class.
"""
def __init__(self, message, status_code=HTTPStatus.INTERNAL_SERVER_ERROR, payload=None):
super().__init__(message)
self.message = message
self.status_code = status_code
self.payload = payload # Additional details for the error
def to_dict(self):
"""Converts the exception to a dictionary for JSON serialization."""
rv = {
"error": {
"code": self.status_code,
"name": self.__class__.__name__,
"message": self.message,
}
}
if self.payload:
rv["error"]["details"] = self.payload
return rv
class BadRequestError(APIError):
"""Raised when the client sends an invalid request."""
def __init__(self, message="Bad request.", payload=None):
super().__init__(message, HTTPStatus.BAD_REQUEST, payload)
class UnauthorizedError(APIError):
"""Raised when authentication is required but missing or invalid."""
def __init__(self, message="Authentication required or invalid credentials.", payload=None):
super().__init__(message, HTTPStatus.UNAUTHORIZED, payload)
class ForbiddenError(APIError):
"""Raised when the client does not have permission to access the resource."""
def __init__(self, message="You do not have permission to perform this action.", payload=None):
super().__init__(message, HTTPStatus.FORBIDDEN, payload)
class NotFoundError(APIError):
"""Raised when a requested resource is not found."""
def __init__(self, message="Resource not found.", payload=None):
super().__init__(message, HTTPStatus.NOT_FOUND, payload)
class ConflictError(APIError):
"""Raised when there's a conflict with the current state of the resource."""
def __init__(self, message="Resource conflict.", payload=None):
super().__init__(message, HTTPStatus.CONFLICT, payload)
class InternalServerError(APIError):
"""
Generic internal server error. Use this when a more specific error
is not applicable, or for unexpected server-side issues.
"""
def __init__(self, message="An unexpected error occurred on the server.", payload=None):
super().__init__(message, HTTPStatus.INTERNAL_SERVER_ERROR, payload)
# You can add more specific exceptions as needed, e.g.,
# class DatabaseError(InternalServerError):
# def __init__(self, message="Database operation failed.", payload=None):
# super().__init__(message, HTTPStatus.INTERNAL_SERVER_ERROR, payload)
utils/logging_config.py - Logging ConfigurationThis utility sets up the application's logger, ensuring consistent log formatting and output.
# utils/logging_config.py
import logging
from logging.handlers import RotatingFileHandler
import os
def configure_logging(app):
"""
Configures the application's logging system.
Logs to console and a rotating file.
"""
if not os.path.exists('logs'):
os.mkdir('logs')
# Set up basic logger
app.logger.setLevel(app.config['LOG_LEVEL'])
# Remove default handlers to prevent duplicate logs if re-initializing
for handler in list(app.logger.handlers):
app.logger.removeHandler(handler)
# Console handler
console_handler = logging.StreamHandler()
console_handler.setLevel(app.config['LOG_LEVEL'])
console_formatter = logging.Formatter(
'[%(asctime)s] %(levelname)s in %(module)s: %(message)s'
)
console_handler.setFormatter(console_formatter)
app.logger.addHandler(console_handler)
# File handler (rotating)
# Max 1MB per file, keep 5 backup files
file_handler = RotatingFileHandler(
os.path.join('logs', app.config['LOG_FILE_PATH']),
maxBytes=1024 * 1024,
backupCount=5
)
file_handler.setLevel(app.config['LOG_LEVEL'])
file_formatter = logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(pathname)s:%(lineno)d - %(message)s'
)
file_handler.setFormatter(file_formatter)
app.logger.addHandler(file_handler)
app.logger.info(f"Logging configured at level: {app.config['LOG_LEVEL']}")
handlers/error_handlers.py - Centralized Error HandlersThis Flask Blueprint registers error handlers for both custom exceptions and standard HTTP errors, ensuring all error responses are standardized.
# handlers/error_handlers.py
import logging
from flask import Blueprint, jsonify
from werkzeug.exceptions import HTTPException
from http import HTTPStatus
from errors.exceptions import APIError, InternalServerError
# Create a Blueprint for error handlers
error_bp = Blueprint('errors', __name__)
logger = logging.getLogger(__name__)
@error_bp.app_errorhandler(APIError)
def handle_api_error(error):
"""
Handles custom APIError exceptions.
Logs the error and returns a standardized JSON response.
"""
# Log the error with specific details for APIError
# For InternalServerError, log the full traceback
if isinstance(error, InternalServerError) or error.status_code >= HTTPStatus.INTERNAL_SERVER_ERROR:
logger.exception(f"API Internal Server Error: {error.message}")
else:
logger.warning(f"API Client Error: {error.status_code} - {error.message} (Payload: {error.payload})")
response = jsonify(error.to_dict())
response.status_code = error.status_code
return response
@error_bp.app_errorhandler(HTTPException)
def handle_http_exception(e):
"""
Handles standard Werkzeug HTTP exceptions (e.g., 404 Not Found, 405 Method Not Allowed).
These are typically raised by Flask itself.
"""
# For internal server errors (5xx), log the exception
if e.code >= HTTPStatus.INTERNAL_SERVER_ERROR:
logger.exception(f"HTTP Server Error: {e.code} - {e.description}")
else:
logger.warning(f"HTTP Client Error: {e.code} - {e.description}")
# Create a standardized error response
error_payload = {
"error": {
"code": e.code,
"name": e.name.replace(" ", ""), # e.g., "Not Found" -> "NotFound"
"message": e.description,
}
}
response = jsonify(error_payload)
response.status_code = e.code
return response
@error_bp.app_errorhandler(Exception)
def handle_unhandled_exception(e):
"""
Handles any unhandled exceptions that are not caught by other handlers.
This is the catch-all for unexpected server errors.
"""
logger.exception(f"Unhandled Exception: {e}") # Log full traceback
# Return a generic internal server error to the client
error = InternalServerError(message="An unexpected server error occurred. Please try again later.")
response = jsonify(error.to_dict())
response.status_code = error.status_code
return response
app.py - Main Flask ApplicationThis file initializes the Flask application, registers the error handlers, and defines example routes to demonstrate error handling.
# app.py
import os
import
Project: Error Handling System
Workflow Step: Review and Documentation (Step 3 of 3)
Date: October 26, 2023
Version: 1.0
This document outlines the detailed specifications and capabilities of the proposed Error Handling System. Designed to significantly enhance system reliability, improve operational efficiency, and provide a superior user experience, this system will empower your teams with proactive error detection, comprehensive contextual data, and streamlined resolution workflows. By transforming reactive firefighting into proactive problem-solving, the Error Handling System ensures critical issues are identified, prioritized, and resolved with maximum efficiency, minimizing downtime and business impact.
The Error Handling System is a robust, centralized solution designed to capture, process, analyze, and manage errors across your entire application ecosystem.
Key Objectives:
The Error Handling System is composed of several interconnected modules designed for scalability, flexibility, and comprehensive error management.
* Function: Captures errors at their point of origin within various application components (frontend, backend services, APIs, databases, infrastructure).
* Mechanism: Utilizes lightweight SDKs or agents integrated into application codebases (e.g., Java, Python, Node.js, React, Angular) and system-level log parsers.
* Key Feature: Minimal performance overhead on the application.
* Function: Standardizes the format of intercepted error data and enriches it with additional context.
* Enrichment Data Includes:
* Full stack trace and error message.
* User identification (anonymized where necessary).
* Request parameters (HTTP headers, body, URL).
* Session information.
* Application version and deployment environment.
* Relevant preceding log entries.
* Server/client-side system metrics at the time of error.
* Key Feature: Ensures consistency and provides a complete picture for debugging.
* Function: Securely stores all processed error data for historical analysis, auditing, and retrieval.
* Technology: Utilizes a scalable, high-performance database or log aggregation system (e.g., Elasticsearch, MongoDB, PostgreSQL).
* Key Feature: Long-term retention with efficient indexing for rapid querying.
* Function: Analyzes incoming errors against predefined rules to classify, de-duplicate, and prioritize them.
* Capabilities:
* Severity Assignment: Automatically tags errors as Critical, Major, Minor, or Warning based on keywords, frequency, or affected components.
* De-duplication: Groups identical or similar errors to prevent alert storms and provide consolidated insights.
* Trend Analysis: Identifies escalating error rates or new error types.
* Impact Analysis: Correlates errors with potential business impact metrics.
* Function: Triggers alerts to relevant stakeholders based on the output of the Rules Engine.
* Configurable Channels: Email, Slack, Microsoft Teams, PagerDuty, SMS, custom webhooks.
* Customizable Rules: Define thresholds (e.g., "5 critical errors in 1 minute," "new error type detected in production"), recipient groups, and escalation paths.
* Key Feature: Intelligent alerting to reduce noise and ensure critical issues receive immediate attention.
* Function: Provides a centralized, visual interface for monitoring, triaging, and managing errors.
* Features:
* Real-time error dashboards.
* Historical error trends and analytics.
* Error search and filtering capabilities.
* Detailed error views with all captured context.
* Error status management (New, Acknowledged, In Progress, Resolved, Ignored).
* User and team-based access controls.
* Function: Facilitates seamless communication and data exchange with external systems (e.g., incident management, issue tracking, CI/CD).
* API: Provides a robust API for programmatic access and integration.
The Error Handling System offers a comprehensive suite of features to ensure effective error management:
The following outlines the typical lifecycle of an error within the system:
* De-duplicates against existing errors.
* Assigns an initial severity level.
* Checks for predefined alert conditions.
* The relevant team acknowledges the error via the Dashboard or integrated tools.
* The error is reviewed, prioritized, and assigned to a specific developer or team.
* Initial investigation begins using the rich contextual data provided.
* A fix is developed, tested, and deployed.
* The system monitors for the recurrence of the resolved error.
* The error status is updated to "Resolved."
* For critical incidents, a post-mortem analysis is conducted.
* Learnings are documented to prevent future occurrences, and system improvements are identified.
* The error record serves as a historical reference.
The Error Handling System is designed for seamless integration with your existing technology stack:
* SDKs/Libraries: For frontend (React, Angular, Vue.js), backend (Java/Spring, Python/Django/Flask, Node.js/Express, .NET), and mobile (iOS, Android) applications.
* Log Aggregators: Elasticsearch, Splunk, Loki, Datadog (for correlating errors with broader log data).
* Metrics Platforms: Prometheus, Grafana (for correlating errors with system performance metrics).
* Platforms: PagerDuty, Opsgenie, VictorOps (for on-call rotations and incident escalation).
* Tools: Jira, Azure DevOps, GitHub Issues (for creating and linking error tickets to development workflows).
* Platforms: Slack, Microsoft Teams, Email (for real-time notifications and team discussions).
* Systems: SSO providers (Okta, Azure AD, Auth0) for secure user access and role-based permissions.
* Tools: Jenkins, GitLab CI, GitHub Actions (for integrating error monitoring into deployment gates).
The system provides powerful reporting and analytics capabilities through its Dashboard & Monitoring Interface:
A successful deployment of the Error Handling System requires a structured approach:
* Phase 1 (Pilot): Start with a critical but contained application or service to validate the system, fine-tune configurations, and gather feedback.
* Phase 2 (Expansion): Gradually roll out to additional services and environments, prioritizing based on business criticality and error volume.
* Phase 3 (Full Adoption): Integrate across the entire application landscape.
* Alerting Rules: Define and continuously refine alert thresholds, escalation policies, and notification channels.
* Severity Mapping: Establish clear criteria for assigning error severities.
* Data Retention: Determine appropriate data retention policies based on compliance and analytical needs.
* Provide comprehensive documentation for developers, operations, and SRE teams on system usage, SDK integration, and workflow best practices.
* Conduct training sessions to ensure all stakeholders are proficient in using the system for error detection, triage, and resolution.
* Ensure the underlying infrastructure for the Error Handling System is scalable to handle anticipated error volumes and data growth.
* Monitor the system's own performance to guarantee reliable error processing.
* Implement robust security measures for data at rest and in transit.
* Adhere to data privacy regulations (e.g., GDPR, CCPA) by anonymizing sensitive user data in error logs where necessary.
* Establish strict access controls based on roles and responsibilities.
* Clearly define ownership for the Error Handling System (e.g., SRE team, Platform team).
* Establish a governance model for managing configurations, integrations, and ongoing enhancements.
The Error Handling System is a foundational component for building and maintaining highly reliable, performant, and user-centric applications. Its comprehensive capabilities for real-time detection, rich contextual data, intelligent alerting, and streamlined workflows will significantly elevate your operational excellence.
**