• Encryption:

* Default Encryption: Enable Server-Side Encryption with S3-managed keys (SSE-S3) by default on the bucket. This encrypts all objects at rest.

* KMS Encryption (Optional): For higher security or compliance needs, use SSE-KMS with customer master keys.

• Version Control: Enable S3 Versioning to protect against accidental deletions or overwrites.
• Lifecycle Policies: Implement policies to transition objects to cheaper storage classes (e.g., S3 Intelligent-Tiering, Glacier) or expire them after a certain period, managing costs effectively.
• Access Logging: Enable S3 Access Logging to another bucket for auditing and security monitoring.
• Object Key Naming: Use a structured naming convention for S3 objects (e.g., uploads/{user_id}/{timestamp}_{original_filename}, or uploads/{unique_file_id}). This aids organization and can prevent naming conflicts.

4.5. Backend Logic (API Endpoints)

• Endpoint: POST /api/upload/initiate

* Request Body: { "filename": "example.png", "contentType": "image/png", "fileSize": 123456 }

* Validation:

* Authenticate and authorize the user.

* Validate contentType and fileSize against allowed configurations.

* Action: Generate an S3 pre-signed PUT URL.

* Response: { "uploadUrl": "https://your-s3-bucket.s3.aws.com/...", "fileId": "unique_id_for_this_file" }

• Endpoint: POST /api/upload/complete (Optional but recommended)

* Purpose: Notify the backend that the client has finished uploading the file to S3.

* Request Body: { "fileId": "unique_id_for_this_file", "s3Key": "uploads/...", "eTag": "file-etag-from-s3" }

* Action:

* Update the file's status in the database (e.g., from "pending" to "uploaded").

* Trigger post-processing (e.g., virus scan, thumbnail generation, metadata extraction).

* Response: { "status": "success", "message": "File upload finalized." }

4.6. Frontend Integration

• File Selection: HTML <input type="file"> element.
• Client-Side Validation: JavaScript to check file size, extension, and MIME type before initiating upload.
• Upload Progress: Display a progress bar using XMLHttpRequest onprogress event or a library (e.g., Axios).
• Error Handling: Provide clear feedback for validation errors, network issues, or S3 upload failures.
• Direct S3 Upload: Use fetch or XMLHttpRequest with the generated pre-signed URL to PUT the file directly to S3.

4.7. Security Considerations

• Authentication & Authorization: All API endpoints must be protected. Only authenticated and authorized users should be able to request upload URLs.
• Pre-signed URLs:

* Limited Lifetime: Set a short expiration time (e.g., 5-15 minutes) for pre-signed URLs.

Specific Operations: Ensure the URL only grants PUT access to a specific* object key.

* Conditional Headers: Include Content-Type and Content-Length in the pre-signed URL generation to ensure S3 validates these headers during upload.

• Server-Side Validation: As noted, this is paramount.
• Virus Scanning: Integrate with an anti-virus solution (e.g., ClamAV, AWS GuardDuty with custom lambda) to scan uploaded files, either synchronously before making them available or asynchronously.
• Content Security Policy (CSP): Implement a strong CSP on your frontend to mitigate XSS attacks and control what resources can be loaded.
• Input Sanitization: If file metadata is stored in a database, sanitize any user-provided inputs (e.g., original filename) to prevent injection attacks.
• Data Encryption: At rest (SSE-S3/KMS) and in transit (HTTPS/TLS).

4.8. Scalability & Reliability

• AWS S3: Inherently highly scalable and durable (11 nines of durability).
• Backend API: Use serverless functions (AWS Lambda) or auto-scaling groups (EC2/ECS) to handle varying load.
• Database: Choose a scalable database solution (e.g., DynamoDB, Aurora).
• CDN (Optional but Recommended for Downloads): For serving files, integrate with Amazon CloudFront to cache content closer to users, improving performance and reducing S3 costs for high-traffic files.

4.9. Error Handling & Logging

• Comprehensive Logging: Implement detailed logging for all API interactions, S3 operations, and validation failures. Use AWS CloudWatch Logs.
• Error Responses: Provide clear, descriptive error messages to the frontend without exposing sensitive backend details.
• Monitoring & Alerting: Set up CloudWatch alarms for critical errors, API latency spikes, or S3 operational issues.

5. Structured Data: Key Configuration Parameters

| Parameter | Type | Description | Default (Test) Value |

| :---------------------- | :------- | :------------------------------------------------------------------------------ | :------------------------------------------------------------ |

| S3_BUCKET_NAME | String | Name of the S3 bucket for file uploads. | pantherahive-collab-uploads-dev |

| S3_REGION | String | AWS region for the S3 bucket. | us-east-1 |

| ALLOWED_MIME_TYPES | Array | List of MIME types permitted for upload. | ["image/jpeg", "image/png", "application/pdf", "text/plain"] |

| MAX_FILE_SIZE_MB | Integer | Maximum allowed file size in megabytes. | 50 (MB) |

| PRESIGNED_URL_EXPIRY | Integer | Expiration time for S3 pre-signed URLs in seconds. | 300 (5 minutes) |

| FILE_KEY_PREFIX | String | S3 object key prefix for organization. | uploads/ |

| FILE_KEY_FORMAT | String | Pattern for S3 object keys (e.g., {prefix}{user_id}/{uuid}_{filename}). | {prefix}{user_id}/{uuid}-{filename} |

| DEFAULT_S3_ACL | String | Default Access Control List for uploaded objects. | private |

| ENABLE_S3_ENCRYPTION | Boolean | Whether to enable server-side encryption by default on S3 bucket (SSE-S3). | true |

| ENABLE_VERSIONING | Boolean | Whether to enable S3 object versioning for the bucket. | true |

| FRONTEND_ALLOWED_ORIGIN | String | Frontend domain allowed to make CORS requests to S3. | https://your-collab-frontend.com |

6. Actionable Recommendations for Next Steps (Step 2: Implement)

Based on this generated design, the following actions are recommended for the implementation phase:

1. AWS S3 Bucket Setup:

* Create the S3 bucket (pantherahive-collab-uploads-dev) in the specified region.

* Configure S3 Bucket Policy (if needed for specific access patterns).

* Set up CORS configuration for your frontend domain.

* Enable default encryption (SSE-S3) and Versioning.

* Configure lifecycle rules for cost optimization.

* Enable S3 Access Logging.

1. IAM Role Creation:

* Create an IAM role for your backend service with the least privilege permissions for S3 (s3:PutObject, s3:GetObject, s3:DeleteObject on the specific bucket).

1. Backend API Development:

* Develop the POST /api/upload/initiate endpoint to authenticate users, validate input, and generate pre-signed S3 PUT URLs.

* Implement robust server-side file type and size validation using the ALLOWED_MIME_TYPES and MAX_FILE_SIZE_MB configurations.

* Develop the POST /api/upload/complete endpoint (optional but recommended) to finalize file metadata in the database and trigger post-processing.

* Integrate a database (e.g., DynamoDB, PostgreSQL) to store file metadata.

1. Frontend Development:

* Implement the file selection UI using <input type="file">.

* Develop client-side validation logic for file types and size.

* Integrate with the backend initiate endpoint to get pre-signed URLs.

* Implement direct file upload to S3 using the pre-signed URL, including progress tracking and error handling.

* Implement robust error handling and user feedback mechanisms.

1. Security Audit & Testing:

* Conduct thorough security testing, including penetration testing, to ensure the system is resilient against common vulnerabilities (e.g., MIME type spoofing, broken access control).

* Test all validation layers (client-side, server-side, S3 conditional headers).

1. Monitoring & Logging:

* Set up comprehensive logging for all components.

* Configure monitoring and alerting for critical metrics and errors.

1. Post-Processing (Optional but recommended):

* Consider implementing S3 event notifications to trigger AWS Lambda functions for tasks like virus scanning, thumbnail generation, or metadata extraction after an upload is complete.

This design provides a solid foundation for building a secure and efficient file upload system within PantheraHive's Collab app.

Workflow Execution Summary: File Upload System

The "File Upload System" workflow (category: Development) has been successfully processed. This documentation outlines the design, implementation considerations, and specific recommendations for building a robust and scalable file upload system leveraging AWS S3, with a focus on defining allowed file types.

The system is designed to provide a secure and efficient mechanism for users to upload files, incorporating best practices for data integrity, security, and cost-effectiveness.

System Architecture Overview

The proposed File Upload System follows a typical client-server architecture, with Amazon S3 serving as the primary, highly available, and durable storage backend.

Key Components:

1. Client-Side (Frontend):

* User Interface (UI) for file selection and upload initiation.

* Client-side validation (initial file type, size checks).

* JavaScript for handling upload progress and direct interaction with AWS S3 (via pre-signed URLs) or the backend API.

1. Server-Side (Backend API):

* RESTful API endpoint(s) to handle file upload requests.

* Authentication and Authorization of upload requests.

* Generation of AWS S3 pre-signed URLs (recommended for direct client-to-S3 uploads).

* Server-side validation (critical for security).

* Processing of file metadata and storage in a database.

* Integration with AWS services (S3, IAM).

1. Storage (AWS S3):

* Primary storage for all uploaded files.

* Provides high durability, availability, scalability, and security features.

* Manages object storage, versioning, lifecycle policies, and access control.

Architectural Flow (Recommended Direct Upload to S3):

1. User Initiates Upload: User selects a file in the frontend.
2. Client Requests Pre-signed URL: Frontend sends a request to the Backend API, including file name, type, and size.
3. Backend Generates Pre-signed URL: Backend authenticates the request, validates basic parameters, and generates a temporary, time-limited S3 pre-signed URL for PUT operation.
4. Client Uploads Directly to S3: Frontend uses the pre-signed URL to directly upload the file to AWS S3.
5. Client Notifies Backend: Upon successful S3 upload, the frontend notifies the Backend API with the S3 object key and other relevant metadata.
6. Backend Processes Metadata: Backend records file metadata (e.g., S3 URL, original filename, uploader, timestamp) in its database.

Core System Features and Configuration

1. File Type Restrictions: "Test Allowed File Types"

The system incorporates robust mechanisms to restrict uploaded files to a predefined set of allowed types. This is crucial for security (preventing malicious executables) and maintaining data integrity.

• Client-Side Validation: Provides immediate feedback to the user, improving user experience. This check is purely for UX and must not be relied upon for security.
• Server-Side Validation: This is the mandatory and authoritative check.

* MIME Type Check: Validate the Content-Type header sent by the client.

* File Extension Check: Cross-reference the file extension with the allowed list.

* Magic Byte Inspection (Recommended): For critical security, inspect the initial bytes of the file content to verify its true file type, as MIME types and extensions can be easily spoofed.

Recommendation: Maintain a centralized whitelist of allowed file types on the backend. For the "Test Allowed File Types" input, this translates to defining specific MIME types and corresponding extensions.

Example Whitelist (Backend Configuration):

[
    { "extension": ".jpg",  "mime_type": "image/jpeg", "magic_bytes_hex": ["FFD8FF", "FFD8FFE0", "FFD8FFE1"] },
    { "extension": ".png",  "mime_type": "image/png",  "magic_bytes_hex": ["89504E470D0A1A0A"] },
    { "extension": ".gif",  "mime_type": "image/gif",  "magic_bytes_hex": ["474946383761", "474946383961"] },
    { "extension": ".pdf",  "mime_type": "application/pdf", "magic_bytes_hex": ["25504446"] },
    { "extension": ".txt",  "mime_type": "text/plain", "magic_bytes_hex": null } // No specific magic bytes for generic text
]

2. Storage Mechanism: AWS S3

AWS S3 is the chosen storage solution, offering significant advantages:

• Scalability: Automatically scales to handle any volume of data.
• Durability: Designed for 99.999999999% (11 nines) durability of objects over a given year.
• Availability: Offers high availability across multiple Availability Zones within a region.
• Security: Comprehensive security features including encryption at rest and in transit, access control, and audit logging.
• Cost-Effectiveness: Pay-as-you-go model with various storage classes to optimize costs based on access patterns.
• Integration: Seamless integration with other AWS services (Lambda, CloudFront, IAM, etc.).

Key S3 Configuration Aspects:

• Bucket Policy & IAM Roles: Strict access control to the S3 bucket using IAM roles for the backend service and, if necessary, bucket policies for specific cross-account or public access (use with extreme caution).
• CORS Configuration: Essential for enabling direct client-to-S3 uploads using pre-signed URLs from a web browser.
• Encryption: Server-Side Encryption (SSE-S3, SSE-KMS, or SSE-C) should be enabled by default for all objects at rest. HTTPS ensures encryption in transit.
• Versioning: Recommended to enable versioning on the S3 bucket to protect against accidental deletions or overwrites.
• Lifecycle Rules: Configure rules to transition objects to cheaper storage classes (e.g., S3 Glacier) or delete old versions after a certain period, optimizing costs.

3. Security & Access Control

• Least Privilege: IAM roles and policies should grant only the minimum necessary permissions to the backend service.
• Pre-signed URLs: Use short-lived pre-signed URLs to grant temporary, limited access to specific S3 objects or operations without exposing AWS credentials.
• Malware Scanning: Integrate with a malware scanning solution (e.g., ClamAV running on Lambda triggered by S3 events, or a third-party service) to scan uploaded files for threats before making them accessible.
• Input Sanitization: Sanitize all user-provided input, especially filenames and metadata, to prevent injection attacks.

4. Scalability & Reliability

• S3's Inherent Scalability: S3 handles storage scaling automatically.
• Backend Scaling: Design the backend API to be stateless and horizontally scalable (e.g., using AWS Auto Scaling Groups, ECS, EKS, or Lambda).
• Event-Driven Processing: Utilize S3 Event Notifications (e.g., s3:ObjectCreated:*) to trigger post-upload processing (e.g., image resizing, metadata extraction, malware scanning) via AWS Lambda, decoupling the upload process from post-processing.

Implementation Details and Recommendations

1. Frontend Integration

• HTML: Use <input type="file" id="fileInput" multiple accept=".jpg,.png,.pdf"> for file selection. The accept attribute provides initial client-side filtering.
• JavaScript:

* Handle change event on the file input.

* Perform client-side validation (file type, size).

* Display upload progress (e.g., using XMLHttpRequest.upload.onprogress for direct S3 uploads).

* Make AJAX calls to the backend to request pre-signed URLs.

* Use fetch or XMLHttpRequest to PUT the file directly to the S3 pre-signed URL.

* Notify the backend upon upload completion.

• UI/UX: Provide clear feedback to users (loading spinners, success/error messages, progress bars).

2. Backend API Design

• Endpoint for Pre-signed URL Request:

* POST /api/upload/presigned-url

* Request Body: { "fileName": "my-document.pdf", "fileType": "application/pdf", "fileSize": 123456 }

* Response: { "uploadUrl": "https://your-bucket.s3.amazonaws.com/...", "fileKey": "unique-id/my-document.pdf" }

* Logic: Authenticate user, validate file type/size against the whitelist, generate S3 pre-signed URL (e.g., using AWS SDK getSignedUrl or createPresignedPost), and return it.

• Endpoint for Upload Completion Notification:

* POST /api/upload/complete

* Request Body: { "fileKey": "unique-id/my-document.pdf", "etag": "s3-etag-value", "originalFileName": "my-document.pdf" }

* Logic: Store fileKey and other metadata in the application database, associate it with the user or relevant entity.

• Server-Side Validation: Absolutely critical. Re-validate file type, size, and potentially content (e.g., magic bytes) upon receiving the pre-signed URL request or post-upload notification.

3. AWS S3 Configuration

• Bucket Naming: Choose a globally unique and descriptive name (e.g., pantherahive-prod-fileuploads).
• Region: Select a region close to your users and backend services for lower latency.
• IAM Policy for Backend Service:

Grant s3:PutObject, s3:GetObject, s3:DeleteObject, s3:ListBucket on the specific bucket and its objects (arn:aws:s3:::your-bucket-name/).

* Grant s3:PutObjectAcl if you need to control object-level ACLs (generally prefer bucket policies).

* Grant s3:GeneratePresignedUrl (or equivalent for createPresignedPost if using that method) for the backend to generate signed URLs.

• CORS Configuration (Example): Add a CORS policy to your S3 bucket to allow PUT requests from your frontend domain.

    <CORSConfiguration>
      <CORSRule>
        <AllowedOrigin>https://your-frontend-domain.com</AllowedOrigin>
        <AllowedMethod>PUT</AllowedMethod>
        <AllowedHeader>*</AllowedHeader>
        <ExposeHeader>ETag</ExposeHeader>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
      </CORSRule>
    </CORSConfiguration>

• Encryption: Enable Server-Side Encryption with S3-managed keys (SSE-S3) as the default.
• Versioning: Enable for data recovery.
• Lifecycle Rules: Implement rules to manage storage costs. For example:

* Transition objects to S3 Standard-IA after 30 days.

* Transition to S3 Glacier Flexible Retrieval after 90 days.

* Delete previous versions after 365 days.

4. File Type Validation Strategy

• Client-side: Basic accept attribute and JavaScript checks (e.g., file.type, file.name.split('.').pop()).
• Server-side (Pre-signed URL generation):

* Validate Content-Type header passed by the client against the whitelist.

* Validate file extension.

• Post-upload (S3 Event Triggered Lambda):

* For advanced security, after the file lands in S3, trigger an AWS Lambda function via S3 Event Notifications.

* This Lambda can download a small portion of the file, perform magic byte inspection, and potentially integrate with a malware scanner. If the file is malicious or incorrect type, it can be quarantined or deleted.

5. Error Handling, Logging, and Monitoring

• Comprehensive Error Handling: Implement robust error handling on both client and server for network issues, validation failures, S3 errors, etc.
• Logging:

* Log all upload attempts (success/failure) with relevant details (user ID, filename, timestamp, IP address, S3 key, error messages) to a centralized logging service (e.g., AWS CloudWatch Logs, Splunk).

* Enable S3 server access logging for detailed requests to your bucket.

• Monitoring:

* Monitor S3 metrics (e.g., NumberOfObjects, BucketSizeBytes, AllRequests, 4xxErrors, 5xxErrors).

* Monitor backend API performance and error rates.

* Set up alarms for critical issues.

6. Security Best Practices

• Secure Pre-signed URLs: Generate them with the shortest possible expiry time (e.g., 5-15 minutes).
• Object Key Randomization: Use a UUID or a combination of user ID and timestamp to generate unique object keys in S3, preventing predictable URLs and collisions.
• Content Security Policy (CSP): Implement a strict CSP on your frontend to mitigate cross-site scripting (XSS) and other client-side attacks.
• Regular Security Audits: Periodically review IAM policies, S3 bucket policies, and application code for vulnerabilities.

7. Cost Management

• S3 Storage Classes: Utilize S3 Standard-IA, One Zone-IA, or Glacier storage classes for objects that are accessed less frequently or for archival purposes, using lifecycle policies.
• Data Transfer: Be mindful of data transfer costs out of AWS.
• Requests: Monitor the number of PUT, GET, and other S3 requests.
• CloudFront: For serving uploaded files, consider using Amazon CloudFront to cache content and reduce S3 data transfer costs for frequently accessed files, while also improving delivery performance.

Actionable Next Steps

To implement the "File Upload System" based on this documentation, proceed with the following steps:

1. Define Exact Allowed File Types: Finalize the precise list of allowed file types (MIME types, extensions, and optionally magic bytes) that your application will support.
2. AWS Infrastructure Setup:

* Create an S3 bucket in your desired AWS region.

* Configure S3 bucket properties: enable versioning, default encryption (SSE-S3), and CORS.

* Create an IAM role for your backend service with the necessary S3 permissions (as outlined in the "Structured Configuration Data" section).

* Consider setting up S3 Event Notifications to trigger Lambda functions for post-upload processing (e.g., malware scanning, image resizing).

1. Backend API Development:

* Implement the API endpoints for requesting pre-signed URLs and confirming upload completion.

* Integrate with the AWS SDK for S3 operations (e.g., getSignedUrl).

* Implement robust server-side validation for file types, sizes, and user authorization.

* Set up logging and error handling.

1. Frontend Development:

* Develop the user interface for file selection, drag-and-drop (optional), and progress display.

* Implement client-side validation and the logic for direct upload to S3 using the pre-signed URLs.

1. Security Integration:

* Integrate a malware scanning solution (e.g., a Lambda-based solution using ClamAV or a third-party service) that scans files immediately after upload to S3.

1. Monitoring and Alerting:

* Configure CloudWatch dashboards and alarms for S3 metrics and backend API health.

* Ensure comprehensive logging is in place for auditing and troubleshooting.

1. Testing: Conduct thorough unit, integration, and security testing of the entire upload workflow.

Structured Configuration Data (Examples)

1. Example AWS IAM Policy for Backend Service

This policy grants the necessary permissions for a backend service to interact with a specific S3 bucket for file uploads. Replace your-upload-bucket-name with your actual bucket name.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:PutObjectAcl",
                "s3:GetObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::your-upload-bucket-name",
                "arn:aws:s3:::your-upload-bucket-name/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::your-upload-bucket-name/*",
            "Condition": {
                "StringLike": {
                    "s3:x-amz-acl": [
                        "private",
                        "public-read",
                        "bucket-owner-full-control"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::your-upload-bucket-name/*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:DeleteObject",
            "Resource": "arn:aws:s3:::your-upload-bucket-name/*"
        }
    ]
}

2. Example S3 Bucket CORS Configuration

This configuration allows PUT requests from https://your-frontend-domain.com for direct browser uploads to S3.

<CORSConfiguration>
  <CORSRule>
    <AllowedOrigin>https://your-frontend-domain.com</AllowedOrigin>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedHeader>*</AllowedHeader>
    <ExposeHeader>ETag</ExposeHeader>
    <ExposeHeader>x-amz-request-id</ExposeHeader>
    <ExposeHeader>x-amz-id-2</ExposeHeader>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
  </CORSRule>
  <!-- Add other rules as needed for GET, POST, DELETE if your application requires them -->
</CORSConfiguration>

3. Recommended File Type Whitelist Structure

This JSON structure provides a robust way to define allowed file types, including extensions, MIME types, and optional magic byte patterns for deeper validation.

[
    {
        "name": "JPEG Image",
        "extension": ".jpg",
        "mime_type": "image/jpeg",
        "magic_bytes_hex": ["FFD8FF"],
        "max_size_mb": 10
    },
    {
        "name": "PNG Image",
        "extension": ".png",
        "mime_type": "image/png",
        "magic_bytes_hex": ["89504E470D0A1A0A"],
        "max_size_mb": 10
    },
    {
        "name": "GIF Image",
        "extension": ".gif",
        "mime_type": "image/gif",
        "magic_bytes_hex": ["474946383761", "474946383961"],
        "max_size_mb": 5
    },
    {
        "name": "PDF Document",
        "extension": ".pdf",
        "mime_type": "application/pdf",
        "magic_bytes_hex": ["25504446"],
        "max_size_mb": 20
    },
    {
        "name": "Microsoft Word Document",
        "extension": ".docx",
        "mime_type": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
        "magic_bytes_hex": ["504B0304"],
        "max_size_mb": 15
    },
    {
        "name": "Plain Text File",
        "extension": ".txt",
        "mime_type": "text/plain",
        "magic_bytes_hex": null,
        "max_size_mb": 2
    }
]