HG Content Generation System - Deployment & Operations Guide¶

Last Updated: 2025-12-30

Table of Contents¶

Overview
Infrastructure
Service Deployment
Environment Variables
Database Configuration
Authentication Setup
Monitoring & Health Checks
Troubleshooting
Rollback Procedures

Overview¶

The HG Content Generation System is a distributed application deployed on Coolify:

Frontend: Next.js 14 application (hgcontent.com)
Backend Services: Python FastAPI services
Content Production Module (CPM) - cpm.hgcontent.com
Instructions Module (IM) - im.hgcontent.com
Strategy Management Module (SMM) - smm.hgcontent.com
Database: PostgreSQL on db-vps (via PgBouncer)
Cache/Queue: Redis container on apps-vps

Architecture Diagram¶

┌─────────────────────────────────────────────────────────────────┐
│                        Coolify (apps-vps)                       │
│                                                                 │
│  ┌─────────────────┐                                            │
│  │    Frontend     │  hgcontent.com                             │
│  │    (Next.js)    │  app.hgcontent.com → redirects to main     │
│  └────────┬────────┘                                            │
│           │                                                     │
│           ▼                                                     │
│  ┌─────────────────┬─────────────────┬─────────────────┐        │
│  │      CPM        │       IM        │      SMM        │        │
│  │   (FastAPI)     │   (FastAPI)     │   (FastAPI)     │        │
│  │ cpm.hgcontent   │ im.hgcontent    │ smm.hgcontent   │        │
│  │   :8000         │    :8001        │    :8003        │        │
│  └────────┬────────┴────────┬────────┴────────┬────────┘        │
│           │                 │                 │                 │
│           ▼                 ▼                 ▼                 │
│  ┌─────────────────────────────────────────────────────┐        │
│  │                  Redis (Queue)                      │        │
│  │             hg-content-redis:6379                   │        │
│  └─────────────────────────────────────────────────────┘        │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                        db-vps                                   │
│  ┌─────────────────────────────────────────────────────┐        │
│  │   PostgreSQL (port 5432) + PgBouncer (port 6543)    │        │
│  │                  hg_content database                │        │
│  └─────────────────────────────────────────────────────┘        │
└─────────────────────────────────────────────────────────────────┘

Infrastructure¶

Server Inventory¶

Server	Purpose	Connection
apps-vps	Application hosting (Coolify)	`ssh root@apps-vps`
db-vps	PostgreSQL + PgBouncer	`ssh root@db-vps`

Service Inventory¶

Service	Container Name	Port	Public URL
Frontend	hg-content-frontend	3000	hgcontent.com
CPM	hg-content-cpm	8000	cpm.hgcontent.com
IM	hg-content-im	8001	im.hgcontent.com
SMM	hg-content-smm	8003	smm.hgcontent.com
Redis	hg-content-redis	6379	Internal only

Service Deployment¶

Deployment Order¶

IMPORTANT: Deploy services in this order to ensure proper dependencies:

Redis (no dependencies)
Backend services: CPM, IM, SMM (depend on Redis + database)
Frontend (depends on all backend services)

1. Redis Container¶

Type: Docker Service (pre-built image)

Image: redis:7-alpine
Service Name: hg-content-redis
Port: 6379 (internal only)
Volume: /data → persistent storage

No environment variables required.

2. CPM Service (Content Production Module)¶

Dockerfile: apps/cpm/Dockerfile

Service Name: hg-content-cpm
Port: 8000
Public Domain: cpm.hgcontent.com
Health Check: /health

3. IM Service (Instructions Module)¶

Dockerfile: apps/im/Dockerfile

Service Name: hg-content-im
Port: 8001
Public Domain: im.hgcontent.com
Health Check: /health

4. SMM Service (Strategy Management Module)¶

Dockerfile: smm/Dockerfile

Service Name: hg-content-smm
Port: 8003
Public Domain: smm.hgcontent.com
Health Check: /health

5. Frontend Service¶

Dockerfile: apps/frontend/Dockerfile

Service Name: hg-content-frontend
Port: 3000
Public Domains: hgcontent.com, app.hgcontent.com
Health Check: /api/health

Note: app.hgcontent.com redirects to hgcontent.com via middleware to ensure session cookies work correctly.

Environment Variables¶

Frontend Environment¶

# Database
DATABASE_URL=postgresql://hg_content_app:PASSWORD@db-vps:6543/hg_content
DATABASE_DIRECT_URL=postgresql://hg_content_app:PASSWORD@db-vps:5432/hg_content

# Backend Services (internal URLs for server-side)
CPM_SERVICE_URL=http://hg-content-cpm:8000
IM_SERVICE_URL=http://hg-content-im:8001
SMM_SERVICE_URL=http://hg-content-smm:8003

# Backend Services (public URLs for client-side)
NEXT_PUBLIC_CPM_API_URL=https://cpm.hgcontent.com
NEXT_PUBLIC_IM_API_URL=https://im.hgcontent.com
NEXT_PUBLIC_SMM_API_URL=https://smm.hgcontent.com

# Authentication (NextAuth)
NEXTAUTH_URL=https://hgcontent.com
NEXTAUTH_SECRET=[secure-random-string]
AUTH_SECRET=[same-as-NEXTAUTH_SECRET]

# Node
NODE_ENV=production
PORT=3000

CPM Environment¶

# Database
DATABASE_URL=postgresql://hg_content_app:PASSWORD@db-vps:6543/hg_content

# LLM Providers (at least one required)
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=AIza...
OPENAI_API_KEY=sk-...
GROQ_API_KEY=gsk_...
OPENROUTER_API_KEY=sk-or-...

# Services
IM_SERVICE_URL=http://hg-content-im:8001
REDIS_URL=redis://hg-content-redis:6379

# Cost Management
DAILY_COST_LIMIT=10.00
MONTHLY_COST_LIMIT=200.00

# Application
ENVIRONMENT=production
LOG_LEVEL=info
PORT=8000

IM Environment¶

# Database
DATABASE_URL=postgresql://hg_content_app:PASSWORD@db-vps:6543/hg_content

# Redis
REDIS_URL=redis://hg-content-redis:6379

# Application
ENVIRONMENT=production
LOG_LEVEL=info
PORT=8001

SMM Environment¶

# Database
DATABASE_URL=postgresql://hg_content_app:PASSWORD@db-vps:6543/hg_content

# Redis
REDIS_URL=redis://hg-content-redis:6379

# Application
ENVIRONMENT=production
LOG_LEVEL=info
PORT=8003

Database Configuration¶

Connection Details¶

Purpose	Host	Port	Connection Type
Application queries	db-vps	6543	PgBouncer (pooled)
Migrations	db-vps	5432	Direct PostgreSQL

Connection String Format¶

# Pooled (for application use)
postgresql://hg_content_app:PASSWORD@db-vps:6543/hg_content

# Direct (for migrations)
postgresql://hg_content_app:PASSWORD@db-vps:5432/hg_content

Database Schema¶

The database uses the following key tables:

Table	Purpose
`auth.users`	User accounts (NextAuth compatible)
`clients`	Client/organization records
`client_users`	User-client associations
`jobs`	Content generation jobs
`strategies`	Client content strategies
`seo_rules`	SEO configuration
`prompt_templates`	Reusable prompt templates
`profiles`	User profile data

See docs/DATABASE_SCHEMA.md for complete schema documentation.

Authentication Setup¶

NextAuth Configuration¶

The frontend uses NextAuth 4.x with JWT strategy:

Credentials Provider: Email/password authentication
JWT Sessions: Stateless sessions with encrypted tokens
Database Integration: Custom adapter for auth.users table

Session Flow¶

1. User submits login form
2. NextAuth validates credentials against auth.users table
3. JWT token created with user ID in 'sub' claim
4. Token stored in HttpOnly cookie (__Secure-next-auth.session-token)
5. API routes verify JWT using getToken() from next-auth/jwt

API Route Authentication¶

API routes authenticate requests by:

// apps/frontend/lib/auth-helper.ts
import { getToken } from 'next-auth/jwt'

export async function getAuthUserId(request: NextRequest): Promise<string | null> {
  // Check x-user-id header (for inter-service calls)
  const userId = request?.headers.get('x-user-id')
  if (userId) return userId

  // Verify NextAuth JWT token
  const secret = process.env.NEXTAUTH_SECRET ?? process.env.AUTH_SECRET
  const token = await getToken({ req: request, secret })
  if (token?.sub) return token.sub

  return null
}

Inter-Service Authentication¶

Backend services authenticate requests via:

x-user-id header: Frontend passes authenticated user ID to backend services
API Keys: Client-specific API keys for external integrations

Monitoring & Health Checks¶

Health Check Endpoints¶

Service	Endpoint	Expected Response
Frontend	`https://hgcontent.com/api/health`	`{"status": "ok"}`
CPM	`https://cpm.hgcontent.com/health`	`{"status": "ok", "service": "cpm"}`
IM	`https://im.hgcontent.com/health`	`{"status": "ok", "service": "im"}`
SMM	`https://smm.hgcontent.com/health`	`{"status": "ok", "service": "smm"}`

Log Access¶

# View container logs
ssh root@apps-vps "docker logs hg-content-frontend --tail 100"
ssh root@apps-vps "docker logs hg-content-cpm --tail 100"
ssh root@apps-vps "docker logs hg-content-im --tail 100"
ssh root@apps-vps "docker logs hg-content-smm --tail 100"

# Follow logs in real-time
ssh root@apps-vps "docker logs -f hg-content-frontend"

Resource Monitoring¶

Access Coolify dashboard on apps-vps for: - CPU usage (target: <70%) - Memory usage (target: <80%) - Container status and restarts - Build logs

Database Monitoring¶

# Check active connections
ssh root@db-vps "sudo -u postgres psql -c 'SELECT count(*) FROM pg_stat_activity;'"

# Check database size
ssh root@db-vps "sudo -u postgres psql -c \"SELECT pg_size_pretty(pg_database_size('hg_content'));\""

Troubleshooting¶

Common Issues¶

1. Service Cannot Start¶

Symptoms: Container exits immediately or health checks fail

Diagnosis:

ssh root@apps-vps "docker logs hg-content-[service] --tail 200"

Common causes: - Missing environment variables - Database connection issues - Port conflicts

2. Database Connection Errors¶

Symptoms: "connection refused" or timeout errors

Diagnosis:

# Check PostgreSQL status
ssh root@db-vps "systemctl status postgresql"

# Check PgBouncer status
ssh root@db-vps "systemctl status pgbouncer"

# Test connection from apps-vps
ssh root@apps-vps "nc -zv db-vps 6543"

3. Authentication Failures (401 errors)¶

Symptoms: API routes return 401 after login

Diagnosis: 1. Check NEXTAUTH_SECRET is set correctly 2. Verify NEXTAUTH_URL matches the actual domain 3. Check browser cookies for session token 4. Review middleware logs

Common causes: - Mismatched NEXTAUTH_SECRET between builds - NEXTAUTH_URL mismatch with actual domain - Missing AUTH_SECRET fallback

4. Inter-Service Communication Failures¶

Symptoms: Frontend can't reach backend services

Diagnosis:

# Check service is running
ssh root@apps-vps "docker ps | grep hg-content"

# Test internal DNS
ssh root@apps-vps "docker exec hg-content-frontend curl http://hg-content-cpm:8000/health"

Common causes: - Services not on same Docker network - Incorrect service URLs in environment - Service not started

Debug Commands¶

# List all hg-content containers
ssh root@apps-vps "docker ps -a | grep hg-content"

# Check container resource usage
ssh root@apps-vps "docker stats --no-stream"

# Inspect container configuration
ssh root@apps-vps "docker inspect hg-content-frontend"

# Check Docker network
ssh root@apps-vps "docker network ls"
ssh root@apps-vps "docker network inspect [network-name]"

Rollback Procedures¶

Quick Rollback¶

If issues arise after deployment:

Revert in Coolify: Roll back to previous deployment in Coolify dashboard

Or redeploy previous commit:

git checkout [previous-commit]
git push origin main --force

Database Rollback¶

If database changes need to be reverted:

# Connect to db-vps
ssh root@db-vps

# Restore from backup (if available)
pg_restore -d hg_content /backups/hg_content_[date].backup

Full Recovery Steps¶

Stop affected services in Coolify
Restore database from backup (if needed)
Redeploy from known-good commit
Verify health checks pass
Test authentication flow
Test content generation

Deployment Checklist¶

Pre-Deployment¶

Database migrations applied (if any)
Environment variables updated in Coolify
Health check endpoints tested locally
Authentication flow verified

Deployment Steps¶

Push changes to main branch
Wait for Coolify auto-build
Monitor build logs for errors
Verify container starts successfully
Check health endpoint responds
Test user authentication
Test content generation

Post-Deployment¶

Monitor logs for errors (15+ minutes)
Verify API routes return 200
Test inter-service communication
Check database queries work

Quick Reference¶

SSH Access¶

ssh root@apps-vps  # Application server
ssh root@db-vps    # Database server

Health Check URLs¶

curl https://hgcontent.com/api/health
curl https://cpm.hgcontent.com/health
curl https://im.hgcontent.com/health
curl https://smm.hgcontent.com/health

Container Logs¶

ssh root@apps-vps "docker logs hg-content-frontend --tail 100"
ssh root@apps-vps "docker logs hg-content-cpm --tail 100"

Version: 2.0.0 Platform: Coolify on apps-vps