homelab/AGENTS.md

# Homelab Service Setup Guide for AI Agents

This document provides patterns, conventions, and best practices for setting up services in this homelab environment. Follow these guidelines when creating new services or modifying existing ones.

## Repository Structure

```
homelab/
├── .claude/               # Claude Code configuration
│   └── skills/           # Custom skills for AI agents
│       └── wiki-docs.md  # Wiki documentation skill
├── compose/
│   ├── core/              # Infrastructure services (Traefik, Authelia, LLDAP)
│   │   ├── traefik/
│   │   ├── authelia/
│   │   └── lldap/
│   ├── services/          # User-facing applications
│   │   ├── service-name/
│   │   │   ├── compose.yaml
│   │   │   ├── .env
│   │   │   ├── .gitignore
│   │   │   ├── README.md
│   │   │   └── QUICKSTART.md
│   ├── media/             # Media-related services
│   │   ├── frontend/      # Media viewers (Jellyfin, Immich)
│   │   └── automation/    # Media management (*arr stack)
│   └── monitoring/        # Monitoring and logging
├── AGENTS.md              # AI agent guidelines (this file)
└── README.md              # Repository overview
```

**External Directories:**
- `/mnt/media/wikijs-content/` - Wiki.js content repository (Git-backed)

## Core Principles

### 1. Domain Convention
- **Primary domain:** `fig.systems`
- **Secondary domain:** `edfig.dev`
- **Pattern:** `service.fig.systems` or `service.edfig.dev`
- **Examples:**
  - `matrix.fig.systems` - Matrix server
  - `auth.fig.systems` - Authelia
  - `books.fig.systems` - BookLore
  - `ai.fig.systems` - Open WebUI

#### DNS and DDNS Setup

**Automatic DNS Resolution:**
- Wildcard DNS records are automatically updated via DDNS Updater
- `*.fig.systems` → Points to current public IP (Cloudflare)
- `*.edfig.dev` → Points to current public IP (Porkbun)
- `fig.systems` (root) → Points to current public IP
- `edfig.dev` (root) → Points to current public IP

**What this means for new services:**
- ✅ DNS is automatic - any `newservice.fig.systems` will resolve to the homelab IP
- ✅ No manual DNS record creation needed
- ✅ Works for all subdomains automatically
- ⚠️ You still need Traefik labels to route traffic to containers (see Traefik Integration section)

**DDNS Updater Service:**
- Location: `compose/services/ddns-updater/`
- Monitors: Public IP changes every 5 minutes
- Updates: Both Cloudflare (fig.systems) and Porkbun (edfig.dev)
- Web UI: https://ddns.fig.systems (local network only)

**Adding a new service:**
1. DNS resolution is already handled by wildcard records
2. Add Traefik labels to your compose.yaml (see Service Setup Pattern below)
3. Start container - Traefik auto-detects and routes traffic
4. Let's Encrypt SSL certificate generated automatically

### 2. Storage Conventions

**Media Storage:** `/mnt/media/`
- `/mnt/media/books/` - Book library
- `/mnt/media/movies/` - Movie library
- `/mnt/media/tv/` - TV shows
- `/mnt/media/photos/` - Photo library
- `/mnt/media/music/` - Music library

**Service Data:** `/mnt/media/service-name/`
```bash
# Example: Matrix storage structure
/mnt/media/matrix/
├── synapse/
│   ├── data/      # Configuration and database
│   └── media/     # Uploaded media files
├── postgres/      # Database files
└── bridges/       # Bridge configurations
    ├── telegram/
    ├── whatsapp/
    └── googlechat/
```

**Always create subdirectories for:**
- Configuration files
- Database data
- User uploads/media
- Logs (if persistent)

### 3. Network Architecture

**External Network:** `homelab`
- All services connect to this for Traefik routing
- Created externally, referenced as `external: true`

**Internal Networks:** `service-internal`
- For multi-container service communication
- Example: `matrix-internal`, `booklore-internal`
- Use `driver: bridge`

```yaml
networks:
  homelab:
    external: true
  service-internal:
    driver: bridge
```

## Service Setup Pattern

### Directory Structure

Every service should have:
```
compose/services/service-name/
├── compose.yaml          # Docker Compose configuration
├── .env                  # Environment variables and secrets
├── .gitignore           # Ignore data directories and secrets
├── README.md            # Complete documentation
├── QUICKSTART.md        # 5-step quick start guide
└── config-files/        # Service-specific configs (optional)
```

### Required Files

#### 1. compose.yaml

**Basic template:**
```yaml
services:
  service-name:
    image: vendor/service:latest
    container_name: service-name
    environment:
      - TZ=${TZ}
      - PUID=${PUID}
      - PGID=${PGID}
      # Service-specific vars
    volumes:
      - /mnt/media/service-name:/data
    restart: unless-stopped
    networks:
      - homelab
    labels:
      # Traefik routing
      traefik.enable: true
      traefik.docker.network: homelab

      # HTTP Router
      traefik.http.routers.service-name.rule: Host(`service.fig.systems`)
      traefik.http.routers.service-name.entrypoints: websecure
      traefik.http.routers.service-name.tls.certresolver: letsencrypt
      traefik.http.services.service-name.loadbalancer.server.port: 8080

      # Homarr Discovery
      homarr.name: Service Name
      homarr.group: Services
      homarr.icon: mdi:icon-name

networks:
  homelab:
    external: true
```

**With database:**
```yaml
services:
  app:
    # ... app config
    depends_on:
      database:
        condition: service_healthy
    networks:
      - homelab
      - service-internal

  database:
    image: postgres:16-alpine  # or mariadb, redis, etc.
    container_name: service-database
    environment:
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_DB: ${DB_NAME}
    volumes:
      - /mnt/media/service-name/db:/var/lib/postgresql/data
    restart: unless-stopped
    networks:
      - service-internal
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${DB_USER}"]
      interval: 10s
      timeout: 5s
      retries: 5

networks:
  homelab:
    external: true
  service-internal:
    driver: bridge
```

#### 2. .env File

**Standard variables:**
```bash
# Domain Configuration
DOMAIN=fig.systems
SERVICE_DOMAIN=service.fig.systems
TRAEFIK_HOST=service.fig.systems

# System
TZ=America/Los_Angeles
PUID=1000
PGID=1000

# Database (if applicable)
DB_USER=service
DB_PASSWORD=<generated-password>
DB_NAME=service

# SMTP Configuration (Mailgun)
SMTP_HOST=smtp.mailgun.org
SMTP_PORT=587
SMTP_USER=noreply@fig.systems
SMTP_PASSWORD=<mailgun-smtp-password>
SMTP_FROM=Service Name <noreply@fig.systems>
# Optional SMTP settings
SMTP_TLS=true
SMTP_STARTTLS=true

# Service-specific secrets
SERVICE_SECRET_KEY=<generated-secret>
```

**Generate secrets:**
```bash
# Random hex (64 chars)
openssl rand -hex 32

# Base64 (32 bytes)
openssl rand -base64 32

# Alphanumeric (32 chars)
openssl rand -base64 24 | tr -d '/+=' | head -c 32
```

#### 3. .gitignore

**Standard pattern:**
```gitignore
# Service data (stored in /mnt/media/)
data/
config/
db/
logs/

# Environment secrets
.env

# Backup files
*.bak
*.backup
```

#### 4. README.md

**Structure:**
```markdown
# Service Name - Brief Description

One-paragraph overview of what the service does.

## Features

- ✅ Feature 1
- ✅ Feature 2
- ✅ Feature 3

## Access

**URL:** https://service.fig.systems
**Authentication:** [Authelia SSO | None | Basic Auth]

## Quick Start

### Deploy
\`\`\`bash
cd /home/eduardo_figueroa/homelab/compose/services/service-name
docker compose up -d
\`\`\`

### First-Time Setup
1. Step 1
2. Step 2
3. Step 3

## Configuration

### Environment Variables
Explain key .env variables

### Storage Locations
- `/mnt/media/service-name/data` - Application data
- `/mnt/media/service-name/uploads` - User uploads

## Usage Guide

Detailed usage instructions...

## Troubleshooting

Common issues and solutions...

## Maintenance

### Backup
Important directories to backup...

### Update
\`\`\`bash
docker compose pull
docker compose up -d
\`\`\`

## Links
- Documentation: https://...
- GitHub: https://...
```

#### 5. QUICKSTART.md

**Fast 5-step guide:**
```markdown
# Service Name - Quick Start

## Step 1: Deploy
\`\`\`bash
cd /path/to/service
docker compose up -d
\`\`\`

## Step 2: Access
Open https://service.fig.systems

## Step 3: Initial Setup
Quick setup steps...

## Step 4: Test
Verification steps...

## Common Commands
\`\`\`bash
# View logs
docker compose logs -f

# Restart
docker compose restart

# Stop
docker compose down
\`\`\`
```

## Traefik Integration

### Basic HTTP Routing

```yaml
labels:
  traefik.enable: true
  traefik.docker.network: homelab

  # Router
  traefik.http.routers.service.rule: Host(`service.fig.systems`)
  traefik.http.routers.service.entrypoints: websecure
  traefik.http.routers.service.tls.certresolver: letsencrypt

  # Service (port)
  traefik.http.services.service.loadbalancer.server.port: 8080
```

### With Custom Headers

```yaml
labels:
  # ... basic routing ...

  # Headers middleware
  traefik.http.middlewares.service-headers.headers.customrequestheaders.X-Forwarded-Proto: https
  traefik.http.middlewares.service-headers.headers.customresponseheaders.X-Frame-Options: SAMEORIGIN

  # Apply middleware
  traefik.http.routers.service.middlewares: service-headers
```

### With Local-Only Access

```yaml
labels:
  # ... basic routing ...

  # Apply local-only middleware (defined in Traefik)
  traefik.http.routers.service.middlewares: local-only
```

### Large Upload Support

```yaml
labels:
  # ... basic routing ...

  # Buffering middleware
  traefik.http.middlewares.service-buffering.buffering.maxRequestBodyBytes: 268435456
  traefik.http.middlewares.service-buffering.buffering.memRequestBodyBytes: 268435456
  traefik.http.middlewares.service-buffering.buffering.retryExpression: IsNetworkError() && Attempts() < 3

  # Apply middleware
  traefik.http.routers.service.middlewares: service-buffering
```

## Authelia OIDC Integration

### 1. Generate Client Secret

```bash
# Generate plain secret
openssl rand -base64 32

# Hash for Authelia
docker exec authelia authelia crypto hash generate pbkdf2 --password 'your-secret-here'
```

### 2. Add Client to Authelia

Edit `/home/eduardo_figueroa/homelab/compose/core/authelia/config/configuration.yml`:

```yaml
identity_providers:
  oidc:
    clients:
      # Your Service
      - client_id: service-name
        client_name: Service Display Name
        client_secret: '$pbkdf2-sha512$310000$...' # hashed secret
        authorization_policy: two_factor
        redirect_uris:
          - https://service.fig.systems/oauth/callback
        scopes:
          - openid
          - profile
          - email
        grant_types:
          - authorization_code
        response_types:
          - code
```

**For public clients (PKCE):**
```yaml
      - client_id: service-name
        client_name: Service Name
        public: true  # No client_secret needed
        authorization_policy: two_factor
        require_pkce: true
        pkce_challenge_method: S256
        redirect_uris:
          - https://service.fig.systems/oauth/callback
        scopes:
          - openid
          - profile
          - email
          - offline_access  # For refresh tokens
        grant_types:
          - authorization_code
          - refresh_token
        response_types:
          - code
```

### 3. Configure Service

**Standard OIDC configuration:**
```yaml
environment:
  OIDC_ENABLED: "true"
  OIDC_CLIENT_ID: "service-name"
  OIDC_CLIENT_SECRET: "plain-secret-here"
  OIDC_ISSUER: "https://auth.fig.systems"
  OIDC_AUTHORIZATION_ENDPOINT: "https://auth.fig.systems/api/oidc/authorization"
  OIDC_TOKEN_ENDPOINT: "https://auth.fig.systems/api/oidc/token"
  OIDC_USERINFO_ENDPOINT: "https://auth.fig.systems/api/oidc/userinfo"
  OIDC_JWKS_URI: "https://auth.fig.systems/jwks.json"
```

### 4. Restart Services

```bash
# Restart Authelia
cd compose/core/authelia
docker compose restart

# Start your service
cd compose/services/service-name
docker compose up -d
```

## SMTP/Email Configuration

### Mailgun SMTP

**Standard Mailgun configuration for all services:**

```bash
# In .env file
SMTP_HOST=smtp.mailgun.org
SMTP_PORT=587
SMTP_USER=noreply@fig.systems
SMTP_PASSWORD=<your-mailgun-smtp-password>
SMTP_FROM=Service Name <noreply@fig.systems>
SMTP_TLS=true
SMTP_STARTTLS=true
```

**In compose.yaml:**
```yaml
environment:
  # SMTP Settings
  SMTP_HOST: ${SMTP_HOST}
  SMTP_PORT: ${SMTP_PORT}
  SMTP_USER: ${SMTP_USER}
  SMTP_PASSWORD: ${SMTP_PASSWORD}
  SMTP_FROM: ${SMTP_FROM}
  # Some services may use different variable names:
  # EMAIL_HOST: ${SMTP_HOST}
  # EMAIL_PORT: ${SMTP_PORT}
  # EMAIL_USER: ${SMTP_USER}
  # EMAIL_PASS: ${SMTP_PASSWORD}
  # EMAIL_FROM: ${SMTP_FROM}
```

**Common SMTP variable name variations:**

Different services use different environment variable names for SMTP configuration. Check the service documentation and use the appropriate format:

| Common Name | Alternative Names |
|------------|-------------------|
| SMTP_HOST | EMAIL_HOST, MAIL_HOST, MAIL_SERVER |
| SMTP_PORT | EMAIL_PORT, MAIL_PORT |
| SMTP_USER | EMAIL_USER, MAIL_USER, SMTP_USERNAME, EMAIL_USERNAME |
| SMTP_PASSWORD | EMAIL_PASSWORD, EMAIL_PASS, MAIL_PASSWORD, SMTP_PASS |
| SMTP_FROM | EMAIL_FROM, MAIL_FROM, FROM_EMAIL, DEFAULT_FROM_EMAIL |
| SMTP_TLS | EMAIL_USE_TLS, MAIL_USE_TLS, SMTP_SECURE |
| SMTP_STARTTLS | EMAIL_USE_STARTTLS, MAIL_STARTTLS |

**Getting Mailgun SMTP credentials:**

1. Log into Mailgun dashboard: https://app.mailgun.com
2. Navigate to **Sending → Domain Settings → SMTP credentials**
3. Use the existing `noreply@fig.systems` user or create a new SMTP user
4. Copy the SMTP password and add it to your service's `.env` file

**Testing SMTP configuration:**

```bash
# Using swaks (SMTP test tool)
swaks --to test@example.com \
  --from noreply@fig.systems \
  --server smtp.mailgun.org:587 \
  --auth LOGIN \
  --auth-user noreply@fig.systems \
  --auth-password 'your-password' \
  --tls
```

## Database Patterns

### PostgreSQL

```yaml
postgres:
  image: postgres:16-alpine
  container_name: service-postgres
  environment:
    POSTGRES_USER: ${DB_USER}
    POSTGRES_PASSWORD: ${DB_PASSWORD}
    POSTGRES_DB: ${DB_NAME}
    POSTGRES_INITDB_ARGS: "--encoding=UTF-8 --lc-collate=C --lc-ctype=C"
  volumes:
    - /mnt/media/service-name/postgres:/var/lib/postgresql/data
  restart: unless-stopped
  networks:
    - service-internal
  healthcheck:
    test: ["CMD-SHELL", "pg_isready -U ${DB_USER}"]
    interval: 10s
    timeout: 5s
    retries: 5
```

### MariaDB

```yaml
mariadb:
  image: lscr.io/linuxserver/mariadb:latest
  container_name: service-mariadb
  environment:
    - PUID=${PUID}
    - PGID=${PGID}
    - TZ=${TZ}
    - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
    - MYSQL_DATABASE=${MYSQL_DATABASE}
    - MYSQL_USER=${MYSQL_USER}
    - MYSQL_PASSWORD=${MYSQL_PASSWORD}
  volumes:
    - /mnt/media/service-name/mariadb:/config
  restart: unless-stopped
  networks:
    - service-internal
  healthcheck:
    test: ["CMD", "mariadb-admin", "ping", "-h", "localhost"]
    interval: 5s
    timeout: 5s
    retries: 10
```

### Redis

```yaml
redis:
  image: redis:alpine
  container_name: service-redis
  command: redis-server --save 60 1 --loglevel warning
  volumes:
    - /mnt/media/service-name/redis:/data
  restart: unless-stopped
  networks:
    - service-internal
  healthcheck:
    test: ["CMD", "redis-cli", "ping"]
    interval: 10s
    timeout: 5s
    retries: 5
```

## Homarr Integration

**Add discovery labels to your service:**
```yaml
labels:
  homarr.name: Display Name
  homarr.group: Services  # or Media, Monitoring, AI, etc.
  homarr.icon: mdi:icon-name  # Material Design Icons
```

**Common groups:**
- `Services` - General applications
- `Media` - Media-related (Jellyfin, Immich)
- `AI` - AI/LLM services
- `Monitoring` - Monitoring tools
- `Automation` - *arr stack

**Find icons:** https://pictogrammers.com/library/mdi/

## Security Best Practices

### 1. Never Commit Secrets

**Always in .gitignore:**
- `.env` files
- Database directories
- Configuration files with credentials
- SSL certificates
- API keys

### 2. Use Authelia for External Access

Services exposed to internet should use Authelia SSO with 2FA.

### 3. Local-Only Services

For sensitive services (backups, code editors), use `local-only` middleware:

```yaml
traefik.http.routers.service.middlewares: local-only
```

### 4. Least Privilege

- Use non-root users in containers (`PUID`/`PGID`)
- Limit network access (internal networks)
- Read-only mounts where possible: `./config:/config:ro`

### 5. Secrets Generation

**Always generate unique secrets:**
```bash
# For each service
openssl rand -hex 32  # Different secret each time
```

## Common Patterns

### Multi-Stage Service Setup

**For services requiring initial config generation:**

1. Generate config:
   ```bash
   docker run --rm -v /path:/data image:latest --generate-config
   ```

2. Edit config files

3. Start service:
   ```bash
   docker compose up -d
   ```

### Bridge/Plugin Architecture

**For services with plugins/bridges:**

```yaml
# Main service
main-app:
  # ... config ...
  volumes:
    - /mnt/media/service/data:/data
    - ./registrations:/registrations:ro  # Plugin registrations

# Plugin 1
plugin-1:
  # ... config ...
  volumes:
    - /mnt/media/service/plugins/plugin-1:/data
  depends_on:
    main-app:
      condition: service_started
  networks:
    - service-internal
```

### Health Checks

**Always include health checks for databases:**
```yaml
healthcheck:
  test: ["CMD-SHELL", "command to test health"]
  interval: 10s
  timeout: 5s
  retries: 5
```

**Then use in depends_on:**
```yaml
depends_on:
  database:
    condition: service_healthy
```

## Troubleshooting Checklist

### Service Won't Start

1. Check logs:
   ```bash
   docker compose logs -f service-name
   ```

2. Verify environment variables:
   ```bash
   docker compose config
   ```

3. Check disk space:
   ```bash
   df -h /mnt/media
   ```

4. Verify network exists:
   ```bash
   docker network ls | grep homelab
   ```

### Can't Access via Domain

1. Check Traefik logs:
   ```bash
   docker logs traefik | grep service-name
   ```

2. Verify service is on homelab network:
   ```bash
   docker inspect service-name | grep -A 10 Networks
   ```

3. Test endpoint directly:
   ```bash
   curl -k https://service.fig.systems
   ```

4. Check DNS resolution:
   ```bash
   nslookup service.fig.systems
   ```

### OIDC Login Issues

1. Verify client secret matches in both Authelia and service
2. Check redirect URI exactly matches in Authelia config
3. Restart Authelia after config changes
4. Check Authelia logs:
   ```bash
   docker logs authelia | grep oidc
   ```

### Database Connection Issues

1. Verify database is healthy:
   ```bash
   docker compose ps
   ```

2. Check database logs:
   ```bash
   docker compose logs database
   ```

3. Test connection from app container:
   ```bash
   docker compose exec app ping database
   ```

4. Verify credentials match in .env and config

## Complete Service Template

See `compose/services/matrix/` for a complete example of:
- ✅ Multi-container setup (app + database + plugins)
- ✅ Authelia OIDC integration
- ✅ Traefik routing
- ✅ Comprehensive documentation
- ✅ Bridge/plugin architecture
- ✅ Health checks and dependencies
- ✅ Proper secret management

## AI Agent Guidelines

When setting up new services:

1. **Always create complete config files in /tmp/** for files requiring sudo access
2. **Follow the directory structure** exactly as shown above
3. **Generate unique secrets** for each service
4. **Create both README.md and QUICKSTART.md**
5. **Use the storage conventions** (/mnt/media/service-name/)
6. **Add Traefik labels** for automatic routing
7. **Include Homarr discovery labels**
8. **Set up health checks** for all databases
9. **Use internal networks** for multi-container communication
10. **Document troubleshooting steps** in README.md

### Files to Always Create in /tmp/

When you cannot write directly:
- Authelia configuration updates
- Traefik configuration changes
- System-level configuration files

**Format:**
```bash
/tmp/service-name-config-file.yml
```

Include clear instructions at the top:
```yaml
# Copy this file to:
# /path/to/actual/location
#
# Then run:
# sudo chmod 644 /path/to/actual/location
# docker compose restart
```

## Claude Code Skills

This repository includes custom skills for Claude Code to enhance productivity and maintain consistency.

### Available Skills

#### wiki-docs (Documentation Management)

**Purpose:** Create and manage markdown documentation files that automatically sync to Wiki.js

**Location:** `.claude/skills/wiki-docs.md`

**When to use:**
- Documenting new services or infrastructure changes
- Creating how-to guides or tutorials
- Recording configuration details for future reference
- Building a knowledge base for the homelab

**Repository:** `/mnt/media/wikijs-content/`
**Wiki URL:** https://wiki.fig.systems
**Git Remote:** `git.fig.systems/eddie/wiki.git`

**How it works:**
1. Markdown files are written to `/mnt/media/wikijs-content/`
2. Files are committed and pushed to the Git repository
3. Wiki.js automatically syncs changes (within 5 minutes)
4. Content appears at https://wiki.fig.systems

**Frontmatter format:**
```yaml
---
title: Page Title
description: Brief description
published: true
date: 2026-03-15T00:00:00.000Z
tags:
  - tag1
  - tag2
---
```

**Example usage:**
```bash
# Create documentation for a service
/mnt/media/wikijs-content/homelab/services/jellyfin.md

# Commit and push
cd /mnt/media/wikijs-content
git pull
git add homelab/services/jellyfin.md
git commit -m "Add: Jellyfin service documentation"
git push
```

**Benefits:**
- Version-controlled documentation
- Accessible via web interface (Wiki.js)
- Searchable and organized
- Supports markdown with frontmatter
- Automatic synchronization

### Using Skills

To invoke a skill in Claude Code, use the appropriate skill when the task matches its purpose. The wiki-docs skill is automatically available for documentation tasks.

## Resources

- **Traefik:** https://doc.traefik.io/traefik/
- **Authelia:** https://www.authelia.com/
- **Docker Compose:** https://docs.docker.com/compose/
- **Material Design Icons:** https://pictogrammers.com/library/mdi/
- **Wiki.js:** https://docs.requarks.io/

---

**Remember:** Consistency is key. Follow these patterns for all services to maintain a clean, predictable, and maintainable homelab infrastructure.