diff --git a/wiki/AWS-Configuration.md b/wiki/AWS-Configuration.md new file mode 100644 index 0000000..3cdd832 --- /dev/null +++ b/wiki/AWS-Configuration.md @@ -0,0 +1,225 @@ +# AWS Configuration + +## Prerequisites + +Before using SGO, ensure you have: + +1. AWS CLI configured with credentials +2. Appropriate IAM permissions +3. MFA device configured (if required by your profiles) + +## AWS Profiles Setup + +SGO reads profiles from `~/.aws/config`. Ensure your AWS configuration files are set up correctly. + +### Basic Profile Configuration + +```ini +[profile my-aws-account] +region = us-west-2 +``` + +### Profile with MFA + +For profiles that require MFA authentication: + +```ini +[profile nonprod-p1p2-admin] +region = us-west-2 +mfa_serial = arn:aws:iam::131340773912:mfa/your-username +``` + +### Multiple Profiles + +You can have multiple profiles in your config file: + +```ini +[default] +region = us-east-1 + +[profile production] +region = us-west-2 +mfa_serial = arn:aws:iam::123456789012:mfa/john.doe + +[profile development] +region = us-west-2 + +[profile staging] +region = us-east-1 +mfa_serial = arn:aws:iam::987654321098:mfa/john.doe +``` + +## MFA Device Setup + +### Finding Your MFA Device ARN + +1. Go to AWS IAM Console +2. Navigate to **Users** → **Your User** → **Security Credentials** +3. Scroll to **Multi-factor authentication (MFA)** +4. Copy the ARN from "Assigned MFA device" + +Example ARN format: +``` +arn:aws:iam::123456789012:mfa/username +``` + +### Adding MFA to Profile + +Add the `mfa_serial` line to your profile in `~/.aws/config`: + +```ini +[profile my-profile] +region = us-west-2 +mfa_serial = arn:aws:iam::123456789012:mfa/username +``` + +## How MFA Works in SGO + +1. The import page shows all profiles from `~/.aws/config` +2. Profiles with `mfa_serial` configured will show an MFA input field +3. Profiles without `mfa_serial` can import without entering a code +4. Enter your current MFA/TOTP code (6 digits) for profiles that require it +5. Click "Start Import" to begin authentication and data import +6. MFA session is valid for 1 hour +7. During the session window (55 minutes), you can refresh without re-entering codes + +### MFA Code Sources + +You can get MFA codes from: +- Authenticator apps (Google Authenticator, Microsoft Authenticator, Authy, etc.) +- Hardware MFA devices +- SMS (if configured) + +**Note**: MFA codes expire every 30 seconds, so enter them promptly. + +## Required IAM Permissions + +Your AWS user/role needs the following permissions to use SGO: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "ec2:DescribeInstances", + "ec2:DescribeSecurityGroups", + "iam:ListAccountAliases", + "sts:GetCallerIdentity" + ], + "Resource": "*" + } + ] +} +``` + +### Permission Breakdown + +- `ec2:DescribeInstances` - List and describe EC2 instances +- `ec2:DescribeSecurityGroups` - List and describe security groups +- `iam:ListAccountAliases` - Get friendly account names +- `sts:GetCallerIdentity` - Get account ID + +## AWS Credentials Location + +### Default Location + +SGO expects AWS credentials at: +- Linux/macOS: `~/.aws/` +- Windows: `%USERPROFILE%\.aws\` + +### Custom Location + +If your AWS credentials are in a non-standard location, specify it in your `.env` file: + +```bash +AWS_CONFIG_PATH=/path/to/custom/.aws +``` + +### Required Files + +Ensure these files exist in your AWS credentials directory: + +1. **`config`** - Contains profile configurations + ```ini + [profile my-profile] + region = us-west-2 + mfa_serial = arn:aws:iam::123456789012:mfa/username + ``` + +2. **`credentials`** - Contains access keys + ```ini + [my-profile] + aws_access_key_id = AKIAIOSFODNN7EXAMPLE + aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY + ``` + +## Testing Your Configuration + +### Verify AWS CLI Access + +```bash +# Test default profile +aws sts get-caller-identity + +# Test specific profile +aws sts get-caller-identity --profile my-profile + +# Test with MFA +aws sts get-caller-identity --profile my-profile +# (will prompt for MFA if configured) +``` + +### Verify Permissions + +```bash +# Test EC2 access +aws ec2 describe-instances --profile my-profile --max-results 1 + +# Test security groups access +aws ec2 describe-security-groups --profile my-profile --max-results 1 +``` + +## Common Configuration Issues + +### No Profiles Found + +**Problem**: Import page shows "No AWS profiles found" + +**Solution**: +- Verify `~/.aws/config` exists and contains profiles +- Check file permissions (should be readable) +- Ensure profiles are properly formatted in config file + +### MFA Authentication Fails + +**Problem**: "MFA authentication failed" error + +**Solution**: +- Verify MFA code is current (not expired) +- Check `mfa_serial` is correct in `~/.aws/config` +- Ensure AWS credentials in `~/.aws/credentials` are valid +- Try generating a new MFA code + +### Permission Denied + +**Problem**: "Access Denied" or "Unauthorized" errors + +**Solution**: +- Verify your IAM user/role has required permissions +- Check if your credentials have expired +- Ensure you're using the correct profile + +### Wrong Region + +**Problem**: Not seeing resources you expect + +**Solution**: +- Verify the `region` setting in your profile +- Remember: EC2 resources are region-specific +- Try setting the region explicitly: + ```ini + [profile my-profile] + region = us-west-2 + ``` diff --git a/wiki/Configuration.md b/wiki/Configuration.md new file mode 100644 index 0000000..3c0b462 --- /dev/null +++ b/wiki/Configuration.md @@ -0,0 +1,217 @@ +# Configuration + +## Environment Variables + +SGO supports configuration through environment variables. Create a `.env` file in the same directory as your `docker-compose.yml` file. + +### Available Options + +| Variable | Description | Required | Default | +|----------|-------------|----------|---------| +| `AWS_CONFIG_PATH` | Absolute path to AWS credentials directory | Yes | None | +| `PUID` | User ID for file permissions | No | `1000` | +| `PGID` | Group ID for file permissions | No | `1000` | +| `DATA_PATH` | Path for database storage (local mode) | No | `./data` | +| `SGO_PORT` | Port to expose on host | No | `5000` | +| `DEBUG` | Enable Flask debug logging | No | `false` | +| `FLASK_ENV` | Flask environment | No | `production` | + +## Data Storage Options + +### Option 1: Docker Volume (Default - Recommended) + +- Data stored in Docker-managed volume `sgo-data` +- Survives container restarts and rebuilds +- Better performance on macOS/Windows +- Use default `docker-compose.yml` + +```bash +docker-compose up --build +``` + +### Option 2: Local Directory + +- Data stored in `./data` directory +- Easy to backup and access +- Better for development +- Use `docker-compose.local.yml`: + +```bash +docker-compose -f docker-compose.local.yml up --build +# or +podman-compose -f docker-compose.local.yml up --build +``` + +Or edit `docker-compose.yml` and swap the volume configuration as indicated in comments. + +## User/Group Configuration + +To avoid permission issues, set `PUID` and `PGID` to match your host user: + +```bash +# Find your IDs +id -u # Your PUID +id -g # Your PGID + +# Add to .env file +echo "PUID=$(id -u)" >> .env +echo "PGID=$(id -g)" >> .env +``` + +## Configuration Examples + +### Example 1: Basic Setup (Default) + +Minimal configuration with Docker volume: + +```bash +# Create .env file +cat > .env << EOF +AWS_CONFIG_PATH=$HOME/.aws +PUID=$(id -u) +PGID=$(id -g) +EOF + +# Run +docker-compose up --build +# or: podman-compose up --build +``` + +### Example 2: Local Data Directory + +Store database in local directory for easy access: + +```bash +# Create .env file +cat > .env << EOF +AWS_CONFIG_PATH=$HOME/.aws +PUID=$(id -u) +PGID=$(id -g) +DATA_PATH=./data +EOF + +# Use the local compose file +docker-compose -f docker-compose.local.yml up --build +``` + +### Example 3: Custom AWS Credentials Location + +If your AWS credentials are in a non-standard location: + +```bash +# Create .env file +cat > .env << EOF +AWS_CONFIG_PATH=/path/to/custom/.aws +PUID=$(id -u) +PGID=$(id -g) +EOF + +docker-compose up --build +``` + +### Example 4: Debug Mode + +Enable detailed logging for troubleshooting: + +```bash +# Create .env file +cat > .env << EOF +AWS_CONFIG_PATH=$HOME/.aws +PUID=$(id -u) +PGID=$(id -g) +DEBUG=true +EOF + +docker-compose up --build +``` + +### Example 5: Custom Port + +Run on a different port (e.g., 8080): + +```bash +# Create .env file +cat > .env << EOF +AWS_CONFIG_PATH=$HOME/.aws +PUID=$(id -u) +PGID=$(id -g) +SGO_PORT=8080 +EOF + +# Access at http://localhost:8080 +docker-compose up --build +``` + +## Quick Reference Commands + +### Essential Commands + +```bash +# Start +docker-compose up --build +# or: podman-compose up --build + +# Stop +docker-compose down +# or: Ctrl+C + +# View logs +docker-compose logs -f + +# Rebuild after changes +docker-compose up --build + +# Remove everything including data +docker-compose down -v +``` + +### Quick .env Setup + +```bash +# Minimal configuration for most users +cat > .env << EOF +AWS_CONFIG_PATH=$HOME/.aws +PUID=$(id -u) +PGID=$(id -g) +EOF + +# Then run +docker-compose up --build +``` + +### Data Location + +**Docker volume (default)**: Managed by Docker, survives rebuilds + +```bash +# Inspect volume +docker volume inspect sgo-data + +# Backup volume +docker run --rm -v sgo-data:/data -v $(pwd):/backup alpine tar czf /backup/sgo-backup.tar.gz -C /data . +``` + +**Local directory**: `./data/aws_export.db` + +```bash +# Use local mode +docker-compose -f docker-compose.local.yml up --build +``` + +## Platform-Specific Notes + +### Windows + +- Use Git Bash, WSL2, or PowerShell for running scripts +- Docker Desktop must be running +- WSL2 backend recommended for better performance + +### macOS + +- Docker Desktop must be running +- Ensure Docker has access to your home directory in Docker Desktop settings + +### Linux + +- You may need to run Docker commands with `sudo` or add your user to the `docker` group +- Podman works without root by default diff --git a/wiki/FAQ.md b/wiki/FAQ.md new file mode 100644 index 0000000..c8362c6 --- /dev/null +++ b/wiki/FAQ.md @@ -0,0 +1,330 @@ +# Frequently Asked Questions (FAQ) + +## General Questions + +### What is SGO? + +SGO (Security Groups Observatory) is a web-based tool for exploring and analyzing AWS EC2 instances and Security Groups. It provides search, filtering, detailed views, and CSV export capabilities for AWS infrastructure. + +### Is SGO free to use? + +Yes, for personal, educational, and non-commercial use. Commercial use by businesses requires a paid license. See [LICENSE](../LICENSE) for details. + +### Can I use SGO in production? + +**No, not recommended.** SGO is designed for local use only and has no authentication mechanisms. It should never be exposed to the internet or production networks. + +### What AWS services does SGO support? + +Currently, SGO supports: +- EC2 Instances +- Security Groups (with ingress/egress rules) + +Future versions may add support for additional AWS services. + +## Installation & Setup + +### Do I need Docker or Podman? + +Not required, but strongly recommended. You can run SGO with local Python, but containers provide easier setup and better isolation. + +### What are PUID and PGID? + +PUID (Process User ID) and PGID (Process Group ID) ensure files created by the container have correct ownership on your host system. Set them to your user's UID/GID to avoid permission issues. + +```bash +# Find your values +id -u # PUID +id -g # PGID +``` + +### Can I use Docker volume or local directory for data? + +Both are supported: +- **Docker volume** (default): Better for macOS/Windows, managed by Docker +- **Local directory**: Better for development, easy to backup + +See [Configuration](Configuration.md#data-storage-options) for details. + +### Where are my AWS credentials stored? + +AWS credentials are NOT stored by SGO. The application reads from your existing `~/.aws/` directory and only keeps temporary session tokens in memory (cached for 55 minutes). + +## AWS & Authentication + +### Do I need MFA? + +Only if your AWS profile requires it. If your `~/.aws/config` has `mfa_serial` configured for a profile, you'll need to provide an MFA code when importing. + +### How long does MFA session last? + +AWS MFA sessions last 1 hour. SGO caches them for 55 minutes, allowing you to refresh data multiple times without re-entering MFA codes. + +### Can I import multiple AWS accounts? + +Yes! SGO supports multiple profiles and can import them in parallel. Select all desired profiles and start imports simultaneously for maximum efficiency. + +### What AWS permissions do I need? + +Minimum required permissions: +- `ec2:DescribeInstances` +- `ec2:DescribeSecurityGroups` +- `iam:ListAccountAliases` +- `sts:GetCallerIdentity` + +See [AWS Configuration](AWS-Configuration.md#required-iam-permissions) for details. + +### Why can't I see my resources? + +Common causes: +1. **Wrong region**: EC2 resources are region-specific +2. **No permission**: Verify IAM permissions +3. **Import failed**: Check import progress logs for errors +4. **Different account**: Ensure you're importing the correct profile + +## Features & Usage + +### How do I search for resources? + +Type in the search box (minimum 2 characters). Search works across: +- EC2 instance names and IDs +- Security group names and IDs +- IP addresses +- Tags + +Enable "Regex" checkbox for pattern matching. + +### What's the difference between EC2 view and SG view? + +- **EC2 view**: Shows instance details with nested Security Groups +- **SG view**: Shows security group details with nested EC2 instances + +Both views are interactive - click any item to see its details. + +### Can I export my data? + +Yes! SGO provides CSV export at multiple levels: +- Search results +- Individual EC2 instance details +- Individual Security Group details +- Security group rules + +All exports include proper CSV formatting and timestamps. + +### Does SGO modify my AWS resources? + +**No.** SGO is read-only and never modifies AWS resources. It only uses `Describe*` and `List*` API calls. + +## Performance + +### How fast is the import? + +Import speed depends on: +- Number of resources in your AWS account +- Network latency to AWS +- Number of accounts being imported + +Parallel import significantly reduces total time by importing multiple accounts simultaneously. + +### Why is search slow with many resources? + +Search is client-side (in your browser) for instant results. With thousands of resources, this can be slower. Tips: +- Use more specific search terms +- Use type filters (EC2 only, SG only) +- Disable regex when not needed + +### Can I improve performance? + +Yes: +- **Use Docker volume** instead of local directory (better I/O) +- **Import selectively**: Only import accounts you need +- **Use specific searches**: Narrow down results +- **Use type filters**: Filter by EC2 or SG only + +## Data & Privacy + +### Is my data sent anywhere? + +**No.** SGO runs entirely locally. Data flows: +1. AWS → Your local SGO instance +2. Stored in local SQLite database +3. Never leaves your machine + +### Can others see my AWS data? + +Only if they have access to: +- Your computer while SGO is running +- Your SQLite database file +- Your Docker volume + +**Never expose SGO to the internet.** + +### How is data stored? + +Data is stored in a local SQLite database: +- Default location: Docker volume `sgo-data` +- Alternative: `./data/aws_export.db` (local mode) + +### Can I backup my data? + +Yes: + +**Docker volume backup**: +```bash +docker run --rm -v sgo-data:/data -v $(pwd):/backup alpine tar czf /backup/sgo-backup.tar.gz -C /data . +``` + +**Local directory backup**: +```bash +cp ./data/aws_export.db ./backups/ +``` + +### How do I clear all data? + +**Remove database and start fresh**: +```bash +# Docker volume mode +docker-compose down -v + +# Local directory mode +rm -rf ./data +``` + +## Troubleshooting + +### Import fails silently + +Check the import progress logs in the UI. Common issues: +- Invalid AWS credentials +- Expired MFA code +- Missing IAM permissions +- Network connectivity + +### Container won't start + +Common causes: +1. Port 5000 already in use +2. Docker/Podman not running +3. Missing `.env` file +4. Invalid AWS_CONFIG_PATH + +See [Troubleshooting](Troubleshooting.md) for detailed solutions. + +### AWS profiles not showing up + +Ensure: +- `~/.aws/config` exists and is readable +- Profiles are properly formatted +- `AWS_CONFIG_PATH` in `.env` is correct +- Container has been restarted after config changes + +## Development & Customization + +### Can I modify SGO? + +Yes, under the license terms: +- ✅ Modify for personal use +- ✅ Contribute improvements back +- ❌ Modify to sell or redistribute commercially + +### Can I contribute to SGO? + +Yes! Contributions are welcome. See [CONTRIBUTING.md](../CONTRIBUTING.md) for guidelines. + +### Does SGO support other clouds? + +Not currently. SGO is AWS-specific. Support for other cloud providers may be added in the future. + +### Can I add custom tags or fields? + +SGO imports all AWS tags automatically. Custom application-level tags are not currently supported but could be added in future versions. + +## Security + +### Is SGO secure? + +SGO is secure for **local use only**: +- ✅ Read-only access to AWS +- ✅ Uses standard AWS SDK authentication +- ✅ Local data storage +- ❌ **NO authentication/authorization** +- ❌ **NOT safe for network exposure** + +### Should I use SGO on a shared server? + +**No.** Use SGO only on your personal workstation. Anyone with access to the running application can view all imported AWS data. + +### Can I add authentication? + +Not currently supported. SGO is designed for single-user local use. Adding authentication would significantly complicate the setup for the primary use case. + +### What about HTTPS? + +Not needed for local use. Since SGO runs on `localhost` and should never be exposed to networks, HTTPS is unnecessary. + +## Updates & Versioning + +### How do I update SGO? + +```bash +# Pull latest changes +git pull + +# Rebuild container +docker-compose down +docker-compose up --build +``` + +### Will updates break my data? + +Database schema is designed for forward compatibility. Major version updates will include migration instructions if needed. + +### How do I check my version? + +Check the repository for version tags or commits: +```bash +git log -1 --oneline +``` + +## License & Commercial Use + +### Can my company use SGO? + +Commercial use requires a paid license. Open an issue in the repository for licensing inquiries. + +### What counts as commercial use? + +Using SGO at a company, for business purposes, or as part of commercial operations requires a license. Personal/educational use is free. + +### Can I sell a modified version? + +No. The license explicitly prohibits modifying SGO for resale or commercial distribution. + +## Getting More Help + +### Where can I get support? + +1. Check this FAQ +2. Read [Troubleshooting](Troubleshooting.md) +3. Search existing issues in the repository +4. Open a new issue with details + +### How do I report a bug? + +Open an issue in the repository with: +- Description of the problem +- Steps to reproduce +- Expected vs actual behavior +- Logs from `docker-compose logs` +- Your environment (OS, Docker version) + +### Can I request features? + +Yes! Open a feature request issue. Include: +- Use case description +- Why the feature is needed +- How you envision it working + +### Is there a community chat? + +Check the repository for links to community discussions or chat platforms. diff --git a/wiki/Home.md b/wiki/Home.md new file mode 100644 index 0000000..e7ba7cc --- /dev/null +++ b/wiki/Home.md @@ -0,0 +1,96 @@ +# SGO: Security Groups Observatory + +A web-based tool for exploring AWS EC2 instances and Security Groups with direct AWS import, MFA support, and CSV export capabilities. + +## ⚠️ Security Warning + +**This application is designed for LOCAL USE ONLY. Do NOT expose it to the internet.** + +- SGO has no authentication or authorization mechanisms +- It provides direct access to your AWS infrastructure data +- It reads AWS credentials from your local system +- Exposing it publicly would allow unauthorized access to sensitive AWS information + +**Always run on localhost (127.0.0.1) only. Never expose port 5000 to external networks.** + +## Features + +- **Direct AWS Import**: Import data directly from AWS using `~/.aws/config` with MFA/OTP support +- **Parallel Import**: Import from multiple AWS accounts simultaneously +- **Search & Filter**: Search by EC2 name, SG name, instance ID, group ID, or IP address +- **Regex Search**: Enable regex checkbox for advanced pattern matching +- **Filter by Type**: View all resources, only EC2 instances, or only Security Groups +- **CSV Export**: Export search results, EC2 details, SG details, and security group rules to CSV +- **Detailed Views**: + - **EC2 View**: Shows EC2 instance details with nested boxes for attached Security Groups + - **Security Group View**: Shows SG details with nested boxes for all attached EC2 instances +- **Security Group Rules**: View and search ingress/egress rules for any security group +- **Statistics Dashboard**: Quick overview of total SGs, EC2s, and accounts + +## Quick Links + +- [Quick Start Guide](Quick-Start.md) +- [Configuration Options](Configuration.md) +- [Usage Guide](Usage.md) +- [AWS Configuration](AWS-Configuration.md) +- [Troubleshooting](Troubleshooting.md) + +## Data Structure + +### Security Groups Table +- Account ID & Name +- Group ID & Name +- Tag Name +- Wave Tag +- Git Repo Tag +- Ingress Rule Count + +### EC2 Instances Table +- Account ID & Name +- Instance ID +- Tag Name +- State (running, stopped, etc.) +- Private IP Address +- Security Groups (IDs and Names) +- Git Repo Tag + +## File Structure + +``` +sgo/ +├── app.py # Flask web application +├── import_from_aws.py # AWS direct import functions +├── import_data.py # CSV to SQLite import (legacy) +├── requirements.txt # Python dependencies +├── Dockerfile # Container image definition +├── docker-compose.yml # Container orchestration (Docker volume) +├── docker-compose.local.yml # Alternative with local directory storage +├── entrypoint.sh # Container entrypoint with PUID/PGID support +├── .dockerignore # Files to exclude from container +├── .env.example # Example environment configuration +├── .gitignore # Git ignore patterns +├── README.md # This file +├── data/ # Local data directory (if using local mode) +│ └── aws_export.db # SQLite database +├── static/ +│ ├── css/ +│ │ └── style.css # Application styles +│ └── images/ +│ └── logo.svg # Application logo +└── templates/ + ├── import.html # Import/profile selection page + └── index.html # Main explorer interface +``` + +## License + +This project is dual-licensed: + +- **FREE** for individual, personal, educational, and non-commercial use +- **PAID LICENSE REQUIRED** for commercial use by businesses and organizations + +You may NOT modify this software for the purpose of selling or commercially distributing it. + +See the [LICENSE](../LICENSE) file for full details. + +For commercial licensing inquiries, please open an issue in the repository. diff --git a/wiki/Quick-Start.md b/wiki/Quick-Start.md new file mode 100644 index 0000000..aec344d --- /dev/null +++ b/wiki/Quick-Start.md @@ -0,0 +1,137 @@ +# Quick Start Guide + +## TL;DR - Get Started in 30 Seconds + +```bash +# 1. Create .env file with your AWS credentials path +cat > .env << EOF +AWS_CONFIG_PATH=$HOME/.aws +PUID=$(id -u) +PGID=$(id -g) +EOF + +# 2. Start the container +docker-compose up --build +# or with Podman: +podman-compose up --build + +# 3. Open browser to http://localhost:5000 + +# 4. Select AWS profiles, enter MFA codes, and import! +``` + +## Container Setup (Recommended) + +The easiest way to run SGO is using Docker or Podman. Works on Linux, macOS, and Windows. + +### Prerequisites + +Install either: +- **Docker**: https://docs.docker.com/get-docker/ +- **Podman**: https://podman.io/getting-started/installation + +### Setup Steps + +1. **Create environment configuration:** + +```bash +# Copy the example file +cp .env.example .env + +# Edit with your settings +nano .env # or your preferred editor +``` + +Or create it manually: + +```bash +cat > .env << EOF +AWS_CONFIG_PATH=$HOME/.aws +PUID=$(id -u) +PGID=$(id -g) +EOF +``` + +2. **Start the application:** + +```bash +# Docker +docker-compose up --build + +# Podman +podman-compose up --build +``` + +3. **Access the application:** + +Open your browser to `http://localhost:5000` + +### Import Data via GUI + +1. Open your browser to `http://localhost:5000` +2. You'll see the **Import Page** with all your AWS profiles +3. **Select profiles**: Check the AWS accounts you want to import +4. **Enter MFA codes**: Paste your MFA/OTP codes for profiles that require authentication +5. **Click "Start Import"**: Watch real-time progress as data is fetched **in parallel** +6. **Auto-redirect**: When complete, you're taken to the Explorer + +**Parallel Import**: All selected profiles are imported simultaneously in separate threads, so total time is the max of any single import, not the sum. This prevents MFA timeout issues. + +### Explore Your Data + +- Search for EC2 instances and Security Groups +- View detailed information +- Inspect security group rules +- Filter and search using regex +- Export data to CSV + +### Refresh Data + +- Click the **Refresh Data** button to refresh data using cached AWS sessions (valid for 55 minutes) +- Click the **Change Profiles** button to switch to different AWS accounts + +## Local Python Setup + +If you prefer to run without containers: + +### 1. Install Dependencies + +```bash +python3 -m venv venv +source venv/bin/activate # On Windows: venv\Scripts\activate +pip install -r requirements.txt +``` + +### 2. Start the Application + +```bash +python app.py +``` + +### 3. Open Browser + +Navigate to `http://localhost:5000` + +## Stopping the Application + +```bash +# Stop with Ctrl+C, or: +docker-compose down # Docker +podman-compose down # Podman + +# To also remove the data volume: +docker-compose down -v +``` + +## Important Notes + +- **⚠️ LOCAL USE ONLY**: Never expose this application to the internet. It has no authentication and provides access to sensitive AWS data. +- **Database Persistence**: When using containers, the database persists in a Docker volume or `./data` directory +- **Session Caching**: AWS sessions are cached for 55 minutes, allowing multiple refreshes without re-authentication +- **Parallel Import**: All selected AWS accounts are imported simultaneously for maximum speed + +## Next Steps + +- [Configuration Options](Configuration.md) - Customize your setup +- [AWS Configuration](AWS-Configuration.md) - Set up MFA and AWS profiles +- [Usage Guide](Usage.md) - Learn how to use SGO features diff --git a/wiki/README.md b/wiki/README.md new file mode 100644 index 0000000..f71c5b1 --- /dev/null +++ b/wiki/README.md @@ -0,0 +1,58 @@ +# SGO Wiki + +This directory contains comprehensive documentation for SGO (Security Groups Observatory). + +## Wiki Pages + +### Getting Started +- **[Home](Home.md)** - Project overview, features, and file structure +- **[Quick Start](Quick-Start.md)** - Get up and running in 30 seconds + +### Configuration +- **[Configuration](Configuration.md)** - Environment variables, data storage options, and configuration examples +- **[AWS Configuration](AWS-Configuration.md)** - AWS profile setup, MFA configuration, and IAM permissions + +### Usage +- **[Usage Guide](Usage.md)** - Complete guide to using SGO features including search, views, and CSV exports +- **[FAQ](FAQ.md)** - Frequently asked questions + +### Help +- **[Troubleshooting](Troubleshooting.md)** - Common issues and solutions + +## Using These Docs + +### For New Users + +1. Start with [Quick Start](Quick-Start.md) to get SGO running +2. Read [AWS Configuration](AWS-Configuration.md) to set up your AWS profiles +3. Check [Usage Guide](Usage.md) to learn the features + +### For Troubleshooting + +1. Check [FAQ](FAQ.md) for common questions +2. Review [Troubleshooting](Troubleshooting.md) for specific error messages +3. If still stuck, open an issue with logs and details + +### For Configuration + +1. See [Configuration](Configuration.md) for all environment variables +2. Review configuration examples for your use case +3. Check platform-specific notes if on Windows/macOS/Linux + +## Contributing to Documentation + +Found an error or want to improve the docs? + +1. Edit the relevant markdown file +2. Submit a pull request +3. Follow the contribution guidelines in [CONTRIBUTING.md](../CONTRIBUTING.md) + +## Publishing to Wiki + +These markdown files can be: +- Used directly in the repository +- Published to GitHub/Codeberg wiki pages +- Rendered with any markdown viewer +- Hosted on documentation platforms + +For Codeberg, these files can be copied to the repository's wiki section. diff --git a/wiki/Troubleshooting.md b/wiki/Troubleshooting.md new file mode 100644 index 0000000..3cc011e --- /dev/null +++ b/wiki/Troubleshooting.md @@ -0,0 +1,442 @@ +# Troubleshooting + +## Container Issues + +### Cannot Connect to Docker Daemon + +**Error**: `Cannot connect to the Docker daemon at unix:///var/run/docker.sock` + +**Solution**: +```bash +# Linux: Start Docker service +sudo systemctl start docker +sudo systemctl enable docker # Start on boot + +# macOS/Windows: Open Docker Desktop application +# Ensure Docker Desktop is running in system tray +``` + +### Port Already in Use + +**Error**: `Bind for 0.0.0.0:5000 failed: port is already allocated` + +**Solution**: + +Option 1 - Use different port: +```bash +# Add to .env file +SGO_PORT=8080 + +# Restart container +docker-compose down && docker-compose up +``` + +Option 2 - Find and stop conflicting service: +```bash +# Linux/macOS: Find process using port 5000 +sudo lsof -i :5000 +sudo kill -9 + +# Windows: Find process using port 5000 +netstat -ano | findstr :5000 +taskkill /PID /F +``` + +### AWS Credentials Not Found in Container + +**Error**: `AWS config file not found` or `Permission denied: '/home/sgo/.aws/config'` + +**Solution**: + +1. Verify `.env` file exists and contains `AWS_CONFIG_PATH`: + ```bash + cat .env + # Should show: AWS_CONFIG_PATH=/home/yourusername/.aws + ``` + +2. Create `.env` file if missing: + ```bash + cat > .env << EOF + AWS_CONFIG_PATH=$HOME/.aws + PUID=$(id -u) + PGID=$(id -g) + EOF + ``` + +3. Verify AWS files exist on host: + ```bash + ls -la ~/.aws/ + # Should show: config, credentials + ``` + +4. Check file permissions: + ```bash + chmod 600 ~/.aws/credentials + chmod 644 ~/.aws/config + ``` + +5. Rebuild container: + ```bash + docker-compose down && docker-compose up --build + ``` + +### Permission Denied Errors + +**Error**: `PermissionError: [Errno 13] Permission denied: '/app/data/aws_export.db'` + +**Solution**: + +1. Set PUID/PGID to match your user: + ```bash + cat > .env << EOF + AWS_CONFIG_PATH=$HOME/.aws + PUID=$(id -u) + PGID=$(id -g) + EOF + ``` + +2. If using local directory mode, fix ownership: + ```bash + sudo chown -R $(id -u):$(id -g) ./data + ``` + +3. Or switch to Docker volume mode (default): + ```bash + # Use default docker-compose.yml + docker-compose down -v + docker-compose up --build + ``` + +### Database Locked Error + +**Error**: `database is locked` or `OperationalError: database is locked` + +**Solution**: + +1. Stop all running containers: + ```bash + docker-compose down + ``` + +2. If using local directory mode, remove lock file: + ```bash + rm -f ./data/aws_export.db-journal + ``` + +3. If problem persists, use Docker volume mode: + ```bash + # Edit docker-compose.yml to use volume instead of bind mount + docker-compose up --build + ``` + +### Container Keeps Restarting + +**Error**: Container exits immediately or restarts constantly + +**Solution**: + +1. Check container logs: + ```bash + docker-compose logs -f + ``` + +2. Look for error messages in logs and address specific issues + +3. Try running in debug mode: + ```bash + # Add to .env + DEBUG=true + + docker-compose down && docker-compose up + ``` + +## AWS Configuration Issues + +### No AWS Profiles Found + +**Error**: "No AWS profiles found" or empty profile list + +**Solution**: + +1. Verify `~/.aws/config` exists: + ```bash + cat ~/.aws/config + ``` + +2. Ensure profiles are properly formatted: + ```ini + [profile my-profile] + region = us-west-2 + + [default] + region = us-east-1 + ``` + +3. Check file permissions: + ```bash + chmod 644 ~/.aws/config + ``` + +4. Restart container to reload profiles: + ```bash + docker-compose restart + ``` + +### MFA Authentication Fails + +**Error**: "MFA authentication failed" or "Invalid MFA code" + +**Solution**: + +1. Verify MFA code is current: + - MFA codes expire every 30 seconds + - Generate a fresh code and try again + +2. Check `mfa_serial` is correct in `~/.aws/config`: + ```ini + [profile my-profile] + region = us-west-2 + mfa_serial = arn:aws:iam::123456789012:mfa/username + ``` + +3. Verify credentials are valid: + ```bash + aws sts get-caller-identity --profile my-profile + ``` + +4. Ensure MFA device is active in AWS IAM Console + +5. Check for clock skew: + - Ensure system time is accurate + - MFA is time-based and requires synchronized clocks + +### Import Fails with Access Denied + +**Error**: "Access Denied" or "UnauthorizedOperation" + +**Solution**: + +1. Verify IAM permissions (see [AWS Configuration](AWS-Configuration.md#required-iam-permissions)) + +2. Test permissions with AWS CLI: + ```bash + aws ec2 describe-instances --profile my-profile --max-results 1 + aws ec2 describe-security-groups --profile my-profile --max-results 1 + ``` + +3. Check if credentials have expired: + - Temporary credentials expire + - Try refreshing credentials + +4. Verify you're using the correct profile + +### Wrong Region / No Resources Found + +**Error**: Import succeeds but no EC2/SG data appears + +**Solution**: + +1. Check region in profile configuration: + ```ini + [profile my-profile] + region = us-west-2 # Ensure this is correct + ``` + +2. Remember: EC2 resources are region-specific + +3. Try specifying region explicitly: + ```bash + aws ec2 describe-instances --profile my-profile --region us-west-2 + ``` + +4. Import from correct region in profile + +## Application Issues + +### Search Returns No Results + +**Problem**: Search box doesn't show any results + +**Solution**: + +1. Verify data was imported successfully: + - Check import progress logs + - Look for "Import complete" messages + +2. Try different search terms: + - Minimum 2 characters required + - Try searching for "*" to see all resources + +3. Check filter buttons: + - Ensure you're not filtering out the type you're searching for + - Try "All Resources" button + +4. Refresh the page: + ``` + Ctrl+F5 or Cmd+Shift+R + ``` + +### Blank Page or UI Not Loading + +**Problem**: Browser shows blank page or incomplete UI + +**Solution**: + +1. Check browser console for JavaScript errors: + - Press F12 to open developer tools + - Look at Console tab for errors + +2. Clear browser cache and reload: + ``` + Ctrl+Shift+R (Windows/Linux) + Cmd+Shift+R (macOS) + ``` + +3. Try a different browser + +4. Check if application is running: + ```bash + docker-compose logs + ``` + +### CSV Export Not Working + +**Problem**: Export button doesn't download file + +**Solution**: + +1. Check browser popup blocker settings + +2. Look for downloads in browser's download location + +3. Try different browser + +4. Check browser console for JavaScript errors (F12) + +### Session Expired / MFA Required Again + +**Problem**: Asked for MFA again before 55 minutes + +**Solution**: + +- AWS sessions expire after 1 hour (we cache for 55 minutes) +- This is expected behavior for security +- Simply enter MFA code again to create new session + +## Performance Issues + +### Slow Import + +**Problem**: Import takes a very long time + +**Solution**: + +1. **Use parallel imports**: + - Select all profiles at once + - Click "Start Import" on each profile + - All imports run simultaneously + +2. **Check network connectivity to AWS**: + ```bash + ping ec2.us-west-2.amazonaws.com + ``` + +3. **Large accounts take longer**: + - Many EC2 instances/security groups increase import time + - This is normal + +### Slow Search + +**Problem**: Search is laggy or slow + +**Solution**: + +1. **Large datasets**: + - Thousands of resources can slow search + - This is a known limitation of client-side search + +2. **Use more specific search terms**: + - Narrow down results with longer search strings + +3. **Use regex carefully**: + - Complex regex patterns can be slow + - Disable regex for simple searches + +## Platform-Specific Issues + +### Windows + +**Problem**: Path issues or permission errors + +**Solution**: + +1. Use WSL2 for better compatibility: + ```powershell + # Install WSL2 and run commands from Linux environment + wsl --install + ``` + +2. Use Git Bash for running commands + +3. Ensure Docker Desktop is using WSL2 backend + +4. Use forward slashes in paths + +### macOS + +**Problem**: Docker can't access files in home directory + +**Solution**: + +1. Open Docker Desktop Settings +2. Go to Resources → File Sharing +3. Add your home directory to shared paths +4. Click "Apply & Restart" + +### Linux (Podman) + +**Problem**: `podman-compose` not found or issues + +**Solution**: + +1. Install podman-compose: + ```bash + # Fedora + sudo dnf install podman-compose + + # Ubuntu/Debian + pip3 install podman-compose + ``` + +2. Use `docker` command instead: + ```bash + # Podman supports docker CLI + alias docker=podman + docker-compose up --build + ``` + +3. Check SELinux contexts (Fedora/RHEL): + ```bash + # Already handled by :z flag in docker-compose.yml + # If issues persist, try: + sudo setenforce 0 # Temporarily permissive mode for testing + ``` + +## Getting Help + +If you continue to experience issues: + +1. **Check application logs**: + ```bash + docker-compose logs -f + ``` + +2. **Enable debug mode** (see [Configuration](Configuration.md#example-4-debug-mode)) + +3. **Open an issue** on the repository with: + - Error messages from logs + - Steps to reproduce + - Your environment (OS, Docker/Podman version) + - Relevant configuration (sanitized) + +4. **Check existing issues** - Your problem may already be solved diff --git a/wiki/Usage.md b/wiki/Usage.md new file mode 100644 index 0000000..214a1fa --- /dev/null +++ b/wiki/Usage.md @@ -0,0 +1,140 @@ +# Usage Guide + +## Search + +### Basic Search + +1. Type in the search box (minimum 2 characters) +2. Results appear instantly as you type +3. Filter by resource type using the buttons: **All Resources** | **EC2 Instances** | **Security Groups** + +### Regex Search + +Enable advanced pattern matching: + +1. Check the "Regex" box to use regular expressions +2. Examples: + - `^prod-.*-\d+$` - Finds names starting with "prod-" and ending with numbers + - `(dev|test|qa)` - Finds names containing dev, test, or qa + - `10\.0\.\d+\.\d+` - Finds IP addresses in the 10.0.x.x range + +## View Details + +### EC2 Instance View + +Click on any EC2 instance from search results to view: + +- **Main card** shows EC2 details: + - Instance ID + - Private IP Address + - State (running, stopped, etc.) + - Account ID and Name + - All AWS Tags +- **Nested cards** show all attached Security Groups with their details + +### Security Group View + +Click on any Security Group from search results to view: + +- **Main card** shows SG details: + - Group ID + - Group Name + - Wave Tag + - Ingress/Egress Rule Counts + - All AWS Tags +- **Nested cards** show all EC2 instances using this Security Group + +## View Security Group Rules + +When viewing security groups (either attached to an EC2 or directly): + +1. Click the **View Rules** button on any security group card +2. A modal opens showing all ingress and egress rules +3. Switch between **Ingress** and **Egress** tabs +4. Use the search box to filter rules by protocol, port, source, or description +5. Rules are displayed in a compact table format with: + - Protocol (TCP, UDP, ICMP, All) + - Port Range + - Source Type (CIDR, Security Group, Prefix List) + - Source (IP range or SG ID) + - Description + +## Navigation + +- Click **← Back to Search** to return to search results +- Perform a new search at any time +- Click outside the rules modal to close it + +## Export to CSV + +SGO provides comprehensive CSV export capabilities in multiple contexts. + +### Search Results Export + +- Click the **💾 Export** button in the view controls (top right) +- Exports all current search results with filters applied +- Includes: + - Type (EC2 or Security Group) + - Name + - ID (Instance ID or Group ID) + - Account + - State (for EC2) + - IP Address (for EC2) + - Security Groups count (for EC2) + - Wave + - Git Repository + - Git Organization + - Git File + +### EC2 Instance Details Export + +- Click the **💾 Export** button in any EC2 detail card +- Exports complete EC2 information including: + - Instance details (ID, name, state, IP, account info) + - All AWS tags + - Attached security groups with their details + +### Security Group Details Export + +- Click the **💾 Export** button in any SG detail card +- Exports complete SG information including: + - Group details (ID, name, wave, rule counts) + - All AWS tags + - Attached EC2 instances with their details + +### Security Group Rules Export + +- Click the **💾 Export** button in the rules modal +- Exports all ingress and egress rules with: + - Rule details (direction, protocol, ports, source) + - Group ID and name + - Account ID and name + - Git file and commit information from tags + +All exports include timestamps in filenames and proper CSV escaping for safe data handling. + +## Import and Refresh + +### Import from AWS + +1. From the Explorer page, click **Change Profiles** (or visit `/` directly) +2. Select AWS profiles to import +3. Enter MFA/TOTP codes for profiles that require authentication +4. Click **Start Import** for each profile +5. Watch real-time progress logs +6. Click **Done - Go to Explorer** when finished + +### Refresh Data + +- Click the **Refresh Data** button on the Explorer page +- Uses cached AWS sessions (valid for 55 minutes) +- No need to re-enter MFA codes during the session cache window +- Imports data from the same profiles used in the last import + +## Tips + +- **Parallel Imports**: Import multiple profiles simultaneously to save time +- **Session Cache**: Refresh multiple times within 55 minutes without re-entering MFA +- **Regex Power**: Use regex patterns for complex searches across your infrastructure +- **CSV Exports**: Export at any level (search results, individual resources, or rules) +- **Quick Navigation**: Use browser back/forward buttons or the built-in navigation links diff --git a/wiki/_Sidebar.md b/wiki/_Sidebar.md new file mode 100644 index 0000000..0fd2071 --- /dev/null +++ b/wiki/_Sidebar.md @@ -0,0 +1,20 @@ +# SGO Wiki + +## Getting Started +- [Home](Home.md) +- [Quick Start](Quick-Start.md) + +## Configuration +- [Configuration Options](Configuration.md) +- [AWS Configuration](AWS-Configuration.md) + +## Using SGO +- [Usage Guide](Usage.md) +- [FAQ](FAQ.md) + +## Help +- [Troubleshooting](Troubleshooting.md) + +## Resources +- [Main Repository](https://codeberg.org/edfig/SGO) +- [Report Issue](https://codeberg.org/edfig/SGO/issues)