Update blog posts: clean up formatting, fix typos, add new posts
- FFB.md: Fix typos, normalize tags/categories to lowercase, replace *** with --- for horizontal rules, trim trailing whitespace - contacting-me.md: Add mailto link for email, improve spacing and formatting - domainandemail.md: Fix typos (regitrar, programmaticaly, Malgun, teir), normalize tags/categories, clean up nested list formatting, remove redundant horizontal rules - rss.md: Reword intro sentence, clarify RSSHub description, fix trailing whitespace, remove duplicate selfhosted tag - stack.md: Convert frontmatter from TOML (+++) to YAML (---), normalize tags/categories, minor copy edits - rosterhash.md: New post about RosterHash fantasy football schedule viewer project - rr-rss-nag.md: New post explaining how to bypass RoyalRoad piracy nags in FreshRSS feeds using CSS selectors - security-groups.md: New post walking through the evolution of formatting AWS security groups in Terraform across three iterations
This commit is contained in:
Eduardo Figueroa
parent
00e856d024
commit
4e5386354f
8 changed files with 636 additions and 90 deletions
|
|
@ -1,41 +1,38 @@
|
||||||
|
|
||||||
|
|
||||||
---
|
---
|
||||||
title: "Fantasy Football Draft Tools"
|
title: "Fantasy Football Draft Tools"
|
||||||
date: "2024-08-29"
|
date: "2024-08-29"
|
||||||
tags:
|
tags:
|
||||||
- FantasyFootball
|
- reference
|
||||||
- Reference
|
- draft
|
||||||
categories:
|
categories:
|
||||||
- fantasyfootball
|
- fantasy-football
|
||||||
---
|
---
|
||||||
|
|
||||||
|
---
|
||||||
***
|
|
||||||
# Draft Tools
|
# Draft Tools
|
||||||
***
|
---
|
||||||
|
|
||||||
### > Beer Sheets
|
### > Beer Sheets
|
||||||
|
|
||||||
I used to recommend and use [Beer Sheets](https://footballabsurdity.com/draft-sheet-form/), a tool I found online and everytime I used it it gave good results. I had gotten good feed back from people I've recommneded it to. The original creator has moved on after the 2023 season and the new team has yet to prove they're as good.
|
I used to recommend and use [Beer Sheets](https://footballabsurdity.com/draft-sheet-form/), a tool I found online that gave good results every time I used it. I've gotten good feedback from people I've recommended it to. The original creator has moved on after the 2023 season and the new team has yet to prove they're as good.
|
||||||
|
|
||||||
#### Usage
|
#### Usage
|
||||||
|
|
||||||
You can go to their main page and just copy over the settings from your league. For examples here's a [direct link](https://footballabsurdity.com/draft-sheet-form/?teams=14&bn=5&qb=1&rb=2&wr=2&rwt=2&patd=6&rutd=6&retd=6&payd=0.04&ruyd=0.1&reyd=0.1&int=-1.0&rec=0.5&fum=-2.0) to the settings my family league is using.
|
You can go to their main page and just copy over the settings from your league. For example here's a [direct link](https://footballabsurdity.com/draft-sheet-form/?teams=14&bn=5&qb=1&rb=2&wr=2&rwt=2&patd=6&rutd=6&retd=6&payd=0.04&ruyd=0.1&reyd=0.1&int=-1.0&rec=0.5&fum=-2.0) to the settings my family league is using.
|
||||||
|
|
||||||
There doesn't seem to be support for PPFD (point per first down) so I just kept it at 1/2 ppr.
|
There doesn't seem to be support for PPFD (point per first down) so I just kept it at 1/2 ppr.
|
||||||
|
|
||||||
***
|
---
|
||||||
|
|
||||||
# Season Long Tools, Podcasts, Youtube
|
# Season Long Tools, Podcasts, Youtube
|
||||||
|
|
||||||
***
|
---
|
||||||
|
|
||||||
## Tools
|
## Tools
|
||||||
|
|
||||||
### > Fantasy Pros
|
### > Fantasy Pros
|
||||||
|
|
||||||
[FantasyPros](https://www.fantasypros.com/). I've used both free and the paid version. The free version is all you need while the paid tools offer quality-of-life (aka just figure it out for me) tools for trading and waiver wire pick ups.
|
[FantasyPros](https://www.fantasypros.com/). I've used both the free and paid versions. The free version is all you need while the paid tools offer quality-of-life (aka just figure it out for me) tools for trading and waiver wire pick ups.
|
||||||
|
|
||||||
The paid version does offer syncing multiple leagues but I just use [multiple throwaway accounts](/posts/domainandemail) to bypass the limit.
|
The paid version does offer syncing multiple leagues but I just use [multiple throwaway accounts](/posts/domainandemail) to bypass the limit.
|
||||||
|
|
||||||
|
|
@ -51,7 +48,7 @@ The paid version does offer syncing multiple leagues but I just use [multiple th
|
||||||
|
|
||||||
### > Late Round with JJ Zachariazon
|
### > Late Round with JJ Zachariazon
|
||||||
|
|
||||||
[LateRound](https://lateround.com/#newsletter) is a podcast which is basically the exact same as the newsletter. Newsletter will usually have +1 bit of info to encourage getting it, e.g podcast covers 10 Waiver Wire Pickups while newsletter has 11. Data backed and pretty accurate.
|
[LateRound](https://lateround.com/#newsletter) is a podcast that covers basically the same content as the newsletter. Newsletter will usually have +1 bit of info to encourage getting it, e.g podcast covers 10 Waiver Wire Pickups while newsletter has 11. Data backed and pretty accurate.
|
||||||
|
|
||||||
### > Rams Brothers
|
### > Rams Brothers
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -4,11 +4,16 @@ description: "If you need to reach me."
|
||||||
date: "2019-02-28"
|
date: "2019-02-28"
|
||||||
---
|
---
|
||||||
|
|
||||||
***
|
---
|
||||||
|
|
||||||
### E-mail
|
### E-mail
|
||||||
Eddie@edfig.dev
|
|
||||||
***
|
[Eddie@edfig.dev](mailto:Eddie@edfig.dev)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
### Seldom used Socials
|
### Seldom used Socials
|
||||||
|
|
||||||
[@edfig@mastodon.social](https://mastodon.social/@edfig)
|
[@edfig@mastodon.social](https://mastodon.social/@edfig)
|
||||||
|
|
||||||
[@edfig.dev](https://bsky.app/profile/edfig.dev)
|
[@edfig.dev](https://bsky.app/profile/edfig.dev)
|
||||||
|
|
|
||||||
|
|
@ -2,14 +2,15 @@
|
||||||
title: "Custom Domain and Emails"
|
title: "Custom Domain and Emails"
|
||||||
date: "2024-10-07"
|
date: "2024-10-07"
|
||||||
tags:
|
tags:
|
||||||
- DNS
|
- dns
|
||||||
|
- email
|
||||||
categories:
|
categories:
|
||||||
- DIY
|
- selfhosted
|
||||||
---
|
---
|
||||||
|
|
||||||
# What is this?
|
# What is this?
|
||||||
|
|
||||||
Let's say you wanted to buy a domain like `edfig.dev`. You can host a personal blog at this address. Once you buy the domain not only can you host content but you with a bit more tinkering you can send and receive emails with the domain.
|
Let's say you wanted to buy a domain like `edfig.dev`. You can host a personal blog at this address. Once you buy the domain, not only can you host content, but with a bit more tinkering you can send and receive emails with it.
|
||||||
|
|
||||||
You can email `eddie@edfig.dev` or `admin@edfig.dev` and that email will make its way to my inbox. You can set up rules to handle specific addresses too.
|
You can email `eddie@edfig.dev` or `admin@edfig.dev` and that email will make its way to my inbox. You can set up rules to handle specific addresses too.
|
||||||
|
|
||||||
|
|
@ -19,33 +20,30 @@ You'll need to create accounts for the following:
|
||||||
|
|
||||||
# Setting it up
|
# Setting it up
|
||||||
|
|
||||||
---
|
|
||||||
## 1. The Domain
|
## 1. The Domain
|
||||||
---
|
|
||||||
You can buy a domain from any regitrar, I recommend [PorkBun](https://porkbun.com/) or [Cloudflare](https://www.cloudflare.com/products/registrar/). I'll be using Porkbun for this discussion.
|
|
||||||
|
|
||||||
Pricing will depend on the name and what TLD (the `@something.com` part). I occasionly run into issues with sites not recognizing `eddie@fig.systems` as a valid email address because it's a lesser known domain.
|
You can buy a domain from any registrar, I recommend [PorkBun](https://porkbun.com/) or [Cloudflare](https://www.cloudflare.com/products/registrar/). I'll be using Porkbun for this discussion.
|
||||||
|
|
||||||
Once you have it you can enter DNS records to to where you host stuff or start using it for email.
|
Pricing will depend on the name and what TLD (the `.com` part). I occasionally run into issues with sites not recognizing `eddie@fig.systems` as a valid email address because it's a lesser known domain.
|
||||||
|
|
||||||
|
Once you have it you can enter DNS records to where you host stuff or start using it for email.
|
||||||
|
|
||||||
---
|
|
||||||
## 2. The Email
|
## 2. The Email
|
||||||
---
|
|
||||||
Email is one of those things you shouldn't host yourself, it's very annoying. But luckily there's services out there that take care of most of the hassle. [MailGun](https://www.mailgun.com/) and [SendGrid](https://sendgrid.com/en-us) are two such services. I'll be using Mailgun here.
|
|
||||||
|
|
||||||
With Malgun I can:
|
Email is one of those things you shouldn't host yourself, it's very annoying. But luckily there are services out there that take care of most of the hassle. [MailGun](https://www.mailgun.com/) and [SendGrid](https://sendgrid.com/en-us) are two such services. I'll be using Mailgun here.
|
||||||
|
|
||||||
|
With Mailgun I can:
|
||||||
- Receive emails at custom email addresses with my domain
|
- Receive emails at custom email addresses with my domain
|
||||||
- Route those emails based on rules
|
- Route those emails based on rules
|
||||||
- - e.g. Emails sent to `no-rreply@edfig.dev` are completely dropped
|
- e.g. Emails sent to `no-reply@edfig.dev` are completely dropped
|
||||||
- Send emails AS those email addresses through gmail
|
- Send emails AS those email addresses through gmail
|
||||||
- - Receive an email at `admin@edfig.dev` at my regular gmail account and reply as `admin@fig.systems`
|
- Receive an email at `admin@edfig.dev` at my regular gmail account and reply as `admin@fig.systems`
|
||||||
- Use their API to programmaticaly send emails
|
- Use their API to programmatically send emails
|
||||||
- Use their SMTP servers to send as custom email addresses
|
- Use their SMTP servers to send as custom email addresses
|
||||||
- - My self hosted services send notification emails as `no-reply@fig.systems` or as `service_name@fig.systems`
|
- My self hosted services send notification emails as `no-reply@fig.systems` or as `service_name@fig.systems`
|
||||||
|
|
||||||
---
|
|
||||||
## 3. Setting up DNS
|
## 3. Setting up DNS
|
||||||
---
|
|
||||||
1. Buy a domain at porkbun.
|
1. Buy a domain at porkbun.
|
||||||
|
|
||||||
Pick your favorite. I'll be using `figgy.foo` for this, there was a good deal on it.
|
Pick your favorite. I'll be using `figgy.foo` for this, there was a good deal on it.
|
||||||
|
|
@ -61,7 +59,7 @@ You'll be provided with records for sending, receiving, and tracking.
|
||||||
|
|
||||||
In Porkbun Domain Management select `DNS` when you hover over your new domain.
|
In Porkbun Domain Management select `DNS` when you hover over your new domain.
|
||||||
|
|
||||||
Copy the entries over. Make sure the Types match and that you leave off the `figgy.foo` portion in the `host` field in porkbun. Anything you add in the host field will automatically append your domain to the end of it. If the field is just `figgy.foo` then the host field blank.
|
Copy the entries over. Make sure the Types match and that you leave off the `figgy.foo` portion in the `host` field in porkbun. Anything you add in the host field will automatically append your domain to the end of it. If the field is just `figgy.foo` then leave the host field blank.
|
||||||
|
|
||||||
Copy all the Value fields from Mailgun to the Answer field in Porkbun and then click on Verify at the top right. You should see the status change to Active.
|
Copy all the Value fields from Mailgun to the Answer field in Porkbun and then click on Verify at the top right. You should see the status change to Active.
|
||||||
|
|
||||||
|
|
@ -69,15 +67,13 @@ This is what your records in porkbun should look like.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
---
|
|
||||||
## 4. Setting up Mailgun
|
## 4. Setting up Mailgun
|
||||||
---
|
|
||||||
|
|
||||||
### Routing emails.
|
### Routing emails.
|
||||||
|
|
||||||
1. Go to Send -> Receiving and Create a Route.
|
1. Go to Send -> Receiving and Create a Route.
|
||||||
|
|
||||||
2. Expression Type -> Match Recepient
|
2. Expression Type -> Match Recipient
|
||||||
|
|
||||||
- Enter `admin@figgy.foo`
|
- Enter `admin@figgy.foo`
|
||||||
|
|
||||||
|
|
@ -86,7 +82,7 @@ This is what your records in porkbun should look like.
|
||||||
4. Set priority to 50 so you have space to add future routes before or after this route.
|
4. Set priority to 50 so you have space to add future routes before or after this route.
|
||||||
5. Add a simple description like "send to gmail" and Create the Route.
|
5. Add a simple description like "send to gmail" and Create the Route.
|
||||||
|
|
||||||
At the free teir you can only have 5 routes total. I only use the following:
|
At the free tier you can only have 5 routes total. I only use the following:
|
||||||
|
|
||||||
1. Match `no-reply@figgy.foo`, Store and Notify and Stop processing.
|
1. Match `no-reply@figgy.foo`, Store and Notify and Stop processing.
|
||||||
2. Match `family@figgy.foo`, forward that email to multiple family members.
|
2. Match `family@figgy.foo`, forward that email to multiple family members.
|
||||||
|
|
@ -95,4 +91,4 @@ At the free teir you can only have 5 routes total. I only use the following:
|
||||||
- much friendlier address than what they make for you.
|
- much friendlier address than what they make for you.
|
||||||
4. A catch all final route that just forwards to my personal address.
|
4. A catch all final route that just forwards to my personal address.
|
||||||
|
|
||||||
Number 4 is where most of the magic and utility of setting all this up happens. I can give out unlimited custom email address and I'll know who sent them by the address. That is if I give out `businessName@fig.systems` I can later use that in gmail to filter, block, or search for anything related to that business. I can even see who sold my info if I start getting spam from that address.
|
Number 4 is where most of the magic and utility of setting all this up happens. I can give out unlimited custom email addresses and I'll know who sent them by the address. That is, if I give out `businessName@fig.systems` I can later use that in gmail to filter, block, or search for anything related to that business. I can even see who sold my info if I start getting spam from that address.
|
||||||
|
|
|
||||||
24
content/posts/rosterhash.md
Normal file
24
content/posts/rosterhash.md
Normal file
|
|
@ -0,0 +1,24 @@
|
||||||
|
---
|
||||||
|
title: "RosterHash: fantasy football schedule viewer"
|
||||||
|
date: "2024-11-20"
|
||||||
|
tags:
|
||||||
|
- fantasy-football
|
||||||
|
- selfhosted
|
||||||
|
categories:
|
||||||
|
- projects
|
||||||
|
---
|
||||||
|
|
||||||
|
I joined another league this year.
|
||||||
|
|
||||||
|
I was losing track of who played when and what league.
|
||||||
|
|
||||||
|
So I made ~~GameTime (nope, that's taken)~~ [RosterHash](https://rosterhash.edfig.dev)!
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Enter your Sleeper username and away you go.
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- Can save up to 4 favorite teams for checking when they play and the score
|
||||||
|
- Shows your players per league per game and color codes it all.
|
||||||
|
- Completed games auto-collapse and get out of the way
|
||||||
116
content/posts/rr-rss-nag.md
Normal file
116
content/posts/rr-rss-nag.md
Normal file
|
|
@ -0,0 +1,116 @@
|
||||||
|
---
|
||||||
|
title: "Bypassing RoyalRoad's piracy nags in RSS Feeds"
|
||||||
|
date: "2024-11-20"
|
||||||
|
tags:
|
||||||
|
- rss
|
||||||
|
- freshrss
|
||||||
|
categories:
|
||||||
|
- selfhosted
|
||||||
|
---
|
||||||
|
# Issue
|
||||||
|
|
||||||
|
[Royal Road](https://www.royalroad.com/home) likes to annoy pirates. This is (arguably) good.
|
||||||
|
|
||||||
|
Royal Road doesn't care if they annoy RSS users. This is **bad**.
|
||||||
|
|
||||||
|
Here's a walkthrough of the problem and the fix.
|
||||||
|
|
||||||
|
### The Problem:
|
||||||
|
|
||||||
|
First, let's look at the full picture of why this is happening.
|
||||||
|
|
||||||
|
The Original Website HTML (Simplified)
|
||||||
|
|
||||||
|
When you visit the Royal Road chapter in your browser, the full page's HTML looks something like this. Your browser loads the <head> section and the <body> section.
|
||||||
|
```HTML
|
||||||
|
|
||||||
|
<html>
|
||||||
|
<head>
|
||||||
|
<style>
|
||||||
|
.cjZhYjNmYjZkZmFjZTQ2YTk4OWQwYjRiMjRjZDQyOGRl {
|
||||||
|
display: none;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
|
||||||
|
...
|
||||||
|
|
||||||
|
<body>
|
||||||
|
<div class="chapter-content">
|
||||||
|
|
||||||
|
<p class="cnMxYzY0ZjllNmVj...">
|
||||||
|
<span style="font-weight: 400">Nathan got the message...</span>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<p class="cnNiYTMwZmE4YjE2..."> </p>
|
||||||
|
|
||||||
|
<p class="cnNiOWQ0MDU1MDA2...">
|
||||||
|
<span style="font-weight: 400">Sarya waved her hand...</span>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<p class="cnM0NjAwNWU4Y2Vl..."> </p>
|
||||||
|
|
||||||
|
<span class="cjZhYjNmYjZkZmFjZTQ2YTk4OWQwYjRiMjRjZDQyOGRl">
|
||||||
|
<br>The narrative has been stolen; if detected on Amazon, report...<br>
|
||||||
|
</span>
|
||||||
|
|
||||||
|
</div>
|
||||||
|
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
On the live website, your browser reads the `<style>` tag in the `<head>` and knows to hide the spam `<span>`. You never see it.
|
||||||
|
|
||||||
|
### What FreshRSS Sees (The Problem)
|
||||||
|
|
||||||
|
I've told FreshRSS to only grab the content from `.chapter-content` which is the actual content of a post. So, FreshRSS requests the page and then scrapes only this part:
|
||||||
|
|
||||||
|
```html
|
||||||
|
|
||||||
|
<p class="cnMxYzY0ZjllNmVj...">
|
||||||
|
<span style="font-weight: 400">Nathan got the message...</span>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<p class="cnNiYTMwZmE4YjE2..."> </p>
|
||||||
|
|
||||||
|
<p class="cnNiOWQ0MDU1MDA2...">
|
||||||
|
<span style="font-weight: 400">Sarya waved her hand...</span>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<p class="cnM0NjAwNWU4Y2Vl..."> </p>
|
||||||
|
|
||||||
|
<span class="cjZhYjNmYjZkZmFjZTQ2YTk4OWQwYjRiMjRjZDQyOGRl">
|
||||||
|
<br>The narrative has been stolen; if detected on Amazon, report...<br>
|
||||||
|
</span>
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
Since FreshRSS never saw the `<head>` or the `<style>` tag, it has no idea it's supposed to hide the spam `<span>`. It just displays all the text it found, resulting in this output in your feed reader:
|
||||||
|
|
||||||
|
Nathan got the message...
|
||||||
|
|
||||||
|
Sarya waved her hand...
|
||||||
|
|
||||||
|
The narrative has been stolen; if detected on Amazon, report...
|
||||||
|
|
||||||
|
This is the core of the issue: the content is hidden by a CSS rule that FreshRSS isn't loading, and the class names are random, so you can't just block the class.
|
||||||
|
|
||||||
|
### The Fix: CSS Selectors
|
||||||
|
|
||||||
|
You need to tell FreshRSS how to remove the unwanted elements based on their structure, not their random class names.
|
||||||
|
|
||||||
|
Go to: **Advanced** -> **CSS selector of the elements to remove**.
|
||||||
|
Paste this in the box:
|
||||||
|
|
||||||
|
```css
|
||||||
|
.chapter-content > span
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
This selector targets any `<span>` element that is a direct child (using `>`) of `.chapter-content`.
|
||||||
|
|
||||||
|
The spam text `<span class="cjZhY...">...</span>` matches this rule.
|
||||||
|
|
||||||
|
The actual story text `<span style="font-weight: 400">...</span>` is safe because it's a "grandchild" (it's inside a `<p>` tag), not a direct child.
|
||||||
|
|
@ -3,7 +3,6 @@ title: "RSS - Still Alive"
|
||||||
date: "2024-10-05"
|
date: "2024-10-05"
|
||||||
tags:
|
tags:
|
||||||
- rss
|
- rss
|
||||||
- selfhosted
|
|
||||||
categories:
|
categories:
|
||||||
- selfhosted
|
- selfhosted
|
||||||
---
|
---
|
||||||
|
|
@ -12,18 +11,17 @@ categories:
|
||||||
|
|
||||||
I like having a centralized curated list of content. I'd rather go to a single page to catch up on new content instead of visiting or remembering to visit a bunch of different sites. I also don't like having to deal with cookies and sites tracking my every move.
|
I like having a centralized curated list of content. I'd rather go to a single page to catch up on new content instead of visiting or remembering to visit a bunch of different sites. I also don't like having to deal with cookies and sites tracking my every move.
|
||||||
|
|
||||||
Intro: [FreshRSS](https://freshrss.org/index.html) and [RSSHub](https://docs.rsshub.app/) with special guest [RSSHub-Radar](https://github.com/DIYgod/RSSHub-Radar)
|
I use: [FreshRSS](https://freshrss.org/index.html), [RSSHub](https://docs.rsshub.app/), and [RSSHub-Radar](https://github.com/DIYgod/RSSHub-Radar)
|
||||||
|
|
||||||
I used to only use RSS for blogs and other text based content but with the above tools I can RSS-ify most anything.
|
I used to only use RSS for blogs and other text based content but with the above tools I can RSS-ify most anything.
|
||||||
|
|
||||||
|
## The Flow of Content
|
||||||
## The Flow of Content ##
|
|
||||||
|
|
||||||
**FreshRSS** is the rss aggregator and can be used as the reader either on desktop or as a PWA on mobile. I host my own at [feeds.fig.systems](https://feeds.fig.systems). It's got a few slick themes and has options to [scrape webpages with x-paths](https://danq.me/2022/09/27/freshrss-xpath/). You provide a URL and use elements to select what you'd like to create your feed from, that's a lot of work per feed.
|
**FreshRSS** is the rss aggregator and can be used as the reader either on desktop or as a PWA on mobile. I host my own at [feeds.fig.systems](https://feeds.fig.systems). It's got a few slick themes and has options to [scrape webpages with x-paths](https://danq.me/2022/09/27/freshrss-xpath/). You provide a URL and use elements to select what you'd like to create your feed from, that's a lot of work per feed.
|
||||||
|
|
||||||
That was my first go at RSS-ifying everything until I learned about **RSSHub** which does the same thing but completely for you. I have my own hosted RSSHub.fig.systems. They provide pre-made "routes" which make turning many common content sources into feeds.
|
That was my first go at RSS-ifying everything until I learned about **RSSHub** which does the same thing but handles it automatically. I host my own instance at RSSHub.fig.systems. They provide pre-made "routes" which make turning many common content sources into feeds.
|
||||||
|
|
||||||
For example I could add the feed ```rsshub.fig.systems/youtube/user/linustechtips/``` to feeds.fig.systems and I'd get a new entry every time that channel had a new feed.
|
For example I could add the feed `rsshub.fig.systems/youtube/user/linustechtips/` to feeds.fig.systems and I'd get a new entry every time that channel uploads a new video.
|
||||||
|
|
||||||
Having to look up the routes can be annoying. Enter **RSSHub-Radar**, a nice browser extension that can automatically detect and provide the route for a given page you'd like to rss-ify. The extension can also be configured to format the url for whatever rss aggregator you use, FreshRSS or otherwise.
|
Having to look up the routes can be annoying. Enter **RSSHub-Radar**, a nice browser extension that can automatically detect and provide the route for a given page you'd like to rss-ify. The extension can also be configured to format the url for whatever rss aggregator you use, FreshRSS or otherwise.
|
||||||
|
|
||||||
|
|
|
||||||
408
content/posts/security-groups.md
Normal file
408
content/posts/security-groups.md
Normal file
|
|
@ -0,0 +1,408 @@
|
||||||
|
---
|
||||||
|
title: "Formatting AWS Security Groups for a VMware Migration"
|
||||||
|
date: "2025-02-05"
|
||||||
|
tags:
|
||||||
|
- terraform
|
||||||
|
- aws
|
||||||
|
- migration
|
||||||
|
categories:
|
||||||
|
- selfhosted
|
||||||
|
---
|
||||||
|
|
||||||
|
# The Problem
|
||||||
|
|
||||||
|
At work we're in the middle of a large lift and shift migration from VMware to AWS (for the same reason everyone is). Hundreds of servers across multiple departments, moved in waves.
|
||||||
|
|
||||||
|
The firewall rules for these servers come from everywhere. Palo Alto firewalls, host-based firewalls, department-specific switches, department-specific IT teams, random appliances that predate much of the current staff. Years of accumulated rules from multiple sources, and now they all need to become AWS security groups.
|
||||||
|
|
||||||
|
I needed to figure out how to format these rules in Terraform so that:
|
||||||
|
1. Coworkers completely new to IaC could read them
|
||||||
|
2. I could maintain them without losing my mind as rule counts climbed
|
||||||
|
3. PRs were reviewable
|
||||||
|
|
||||||
|
This is how the format evolved over three iterations.
|
||||||
|
|
||||||
|
# Iteration 1: Inline Rules
|
||||||
|
|
||||||
|
The most straightforward way to write a security group. Everything in one block.
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
resource "aws_security_group" "web_server" {
|
||||||
|
name = "web-server"
|
||||||
|
description = "SG for web-server"
|
||||||
|
vpc_id = var.vpc_id
|
||||||
|
|
||||||
|
ingress {
|
||||||
|
description = "HTTPS from campus"
|
||||||
|
from_port = 443
|
||||||
|
to_port = 443
|
||||||
|
protocol = "tcp"
|
||||||
|
cidr_blocks = ["10.0.0.0/24"]
|
||||||
|
}
|
||||||
|
|
||||||
|
ingress {
|
||||||
|
description = "SSH from admin subnet"
|
||||||
|
from_port = 22
|
||||||
|
to_port = 22
|
||||||
|
protocol = "tcp"
|
||||||
|
cidr_blocks = ["10.100.0.0/24"]
|
||||||
|
}
|
||||||
|
|
||||||
|
egress {
|
||||||
|
description = "Allow all outbound"
|
||||||
|
from_port = 0
|
||||||
|
to_port = 0
|
||||||
|
protocol = "-1"
|
||||||
|
cidr_blocks = ["0.0.0.0/0"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
This works fine for a server with 3-4 rules and is the first example you usually come across if you search for "ec2 firewalls". It's easy to read and easy to explain to someone who's never seen Terraform before.
|
||||||
|
|
||||||
|
The problem is that any change to any inline rule forces Terraform to evaluate the entire security group. Add a CIDR to one ingress block and the plan output gets noisy. It also doesn't play well with `for_each` if you want to loop over CIDRs for a single port.
|
||||||
|
|
||||||
|
# Iteration 2: Separate Rule Resources
|
||||||
|
|
||||||
|
Breaking the rules out into their own resources using `aws_vpc_security_group_ingress_rule` and `aws_vpc_security_group_egress_rule`.
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
resource "aws_security_group" "web_server" {
|
||||||
|
description = "SG for web-server"
|
||||||
|
vpc_id = var.vpc_id
|
||||||
|
|
||||||
|
tags = {
|
||||||
|
Name = "web-server"
|
||||||
|
Source = "Palo Alto Firewall"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Egress
|
||||||
|
resource "aws_vpc_security_group_egress_rule" "web_server_allow_all_outbound" {
|
||||||
|
security_group_id = aws_security_group.web_server.id
|
||||||
|
ip_protocol = "-1"
|
||||||
|
cidr_ipv4 = "0.0.0.0/0"
|
||||||
|
|
||||||
|
tags = {
|
||||||
|
Name = "allow-all-outbound"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# HTTPS from campus
|
||||||
|
resource "aws_vpc_security_group_ingress_rule" "web_server_https_443" {
|
||||||
|
for_each = var.https_443_cidrs
|
||||||
|
security_group_id = aws_security_group.web_server.id
|
||||||
|
cidr_ipv4 = each.key
|
||||||
|
description = each.value
|
||||||
|
ip_protocol = "tcp"
|
||||||
|
from_port = 443
|
||||||
|
to_port = 443
|
||||||
|
|
||||||
|
tags = {
|
||||||
|
Name = "HTTPS-443-${replace(each.key, "/", "-")}"
|
||||||
|
Rule = "tcp-443"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# SSH from admin subnet
|
||||||
|
resource "aws_vpc_security_group_ingress_rule" "web_server_ssh_22" {
|
||||||
|
for_each = var.ssh_22_cidrs
|
||||||
|
security_group_id = aws_security_group.web_server.id
|
||||||
|
cidr_ipv4 = each.key
|
||||||
|
description = each.value
|
||||||
|
ip_protocol = "tcp"
|
||||||
|
from_port = 22
|
||||||
|
to_port = 22
|
||||||
|
|
||||||
|
tags = {
|
||||||
|
Name = "SSH-22-${replace(each.key, "/", "-")}"
|
||||||
|
Rule = "tcp-22"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
With variables like:
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
variable "https_443_cidrs" {
|
||||||
|
type = map(string)
|
||||||
|
default = {
|
||||||
|
"10.0.0.0/24" = "Campus network"
|
||||||
|
"10.100.0.0/24" = "Admin subnet"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
variable "ssh_22_cidrs" {
|
||||||
|
type = map(string)
|
||||||
|
default = {
|
||||||
|
"10.100.0.0/24" = "Admin subnet"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
This is better. Each rule is its own resource so Terraform plans are cleaner. Adding a CIDR to a port only shows that one rule changing. The `for_each` on a map of CIDR-to-description means you can see at a glance what each IP range is for.
|
||||||
|
|
||||||
|
I used this format for the 2nd wave. It worked. But by the next few waves we were moving more servers per wave and each server had its own set of variables. The variable files were getting long and hard to cross-reference with the rules.
|
||||||
|
|
||||||
|
Everything was also moved into a `$WORKSPACE/modules/security-groups/` directory to keep it organized. One file per server's rules, one file per server's variables.
|
||||||
|
|
||||||
|
# Iteration 3: Locals with Structured Data
|
||||||
|
|
||||||
|
By the time we were moving double digit servers per wave, the variable-per-port approach was getting hard to maintain. Too many variable files, too much scrolling back and forth to understand what a server's rules actually looked like.
|
||||||
|
|
||||||
|
I switched to using `locals` with a structured list. All the rules for a server live in one block. Each entry defines the port, protocol, and every CIDR that needs access on that port.
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
locals {
|
||||||
|
web_server_ports = [
|
||||||
|
# HTTPS
|
||||||
|
{
|
||||||
|
protocol = "tcp"
|
||||||
|
from = 443
|
||||||
|
to = 443
|
||||||
|
name = "https-443"
|
||||||
|
cidrs = {
|
||||||
|
"10.0.0.0/24" = "Campus network"
|
||||||
|
"10.100.0.0/24" = "Admin subnet"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
# SSH
|
||||||
|
{
|
||||||
|
protocol = "tcp"
|
||||||
|
from = 22
|
||||||
|
to = 22
|
||||||
|
name = "ssh-22"
|
||||||
|
cidrs = {
|
||||||
|
"10.100.0.0/24" = "Admin subnet"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
# RDP
|
||||||
|
{
|
||||||
|
protocol = "tcp"
|
||||||
|
from = 3389
|
||||||
|
to = 3389
|
||||||
|
name = "rdp-3389"
|
||||||
|
cidrs = {
|
||||||
|
"10.100.0.0/24" = "Admin subnet"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
# HTTP
|
||||||
|
{
|
||||||
|
protocol = "tcp"
|
||||||
|
from = 80
|
||||||
|
to = 80
|
||||||
|
name = "http-80"
|
||||||
|
cidrs = {
|
||||||
|
"10.0.0.0/24" = "Campus network"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
# Flatten into individual rules
|
||||||
|
web_server_rules = flatten([
|
||||||
|
for port_config in local.web_server_ports : [
|
||||||
|
for cidr, description in port_config.cidrs : {
|
||||||
|
key = "${port_config.name}-${replace(cidr, "/", "-")}"
|
||||||
|
protocol = port_config.protocol
|
||||||
|
from_port = port_config.from
|
||||||
|
to_port = port_config.to
|
||||||
|
cidr = cidr
|
||||||
|
description = description
|
||||||
|
rule_name = port_config.name
|
||||||
|
}
|
||||||
|
]
|
||||||
|
])
|
||||||
|
|
||||||
|
# How many rules total
|
||||||
|
web_server_total_rule_count = length(local.web_server_rules)
|
||||||
|
|
||||||
|
# How many SGs needed (AWS has a rules-per-SG limit)
|
||||||
|
web_server_sg_count = max(1, ceil(local.web_server_total_rule_count / var.max_rules_per_sg))
|
||||||
|
|
||||||
|
# Chunk rules across SGs
|
||||||
|
web_server_rules_chunked = {
|
||||||
|
for sg_index in range(local.web_server_sg_count) : sg_index => [
|
||||||
|
for rule_index in range(
|
||||||
|
sg_index * var.max_rules_per_sg,
|
||||||
|
min((sg_index + 1) * var.max_rules_per_sg, local.web_server_total_rule_count)
|
||||||
|
) : local.web_server_rules[rule_index]
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The security group itself handles overflow automatically. If a server has more rules than AWS allows per SG, it creates additional SGs and distributes the rules across them. Neither I nor anyone in my team had to count rules to make sure they were split across security groups evenly. It all gets generated dynamically.
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
# Primary SG
|
||||||
|
resource "aws_security_group" "web_server" {
|
||||||
|
name = "web-server"
|
||||||
|
description = "SG for web-server"
|
||||||
|
vpc_id = var.vpc_id
|
||||||
|
|
||||||
|
lifecycle {
|
||||||
|
create_before_destroy = true
|
||||||
|
}
|
||||||
|
|
||||||
|
tags = {
|
||||||
|
Name = "web-server"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Overflow SGs (created only if needed)
|
||||||
|
resource "aws_security_group" "web_server_overflow" {
|
||||||
|
for_each = { for idx in range(1, local.web_server_sg_count) : idx => idx }
|
||||||
|
|
||||||
|
name = "web-server-overflow-${each.value}"
|
||||||
|
description = "SG for web-server (Overflow ${each.value})"
|
||||||
|
vpc_id = var.vpc_id
|
||||||
|
|
||||||
|
lifecycle {
|
||||||
|
create_before_destroy = true
|
||||||
|
}
|
||||||
|
|
||||||
|
tags = {
|
||||||
|
Name = "web-server-overflow-${each.value}"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Egress (primary SG only)
|
||||||
|
resource "aws_vpc_security_group_egress_rule" "web_server_allow_all_outbound" {
|
||||||
|
security_group_id = aws_security_group.web_server.id
|
||||||
|
ip_protocol = "-1"
|
||||||
|
cidr_ipv4 = "0.0.0.0/0"
|
||||||
|
|
||||||
|
tags = {
|
||||||
|
Name = "allow-all-outbound"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Ingress for primary SG
|
||||||
|
resource "aws_vpc_security_group_ingress_rule" "web_server_ingress" {
|
||||||
|
for_each = {
|
||||||
|
for rule in local.web_server_rules_chunked[0] :
|
||||||
|
rule.key => rule
|
||||||
|
}
|
||||||
|
|
||||||
|
security_group_id = aws_security_group.web_server.id
|
||||||
|
cidr_ipv4 = each.value.cidr
|
||||||
|
description = each.value.description
|
||||||
|
ip_protocol = each.value.protocol
|
||||||
|
from_port = each.value.protocol == "-1" ? null : each.value.from_port
|
||||||
|
to_port = each.value.protocol == "-1" ? null : each.value.to_port
|
||||||
|
|
||||||
|
tags = {
|
||||||
|
Name = each.value.key
|
||||||
|
Rule = each.value.rule_name
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Ingress for overflow SGs
|
||||||
|
resource "aws_vpc_security_group_ingress_rule" "web_server_overflow_ingress" {
|
||||||
|
for_each = merge([
|
||||||
|
for sg_index, sg in aws_security_group.web_server_overflow : {
|
||||||
|
for rule in local.web_server_rules_chunked[sg_index] :
|
||||||
|
"${sg_index}-${rule.key}" => {
|
||||||
|
sg_id = sg.id
|
||||||
|
cidr = rule.cidr
|
||||||
|
description = rule.description
|
||||||
|
protocol = rule.protocol
|
||||||
|
from_port = rule.from_port
|
||||||
|
to_port = rule.to_port
|
||||||
|
key = rule.key
|
||||||
|
rule_name = rule.rule_name
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]...)
|
||||||
|
|
||||||
|
security_group_id = each.value.sg_id
|
||||||
|
cidr_ipv4 = each.value.cidr
|
||||||
|
description = each.value.description
|
||||||
|
ip_protocol = each.value.protocol
|
||||||
|
from_port = each.value.protocol == "-1" ? null : each.value.from_port
|
||||||
|
to_port = each.value.protocol == "-1" ? null : each.value.to_port
|
||||||
|
|
||||||
|
tags = {
|
||||||
|
Name = each.value.key
|
||||||
|
Rule = each.value.rule_name
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Adding a new server means copying the template, doing a find-and-replace on the server name, and filling in the `ports` list. The SG resource, egress, overflow, and ingress logic are all identical across servers. The only thing that changes is the data in `locals`.
|
||||||
|
|
||||||
|
The big win for PR reviews is that the `ports` local reads like a table. You can look at it and immediately see what ports are open and to whom without having to mentally reconstruct it from scattered variable files.
|
||||||
|
|
||||||
|
|
||||||
|
# Standard Security Groups
|
||||||
|
|
||||||
|
While all the above handles per-server rules, we noticed early on that a lot of rules were the same across every server. RDP from the admin subnet, SSH from the admin subnet, ICMP from campus, etc. Every single server had these and we were duplicating them everywhere.
|
||||||
|
|
||||||
|
So we created a separate shared module: `$ROOT_OF_MONOREPO/modules/standard-securitygroups`. It only takes a `vpc_id` as input and creates a set of reusable security groups that any server can reference.
|
||||||
|
|
||||||
|
It does stuff like create our 3 admin groups:
|
||||||
|
|
||||||
|
- **default_admin** — ICMP and monitoring/backup access. No remote access.
|
||||||
|
- **linux_admin** - SSH mostly
|
||||||
|
- **windows_admin** - All the lovely SCCM/WSUS/SMB cruft from admin networks.
|
||||||
|
|
||||||
|
The key difference from per-server groups is that it uses managed prefix lists to centralize the IP ranges. Instead of hardcoding CIDRs in every rule, the rules reference a prefix list.
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
resource "aws_ec2_managed_prefix_list" "linux_admin_access" {
|
||||||
|
name = "server-admin-access"
|
||||||
|
address_family = "IPv4"
|
||||||
|
max_entries = 5
|
||||||
|
|
||||||
|
entry {
|
||||||
|
cidr = "10.0.0.0/24"
|
||||||
|
description = "Dept A linux Admin"
|
||||||
|
}
|
||||||
|
|
||||||
|
entry {
|
||||||
|
cidr = "10.100.0.0/24"
|
||||||
|
description = "Dept B linux Admin"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Then the rules reference the prefix list instead of individual CIDRs:
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
resource "aws_vpc_security_group_ingress_rule" "linux_admin_ssh" {
|
||||||
|
security_group_id = aws_security_group.linux_admin.id
|
||||||
|
prefix_list_id = aws_ec2_managed_prefix_list.server_admin_access.id
|
||||||
|
ip_protocol = "tcp"
|
||||||
|
from_port = 22
|
||||||
|
to_port = 22
|
||||||
|
|
||||||
|
tags = {
|
||||||
|
Name = "SSH-22-admin-access"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
When a new admin subnet needs access, you add one entry to the prefix list and every security group that references it picks it up. No touching individual server rules.
|
||||||
|
|
||||||
|
A server ends up with its per-server SG for application-specific rules and one or more standard SGs for the common stuff:
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
vpc_security_group_ids = [
|
||||||
|
module.security_groups.web_server_sg_id,
|
||||||
|
module.standard_securitygroups.windows_admin_security_group_id
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
This keeps the per-server rule files focused on what's actually unique to that server.
|
||||||
|
|
||||||
|
# What's Next
|
||||||
|
|
||||||
|
The standard module handles the baseline admin access that every server gets. The next step is creating standard service-level and department-service-level SGs.
|
||||||
|
|
||||||
|
A generic `db-sg` would cover common database ports that most database servers need. But a `math-db-sg` would layer on department-specific rules for the math department's network ranges, their specific application servers, and their particular inter-database communication patterns. Same idea for web servers, app servers, etc.
|
||||||
|
|
||||||
|
The goal is to get to a point where standing up a new server means picking from a menu of standard SGs rather than writing rules from scratch every time.
|
||||||
|
|
||||||
|
# What I'd Do Differently
|
||||||
|
|
||||||
|
Not much honestly. The progression made sense given the constraints. We didn't know how many servers we'd be moving per wave at the start and the format evolved as the workload scaled. The template approach with find-and-replace is simple enough that even the folks brand new to Terraform are following along.
|
||||||
|
|
@ -1,24 +1,27 @@
|
||||||
+++
|
---
|
||||||
title = 'Self Hosting'
|
title: "Self Hosting"
|
||||||
date = 2024-08-19T10:57:05-07:00
|
date: "2024-08-19T10:57:05-07:00"
|
||||||
tags = ['server','selfhosted']
|
tags:
|
||||||
categories = ['server', 'selfhosted', 'unraid']
|
- server
|
||||||
+++
|
- unraid
|
||||||
|
categories:
|
||||||
|
- selfhosted
|
||||||
|
---
|
||||||
|
|
||||||
This is just a general overview of what I'm running. Updated sporadically, only includes things I'm actively using or have deployed.
|
This is just a general overview of what I'm running. Updated sporadically, only includes things I'm actively using or have deployed.
|
||||||
|
|
||||||
## Networking ##
|
## Networking
|
||||||
|
|
||||||
### In Production ###
|
### In Production
|
||||||
|
|
||||||
* Unifi
|
* Unifi
|
||||||
* UDM Pro
|
* UDM Pro
|
||||||
* 3 APs
|
* 3 APs
|
||||||
* PoE and non-PoE switches
|
* PoE and non-PoE switches
|
||||||
* A certain number and variety of cameras
|
* Various cameras
|
||||||
|
|
||||||
|
## Services
|
||||||
|
|
||||||
## Services ##
|
|
||||||
Self-hosted:
|
Self-hosted:
|
||||||
|
|
||||||
* NGINX Proxy Manager
|
* NGINX Proxy Manager
|
||||||
|
|
@ -27,10 +30,9 @@ Self-hosted:
|
||||||
* [Recipes](https://recipes.fig.systems). Recipes I've saved.
|
* [Recipes](https://recipes.fig.systems). Recipes I've saved.
|
||||||
* [DokuWiki](https://wiki.fig.systems). Personal wiki.
|
* [DokuWiki](https://wiki.fig.systems). Personal wiki.
|
||||||
* [MicroBin](https://bin.fig.systems). "paste bin" share.
|
* [MicroBin](https://bin.fig.systems). "paste bin" share.
|
||||||
* [FreshRSS](https://feeds.fig.systems). RSS agregator
|
* [FreshRSS](https://feeds.fig.systems). RSS aggregator
|
||||||
* [Forgejo](https://git.fig.systems). Git repos.
|
* [Forgejo](https://git.fig.systems). Git repos.
|
||||||
|
|
||||||
## Cloud ##
|
## Cloud
|
||||||
|
|
||||||
Came from an azure shop years ago, current aws. I've had personal items on Oracle and AWS but pulled it all down to this server.
|
|
||||||
|
|
||||||
|
I came from an Azure shop years ago, currently on AWS. I've had personal projects on Oracle and AWS but pulled everything down to my local server.
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue