Task Summary: Insights Hub Export + Gitea Push Fix

We created a Python script to support your Insights Hub / Portable Identity Document workflow.

The script:

C:\projects\ES\eddie-soehnel-portable-identity-document-OPEN\scripts\insights-hub-posts-last-6-months.py

pulls matching JSON records from:

C:\Users\edsoe\My Drive\Personal Organization\JSON_Database\hrecords

filters for records where:

"HubTags": ["External Platform Posts"]

copies matching JSON files into:

C:\projects\ES\eddie-soehnel-portable-identity-document-OPEN\data\insights-hub\hrecords

copies referenced media files from:

C:\Users\edsoe\My Drive\Personal Organization\JSON_Database\files

into:

C:\projects\ES\eddie-soehnel-portable-identity-document-OPEN\data\insights-hub\files

and generates a Markdown digest:

C:\projects\ES\eddie-soehnel-portable-identity-document-OPEN\data\insights-hub\insights-hub-posts-last-6-months.md

with the title:

# Insights Hub Posts - Last 6 Months

The Markdown output is sorted most recent first, covers roughly the last 6 months, and includes each post’s title, summary, and referenced image/file where available.

Problem 1: Git push failed with HTTP 413

When syncing to your Gitea repo:

https://projects.eddiesoehnel.com/adminprojects/eddie-soehnel-portable-identity-document-OPEN

Git failed with:

error: RPC failed; HTTP 413 curl 22 The requested URL returned error: 413
fatal: the remote end hung up unexpectedly

The key issue was:

HTTP 413 = Payload Too Large

Your push included a large Git pack, around:

118.25 MiB

because the new Insights Hub export included many image and PDF files under:

data/insights-hub/files/
Problem 2: Misleading “Everything up-to-date” message

Git also printed:

Everything up-to-date

after the failed push.

We treated that as unreliable because the same output also included:

fatal: the remote end hung up unexpectedly

The local branch remained ahead of origin, confirming the push had not actually completed.

Workaround Attempt 1: Exclude media from Git

We temporarily added this to .gitignore:

data/insights-hub/files/

Then we reset and recommitted the lightweight files only:

.gitignore
scripts/insights-hub-posts-last-6-months.py
data/insights-hub/hrecords/
data/insights-hub/insights-hub-posts-last-6-months.md

That allowed Git to avoid the large media payload.

However, this created a new issue: the generated six-month Markdown file referenced local media files, but those files were not present on the Gitea/server side because Git was ignoring them.

Problem 3: Markdown referenced missing media

Since the Markdown file displays images/files from:

data/insights-hub/files/

the media files do need to exist wherever the repo is served or rendered.

So excluding the media folder solved the Git push problem but broke the completeness of the published Markdown output.

Final Fix: Increase Caddy request body limit

We inspected your Caddyfile and found the Gitea reverse proxy block:

see caddyfile for the new addition

We added a larger request body limit:

Then Caddy was validated and reloaded:

sudo caddy validate --config /etc/caddy/Caddyfile
sudo systemctl reload caddy

After that, the push worked.

Cloudflare Change

You also turned off the Cloudflare proxy for:

projects.eddiesoehnel.com

and set it to DNS only.

That was the correct move because Git pushes over HTTPS can hit Cloudflare request-body limits. For a self-hosted Gitea server, the cleaner current setup is:

Cloudflare DNS only → Caddy HTTPS reverse proxy → Gitea VM

rather than:

Cloudflare proxy → Caddy → Gitea
Current Recommended State

Keep:

projects.eddiesoehnel.com = DNS only

Keep this in the Caddyfile:

request_body {
    max_size 1GB
}

Allow the media files in Git if the Markdown digest depends on them being present in the repo.

Longer term, the cleaner architecture would be to separate Git-tracked content from large media deployment:

Git:
- JSON
- Markdown
- scripts

Separate media deployment:
- rsync
- SFTP
- object storage
- static media volume
- Git LFS

But for the current workflow, the practical fix is:

Track the media in Git + raise Caddy upload limit + keep Cloudflare DNS-only for Gitea.