- Go 57.8%
- TypeScript 39.6%
- Python 0.8%
- Shell 0.8%
- PLpgSQL 0.6%
- Other 0.3%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
|
Some checks failed
Deployment Non-Production / check-inputs (push) Successful in 0s
PR & Branch Check / Backend Go Verification & Coverage Gate (push) Failing after 15s
PR & Branch Check / Frontend Next.js Lint & Build (push) Successful in 1m39s
Deployment Non-Production / build (push) Successful in 2m37s
Deployment Non-Production / deploy (push) Successful in 0s
|
||
| .forgejo/workflows | ||
| .github | ||
| docs/standards | ||
| scripts | ||
| services | ||
| sql | ||
| web | ||
| .cursorrules | ||
| .dockerignore | ||
| .env.example | ||
| .gitignore | ||
| .golangci.yml | ||
| .nvmrc | ||
| AGENTS.md | ||
| audit_report.md | ||
| CLAUDE.md | ||
| docker-compose.dev.yml | ||
| docker-compose.yml | ||
| Dockerfile | ||
| GEMINI.md | ||
| go.mod | ||
| go.sum | ||
| historical_audit_report.md | ||
| ingester | ||
| main | ||
| README.md | ||
| renovate.json | ||
| SETUP_AND_BACKFILL_GUIDE.md | ||
| walkthrough_go_tests.go | ||
Sports-Ingestion
A robust, enterprise-grade Go data pipeline and analytics platform for ingesting, normalizing, stitching, and storing NFL player statistics across multiple data sources:
- Box Scores & Counting Stats: ESPN and Sleeper API
- Advanced EPA & Passing/Receiving Metrics: nflverse (EPA, CPOE, Air Yards, Target Share, WOPR)
- Advanced Defensive Metrics: Pro Football Reference / nflverse (Pressures, Hurries, Blitzes, Sacks, Coverage Passer Rating)
- Film & Analytics Grades: Pro Football Focus (PFF overall, passing, rushing, receiving, run defense, and coverage grades)
- Canonical Player Identity & Deduplication: Unified player IDs, suffix/punctuation normalization (Jr., III, II, Sr.), and deduplication
🚀 New to the project or provisioning a fresh database?
Check out the complete step-by-step onboarding guide:SETUP_AND_BACKFILL_GUIDE.md
It covers database initialization viasql/init_complete_schema.sql, automated backfill viascripts/bootstrap_all.sh, and running the full stack.
Prerequisites
- Go 1.26+
- Node.js 26+ (LTS, specified in
.nvmrc) & npm 11+ - PostgreSQL database (Zeta or local)
- Set the
DATABASE_URLenvironment variable:export DATABASE_URL="postgres://user:pass@localhost:5432/sports"
Running the Application Stack & 24/7 Deployment
The system is architected for simple 24/7 deployment with just two services:
- Go API & Embedded Ingestion Scheduler (
services/api):- Listens on port
8080and serves the REST endpoints (includingGET /api/healthfor liveness/readiness probes) to the frontend. - Embedded Tuesday Ingestion Worker: By default, automatically runs the Tuesday 07:00 weekly ingestion pipeline in a background goroutine (sharing a single unified PostgreSQL connection pool), eliminating the need to deploy and orchestrate a separate ingester worker container!
- Can be toggled via
ENABLE_SCHEDULER:ENABLE_SCHEDULER=true(default): Runs both the HTTP API server and the Tuesday weekly sync.ENABLE_SCHEDULER=false: Runs in pure HTTP mode without starting the background cron worker (useful for staging or horizontal API replicas).
- Listens on port
- Next.js Web Frontend (
web): Next.js 16 interactive UI running on port3000that proxies/api/*requests tohttp://localhost:8080.
To run the full stack locally:
# Terminal 1 — Start the Go API Backend (includes embedded Tuesday scheduler)
export DATABASE_URL="postgres://user:password@localhost:5432/zeta?sslmode=disable"
go run ./services/api
# Terminal 2 — Start the Next.js Web Frontend
cd web
npm run dev
Then navigate to http://localhost:3000 in your browser.
Running with Docker Compose & Coolify Deployment
A multi-container Docker Compose definition is provided at the repository root (docker-compose.yml) to run both the Go API (api) and Next.js frontend (web) together under an isolated bridge network.
# 1. Copy sample environment variables
cp .env.example .env
# 2. Run the application stack in Docker
docker compose up -d
# 3. Check health and logs
docker compose ps
docker compose logs -f
- Coolify Deployment: Coolify manages the multi-container stack via
docker-compose.yml. Forgejo Actions pushes tagged images (latestand${GITHUB_SHA:0:8}) toregistry.rcslabs.devon every push tomainand invokes the Coolify automated deployment webhook (COOLIFY_WEBHOOK_URL). - Network & Port Routing: The Next.js web frontend listens on internal port
3000and is published to host port16969(16969:3000). All browser requests to/api/*are reverse-proxied internally tohttp://api:8080. Navigate tohttp://localhost:16969(or your Coolify domain mapped to port 3000/16969).
Standalone Ingester CLI Reference
The ingester binary provides automated daemon scheduling alongside granular, modular CLI flags for targeted backfills and one-off ingestion tasks.
CLI Flags Summary
| Flag | Type | Default | Description |
|---|---|---|---|
-run-now |
bool |
false |
Trigger the unified weekly pipeline immediately across all data sources (ESPN, Sleeper, nflverse, PFR defense, PFF grades, and DQ report) and exit. |
-week |
int |
0 (dynamic) |
Specific NFL week to target for -run-now or sync (1–18). Defaults to the active NFL week resolved dynamically via Sleeper API or calendar rules. |
-season |
int |
0 |
Specific NFL season to target for backfill, stats ingestion, or historical PFF grade scraping (e.g. 2010 or 2026). Automatically sets -start and -end if not specified. |
-backfill |
bool |
false |
Run historical backfill of box score stats across a year range instead of starting the daemon. |
-start |
int |
2026 |
Start year for backfill / ingestion commands. |
-end |
int |
2026 |
End year for backfill / ingestion commands. |
-source |
string |
"all" |
Specific source to backfill ("all", "sleeper", or "espn"). |
-sync-roster |
bool |
false |
Sync official master NFL roster directory (positions, PFF IDs, ESPN IDs, GSIS IDs, latest teams, headshot URLs) from nflverse and exit. |
-sync-players |
bool |
false |
Sync latest player roster directory from Sleeper into the players table and exit. |
-sync-pff |
bool |
false |
Crawl PFF sitemap to map player slugs & IDs to canonical database players and exit. |
-scrape-pff |
bool |
false |
Scrape PFF grades for active players and exit. |
-pff-limit |
int |
60 |
Maximum number of active players to scrape PFF grades for (0 for unlimited). |
-ingest-nflverse |
bool |
false |
Ingest advanced EPA, CPOE, Air Yards, and success rate metrics from nflverse for the specified year range and exit. |
-sync-defense |
bool |
false |
Ingest advanced defensive metrics (pressures, hurries, blitzes, coverage) from PFR (2018–present) and exit. |
-sync-schedule |
bool |
false |
Sync weekly NFL schedules, kickoff times, networks, odds, and game results from ESPN and exit. |
-backfill-odds |
bool |
false |
Backfill closing betting odds (spread and over/under total) from canonical NFL games registry into nfl_games and exit. |
-backfill-games |
bool |
false |
Backfill historical NFL regular season games, scores, and closing betting lines (1999–2023) from canonical games registry and exit. |
-compute-ratings |
bool |
false |
Compute and persist proprietary True Grade (TG) (0–100 efficiency rate) and True Value (TV) (net EPA / havoc added above replacement) for offensive and defensive players across seasons (e.g. -season 2026 or -start 1999 -end 2025) and exit. |
-dq-report |
bool |
false |
Run the cross-source Data Quality comparison report against the database and exit. |
-cross-audit |
bool |
false |
Run comprehensive cross-source accuracy audit across database, external ground truth (Yahoo/GSIS), and live deployment, then exit. |
-audit-output |
string |
"audit_report.md" |
Output markdown file path for the cross-audit scorecard report. |
-audit-live |
bool |
true |
Include live web deployment contract checks in cross-audit. |
-audit-historical |
bool |
false |
Expand cross-audit downward across historical seasons back to 1999. |
How to Run & Usage Modes
1. Unified Weekly Ingestion (On-Demand)
To run the complete multi-source weekly ingestion pipeline immediately without waiting for the scheduled Tuesday cron job:
# Automatically resolves current live NFL Season & Week (e.g., Season 2026 Week 5)
go run services/ingester/cmd/ingester/main.go -run-now
# Target a specific season and week on-demand
go run services/ingester/cmd/ingester/main.go -run-now -season 2024 -week 18
The unified pipeline executes in 8 sequential phases:
- [Phase 1/8] Player Directory Sync: Syncs player metadata from Sleeper to register any rookie or practice-squad activations.
- [Phase 2/8] Box Score Stats: Scrapes completed week games from ESPN and Sleeper into
weekly_stats. - [Phase 3/8] Advanced EPA Metrics: Pulls updated season EPA, CPOE, Air Yards, and target shares into
player_nflverse_stats. - [Phase 4/8] Advanced Defense: Scrapes updated PFR pressures, blitzes, sacks, tackles, and coverage metrics into
player_defense_advanced_stats. - [Phase 5/8] PFF Grades Refresh: Updates PFF grades and ranks for active players with polite rate limiting.
- [Phase 6/8] True Grade & Value Calculation: Computes and stores proprietary True Grade efficiency ratings and cumulative True Value EPA points added.
- [Phase 7/8] NFL Games & Matchup Schedule Sync: Syncs schedules, broadcast networks, Vegas spreads/totals, and final scores into
nfl_gamesfor the current and upcoming weeks. - [Phase 8/8] Data Quality Audit: Runs cross-source discrepancy analysis and logs a DQ report.
2. Daemon Mode (Scheduled Automation)
By default (without flags), the application runs as a persistent daemon. It registers a cron job that executes every Tuesday at 07:00:
go run services/ingester/cmd/ingester/main.go
Key Scheduler Capabilities:
- Dynamic NFL State Resolution: Queries Sleeper's state endpoint (
/v1/state/nfl) to determine the exact active season, week, and postseason status in real time. - Calendar-Aware Fallback: If offline or during network interruptions, deterministic date math calculates the current NFL season and week (including handling January/February playoff transitions belonging to the previous calendar year).
- Concurrency Guard: Protected by an execution mutex (
TryLock) preventing overlapping runs if an ingestion takes longer than anticipated. - Resilient Timeouts: Each scheduled execution runs within a 45-minute bounded context.
3. Historical Backfill Modes
Box Score Stats (ESPN & Sleeper)
# Backfill all sources for 2023
go run services/ingester/cmd/ingester/main.go -backfill -start 2023 -end 2023
# Backfill a specific source for a range of years
go run services/ingester/cmd/ingester/main.go -backfill -source espn -start 2020 -end 2024
Advanced EPA & Passing/Receiving (nflverse)
# Ingest advanced EPA and CPOE stats for a single season
go run services/ingester/cmd/ingester/main.go -ingest-nflverse -season 2026
# Or ingest across a historical season range
go run services/ingester/cmd/ingester/main.go -ingest-nflverse -start 1999 -end 2026
Advanced Defensive Stats (PFR)
# Ingest PFR advanced defense metrics (pressures, hurries, blitzes, coverage)
go run services/ingester/cmd/ingester/main.go -sync-defense
PFF Player Directory & Grades
# Step 1: Map player names to PFF directory IDs via sitemap
go run services/ingester/cmd/ingester/main.go -sync-pff
# Step 2: Scrape PFF grades for top active players (default 60)
go run services/ingester/cmd/ingester/main.go -scrape-pff -pff-limit 60
# Step 3: Target un-scraped players from a specific historical season (prioritizing modern active veterans)
go run services/ingester/cmd/ingester/main.go -scrape-pff -season 2010 -pff-limit 50
# Or scrape all mapped players without a limit
go run services/ingester/cmd/ingester/main.go -scrape-pff -pff-limit 0
4. Data Quality (DQ) Validation & Auto-Remediation Mode
Analyze database records across sources (e.g. ESPN vs Sleeper variance) and run automated remediations (e.g. clearing placeholder scores on unplayed scheduled games, normalizing legacy franchise aliases such as "LA"/"STL" to "LAR", "OAK" to "LV", "SD" to "LAC", "WSH" to "WAS", "JAC" to "JAX", and "PHX" to "ARI"):
go run services/ingester/cmd/ingester/main.go -dq-report
5. Proprietary True Grade & True Value Rating Engine
Compute and persist Bayesian composite efficiency grades (TG, 0–100) and cumulative value points added above replacement (TV, +EPA) for offensive players (QBs, RBs, WRs, TEs):
# Compute True Grades for the current 2026 season
go run services/ingester/cmd/ingester/main.go -compute-ratings -season 2026
# Backfill True Grades across historical seasons (1999–2025)
go run services/ingester/cmd/ingester/main.go -compute-ratings -start 1999 -end 2025
6. NFL Schedules, Matchup Odds & Game Results Ingestion
Sync weekly NFL schedules, kickoff times, broadcast networks, Vegas spreads, over/under totals, and official game results into nfl_games. Ingesting historical seasons (e.g. 2024–2025) powers complete historical team standings (PCT, HOME, AWAY, DIV, CONF, STRK) and True Opponent Defensive EPA calculations on the /teams page:
# Sync entire 2026 season schedule (Weeks 1–18, completed scores and upcoming lines)
go run services/ingester/cmd/ingester/main.go -sync-schedule -season 2026
# Ingest historical season game schedules for full team records & telemetry (2024-2025)
go run services/ingester/cmd/ingester/main.go -sync-schedule -season 2025
go run services/ingester/cmd/ingester/main.go -sync-schedule -season 2024
# Backfill complete historical regular season games (1999–2023) from canonical registry
go run services/ingester/cmd/ingester/main.go -backfill-games -start 1999 -end 2023
# Target a specific week
go run services/ingester/cmd/ingester/main.go -sync-schedule -season 2026 -week 6
# Backfill canonical closing spreads and over/under totals for completed games
go run services/ingester/cmd/ingester/main.go -backfill-odds -season 2026
7. Cross-Source Accuracy & Integrity Audit
To run a comprehensive verification comparing the database, live application contracts, and external ground truth (Yahoo Sports official telemetry & canonical NFL GSIS registry):
# Audit active 2026 season against live deployment and external ground truth
go run services/ingester/cmd/ingester/main.go -cross-audit -season 2026
# Output to custom markdown report file
go run services/ingester/cmd/ingester/main.go -cross-audit -season 2026 -audit-output audit_report.md
# Audit historical seasons downward to 1999
go run services/ingester/cmd/ingester/main.go -cross-audit -season 2026 -audit-historical
Development & Testing
This project enforces strict CI validation and automated operational health audits via Forgejo Actions:
- Test Coverage Gate: Enforces a strict minimum of 90.0% filtered unit test coverage across backend services.
- In-Memory Testing: Scheduler and pipeline tests use socketless mock transports to guarantee deterministic, sandbox-safe execution.
- Frontend Verification: Runs ESLint and Next.js production builds on every PR and push to
main. - Automated Security Auditing: Runs daily dependency CVE checks with auto-filing of Forgejo issues.
- Automated Data Quality Auditing: Runs scheduled weekly database reconciliation audits with auto-filing of Forgejo issues.
Run tests and verification locally:
# Run all unit tests
CGO_ENABLED=0 go test -v ./services/ingester/... ./services/api/...
# Verify backend compilation and linting
go build ./...
golangci-lint run ./...
# Verify frontend Next.js lint and production build
cd web && npm run lint
cd web && npm run build
Forgejo Actions Workflows
The repository includes three production Forgejo Actions in .forgejo/workflows/:
| Workflow File | Trigger | Description |
|---|---|---|
pr-check.yml |
pull_request, push: main |
Parallel verification: (1) Go 1.27.3+ govulncheck, golangci-lint, 90.0% coverage gate, binary build; (2) Node.js 26 ESLint & Next.js production build. |
security-scan.yml |
Daily at 5:00 PM EDT (0 21 * * *), workflow_dispatch |
Automated daily security scan (govulncheck + npm audit). If CVEs are detected, automatically opens or updates a deduplicated Forgejo issue with actionable fix instructions; automatically resolves open issues once all scans pass cleanly. |
dq-audit.yml |
Tuesdays at 9:00 AM EDT (0 13 * * 2), workflow_dispatch |
Scheduled post-sync database quality & cross-source reconciliation audit. Runs main.go -dq-report auto-remediation and main.go -cross-audit against Yahoo Sports/GSIS, posting a rich markdown scorecard to the Actions summary and auto-filing Forgejo issues if anomalies are detected. |