No description
  • Go 57.8%
  • TypeScript 39.6%
  • Python 0.8%
  • Shell 0.8%
  • PLpgSQL 0.6%
  • Other 0.3%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
John Parkhurst 381bd371bc
Some checks failed
Deployment Non-Production / check-inputs (push) Successful in 0s
PR & Branch Check / Backend Go Verification & Coverage Gate (push) Failing after 15s
PR & Branch Check / Frontend Next.js Lint & Build (push) Successful in 1m39s
Deployment Non-Production / build (push) Successful in 2m37s
Deployment Non-Production / deploy (push) Successful in 0s
Scheduler Change
2026-11-11 19:21:57 -05:00
.forgejo/workflows Scheduler Change 2026-11-11 19:21:57 -05:00
.github Documentations and Standards 2026-10-14 01:01:36 -04:00
docs/standards Documentations and Standards 2026-10-14 01:01:36 -04:00
scripts Pending changes 2026-10-27 19:07:18 -04:00
services Scheduler Change 2026-11-11 19:21:57 -05:00
sql Pending changes 2026-10-27 19:07:18 -04:00
web . 2026-11-07 18:49:18 -05:00
.cursorrules Documentations and Standards 2026-10-14 01:01:36 -04:00
.dockerignore Test 2026-10-10 15:47:27 -04:00
.env.example Teamcity->Coolify 2026-11-07 16:06:34 -05:00
.gitignore Custom Rating 2026-10-13 23:42:26 -04:00
.golangci.yml If to Switch 2026-10-12 14:52:44 -04:00
.nvmrc Security 2026-10-14 00:01:31 -04:00
AGENTS.md Security vuln 2026-10-27 18:42:25 -04:00
audit_report.md Teamcity->Coolify 2026-11-07 16:06:34 -05:00
CLAUDE.md Documentations and Standards 2026-10-14 01:01:36 -04:00
docker-compose.dev.yml Docker Compose 2026-11-07 17:26:08 -05:00
docker-compose.yml . 2026-11-07 18:49:18 -05:00
Dockerfile Security vuln 2026-10-27 18:42:25 -04:00
GEMINI.md Documentations and Standards 2026-10-14 01:01:36 -04:00
go.mod Security vuln 2026-10-27 18:42:25 -04:00
go.sum version bump 2026-10-01 21:09:07 -04:00
historical_audit_report.md Data quality includes extra audit 2026-10-14 23:03:54 -04:00
ingester Unit tests, Actions, Sleeper 2026-10-01 19:49:05 -04:00
main DockerFiles 2026-10-10 13:46:14 -04:00
README.md Scheduler Change 2026-11-11 19:21:57 -05:00
renovate.json Add renovate.json 2026-10-02 00:16:20 +00:00
SETUP_AND_BACKFILL_GUIDE.md Scheduler Change 2026-11-11 19:21:57 -05:00
walkthrough_go_tests.go Unit tests, Actions, Sleeper 2026-10-01 19:49:05 -04:00

Sports-Ingestion

A robust, enterprise-grade Go data pipeline and analytics platform for ingesting, normalizing, stitching, and storing NFL player statistics across multiple data sources:

  • Box Scores & Counting Stats: ESPN and Sleeper API
  • Advanced EPA & Passing/Receiving Metrics: nflverse (EPA, CPOE, Air Yards, Target Share, WOPR)
  • Advanced Defensive Metrics: Pro Football Reference / nflverse (Pressures, Hurries, Blitzes, Sacks, Coverage Passer Rating)
  • Film & Analytics Grades: Pro Football Focus (PFF overall, passing, rushing, receiving, run defense, and coverage grades)
  • Canonical Player Identity & Deduplication: Unified player IDs, suffix/punctuation normalization (Jr., III, II, Sr.), and deduplication

🚀 New to the project or provisioning a fresh database?
Check out the complete step-by-step onboarding guide: SETUP_AND_BACKFILL_GUIDE.md
It covers database initialization via sql/init_complete_schema.sql, automated backfill via scripts/bootstrap_all.sh, and running the full stack.


Prerequisites

  • Go 1.26+
  • Node.js 26+ (LTS, specified in .nvmrc) & npm 11+
  • PostgreSQL database (Zeta or local)
  • Set the DATABASE_URL environment variable:
    export DATABASE_URL="postgres://user:pass@localhost:5432/sports"
    

Running the Application Stack & 24/7 Deployment

The system is architected for simple 24/7 deployment with just two services:

  1. Go API & Embedded Ingestion Scheduler (services/api):
    • Listens on port 8080 and serves the REST endpoints (including GET /api/health for liveness/readiness probes) to the frontend.
    • Embedded Tuesday Ingestion Worker: By default, automatically runs the Tuesday 07:00 weekly ingestion pipeline in a background goroutine (sharing a single unified PostgreSQL connection pool), eliminating the need to deploy and orchestrate a separate ingester worker container!
    • Can be toggled via ENABLE_SCHEDULER:
      • ENABLE_SCHEDULER=true (default): Runs both the HTTP API server and the Tuesday weekly sync.
      • ENABLE_SCHEDULER=false: Runs in pure HTTP mode without starting the background cron worker (useful for staging or horizontal API replicas).
  2. Next.js Web Frontend (web): Next.js 16 interactive UI running on port 3000 that proxies /api/* requests to http://localhost:8080.

To run the full stack locally:

# Terminal 1 — Start the Go API Backend (includes embedded Tuesday scheduler)
export DATABASE_URL="postgres://user:password@localhost:5432/zeta?sslmode=disable"
go run ./services/api

# Terminal 2 — Start the Next.js Web Frontend
cd web
npm run dev

Then navigate to http://localhost:3000 in your browser.

Running with Docker Compose & Coolify Deployment

A multi-container Docker Compose definition is provided at the repository root (docker-compose.yml) to run both the Go API (api) and Next.js frontend (web) together under an isolated bridge network.

# 1. Copy sample environment variables
cp .env.example .env

# 2. Run the application stack in Docker
docker compose up -d

# 3. Check health and logs
docker compose ps
docker compose logs -f
  • Coolify Deployment: Coolify manages the multi-container stack via docker-compose.yml. Forgejo Actions pushes tagged images (latest and ${GITHUB_SHA:0:8}) to registry.rcslabs.dev on every push to main and invokes the Coolify automated deployment webhook (COOLIFY_WEBHOOK_URL).
  • Network & Port Routing: The Next.js web frontend listens on internal port 3000 and is published to host port 16969 (16969:3000). All browser requests to /api/* are reverse-proxied internally to http://api:8080. Navigate to http://localhost:16969 (or your Coolify domain mapped to port 3000/16969).

Standalone Ingester CLI Reference

The ingester binary provides automated daemon scheduling alongside granular, modular CLI flags for targeted backfills and one-off ingestion tasks.

CLI Flags Summary

Flag Type Default Description
-run-now bool false Trigger the unified weekly pipeline immediately across all data sources (ESPN, Sleeper, nflverse, PFR defense, PFF grades, and DQ report) and exit.
-week int 0 (dynamic) Specific NFL week to target for -run-now or sync (1–18). Defaults to the active NFL week resolved dynamically via Sleeper API or calendar rules.
-season int 0 Specific NFL season to target for backfill, stats ingestion, or historical PFF grade scraping (e.g. 2010 or 2026). Automatically sets -start and -end if not specified.
-backfill bool false Run historical backfill of box score stats across a year range instead of starting the daemon.
-start int 2026 Start year for backfill / ingestion commands.
-end int 2026 End year for backfill / ingestion commands.
-source string "all" Specific source to backfill ("all", "sleeper", or "espn").
-sync-roster bool false Sync official master NFL roster directory (positions, PFF IDs, ESPN IDs, GSIS IDs, latest teams, headshot URLs) from nflverse and exit.
-sync-players bool false Sync latest player roster directory from Sleeper into the players table and exit.
-sync-pff bool false Crawl PFF sitemap to map player slugs & IDs to canonical database players and exit.
-scrape-pff bool false Scrape PFF grades for active players and exit.
-pff-limit int 60 Maximum number of active players to scrape PFF grades for (0 for unlimited).
-ingest-nflverse bool false Ingest advanced EPA, CPOE, Air Yards, and success rate metrics from nflverse for the specified year range and exit.
-sync-defense bool false Ingest advanced defensive metrics (pressures, hurries, blitzes, coverage) from PFR (2018–present) and exit.
-sync-schedule bool false Sync weekly NFL schedules, kickoff times, networks, odds, and game results from ESPN and exit.
-backfill-odds bool false Backfill closing betting odds (spread and over/under total) from canonical NFL games registry into nfl_games and exit.
-backfill-games bool false Backfill historical NFL regular season games, scores, and closing betting lines (1999–2023) from canonical games registry and exit.
-compute-ratings bool false Compute and persist proprietary True Grade (TG) (0–100 efficiency rate) and True Value (TV) (net EPA / havoc added above replacement) for offensive and defensive players across seasons (e.g. -season 2026 or -start 1999 -end 2025) and exit.
-dq-report bool false Run the cross-source Data Quality comparison report against the database and exit.
-cross-audit bool false Run comprehensive cross-source accuracy audit across database, external ground truth (Yahoo/GSIS), and live deployment, then exit.
-audit-output string "audit_report.md" Output markdown file path for the cross-audit scorecard report.
-audit-live bool true Include live web deployment contract checks in cross-audit.
-audit-historical bool false Expand cross-audit downward across historical seasons back to 1999.

How to Run & Usage Modes

1. Unified Weekly Ingestion (On-Demand)

To run the complete multi-source weekly ingestion pipeline immediately without waiting for the scheduled Tuesday cron job:

# Automatically resolves current live NFL Season & Week (e.g., Season 2026 Week 5)
go run services/ingester/cmd/ingester/main.go -run-now

# Target a specific season and week on-demand
go run services/ingester/cmd/ingester/main.go -run-now -season 2024 -week 18

The unified pipeline executes in 8 sequential phases:

  1. [Phase 1/8] Player Directory Sync: Syncs player metadata from Sleeper to register any rookie or practice-squad activations.
  2. [Phase 2/8] Box Score Stats: Scrapes completed week games from ESPN and Sleeper into weekly_stats.
  3. [Phase 3/8] Advanced EPA Metrics: Pulls updated season EPA, CPOE, Air Yards, and target shares into player_nflverse_stats.
  4. [Phase 4/8] Advanced Defense: Scrapes updated PFR pressures, blitzes, sacks, tackles, and coverage metrics into player_defense_advanced_stats.
  5. [Phase 5/8] PFF Grades Refresh: Updates PFF grades and ranks for active players with polite rate limiting.
  6. [Phase 6/8] True Grade & Value Calculation: Computes and stores proprietary True Grade efficiency ratings and cumulative True Value EPA points added.
  7. [Phase 7/8] NFL Games & Matchup Schedule Sync: Syncs schedules, broadcast networks, Vegas spreads/totals, and final scores into nfl_games for the current and upcoming weeks.
  8. [Phase 8/8] Data Quality Audit: Runs cross-source discrepancy analysis and logs a DQ report.

2. Daemon Mode (Scheduled Automation)

By default (without flags), the application runs as a persistent daemon. It registers a cron job that executes every Tuesday at 07:00:

go run services/ingester/cmd/ingester/main.go

Key Scheduler Capabilities:

  • Dynamic NFL State Resolution: Queries Sleeper's state endpoint (/v1/state/nfl) to determine the exact active season, week, and postseason status in real time.
  • Calendar-Aware Fallback: If offline or during network interruptions, deterministic date math calculates the current NFL season and week (including handling January/February playoff transitions belonging to the previous calendar year).
  • Concurrency Guard: Protected by an execution mutex (TryLock) preventing overlapping runs if an ingestion takes longer than anticipated.
  • Resilient Timeouts: Each scheduled execution runs within a 45-minute bounded context.

3. Historical Backfill Modes

Box Score Stats (ESPN & Sleeper)

# Backfill all sources for 2023
go run services/ingester/cmd/ingester/main.go -backfill -start 2023 -end 2023

# Backfill a specific source for a range of years
go run services/ingester/cmd/ingester/main.go -backfill -source espn -start 2020 -end 2024

Advanced EPA & Passing/Receiving (nflverse)

# Ingest advanced EPA and CPOE stats for a single season
go run services/ingester/cmd/ingester/main.go -ingest-nflverse -season 2026

# Or ingest across a historical season range
go run services/ingester/cmd/ingester/main.go -ingest-nflverse -start 1999 -end 2026

Advanced Defensive Stats (PFR)

# Ingest PFR advanced defense metrics (pressures, hurries, blitzes, coverage)
go run services/ingester/cmd/ingester/main.go -sync-defense

PFF Player Directory & Grades

# Step 1: Map player names to PFF directory IDs via sitemap
go run services/ingester/cmd/ingester/main.go -sync-pff

# Step 2: Scrape PFF grades for top active players (default 60)
go run services/ingester/cmd/ingester/main.go -scrape-pff -pff-limit 60

# Step 3: Target un-scraped players from a specific historical season (prioritizing modern active veterans)
go run services/ingester/cmd/ingester/main.go -scrape-pff -season 2010 -pff-limit 50

# Or scrape all mapped players without a limit
go run services/ingester/cmd/ingester/main.go -scrape-pff -pff-limit 0

4. Data Quality (DQ) Validation & Auto-Remediation Mode

Analyze database records across sources (e.g. ESPN vs Sleeper variance) and run automated remediations (e.g. clearing placeholder scores on unplayed scheduled games, normalizing legacy franchise aliases such as "LA"/"STL" to "LAR", "OAK" to "LV", "SD" to "LAC", "WSH" to "WAS", "JAC" to "JAX", and "PHX" to "ARI"):

go run services/ingester/cmd/ingester/main.go -dq-report

5. Proprietary True Grade & True Value Rating Engine

Compute and persist Bayesian composite efficiency grades (TG, 0–100) and cumulative value points added above replacement (TV, +EPA) for offensive players (QBs, RBs, WRs, TEs):

# Compute True Grades for the current 2026 season
go run services/ingester/cmd/ingester/main.go -compute-ratings -season 2026

# Backfill True Grades across historical seasons (1999–2025)
go run services/ingester/cmd/ingester/main.go -compute-ratings -start 1999 -end 2025

6. NFL Schedules, Matchup Odds & Game Results Ingestion

Sync weekly NFL schedules, kickoff times, broadcast networks, Vegas spreads, over/under totals, and official game results into nfl_games. Ingesting historical seasons (e.g. 2024–2025) powers complete historical team standings (PCT, HOME, AWAY, DIV, CONF, STRK) and True Opponent Defensive EPA calculations on the /teams page:

# Sync entire 2026 season schedule (Weeks 1–18, completed scores and upcoming lines)
go run services/ingester/cmd/ingester/main.go -sync-schedule -season 2026

# Ingest historical season game schedules for full team records & telemetry (2024-2025)
go run services/ingester/cmd/ingester/main.go -sync-schedule -season 2025
go run services/ingester/cmd/ingester/main.go -sync-schedule -season 2024

# Backfill complete historical regular season games (1999–2023) from canonical registry
go run services/ingester/cmd/ingester/main.go -backfill-games -start 1999 -end 2023

# Target a specific week
go run services/ingester/cmd/ingester/main.go -sync-schedule -season 2026 -week 6

# Backfill canonical closing spreads and over/under totals for completed games
go run services/ingester/cmd/ingester/main.go -backfill-odds -season 2026

7. Cross-Source Accuracy & Integrity Audit

To run a comprehensive verification comparing the database, live application contracts, and external ground truth (Yahoo Sports official telemetry & canonical NFL GSIS registry):

# Audit active 2026 season against live deployment and external ground truth
go run services/ingester/cmd/ingester/main.go -cross-audit -season 2026

# Output to custom markdown report file
go run services/ingester/cmd/ingester/main.go -cross-audit -season 2026 -audit-output audit_report.md

# Audit historical seasons downward to 1999
go run services/ingester/cmd/ingester/main.go -cross-audit -season 2026 -audit-historical

Development & Testing

This project enforces strict CI validation and automated operational health audits via Forgejo Actions:

  • Test Coverage Gate: Enforces a strict minimum of 90.0% filtered unit test coverage across backend services.
  • In-Memory Testing: Scheduler and pipeline tests use socketless mock transports to guarantee deterministic, sandbox-safe execution.
  • Frontend Verification: Runs ESLint and Next.js production builds on every PR and push to main.
  • Automated Security Auditing: Runs daily dependency CVE checks with auto-filing of Forgejo issues.
  • Automated Data Quality Auditing: Runs scheduled weekly database reconciliation audits with auto-filing of Forgejo issues.

Run tests and verification locally:

# Run all unit tests
CGO_ENABLED=0 go test -v ./services/ingester/... ./services/api/...

# Verify backend compilation and linting
go build ./...
golangci-lint run ./...

# Verify frontend Next.js lint and production build
cd web && npm run lint
cd web && npm run build

Forgejo Actions Workflows

The repository includes three production Forgejo Actions in .forgejo/workflows/:

Workflow File Trigger Description
pr-check.yml pull_request, push: main Parallel verification: (1) Go 1.27.3+ govulncheck, golangci-lint, 90.0% coverage gate, binary build; (2) Node.js 26 ESLint & Next.js production build.
security-scan.yml Daily at 5:00 PM EDT (0 21 * * *), workflow_dispatch Automated daily security scan (govulncheck + npm audit). If CVEs are detected, automatically opens or updates a deduplicated Forgejo issue with actionable fix instructions; automatically resolves open issues once all scans pass cleanly.
dq-audit.yml Tuesdays at 9:00 AM EDT (0 13 * * 2), workflow_dispatch Scheduled post-sync database quality & cross-source reconciliation audit. Runs main.go -dq-report auto-remediation and main.go -cross-audit against Yahoo Sports/GSIS, posting a rich markdown scorecard to the Actions summary and auto-filing Forgejo issues if anomalies are detected.