Agentic Market-Data Pipeline
An end-to-end system that ingests domain sources, extracts structured signal with an LLM, generates enriched commentary, and self-publishes across channels on a schedule. Built and operated solo, running in production.
Book a callRuns unattended on a daily schedule, with a hard stop on stale data and graceful fallback at every step.
Grounded commentary, published automatically
A live financial-data product needed daily, information-rich market commentary published automatically, without a human writing each update, and without ever publishing stale or low-quality output.
The hard part isn’t generating text. It’s grounding that text in fresh, real-world signal, and making the whole chain fail safely when a source is empty or an upstream step breaks.
- Hard gate
Data refresh
Pulls latest data. Hard gate — if it fails, the whole run aborts (never publish stale).
Signal extraction
Scrapes sources; LLM extracts & consolidates drivers; filters to multi-source signal.
Enriched generation
Generates a grounded note from the extracted facts; publishes via API.
Automated distribution
Composes a post + live chart screenshot; publishes, with graceful fallback.
What I built
A single orchestrated pipeline that runs end to end on a schedule.
1. Data refresh (hard gate): Pulls and upserts the latest market data into the database. If this step fails, the entire run aborts, so nothing stale is ever published.
2. Signal extraction: Scrapes the day’s domain sources, runs an LLM extraction-and-consolidation pass to pull out structured catalysts and causal drivers, and filters down to multi-source, higher-confidence items only.
3. Enriched generation: Generates a structured note that folds the extracted drivers into a clean read of the current data, grounded strictly in the extracted facts, then publishes it to the product through an API.
4. Automated distribution: Composes a channel-appropriate post, captures a live chart screenshot via resilient headless browser automation, and publishes a thread, with graceful fallback if any step degrades.
Every stage degrades gracefully: on a day with no usable signal, the note still publishes from fresh data. The system never blocks on a missing input.
Engineering decisions that mattered
• Fail-safe ordering over completeness: The data-refresh gate guarantees freshness; everything downstream is best-effort. Publishing a clean baseline beats publishing a stale or half-broken enrichment.
• Fact discipline in the LLM layer: A multi-source confidence filter plus strict “use only the extracted drivers” prompting stops the model from inventing precise numbers or pulling in unrelated headlines, the exact failure mode that makes auto-generated market content dangerous.
• Local orchestration over cloud routines: I evaluated a cloud-scheduled split and rejected it. The browser-automation and local-environment requirements made a single local orchestrator on a scheduler simpler and more reliable than a distributed handoff.
• Cheap to run: On edge infrastructure, the always-on cost is a few dollars a month.
Stack
Python · OpenAI / Claude (LLM orchestration) · Model Context Protocol (MCP) · Cloudflare Workers · edge SQL (libSQL) · headless browser automation · scheduled task runner.
Outcome
Runs unattended on a daily schedule: refreshes data, extracts and grounds the day’s drivers, publishes an enriched note to the product, and distributes a chart-backed post, every step logged, with a hard stop on stale data.
What this means I can build for you
The same pattern is reusable for any product that needs to turn raw sources into structured, grounded, auto-published output:
source ingestion → LLM extraction with fact discipline → enriched generation → MCP/API publishing → scheduled, fail-safe automation.
If you want an agentic content or data pipeline, or an existing API wrapped as an MCP server your AI tools can actually drive, this is squarely what I do.