Inspiration
Every on-call engineer and data platform lead knows the dread of being away from their laptop—traveling, on a weekend trip, or outside regular office hours—when critical batch ETL pipelines crash. Traditionally, an on-call alert like PagerDuty means having to find stable Wi-Fi, open a laptop, decipher stack traces across terminal logs, dig through Jira tickets and Confluence runbooks, and manually patch database schemas while incident timers tick.
We asked ourselves: What if an engineer on the road could resolve a production data pipeline failure in 60 seconds simply by answering a phone call?
With OnTripFix, the data platform doesn't just alert—it actively investigates, consults knowledge bases, dials the on-call engineer via Call-E Voice AI, validates their identity securely, summarizes the proposed fix, captures spoken authorization, and automatically executes deterministic schema healing with zero laptop required.
YouTube Video Link - https://youtu.be/SCRXJr_r0qA
What it does
OnTripFix is an autonomous incident response and auto-remediation voice agent for Apache Airflow data pipelines:
- Incident Interception: When an Airflow DAG task fails (e.g., Sunday night Retail Inventory ETL failing due to an unexpected missing column
inventory_status), Airflow'son_failure_callbackimmediately posts the error payload to OnTripFix's FastAPI incident gateway. - Context Enrichment & Knowledge Retrieval:
- Jira REST API: Searches historical incident tickets and JQL for matching bugs (or falls back to previous incident matches like
RETAIL-4021). - Confluence Playbooks & Team Calendar: Looks up standard operating procedures and queries the active on-call schedule to identify the responsible engineer.
- AI Bug Diagnosis: If no existing playbook exists, LangChain AI diagnosis generates the necessary deterministic SQL remediation script.
- Jira REST API: Searches historical incident tickets and JQL for matching bugs (or falls back to previous incident matches like
- Interactive Call-E Voice AI Escalation:
- Places an outbound voice call to the on-call engineer's mobile device.
- Security First: Prompts the engineer to speak their Employee Security User ID before any operational details or remediation steps can be approved.
- Summarizes the failing DAG, task, Jira reference, and proposed SQL fix (
ALTER TABLE daily_store_inventory_agg ADD COLUMN inventory_status TEXT...). - Captures spoken approval and resolution instructions as structured JSON via Call-E's
result_schema.
- LangGraph StateGraph Auto-Remediation:
- Parses the spoken resolution instructions.
- Executes the DDL patch safely on the SQLite data store.
- Triggers an automated Airflow DAG task retry.
- Validates post-remediation data load integrity (verifying records processed, stock levels, and revenue aggregations).
- Transitions the corresponding Jira issue status to Done with full audit logs.
- Call-E Confirmation Voice Call: Dials the engineer back with a voice confirmation stating the pipeline is 100% green for Monday morning business operations.
- Real-time Telemetry Dashboard: A React frontend providing live queue inspection (
Error Queue,Resolution Queue), incident audits, and active on-call roster management.
How we built it
- Voice AI Agent: Integrated the Call-E Python SDK / REST API (
api.heycall-e.com/v1/calls) with strict E.164 phone formatting, result schemas for structured decision extraction, and regional/multilingual localization (supporting US English, India English, Hindi, and Tamil). - Remediation Engine: Built with LangGraph StateGraph and Google Gemini (Gemini 2.5 Flash) tool binding to parse spoken instructions, apply deterministic SQLite patches, trigger Airflow retries, and validate database consistency.
- Workflow & Webhook Server: Developed using FastAPI with background threading queues (
Error Queueconsumer andResolution Queueworker) for resilient, non-blocking telemetry enrichment and dispatch. - Observability & Ticketing: Implemented bi-directional Atlassian Jira REST API integrations (comment logging, transition updates) and Confluence runbook matching with local fallback safety layers.
- Frontend Dashboard: Built with React and Vite, featuring glassmorphism aesthetics, live queue inspection, audio call logs, and incident telemetry.
- Pipeline Simulation: Engineered an Apache Airflow DAG (
retail_inventory_etl) and end-to-end Python simulation harness (simulate_e2e_incident.py) reproducing production database crashes, automated triage, and verified recovery.
Challenges we ran into
- Deterministic Execution vs. LLM Non-Determinism: Voice instructions from an engineer must be translated into precise, syntactically correct SQL. We designed LangGraph nodes with strict tool-calling validation rather than allowing raw text generation directly against the production database.
- Security & Authentication over Voice: Preventing unauthorized voice approvals required embedding a mandatory User ID challenge step into the Call-E call task and enforcing
user_id_validated: Truein theresult_schemabefore any remediation node could trigger. - Public Privacy & Telephony Standards: Balancing live Call-E testing with open-source reference repository rules required sanitizing personal numbers, implementing phone masking (
mask_phone_number()), isolating credentials into gitignored sample templates (*.json.example), and ensuring default dry-run mock paths work offline without live telecom costs.
Accomplishments that we're proud of
- True Hands-Free Resolution: Complete end-to-end incident handling from Airflow crash to verified database repair in under 60 seconds without typing a single terminal command.
- Security-Gated Human-in-the-Loop: Autonomous remediation without blind agent actions—every state change requires authenticated human voice authorization.
- Production-Grade Architecture: Decoupled async queues, automatic fallback mechanisms (graceful degradation if Jira or Confluence APIs are unreachable), and localized multi-language on-call roster configurations.
- Seamless Developer Experience: Runs with one command (
python apps/python/ontripfix/simulate_e2e_incident.py) and passes a 10-test automated verification suite (pytest apps/python/ontripfix/tests/test_flow.py).
What we learned
- Structured Voice Extraction is Transformative: Call-E's server-side
result_schemavalidation removes the need for secondary prompt chains—spoken confirmations naturally translate into typed, reliable data models. - Progressive Fallbacks are Essential for Reliability: In on-call systems, external dependencies (calendars, ticketing systems) can fail simultaneously with pipelines. Building deterministic local fallback layers ensures the system stays operational under severe outages.
- Voice is the Ultimate Low-Latency Mobile Interface: For urgent operational approvals, a direct telephone call provides vastly higher responsiveness than Slack pings or email alerts.
What's next for OnTripFix
- Multi-Cloud Data Warehouse Support: Extending LangGraph database patch tools to support Snowflake, Google BigQuery, and Amazon Redshift.
- Smart Escalation Ladders: Automatically cascading to secondary and tertiary on-call leads if the primary engineer does not answer or declines the fix within a designated timeout window.
- Interactive Multi-Turn Voice Runbooks: Enabling engineers on the call to ask Call-E questions (e.g., "What was the error log from task 3?" or "Have we seen this error in the last 30 days?") before granting remediation approval.

