Pro Church Tools — Company OS

Audit Evidence Pipeline — From Audit Submission to Show + Podcast Source Material

10 min read

Audit Evidence Pipeline

For agents and team: How every audit submission becomes reusable evidence — timestamped moments + notes + production briefs — that feeds the Church Chaos YouTube show (visit briefs) and the Pro Church Tools Show (reactive podcast segment library). Connective tissue between README and the content production playbooks.

The two halves of every audit

Every audit produces two outputs from the same source (3 transcripts + website + form fields):

  1. The public audit — customer deliverable. CCI score, four-dimension breakdown, qualitative findings, Path Forward copy.
  2. The private evidence graph — content asset. Timestamped citations, qualitative observations, segment-tag candidates.

Surfaces involved

SurfaceWhat it isRole in this pipeline
The Comms Dept adminFirestore (churches/*, auditMoments/*, segmentLibrary/*, productionBriefs/*, weeklyBriefs/*) + Cloud Functions + admin app at admin.thecommsdept.comWhere evidence is extracted, tagged, and assembled into briefs.
Audit transcription pipelineSupadata + AssemblyAI via transcribeServices.jsStores per-service { text, segments[], durationMs, method, sourceUrl }. Sentence-level timestamps are the foundation everything else hangs off of.
Claude API@anthropic-ai/sdkPowers moment extraction, segment matching, and brief generation.
The Fort (Nucleus)ExternalSource of the church’s audit submission + service URLs.

The data model

auditMoments/{momentId}

The moment-level evidence graph. Every moment is a discrete beat from a service recording — an announcement, a transition, a giving ask, a welcome — anchored to a specific timestamp range and source service file.

auditMoments/{momentId}
  churchId, churchName        — denormalized for query convenience
  audioSourceFile             — service1 / service2 / service3
  sourceUrl                   — the original YouTube/audio URL
  startMs, endMs              — sentence-level timestamps in source
  transcript                  — verbatim text of the moment
  dimension                   — announcement_avalanche | where_do_i_go | sunday_savior | mixed_signals | null
  severity                    — low | moderate | high | critical | positive
  kind                        — announcement | transition | prayer | giving_ask | welcome | sermon_setup | response_call | scripture_reading | song_intro | communion | pastor_quote | visitor_callout | next_step_callout | volunteer_ask | small_group_ask | other
  tags[]                      — open-ended array of segment slugs (e.g., 'quietly_crushing_it')
  notes                       — qualitative analyst observation
  source                      — auto (extracted) | manual (admin-tagged)
  metadata{}                  — flexible per-segment payload
  createdAt, updatedAt

The tags[] array is the open-ended hook. Anything new gets added to a moment’s tags later via the matcher; the schema doesn’t lock you into a fixed set of segments.

segmentLibrary/{segmentSlug}

Segments — podcast and show buckets — are first-class data, not hardcoded.

segmentLibrary/{segmentSlug}
  name                        — display name, e.g., "Quietly Crushing It"
  description                 — what the segment is about
  matchCriteria               — text spec used by the matcher to identify candidate moments
  cadence                     — weekly | rotational | monthly | quarterly
  active                      — bool. inactive segments don't surface in briefs
  createdAt, lastReviewed

Add a doc here → segment exists. No code changes. New segments can match against every past audit retroactively (see Stage 5 below).

productionBriefs/{churchId}

One per church. Cached output of generateProductionBrief. Markdown, generated/regenerated on demand.

productionBriefs/{churchId}
  churchId, churchName
  markdown                    — the full populated production brief (sections 1-11)
  momentsCount                — how many moments fed this brief
  model                       — claude-sonnet-4-...
  generatedAt, generatedBy

weeklyBriefs/{periodIso}

One per week. Output of the Friday cron runWeeklyBrief.

weeklyBriefs/{periodIso}
  periodIso                   — date string like '2026-05-01'
  fromIso, toIso              — window covered
  markdown                    — the full populated weekly brief
  totalCandidates             — count of tagged moments in the window
  segmentsWithCandidates      — count of unique segments represented
  momentCount                 — total tagged moments
  generatedAt

Visit lifecycle (on the church doc)

For churches earmarked for an in-person visit (Church Chaos YouTube show):

church.visit.{
  status                      — candidate | scheduled | in_flight | completed
  candidateMarkedAt/By
  scheduledDate, scheduledAt/By
  crew{ host, producer, third }
  beats{                      — the 9 investigative pre-visit beats
    first_impression{ status, findings, updatedAt, updatedBy },
    google_business_profile{ ... },
    new_visitor_email{ ... },
    planning_to_visit{ ... },
    phone_call_test{ ... },
    kids_ministry{ ... },
    giving_flow{ ... },
    small_groups_signup{ ... },
    volunteer_ask{ ... },
  }
  completedAt/By
  followUpDueAt               — auto-set to completedAt + 90 days
  followUpDone
  notes
}

The lifecycle, stage by stage

Stage 0 — Audit transcribed with timestamps

What happens. transcribeServices pulls each service URL through Supadata (YouTube native captions or AI fallback) or AssemblyAI (non-YouTube). Output per service: { text, segments[], durationMs, method, sourceUrl }, where segments[] is sentence-level ({ startMs, endMs, text }). Sentence-level timestamps are required downstream for clip extraction (production briefs and podcast source tape).

AssemblyAI word_boost is extended with church vocabulary (foyer, narthex, vestibule, lectern, doxology, eucharist, etc.) to reduce phonetic mishears at transcription time. A deterministic post-extraction quote filter catches “forer”-style mishears the model occasionally ships despite the artifact-check rule.

Code: functions/src/transcription/transcribeServices.js. Shape helpers transcriptText, transcriptSegments, isTranscriptAvailable tolerate legacy plain strings on pre-Phase-0 audits.

Transcription-failed auto-handler. When the Mac Mini retry pass finishes and ≥1 service is still permanently dead, the audit auto-flips to rejected with reason transcription_failed, a HelpScout draft is created naming the failed link(s), and Slack pings.

Stage 1 — Public audit produced

What happens. analyzeChurch (called from onTranscriptsAdded) feeds the website content + transcripts to Claude. The model’s job is bucketing (Low/Medium/High per indicator), not prose — most indicators are deterministic and the model is overridden if it disagrees with the computed value. D4’s binary indicators (service times on homepage, address on homepage) are vision-detected via a Firecrawl full-page screenshot + Claude vision pass. Evidence prose is rendered server-side from indicator-name + bucket templates, not generated by the model. Crawler preserves footer + header (most church addresses live in the footer). Multi-campus references in announcements (“Briargate folks, see you Sunday”) are excluded from destinations extraction.

After scoring, analyzeChurch runs Gate 2 validation + sanity gates (see below), then writes the audit data to the church doc. Audit summary is capped at 350 chars / 2 sentences.

Sanity gates — auto-hold before delivery. Six flags: noAnnouncements (total = 0), shortServices (avg < 50 min), extremeAvgTime (outside [30s, 25min]), extremeAvgAnnouncements (outside [1, 30]), extremeDestinations (outside [1, 30]), missingLists/missingDimensions (malformed extraction). Sermon-only flags auto-create a HelpScout draft asking the church to re-upload full services + Slack ping with deeplink to admin task.

Rejection workflow. Two paths: Correctable (sermon-only, transcription failed, dead link) — admin emails via auto-created HelpScout draft, marks rejected, church can resubmit (new church record). Terminal (non-English, not-a-church-service, gaming) — admin clicks “Block from resubmitting,” sets audit.blockResubmit = true. Intake’s duplicate check then refuses any future submission from that email or domain. Reversible. The duplicate check blocks: in-progress audits, audits delivered within last 6 months, and blockResubmit === true.

Code: functions/src/audit/analyzeChurch.js. Methodology canonical at church-chaos-index.

Stage 2 — Moments extracted (the evidence graph populates)

What happens. After analyzeChurch succeeds, it calls extractAuditMoments as a best-effort post-step (failures don’t block audit delivery). The extractor walks the timestamped segments through Claude, asking for 10–25 noteworthy moments — announcements, transitions, prayers, giving asks, etc. — each with kind, dimension, severity, audioSourceFile, startMs, endMs, transcript, and note.

Output writes to auditMoments/{id} docs. Idempotent on re-run: deletes prior auto-extracted moments for the church before writing new ones; manually-tagged moments (source: 'manual') are preserved.

Failure surfacing. If extraction fails inline during analysis, church.auditMoments.lastError + lastErrorAt are stamped. The admin AuditMomentsSection surfaces this with a “click Re-extract to retry” hint.

Code: functions/src/audit/extractAuditMoments.js (the extractor) + functions/src/audit/auditMomentsApi.js (the manual-trigger callable).

Stage 3 — Manual tagging + private notes

What happens. Admins review moments per-church on the church detail page (AuditMomentsSection component). Each moment has an inline chip-toggle tag editor that lets the admin add or remove segment tags. Private notes can also be added alongside the standard church record.

This stage is the human-in-loop hook for everything the AI matcher misses or misclassifies.

Stage 4 — Visit pipeline (for show candidates)

What happens. When a church looks like a show candidate, an admin clicks “Mark as visit candidate” on the church detail page. That puts the church on the /visits pipeline at /admin.thecommsdept.com/visits — a status-grouped table with drive-time sorting (closer churches surface first, using the geocoded church.location.distanceKm).

The producer then runs the 9 investigative pre-visit beats against this church (first impression test, GBP check, new visitor email sequence, phone call test, kids ministry, giving flow, small groups signup, volunteer ask, planning-to-visit outreach). Each beat has a status (pending → in progress → done → skipped) and a findings text field on church.visit.beats.{slug}. Findings get used in the Production Brief and on the Visit Day Card.

When the visit is completed, the cron sweepVisitFollowUps fires a visit_follow_up task 90 days later (per production-brief-template §11 — “best window for return visit: 90–120 days post-makeover delivery”). That surfaces in the notification bell.

Code: functions/src/audit/visitFollowUp.js (cron) + apps/admin/src/pages/VisitsPipelinePage.jsx + VisitDayCardPage.jsx + components/churches/detail/VisitSection.jsx + VisitBeatsSection.jsx.

Stage 5 — Segment matcher (the backfill mechanism)

What happens. When a new segment is added to segmentLibrary (e.g., a year from now you invent “The Sunday Stutter” as a new podcast segment), an admin clicks “Run matcher” on that segment in the /segments page. The matcher iterates every church’s moments, asks Claude which moments fit the segment’s matchCriteria, and applies the segment’s slug as a tag on matching moments.

This is the “backfill across all past audits” capability the strategy doc calls for. New segments don’t need re-transcription, re-extraction, or schema changes — just a Claude pass over the existing moments.

Code: functions/src/audit/auditMomentsApi.js (matchSegmentToMoments callable).

Stage 6 — Brief generators

Two surfaces, both reading from the same evidence graph:

Production Brief — for an in-person visit. Per-church, on-demand. Reads the audit + this church’s moments + private notes, asks Claude to populate the production-brief-template (sections 1–11). Cached in productionBriefs/{churchId}. Triggered from the AuditMomentsSection on the church detail page or from the Visit Day Card. Renders as markdown.

Weekly Brief — for the podcast. Cross-church, scheduled. Runs every Friday 9am Eastern via runWeeklyBrief (cron). Pulls all moments tagged in the prior 7 days, groups by segment (per podcast-operating-doc § “The Weekly Brief”), renders a candidate slate. Stored in weeklyBriefs/{periodIso}. Slack ping when ready. Manual generation also available via generateWeeklyBriefNow callable.

Code: functions/src/audit/productionBriefApi.js + functions/src/audit/weeklyBrief.js.

Stage 7 — Backfill timestamps for legacy audits

What happens. Audits delivered before Phase 0 stored transcripts as plain strings (no timestamps). Those audits can be re-transcribed against their stored serviceLinks URLs to capture timestamps + segments via the reTranscribeWithTimestamps callable. After re-transcription, moments are re-extracted automatically.

Surfaced as a “Re-transcribe with timestamps” button in TranscriptManager on the church detail page. Lazy: one church at a time, admin-triggered.

Code: functions/src/audit/transcriptBackfillApi.js.

Open questions

  • Audio clip extraction. The pipeline produces timestamps + source URLs but doesn’t actually pull the clipped audio. The producer still has to manually pull each watch-back clip from the source recording. A future improvement: a callable that uses ffmpeg or yt-dlp to extract a clipped MP3 for any given moment, stored in Cloud Storage and referenced from the moment doc.
  • Cost guards. Every audit costs ~$0.05 of moment extraction. Each matcher run is N × ~$0.01 across N churches. Each brief is ~$0.10. At low volume fine; at thousands of audits worth a monthly cap or daily-cost alerting.
  • Episode lifecycle tracking. Visit pipeline tracks pre-production through visit-day. Post-visit (footage tagging, cut tracking, episode shipped, follow-up window) isn’t modeled yet.

Source of truth (code paths)

  • functions/src/transcription/transcribeServices.js — Stage 0: timestamp capture + shape helpers
  • functions/src/audit/analyzeChurch.js — Stage 1: public audit + inline trigger of Stage 2
  • functions/src/audit/extractAuditMoments.js — Stage 2: moment extraction
  • functions/src/audit/auditMomentsApi.js — Stages 3 & 5: manual extract callable + segment matcher
  • functions/src/segments/seedSegments.js — Stage 5 seed (one-time starter library)
  • functions/src/audit/visitFollowUp.js — Stage 4: 90-day follow-up cron
  • functions/src/audit/productionBriefApi.js — Stage 6: production brief generator
  • functions/src/audit/weeklyBrief.js — Stage 6: weekly podcast brief (cron + callable)
  • functions/src/audit/transcriptBackfillApi.js — Stage 7: timestamp backfill
  • Admin UI: apps/admin/src/pages/{SegmentsPage,WeeklyBriefPage,VisitsPipelinePage,VisitDayCardPage}.jsx + components/churches/detail/{AuditMomentsSection,VisitSection,VisitBeatsSection}.jsx

Graph View

Backlinks