Case study
AI-assisted delivery inside the PBW codebase.
This case study documents PBW's branch-based, review-centered workflow for using Codex-style software agents on a real production website. The point is not autonomous shipping. The point is faster exploration with tighter review, clearer repository instructions, and controlled deployment discipline.
Baseline to current state
The workflow changed from isolated edits to managed implementation loops.
PBW's current repo instructions define a clear senior-engineer posture: read first, preserve existing design patterns, avoid destructive git actions, verify routes, and keep public claims grounded.
1. Brief
Convert a market or site objective into a scoped tranche with acceptance criteria.
2. Repo read
Inspect routes, views, shared context, CSS, analytics, and existing conventions before editing.
3. Branch change
Implement in a feature branch, staying close to the existing architecture.
4. Verification
Run syntax checks, route audits, screenshots, and targeted tests according to risk.
5. Review
Human review decides what ships, what is cut, and which claims need more evidence.
Before / after artifact
One route family, one real defect, and one verification loop.
The point of the workflow is not that an agent gets to decide readiness. The point is that it can trace a business objective or defect across the codebase, draft a bounded remediation, and produce evidence for human review.
| Artifact | Before | Codex-assisted remediation | Verification result |
|---|---|---|---|
| AI portfolio route family Route evidence |
AI implementation work existed as strategy notes, prompt cycles, and internal operating practice, not as an indexed public proof layer. | routes/ai-portfolio.js now owns the portfolio routes, metadata, schema inputs, and supporting page collection. | Routes are discoverable through navigation, breadcrumbs, sitemap generation, canonical metadata, and GA4/first-party event tracking. |
| Document-slot defect Bug evidence |
Admin document uploads can fail when a requested label is absent from the centralized document registry, returning Unknown document label. | The workflow traces the failure through routes/docs-admin.js, the public /docs/:label download path, and product pages that depend on those slots. | The release standard is: label exists, upload route accepts the slot, public download resolves, and no legacy proxy path reappears. |
| Digital product build Business evidence |
A consulting thesis can remain abstract if it is not converted into a priced asset, buyer journey, fulfillment path, and measurable funnel. | The AIM vs CRD Decision Kit and OMS Playbook Part 2 work connect product copy, Stripe checkout, document slots, fulfillment emails, gates, and nurture. | The evidence is not just a page. It is product configuration, paid-delivery plumbing, analytics hooks, and operational controls around buyer access. |
| Nurture delivery control Control evidence |
Email automation risk was not only copy quality. It included send enablement, provider configuration, logging, suppression, and scheduler boundaries. | The agent can map lib/email/send.js, nurture cron routes, unsubscribe checks, and outbound logging so the human owner can review operational risk. | Controls remain explicit: send disabled unless configured, in-process crons gated, outbound attempts logged, and purchaser/global suppression checked before sending. |
Representative workflow evidence
Where the agent helped, and where human control stayed in charge.
Each block shows the operating loop: define the problem, use AI for repo analysis and draft implementation work, keep human review over claims and risk, then verify the outcome.
EJS route and schema issue
- Problem
- AI portfolio pages needed clean route ownership, metadata, breadcrumbs, schema inputs, and analytics without becoming orphan content.
- AI-assisted action
- Repo analysis across Express routes, EJS templates, shared inner-layout context, sitemap generation, and GA4 event hooks.
- Human control
- Review of page claims, route boundaries, naming, and whether the section stayed inside PBW's brand voice.
- Outcome
- Cleaner route family with clearer proof architecture, canonical metadata, and portfolio-specific event tracking.
Admin workflow and document control
- Problem
- Document delivery needed a source-of-truth model so paid products and lead magnets could resolve through labeled slots.
- AI-assisted action
- Trace admin upload routes, public download aliases, R2 helpers, site document seeds, and product fulfillment dependencies.
- Human control
- Confirm which document labels are legitimate business artifacts and reject legacy or proxy-dependent paths.
- Outcome
- Reduced ambiguity around delivery ownership, upload slots, and public product download behavior.
SEO route hygiene
- Problem
- New proof content needed discoverability without diluting the core PBW consulting site or creating duplicate route surfaces.
- AI-assisted action
- Inspect route manifests, static SEO routes, navigation placement, footer links, canonical URLs, and internal cross-links.
- Human control
- Keep the AI portfolio commercially connected but prevent page-level CTAs from collapsing back into generic OMS lead generation.
- Outcome
- Safer claim framing, AI-specific CTAs, and cleaner discovery for reviewers and buyers.
Nurture delivery control
- Problem
- Email automation needed operational guardrails around provider ownership, scheduler activation, suppression, and logging.
- AI-assisted action
- Map send facade behavior, automation jobs, queue tables, global unsubscribe checks, and deployment defaults.
- Human control
- Review what may run by default, what requires PBW-owned credentials, and which logs or tokens must never be exposed.
- Outcome
- Clearer deployment control, safer instrumentation, and less ambiguity around live-send readiness.
Prompting patterns
Patterns that improved outcomes.
The strongest prompts behaved less like orders and more like operating procedures.
Constrain by architecture
Ask the agent to read the route, view, and CSS patterns before choosing where new work belongs.
Separate evidence from copy
Require claims to be tied to repo surfaces, observed workflow, or explicitly synthetic examples.
Define verification before polish
State which checks matter: route audits, syntax, responsive screenshots, or targeted unit tests.
Redacted agent prompt
Read the current Express/EJS routing structure before proposing changes. Identify the smallest safe route/view/module boundary. Do not change production secrets, destructive git operations, or client-facing claims. Return implementation plan, risks, files touched, and verification commands.
Strategy-to-build prompt
Review the attached feedback on the AI portfolio page. Assess it, plan a meaningful implementation, and improve the page in a detailed, sophisticated way. Use proof from the PBW business/product work, not generic AI claims.
Repository-bounded prompt
Read routes/ai-portfolio.js, views/ai-portfolio/*.ejs, public/css/ai-portfolio.css, and the local AI implementation plan first. Add evidence-bearing sections only. Acceptance criteria: AI-specific CTA, proof artifacts, no unsupported production claims, route verification passes.
Review-control prompt
Review the change as a skeptical financial-services buyer and production maintainer. Flag claims without receipts, route regressions, accessibility issues, and any language implying AI can approve compliance, UAT, investment, or production decisions.
Verification prompt
Run the narrow checks that prove this release: static route verification, public route rendering, CSV asset availability, and browser inspection at desktop and mobile widths. Report remaining risk instead of treating tool output as approval.
Workflow maturity
What changed about the way prompts are written.
The improvement is less about clever wording and more about turning a prompt into an executable control packet.
| Earlier pattern | Current pattern | Why it matters |
|---|---|---|
| "Improve this page." | "Read the route, templates, CSS, analytics hooks, and feedback file. Add proof artifacts without changing the site architecture." | The agent has a boundary, and the reviewer can tell whether it stayed inside it. |
| "Make the business more marketable." | "Tie the page to real PBW products: Decision Kit, Playbook Part 2, document delivery, nurture controls, analytics, and product fulfillment." | The AI portfolio becomes proof of business execution rather than a detached AI essay. |
| "Make it sound more professional." | "Tie claims to local artifacts, synthetic examples, or explicitly labeled implementation patterns." | Public copy becomes inspectable instead of merely polished. |
| "Run tests." | "Run route verification, inspect rendered pages, and state any checks that could not be completed." | Verification becomes evidence-bearing, not ceremonial. |
Failure modes and controls
The workflow assumes the agent can be wrong.
| Failure mode | Correction | Control retained |
|---|---|---|
| Over-broad refactor when a local change would work. | Re-scope to the smallest route/view/module boundary. | Human owner preserves architecture and commercial intent. |
| Public copy overstates AI or production capability. | Rewrite around current evidence, synthetic labels, and review gates. | Claim discipline before launch. |
| UI polish creates route, mobile, or accessibility regressions. | Use browser verification, screenshot review, and stable responsive constraints. | Visual QA remains part of the release loop. |
| Tool output is treated as approval. | Convert output into recommendations, tests, or drafts requiring review. | No silent production, compliance, or client-impacting decisions. |
Correction log
Examples of AI output that should be corrected, not shipped.
Scope drift
Bad output: proposed a broad visual redesign for a proof page. Correction: preserve the existing AI portfolio design system and add only evidence-bearing sections.
Overstated authority
Bad output: implied AI could approve UAT exit or compliance readiness. Correction: require explicit operations, compliance, and project-owner review gates.
False receipt
Bad output: treated internal file references as public proof. Correction: expose sanitized artifacts, route-backed examples, and a downloadable eval pack.
Security boundaries
Agent use is bounded by repository, data, and deployment controls.
What the agent may do
- Read public repository code and local project documentation.
- Draft scoped code changes and update public templates.
- Run non-destructive verification commands.
- Summarize risks and test gaps for human review.
What remains human-controlled
- Secrets, production credentials, and client confidential data.
- Deployment approval and public claim review.
- Commercial positioning and final editorial judgment.
- Any destructive git or filesystem action.
Translate this workflow discipline into your own operating model.
PBW can help frame where AI belongs in a financial-services workflow, which evidence it must use, and how review should operate before production rollout.
Next: financial-services AI evals