Beyond the Leaderboard: Building a Codex-Powered Improvement Harness on ViralBench

ViralBench puts AI models inside a live attention market. I am building the engineering and evaluation layer around it so Codex can inspect failures, implement bounded changes, run replay tests, and promote improvements only when the evidence supports them.

Research notes · AI Systems Engineering · Published July 9, 2026 · By Sulayman Bowles

ViralBench gives AI models a live market, multimodal tools, and a measurable objective. I am building the layer above it: an evidence-backed system in which Codex can diagnose the agent, change its harness, replay past conditions, and test improvements without being allowed to grade or deploy its own work.

Most AI benchmarks stop when the model returns an answer.

ViralBench starts after that point. It gives frontier models real TikTok accounts, access to current social-content intelligence, the ability to inspect reference posts, an image generator, a slideshow renderer, and a publishing tool. The models are asked to maximize views in the fitness category. They run repeatedly, see prior account performance, and make another decision.

That is a much more demanding environment than a static prompt set. The agent has to search, interpret visual material, choose a content thesis, allocate a limited tool budget, generate an artifact, inspect its own work, publish it, and learn from delayed market feedback. A fluent answer is not enough. The work has to survive contact with an external system.

After reading the live methodology and auditing the code used to run the agent, I think the most valuable part of ViralBench is not the model leaderboard. It is the environment beneath the leaderboard.

The open-source implementation is a compact, working control loop for autonomous marketing. My next step is to turn that loop into a proper experimental harness and place Codex above it as a bounded engineering agent.

Codex will not be asked to write a better TikTok carousel directly. It will be asked to improve the system that researches, creates, reviews, and publishes the carousel. It will inspect traces, form a narrow hypothesis, modify a defined part of the repository, run offline replay tests, produce a reviewable patch, and submit a candidate for a controlled live trial.

The distinction matters:

The ViralBench agent tries to make a successful post. Codex tries to make the ViralBench agent more reliable and effective.

That outer loop is the project I am building.

ViralBench is a live agent environment

The public ViralBench experiment currently compares GPT-5.5, Claude Opus 4.8, and Kimi. Each model receives two warmed-up TikTok accounts. It runs at approximately 7 AM and 5 PM Central Time, with one stated objective: get as many views in the fitness space as possible.¹

The repository I reviewed is the standalone handoff version of that system. It is deliberately compact: one 937-line TypeScript file contains the prompt, agent loop, API clients, media handling, account telemetry, memory, rendering, publishing, and local run persistence.² The public handoff is configured for one account rather than the two-account live competition, and it leaves a reviewable draft by default instead of automatically scheduling the post.

At runtime, the agent receives up to 18 rounds. Within those rounds it can use five tools:

The agent is vision-capable. TikTok and Instagram URLs are resolved into actual images, resized, labeled, and passed back into the model’s context. Generated images receive their own labels and can become references for later generations. This creates a rudimentary visual working memory inside a single run.

The loop also has real resource constraints. It allows no more than six Lightreel research calls per run, one research call in any given round, up to 12 viewed reference images per round, and a fixed 18-round working budget. The main model is called through OpenRouter with an append-only conversation history. Actions in the same round are executed concurrently.

The resulting workflow is closer to an operating process than a content-generation prompt:

Account performance and prior notes
                ↓
Current-culture research
                ↓
Full-post visual inspection
                ↓
Content thesis and slide design
                ↓
Image generation and image-to-image adaptation
                ↓
Rendered slideshow preview
                ↓
Revision
                ↓
Draft or scheduled post
                ↓
Delayed TikTok metrics

That loop contains several forms of uncertainty at once.

The model does not know which retrieved examples are representative. It cannot observe TikTok’s distribution mechanism. It receives sparse feedback from a small number of posts. A strong post can receive weak initial distribution; a mediocre one can catch a transient wave. The model also has to decide when to repeat a format, when to alter it, and when to abandon it.

This is why ViralBench is interesting. The agent is not merely generating media. It is operating under partial observability, limited actions, a changing external environment, and delayed rewards.

What the current code gets right

The implementation has several choices worth preserving.

First, it forces external research. The system prompt explicitly tells the model not to treat its training data as a reliable guide to current culture. The research budget is limited, so the agent must decide what to ask rather than calling retrieval indefinitely.

Second, it makes the model inspect complete artifacts. The model can view the source carousel, generate its own images, and then see the final text-on-image composition. That is more useful than asking a text-only critic whether a visual asset is good.

Third, it keeps the loop small enough to understand. The repository is not hidden behind a large framework. The control flow, tools, limits, failure handling, and memory are visible in one file. For a public handoff, that is an advantage.

Fourth, it puts a real outcome on the other side of the agent. Views are imperfect, but they are not an LLM judge score. The platform supplies the response.

Finally, the default public configuration stops at a draft. The live benchmark can operate autonomously, but the handoff code preserves a human-review boundary. That is the correct starting posture for anyone adapting the system to a company or client account.

The code is therefore a strong experimental seed. It already contains the inner loop. What it does not yet contain is the infrastructure needed to make trustworthy claims about improvement.

The leaderboard is a system result, not a pure model score

ViralBench describes itself as an AI marketing benchmark. That is directionally right, but the object being measured needs to be stated precisely.

A post’s result is produced by a full stack:

result = f(
  language model,
  system prompt,
  tool definitions,
  retrieval quality,
  reference-content selection,
  image model,
  account history,
  memory,
  posting time,
  publishing infrastructure,
  current TikTok conditions,
  random distribution effects
)

Changing the base model can change the result. So can changing the Lightreel query, the image generator, the prompt’s copying instructions, the account’s recent history, the available sound, or the time at which the post enters the feed.

This means the leaderboard is best understood as a live system benchmark or field trial of agent stacks. It is not a clean isolation of model intelligence.

That does not weaken the experiment. In many commercial settings, system performance is exactly what matters. A company does not buy an abstract model capability; it deploys a model inside prompts, tools, data, permissions, interfaces, and controls. The assembled system succeeds or fails.

The distinction becomes important when trying to improve it. If one model wins on a given week, the correct conclusion is not automatically that it is the best marketing reasoner. It may have used the available harness more effectively. It may have benefited from a format that happened to work for one account. It may have been more willing to imitate a current winner. It may simply have received a favorable distribution event.

For my build, the unit of analysis will therefore be the model-plus-harness configuration, not the model name alone.

Every result needs to be tied to a versioned configuration. Every configuration change needs a stated hypothesis. Every promotion decision needs to distinguish observed lift from a lucky outlier.

What the code audit reveals

The current file is described in its own header as a self-contained handoff. I would not expect a handoff script to contain a full experimentation platform. Still, the exact gaps matter because they define the work Codex must eventually be able to perform.

1. The run record is too thin to explain success or failure

The agent writes a local .marketing-agent-runs.json file. Each entry preserves the goal, published posts, final answer, reflection note, and round count.

It does not preserve the complete message history, individual tool inputs and outputs, Lightreel source material, viewed-media lineage, image prompts, preview assets, model usage, latency, cost, configuration hash, Git commit, or account-metric snapshots.

A post can therefore outperform without leaving enough evidence to explain why. Conversely, a run can fail because of retrieval, image generation, publishing, or model behavior without a durable trace that separates those causes.

For an improvement harness, the trace is not debugging exhaust. It is the primary dataset.

2. Several important constraints exist only as prompt prose

The prompt tells the agent to preview before publishing, use one distinct image per slide, publish exactly one post per account, and finish only after completing the work.

The runtime enforces a maximum number of posts per account. It does not enforce a minimum before accepting the final answer. It does not require a successful preview token before publication. It does not reject duplicate image URLs. It does not validate slide count, copy length, text fit, unsupported claims, or whether every image came from the current generation lineage.

This is a common agent-design failure mode: a policy is written as an instruction and treated as if it were an invariant.

Instructions influence model behavior. Invariants should be enforced by code.

3. Visual review is incomplete for longer carousels

The preview function renders no more than six slides, even when the candidate contains more. The agent can therefore publish a longer carousel without seeing the final slides as composed assets.

A harness should either render every slide or explicitly block publication when complete visual review has not occurred.

4. Cross-run memory is compressed into free-form notes

The only durable strategic memory loaded into a future run is up to ten reflection notes written by the model itself.

This is cheap and useful, but it mixes facts, interpretations, and narrative into one unstructured field. There is no guarantee that the note accurately reflects the trace. There is no stable representation of tested hypotheses, confidence, failed formats, account-specific effects, or unresolved questions.

The model is effectively writing its own institutional memory without an evidence check.

5. Account telemetry has attribution edge cases

For recent posts with a public URL, the code refreshes TikTok metrics through ScrapeCreators. When the URL is missing, it attempts to match a profile post by a normalized prefix of the caption.

That fallback is practical, but two similar captions can create a false match. A production experiment ledger needs durable post IDs and explicit metric provenance.

6. Same-round parallelism creates hidden execution semantics

Multiple actions are run with Promise.all. That is efficient when the actions are independent. It also means one image generation in a round cannot reliably depend on the label produced by another generation in that same round. The second action must wait for a later round to reference the first result.

The prompt does not formally express this dependency rule. A typed planner or action graph would make it explicit.

7. The content-acquisition strategy creates provenance risk

The system prompt tells the agent to use real winning posts, closely adapt their language, and in some cases mirror their slide-by-slide structure. That may improve short-term performance, but it collapses several distinct behaviors into one instruction:

• learning a general format;
• adapting a common hook pattern;
• paraphrasing a specific creator’s copy;
• reproducing a distinctive sequence or visual treatment.

Those are not equivalent. The current system does not retain a source graph or calculate similarity between the references and the final artifact.

A durable marketing agent should be able to learn the mechanism behind a successful post without reproducing another creator’s work.

8. The package is not fully reproducible as shipped

The source imports sharp for image resizing, and the file header instructs the user to install it. The checked-in package.json does not declare sharp as a dependency.

This is a small issue, but it is exactly the kind of issue that matters once Codex is expected to run unattended tests in a clean environment. Reproducibility starts with a repository that installs and executes from its manifest.

None of these observations invalidate ViralBench. They show the difference between a working agent and an experimental system that can support causal claims, regression testing, controlled autonomy, and repeatable engineering.

What I mean by “Codex as a harness”

The word harness is used in several overlapping ways in AI engineering.

OpenAI describes the Codex harness as the agent loop and execution logic that connects a model to instructions, tools, filesystem operations, and feedback.³ OpenAI’s more recent work on harness engineering expands the concept further: the repository, documentation, tests, review process, execution environment, and validation rules all shape what the agent can reliably do.⁴

In my design, ViralBench becomes an executable marketing environment that Codex can work on. Codex sits outside the content-production loop as an engineering agent.

There are four possible roles Codex could play:

The last boundary is the most important.

Codex should not be able to modify the evaluator, run the treatment, inspect the result, redefine success, and deploy itself. That would create a direct path to metric gaming and accidental policy erosion.

Instead, the system should separate five functions:

1. The inner agent researches, creates, previews, and submits a post.
2. The recorder captures raw evidence, actions, artifacts, and outcomes.
3. The evaluator applies versioned checks and statistical rules.
4. Codex diagnoses a bounded issue and produces a patch in an isolated worktree.
5. The release controller decides whether the candidate can enter replay, draft canary, or live deployment.

The control flow looks like this:

                  LIVE ENVIRONMENT
      Lightreel / TikTok / Doublespeed / media APIs
                         ↑       ↓
                ┌───────────────────┐
                │ ViralBench agent  │
                │   inner loop      │
                └───────────────────┘
                         ↓
          immutable traces, sources, artifacts,
            configuration, cost, metric snapshots
                         ↓
                ┌───────────────────┐
                │ Evaluation layer  │
                │ checks + analysis │
                └───────────────────┘
                         ↓
              bounded experiment candidate
                         ↓
                ┌───────────────────┐
                │ Codex worktree    │
                │ plan / patch/test │
                └───────────────────┘
                         ↓
             offline replay and regression gate
                         ↓
               human-approved draft canary
                         ↓
                 locked online evaluator
                         ↓
                   promote or revert

This is not unrestricted recursive self-improvement. It is a controlled software-improvement loop with an agent doing much of the engineering work.

The evidence layer comes before the Codex layer

My work on Atlas has pushed me toward a consistent design rule: raw observations and derived judgments should not occupy the same layer.

In a technical SEO or retrieval audit, the raw HTTP response, rendered DOM, sitemap entry, schema block, and search result are evidence. An issue label, quality score, or recommendation is an interpretation built from that evidence. The interpretation can change. The underlying observation should remain available.

ViralBench needs the same separation.

For every run, the system should preserve an immutable evidence bundle containing:

• the Git commit and experiment manifest;
• model provider, model version, reasoning settings, and sampling parameters;
• the exact system prompt and tool schemas;
• account state at the beginning of the run;
• every Lightreel query and response;
• every source URL and retrieved media item;
• hashes of source and generated images;
• image prompts and reference-image lineage;
• every preview render;
• the final slide data and caption;
• publishing response and stable TikTok post ID;
• token usage, API cost, latency, and tool errors;
• metrics captured at fixed post-publication windows;
• human review decisions and reasons.

Derived outputs would live separately:

• “insufficient research breadth”;
• “slide 1 text is likely unreadable”;
• “final copy is too similar to source post X”;
• “the treatment improved account-normalized 72-hour views”;
• “the agent repeated a failed format despite contrary evidence.”

Every derived finding should reference the evidence IDs that support it. Codex can propose a new finding or a code change. It cannot rewrite the raw trace.

This evidence spine serves four purposes at once.

It makes a failure reproducible. It creates training data for future evaluators. It gives Codex enough context to investigate a specific problem. It also prevents the system from turning a model-written narrative into accepted truth.

The experiment is the unit of change

Codex should never receive a task such as “make the agent more viral.” That is too broad to test and too easy to satisfy cosmetically.

A useful task starts with a falsifiable hypothesis and a protected set of variables.

For example:

experiment_id: visual-critic-v1
hypothesis: >
  A separate visual-critique pass after rendering will reduce
  text-overflow, repeated-image, and synthetic-looking-image failures
  without lowering 72-hour account-normalized views.

baseline_commit: 5f5f57e251023ceb37961c0fc2c808f67ceb71eb
editable_surfaces:
  - agent/critics/visual.ts
  - evals/visual/*
  - experiments/visual-critic-v1.yaml

protected_surfaces:
  - evals/online/primary-metric.ts
  - policies/publish-gateway.ts
  - traces/raw/*
  - secrets/*

offline_gates:
  - all_slides_rendered
  - no_duplicate_images
  - no_text_overflow
  - source_lineage_complete
  - historical_replay_pass_rate >= 0.95

live_design:
  unit: post
  blocking:
    - account
    - posting_slot
    - calendar_week
  primary_window: 72h
  minimum_pairs: 20

primary_metric: account_adjusted_log_views_72h
guardrails:
  - publish_failure_rate <= baseline
  - median_cost_per_post <= baseline * 1.15
  - policy_violation_count == 0
  - source_similarity_high_risk_count == 0

promotion_rule: >
  Pass every hard guardrail and show a positive primary-metric estimate
  with the pre-registered uncertainty threshold.

The point is not this exact schema. The point is that the hypothesis, editable scope, validation commands, metrics, guardrails, and stopping rule are defined before Codex edits the repository or the live result is known.

This makes the output reviewable. It also makes failure useful. A rejected experiment still adds evidence about what did not work.

Offline replay should absorb most iteration

Live TikTok trials are slow and noisy. They also consume accounts, publishing slots, API spend, and attention. Codex should not need a real post every time it changes a parser or prompt.

The first major addition should therefore be a replay environment.

Historical tool calls can be converted into versioned fixtures:

• frozen Lightreel responses;
• saved TikTok and Instagram media metadata;
• locally stored reference images or content-addressed snapshots;
• Doublespeed style presets and render responses;
• account-statistics snapshots;
• image-generation stubs or a fixed asset bank;
• publishing mocks;
• known failure responses from each dependency.

The inner agent can then be run repeatedly against the same world state. That makes prompt changes, tool-schema changes, memory changes, and model substitutions comparable.

Offline replay cannot predict views. It can test everything that should be true before a post deserves a live trial.

Deterministic checks

Some failures should be scored without another model:

• Did the agent remain within its research and round budgets?
• Did it inspect source material before generating?
• Was every generated asset linked to a valid reference or explicit text-only path?
• Was every slide rendered?
• Were duplicate image URLs rejected?
• Did publication occur only after a passing preview?
• Did the final account count match the experiment contract?
• Were claims, source IDs, and artifact hashes complete?
• Did the agent recover correctly from a tool timeout?
• Did it leave the repository and fixture state clean?

Rubric-based checks

Other qualities are not fully reducible to deterministic rules:

• Is the first slide visually legible at mobile size?
• Does the carousel maintain a coherent subject across slides?
• Does the copy preserve the mechanism of a reference without reproducing it?
• Does the post contain unsupported fitness or health claims?
• Does the final creative look internally consistent?

These can use model-based graders, but the grader should receive the evidence and a narrow rubric. Its output should be treated as a scored judgment, not ground truth.

Adversarial and fault-injection cases

A real harness also needs failure tests:

• Lightreel returns a plausible but source-poor answer.
• ScrapeCreators resolves only half of a carousel.
• A reference image is too large or corrupt.
• Image generation returns the same asset twice.
• The renderer clips the last line of text.
• Sound extraction fails.
• Doublespeed creates a draft but returns no public review URL.
• TikTok metrics arrive late.
• Two profile posts have nearly identical caption prefixes.
• The model attempts to finish before publishing the required artifact.

The agent should fail safely and leave enough trace to diagnose what happened.

OpenAI’s own guidance on agent evals uses a similar structure: prompt, captured run, trace and artifacts, checks, and a comparable score.⁵ That is the standard I want to apply here.

Live evaluation needs experimental discipline

Once a candidate passes replay, it can enter a controlled live trial. This is where ViralBench’s real-market advantage becomes valuable, but it is also where weak measurement can create false confidence.

Views are noisy and path-dependent

TikTok outcomes are affected by account history, current content supply, audience composition, timing, trend decay, sound, source similarity, and platform distribution. The same creative cannot be posted twice to the same audience under identical conditions.

The accounts also change because the agent acts on them. A successful post can alter the audience and distribution of later posts. The experiment is therefore sequential, not a collection of independent static rows.

A simple average of raw views across two groups will often be misleading, especially with small samples and rare large winners. Online business metrics commonly exhibit skew and heavy tails, which is why robust estimators, covariate adjustment, and longitudinal methods are often used in controlled experiments.⁶

The trial should be blocked and paired

With a limited number of accounts and posting slots, I would use a blocked design rather than comparing all treatment posts with all historical control posts.

Each candidate should be paired against a contemporaneous baseline while controlling, as far as the system allows, for:

• account;
• morning or evening slot;
• day or calendar week;
• broad content lane;
• recent account performance;
• number of slides;
• use of sound;
• paid or organic distribution, if that ever changes.

A switchback or crossover design can alternate baseline and treatment configurations across comparable account-slot blocks. Pre-experiment account performance can be used as a covariate, in the spirit of variance-reduction methods such as CUPED, rather than pretending every account begins from the same level.⁷

Measurement windows should be fixed in advance

The ledger should capture views and engagement at fixed horizons, for example:

• one hour;
• six hours;
• 24 hours;
• 72 hours;
• seven days.

The primary horizon should be chosen before the experiment starts. A result should not be called a win because its best-looking window happened to be six hours rather than 72.

I would use an account-adjusted transformation such as log(1 + views) for the primary analysis, alongside the raw distribution. The log scale reduces the ability of one extreme viral post to dominate the estimate. Median and trimmed-mean comparisons can be reported as robustness checks.

The system should still display every raw post result. Robust statistics are not a reason to hide outliers; they are a way to keep one outlier from becoming the entire conclusion.

Hard guardrails should precede optimization

The evaluator should not collapse every concern into one weighted score. A sufficiently large view count should not compensate for a copyright risk, a misleading health claim, or a broken disclosure.

The optimization problem is better stated as:

maximize expected account-adjusted views at the locked horizon

subject to:
- platform and disclosure requirements pass
- source-provenance checks pass
- factual and health-claim checks pass
- publishing and rollback checks pass
- cost remains within budget
- human approval is present where required

Only feasible candidates compete on the primary outcome.

A/B tests and bandits answer different questions

A multi-armed bandit may eventually be useful for allocating more posts to strategies that appear to work. It is not a substitute for causal evaluation.

An A/B or switchback trial asks: did this change improve the expected result under comparable conditions?

A bandit asks: given current beliefs, where should the next unit of traffic or publishing capacity go to maximize reward while still learning?

The first is designed for inference. The second is designed for allocation. I would use controlled trials to validate harness changes, then use bounded bandit logic for choosing among proven content strategies. Mixing the two too early can produce a system that earns more views but cannot explain why.

Codex should receive bounded engineering tasks

Codex can inspect, edit, and execute code through the CLI, non-interactive codex exec, the TypeScript SDK, or a larger orchestration layer.⁸ The SDK is particularly useful when a service needs to start and resume coding threads programmatically. Worktrees and sandbox policies provide practical isolation for parallel experiments and restricted write access.⁹

The integration should begin with the simplest reliable surface.

A first version could use codex exec in CI with a structured output schema. Later, the Codex SDK can manage persistent experiment threads, staged tasks, and richer orchestration.

Each task should specify:

• a single concrete goal;
• the working directory and base commit;
• task mode;
• files that may be edited;
• protected paths;
• raw evidence inputs;
• known failure reproduction;
• required tests and replay suites;
• artifact locations;
• output schema;
• done criteria;
• conditions that require stopping rather than guessing.

The task modes I plan to support are:

“Deploy” should be a narrow operational mode, not permission to decide that deployment is warranted.

A representative Codex handoff might read:

Mode: Implement
Goal: Make a successful full-slideshow preview a hard prerequisite for publishing.

Evidence:
- Trace VB-2026-07-08-017 published eight slides after previewing only six.
- The current preview function caps max_slides at six.
- publish_slideshow has no preview receipt check.

Editable paths:
- agent/rendering/**
- agent/publishing/**
- evals/preview-gate/**

Do not edit:
- evals/online/**
- policies/**
- traces/raw/**
- experiment promotion thresholds

Verification:
- npm test
- npm run eval:preview-gate
- npm run replay -- --case VB-2026-07-08-017

Done when:
- every slide is rendered or publication is blocked;
- a preview receipt is bound to the exact slide payload;
- the historical failure becomes a passing regression case;
- RESULTS.md contains commands, results, changed files, and remaining risks.

That is a task Codex can execute and a human can review. “Improve visual quality” is not.

The repository itself must become legible to Codex

Codex performance depends on what the repository makes available. OpenAI’s harness-engineering work makes a useful point: information that lives only in chat threads, documents, or a person’s memory is effectively invisible to the agent while it works.⁴

I would restructure the current handoff into explicit modules and versioned artifacts:

/agent
  /policy              system prompts, action schemas, run policy
  /runtime             loop, state machine, dependency handling
  /memory              structured cross-run state

/tools
  /lightreel
  /media
  /images
  /rendering
  /publishing
  /metrics

/experiments
  /manifests            hypotheses, protected variables, promotion rules
  /candidates           Codex-ready tasks and results

/replay
  /fixtures             frozen external responses and assets
  /worlds               named historical environment states
  /faults               dependency and data failure scenarios

/evals
  /deterministic
  /rubrics
  /visual
  /online
  /regression

/traces
  /raw                  append-only event records
  /indexes              searchable run and artifact indexes

/artifacts
  /source-media
  /generated-media
  /previews
  /published

/reports
  /runs
  /experiments
  /leaderboard

/codex
  AGENTS.md
  /skills
  /schemas
  /hooks
  /task-templates

/policies
  publish-gateway.ts
  provenance.ts
  platform-compliance.ts
  health-claims.ts
  cost-limits.ts

AGENTS.md should be a map, not a giant manual. It should point Codex to architecture, evidence contracts, task modes, validation commands, and protected surfaces. Detailed decisions belong in versioned documents and schemas close to the code they govern.

Deterministic hooks can check that a Codex task includes an experiment ID, that protected files were not altered, that generated results reference a trace, and that required evals ran before a pull request is accepted. The agent can reason about the change; the repository can mechanically enforce the contract.

Memory should become structured belief, not autobiography

The current note mechanism asks the agent to leave a short reflection for its future self. That is a useful bootstrap, but a stronger memory layer should separate:

• observation;
• hypothesis;
• decision;
• result;
• confidence;
• scope;
• expiration.

For example:

{
  "belief_id": "BEL-0041",
  "account": "fitness-account-a",
  "claim": "Beginner mistake carousels outperform motivation quotes in the evening slot",
  "evidence_runs": ["RUN-0108", "RUN-0112", "RUN-0117"],
  "counterevidence_runs": ["RUN-0120"],
  "confidence": 0.63,
  "valid_from": "2026-07-01",
  "review_after": "2026-07-15",
  "status": "provisional"
}

The model can still write a narrative note. The system should derive or propose structured beliefs from the trace and require the evidence links to be valid.

This also guards against stale strategy. Social conditions change quickly. A belief should expire or lose weight when its evidence is old, the account changes, or new results contradict it.

The goal is not to create an ever-growing memory store. It is to create a small, contestable set of beliefs the agent can use without mistaking repetition for truth.

Provenance is a product requirement, not an afterthought

The current agent’s strategy is deliberately aggressive about adapting winning content. That makes provenance one of the first systems I would add.

Every final slide should have a lineage record:

final slide
  ├── generated image hash
  │     ├── generation prompt
  │     ├── image model and version
  │     └── reference source image hash
  ├── overlay text
  │     ├── source post(s) consulted
  │     ├── similarity score
  │     └── transformation rationale
  └── visual structure
        ├── source pattern tags
        └── originality / duplication checks

This lets the system distinguish between learning and copying.

A common pattern such as “problem, explanation, correction, payoff” is not owned by one creator. A distinctive sequence of sentences, examples, and visual compositions may be much closer to reproduction. The evaluator should detect that difference and route high-risk cases for review.

The same layer should manage AI-generated-content disclosure. TikTok currently requires creators to label realistic AI-generated images, audio, and video, and may automatically label content through Content Credentials.¹⁰ ViralBench’s image prompts explicitly seek realistic, native-looking imagery. A publishing harness should therefore carry disclosure metadata as part of the artifact and fail closed when the required setting cannot be verified.

For fitness content, factual provenance matters as well. A system optimized only for views can discover that extreme nutrition claims, impossible transformations, or fear-based health advice attract attention. The solution is not a vague “be safe” sentence in the prompt. Claims that touch health, injury, supplementation, or disease should pass a separate evidence and policy gate before publication.

“Maximize views” is a useful benchmark objective and an incomplete business objective

The current goal is unusually explicit: views are the only objective, and brand safety, selling, and follower count do not matter.

That sharp objective is one reason the experiment is revealing. It removes the model’s ability to hide behind a balanced but unmeasurable marketing answer. The agent has a number to pursue.

It also creates predictable behavior. Any unconstrained optimizer will search for the cheapest route to the metric. In this environment that can mean copied hooks, outrage, exaggerated promises, generic high-arousal content, or short-term account tactics that damage future performance.

I would keep two modes separate.

Benchmark mode

The purpose is to study attention capture under controlled conditions.

The objective remains account-adjusted views, but hard constraints cover platform policy, disclosure, provenance, factuality, cost, and publishing integrity. This preserves the clarity of the benchmark without rewarding obvious rule-breaking.

Enterprise mode

The objective reflects the actual business system:

• qualified reach;
• attributable site visits;
• assisted conversions;
• customer acquisition cost;
• lead quality;
• retention or repeat engagement;
• brand and compliance constraints.

Enterprise mode should not simply replace views with a large weighted score. Different metrics operate at different horizons and have different causal relationships. The harness should distinguish leading indicators from final outcomes and record attribution uncertainty.

A post can generate large reach and no commercial value. It can also generate a small number of views from the exact audience a company needs. Those are different optimization problems.

ViralBench is a good laboratory because it begins with a clean, visible reward. The commercial system should be built by adding constraints and downstream evidence, not by pretending views are revenue.

The first build sequence

I would not begin by letting Codex rewrite the prompt. The first work is to make the existing baseline reproducible and observable.

Phase 0: Freeze and document the baseline

• Pin the current commit and dependency versions.
• Add the missing sharp dependency and a lockfile.
• Record the exact live and handoff configurations separately.
• Create an architecture map and a canonical run schema.
• Save a small set of representative historical runs as reference cases.

The output of this phase is a baseline that can be installed and replayed.

Phase 1: Add complete traces and hard runtime invariants

• Replace the thin snapshot with append-only event traces.
• Hash prompts, tool schemas, source assets, generated assets, and final payloads.
• Require full preview before publication.
• Enforce minimum and maximum post counts.
• Reject duplicate images and malformed slide payloads.
• Bind a preview receipt to the exact content being published.
• Persist stable post IDs and fixed-window metric snapshots.

The output is an agent whose behavior can be explained after the fact.

Phase 2: Split the monolith and build replay

• Extract provider and tool adapters behind typed interfaces.
• Add mock implementations and frozen fixtures.
• Build deterministic checks and regression cases.
• Add fault injection for each external dependency.
• Preserve the original single-file behavior as a compatibility baseline.

The output is a local environment in which changes can be tested without publishing.

Phase 3: Introduce Codex as a reviewed engineering agent

• Add AGENTS.md, task templates, output schemas, and protected-path rules.
• Run Codex in isolated worktrees with workspace-scoped permissions.
• Start with Plan, Review, Debug, and Implement modes.
• Require a human to approve every patch and experiment manifest.
• Store Codex transcripts, diffs, test results, and review decisions.

The output is faster engineering, not autonomous deployment.

Phase 4: Add controlled draft canaries

• Allow a passing candidate to create a draft on a designated test account.
• Run visual, provenance, claim, and disclosure checks on the final artifact.
• Require human approval to publish.
• Collect fixed-window outcomes through the locked evaluator.
• Compare treatment against a concurrent blocked baseline.

The output is an online experiment with a reliable rollback point.

Phase 5: Automate promotion within narrow boundaries

Only after the offline and online gates have a strong record should the system promote low-risk changes automatically.

Early candidates for bounded automation might include:

• parser fixes;
• retry and timeout handling;
• trace completeness;
• renderer validation;
• budget enforcement;
• memory retrieval changes that pass replay;
• switching among already-approved prompt variants.

High-impact changes should remain reviewed:

• publishing permissions;
• content-policy logic;
• provenance thresholds;
• health-claim policy;
• primary metrics;
• promotion rules;
• production credentials;
• changes that expand Codex’s editable scope.

Autonomy should grow by demonstrated reliability in a defined surface, not by granting broad permission and hoping the agent behaves.

The experiments I would run first

Once the trace and replay layers exist, several questions become testable.

Does a separate visual critic improve the final artifact?

The current agent both creates and judges its slideshow. A separate critic may catch repeated imagery, inconsistent people, text overflow, or artificial-looking compositions. It may also add cost and over-edit good work.

This is a clean offline-first experiment because many visual defects can be measured before publication.

Does structured memory outperform free-form notes?

A treatment could replace or supplement the ten narrative notes with evidence-linked beliefs, account-specific hypotheses, and expiration dates.

Offline replay can test whether the agent uses the relevant memory and ignores stale or contradictory beliefs. Live trials can test whether that produces more consistent account-adjusted performance.

Is candidate generation better than single-path creation?

The agent currently tends to commit to a strategy within one run. Another design would generate several cheap concepts, score them against current evidence, and spend image-generation budget only on the strongest candidate.

The risk is that a critic rewards generic consensus and removes unusual ideas with high upside. The experiment should measure diversity, cost, and live outcome rather than assuming more candidates are better.

How much research is enough?

Six Lightreel calls is a fixed budget, not a proven optimum. The harness can test whether four, six, or eight calls change source breadth, decision quality, cost, and final views.

The relevant variable may not be call count. It may be query diversity, number of full posts inspected, or whether the final copy receives a separate evidence-grounded review.

Should planning and execution use different models?

One model could select the thesis and sources; another could produce and inspect the creative. This may improve specialization or simply introduce handoff loss.

The harness should compare total system performance and trace quality, not isolated model benchmark scores.

Does source similarity predict either performance or risk?

The current prompt assumes close adaptation improves views. A source-lineage system makes that hypothesis measurable. The experiment can compare originality bands against view performance, reviewer concern, and platform outcomes.

That question is more useful than a binary argument about copying. It can reveal where format learning stops adding value and starts creating duplication risk.

What this becomes beyond TikTok

The immediate environment is a TikTok carousel agent. The larger design is a reusable pattern for autonomous marketing systems.

The same outer loop can apply to:

• SEO and AEO recommendations generated from public-web evidence;
• paid-media agents that propose and test campaign changes;
• sales-content agents that build materials from product and customer evidence;
• brand-monitoring agents that investigate emerging narratives;
• content operations that generate, review, publish, and learn across channels.

The common structure is not “AI creates marketing.” It is:

1. observe an environment;
2. preserve the evidence;
3. make a bounded decision;
4. act through controlled tools;
5. capture the trace and artifact;
6. measure the external result;
7. turn recurring failures into evals;
8. give Codex a narrow engineering task;
9. validate before promotion.

That is also where this work connects to Atlas. In both systems, a model is not the final authority. It proposes interpretations and actions over an evidence base. Raw observations remain inspectable. Derived claims must point back to them. The evaluator is separate from the generator. A recommendation that cannot show its evidence is not ready to execute.

The durable asset is the learning system

A successful post is useful. A repeatable explanation of why the system improved is more valuable.

The first can disappear with a trend. The second accumulates into infrastructure: traces, replay worlds, regression cases, source graphs, task contracts, experiment results, and release controls.

That is the real opportunity in building on ViralBench.

The public project already demonstrates that an AI model can operate a live multimodal marketing loop. The next question is whether that loop can be improved systematically without confusing randomness for progress, imitation for insight, or autonomy for control.

My build is aimed at that question.

ViralBench supplies the live environment. Codex supplies the engineering capacity. The harness between them will decide whether the result is merely faster content production or a credible system for learning how autonomous agents improve.

Frequently asked questions

What is ViralBench?

ViralBench is a live AI marketing benchmark in which frontier models operate TikTok accounts and try to maximize views in the fitness category. The agents can research current content, inspect TikTok and Instagram posts, generate images, preview carousels, and publish through external tools.

What is an AI agent harness?

An agent harness is the operating system around a model: instructions, tools, schemas, memory, permissions, execution loop, tests, traces, validation rules, and deployment boundaries. The same model can perform very differently in two harnesses because the available context and feedback are different.

How would Codex use ViralBench as a harness?

Codex would operate outside the content agent. It would inspect failed runs and evaluation reports, modify a bounded part of the ViralBench repository, run replay and regression tests, and produce a candidate patch for review. ViralBench would provide the executable environment and live outcome data.

Is ViralBench measuring the model or the whole system?

The live result reflects the whole model-plus-harness stack: base model, prompt, research layer, image model, tools, account state, timing, memory, and platform conditions. Model comparisons remain useful, but they should be described as comparisons of configured agent systems rather than isolated intelligence scores.

Can an AI agent safely improve another AI agent?

It can improve the software around another agent when the work is bounded and independently evaluated. The builder should not control the raw evidence, success metric, evaluator, publishing credentials, and deployment decision at the same time. Sandboxes, protected paths, locked evals, human review, and rollback rules are part of the design.

Why not optimize only for views?

Views provide a clear benchmark reward, but an unconstrained views objective can favor copying, sensational claims, or short-term tactics. A serious system should maximize the chosen outcome only after platform, provenance, factuality, cost, and publishing constraints pass.

Why is offline replay necessary if TikTok is the real evaluator?

Replay catches process failures cheaply and repeatedly: missing research, broken tools, incomplete previews, duplicate assets, stale memory, and policy violations. It cannot predict TikTok distribution, so passing replay is a prerequisite for a live test, not a substitute for one.

Sources and technical notes

This article is based on the live ViralBench methodology and a code audit of repository commit 5f5f57e251023ceb37961c0fc2c808f67ceb71eb, supplied directly for review.

1. ViralBench methodology and live experiment. ↩
2. ViralBench repository and the audited marketing-agent.ts commit. ↩
3. OpenAI, “Unrolling the Codex agent loop”, January 23, 2026. ↩
4. OpenAI, “Harness engineering: leveraging Codex in an agent-first world”, February 11, 2026. ↩
5. OpenAI Developers, “Testing Agent Skills Systematically with Evals” and “Build an Agent Improvement Loop with Traces, Evals, and Codex”. ↩
6. Chen Wei et al., “Beyond Basic A/B Testing: Improving Statistical Efficiency for Business Growth”, revised December 27, 2025. ↩
7. Microsoft Research, “Deep Dive Into Variance Reduction”. ↩
8. OpenAI Developers, Codex SDK and Codex non-interactive mode. ↩
9. OpenAI Developers, Codex sandboxing and Codex worktrees. ↩
10. TikTok Support, “About AI-generated content”. ↩