Solving core problems in clinical research requires stitching together disparate contexts and systems sitting at the intersection of two massive industries: Healthcare and Life Sciences, each of which have multiple enterprise software platforms, data sources, and ontologies. We started with a product thesis that you need an end-to-end approach mirroring the stages of clinical research: (1) Protocol Design and Study Planning, (2) identifying patients for relevant studies, and (3) collecting and preparing research-grade data.

We needed to achieve this at a large scale and across heterogeneous data sources: we integrate with some of the largest health systems (including EHRs, CTMSs, labs, genomics, billing) and Life Sciences companies (including EDCs, CDWs, PDF and Structured Digital Protocols). We also needed a high accuracy bar across the entire chain of personas, who are all experts in their fields: Principal Investigators, Research Coordinators, Clinicians (in health system clinical settings), Study Ops, Regulatory, Pharmacovigilance, Medical Monitors, Data Managers, and Biostatisticians (in Life Sciences).

Additional core requirements of a production grade AI system: orchestrating data pipelines, retrieval, model inference, agentic coordination, deterministic logic and tool calling, failure recovery, observability, traceability, and human review. The system must trace every output to the exact version of the algorithm and source evidence that produced it. It must avoid rerunning expensive work without reusing stale results. And it must do all of this within healthcare's strict data, security, and governance constraints.

Meet Matcha: our in-house AI orchestration platform. This post will focus on Matcha's performance optimization, modularity, and support for traceability.

AI systems extend beyond a LLM

Language models are only one layer of a production grade AI product. Around the model itself sits a larger system that determines whether a task is repeatable, efficient, observable, and governable.

For healthcare tasks, the workflow begins well before inference. Patient documents must be loaded and normalized. Text must be segmented without losing its relationship to the source. Segments must be embedded and retrieved against a criterion-specific query. Evidence must remain linked to document identifiers and character offsets. Some criteria failures must be isolated to the affected item rather than corrupting an entire cohort. Outputs must then be materialized for downstream applications and analysis.

Matcha instead makes the workflow explicit.

Figure 1. Matcha makes each computational boundary explicit and reusable.

This pattern creates boundaries within which we can define identity, cache safely, measure behavior, handle errors, and preserve provenance.

A small component contract

Matcha is primarily a Python platform built around a deliberately small component interface. A component accepts a batch of inputs and returns a corresponding batch of outputs:

class Component[In, Out](Protocol):
    @property
    def spec(self) -> AlgorithmSpec[Out]: ...

    def get_cache_key(self, item: In) -> str: ...

    def __call__(self, inputs: Sequence[In]) -> Sequence[Out]: ...

The small surface is intentional.

Matcha components are batch-first, structurally typed, and operationally composable. A single item is treated as a batch of one, allowing the same clinical logic to run in tests, notebooks, local processes, or distributed Ray jobs without knowing its execution environment. Inputs rely on structural typing, while outputs use concrete dataclasses that provide stable, serializable contracts for caching and analytics. Operational concerns, including caching, metrics, retries, analytics, and distributed execution, wrap the underlying business logic, keeping the core implementation focused and testable.

This "functional core, imperative shell" pattern is central to Matcha. Deterministic logic (policy construction, cache resolution, pipeline identity, serialization rules) can be tested without infrastructure. The shell handles storage, APIs, concurrency, retries, and telemetry.

Algorithm identity is a first-class object

Caching an AI result safely requires a stronger question than "have we seen this input before?" The actual question is:

Have we seen the same semantically relevant input under a compatible version of the algorithm that produced this output?

Matcha represents the answer with an immutable algorithm specification. A specification contains the parameters that define the behavior of a component and generates a deterministic algorithm_key.

Conceptually, this becomes an identity such as:

@dataclass(frozen=True)
class RetrievalSpecV1(AlgorithmSpecBase):
    output_type = RetrievedEvidence
    top_k: int = 10

RetrievalSpecV1!top_k=10

Matcha treats each computation's configuration and relevant inputs as part of its identity. Model settings, prompts, schemas, document-processing parameters, and component-specific input properties produce a deterministic cache key. When any behaviorally meaningful value changes, Matcha recomputes the result instead of returning an incompatible cached output.

The effective lookup is:

(algorithm_key, semantic_input_key) -> typed output

This is how Matcha implements a practical rule we care about deeply: do not ask the same question twice when the underlying inputs and relevant computation have not changed.

Compatibility is explicit

Strict cache invalidation can often be unnecessarily expensive. An agentic call may receive a new name or output schema even when an older result remains semantically valid. Matcha addresses this with explicit equivalence declarations. A new function may determine that one or more older versions are acceptable fallbacks. If the output type changed, the equivalence can include a deterministic translation from the old schema to the new one.

In our patient-matching pipeline that compatibility policy is visible directly in the stage definition:

CURRENT_INFERENCE_STAGE = PatientMatchingStageConfig(
    stage_id="inference",
    spec=SPEC_INF_V3,
    deps=("retrievals",),
    equivalences=(
        Equivalence(SPEC_INF_V2),
        Equivalence(SPEC_INF_V1),
    ),
)

Matcha resolves cached results in a deliberate order: the current specification first, followed by explicitly compatible predecessors. Compatibility is always developer-defined, never inferred from similar schemas, and may include a conversion step. Successful resolutions are retained for faster future lookups, while stale mappings trigger a new ordered search and repair.

This turns algorithm evolution into a controlled compatibility problem rather than a choice between indefinite staleness and full recomputation.

Three cache layers, one policy

Matcha's component cache separates backend mechanics from typed materialization and lookup policy:

• A scoped key-value transport handles storage operations.
• An object layer serializes and validates typed outputs.
• A policy layer implements primary lookup, ordered equivalence fallback, translation, and warm resolution.

Because the policy is independent of storage, the same behavior can run against any set of data stores: an in-memory dictionary during a unit test, a persistent local cache or file during development, DynamoDB for distributed workloads, S3 for object-backed storage, or an S3-and-DynamoDB combination for large values.

Matcha treats malformed or incompatible cached values as errors rather than silent misses. It also deduplicates repeated inputs within a batch, computes each unique result once, and records failures as structured errors rather than caching them as successful outputs.

Component reuse beyond pipeline completion

Component caching solves one problem: avoiding repeated work inside a pipeline. We also need to address caching across a complete pipeline output.

Matcha has a second, distinct mechanism: the pipeline sink.

The pipeline sink tracks completion per stable pipeline input key. Its descriptor represents the computational stages as a directed acyclic graph. Each stage records its current algorithm, ordered compatible predecessors, and dependencies.

The resulting definition is intentionally declarative:

PipelineDescriptor(
    pipeline_id="patient-matching-v2",
    stages=(
        StageDescriptor(
            stage_id="doc-chunk-embeddings",
            primary_algorithm_key=embedding_spec.algorithm_key,
        ),
        StageDescriptor(
            stage_id="retrievals",
            primary_algorithm_key=retrieval_spec.algorithm_key,
            deps=("doc-chunk-embeddings",),
        ),
        StageDescriptor(
            stage_id="inference",
            primary_algorithm_key=inference_spec.algorithm_key,
            deps=("retrievals",),
        ),
    ),
)

This serves as a compact statement of computational identity: which algorithm produces each stage and upstream dependencies. Matcha validates the descriptor, rejects unknown dependencies and cycles, and derives the pipeline policy from the complete graph.

From that descriptor, Matcha derives deterministic policy fingerprints:

• A stage policy describes the current and compatible algorithms for one stage.
• A pipeline policy describes a concrete assignment of algorithms across the full graph.
• A resolution policy describes the compatibility search space for the pipeline.

On a rerun, the sink asks whether an acceptable materialization already exists. The warm path follows a previously resolved pipeline policy. The cold path searches bounded combinations of compatible stage versions. Only inputs without an acceptable completed output proceed to compute.

Outputs are written before completion is recorded. This ordering is important: a worker that fails during materialization cannot leave behind a false "complete" marker. The design uses at-least-once materialization, so downstream readers can deduplicate by stable identities rather than relying on fragile exactly-once assumptions.

Component caches and the pipeline sink work together but remain intentionally separate:

component cache  = reuse an intermediate computation
pipeline sink    = reuse a completed, materialized workflow result
analytics sink   = preserve operational and analytical records

Keeping these concerns distinct makes both their semantics and their failure modes easier to reason about.

Pipelines in practice

Matcha turns enterprise source documents into traceable, task-specific evidence by chunking and embedding content, retrieving the most relevant passages, and producing schema-constrained outputs. Source identity, offsets, model configuration, and request metadata remain attached throughout, while prerequisite checks can stop unnecessary downstream inference.

This is a practical example of intelligent model use: the platform does not send every possible question to the most expensive stage simply because it can. Deterministic orchestration reduces avoidable model calls while preserving a record of why processing continued, stopped, or was routed for review.

The gate itself is conventional, inspectable software:

def decide_workflow_gate(
    *,
    required_conditions: int,
    satisfied_conditions: int,
) -> str:
    if satisfied_conditions < required_conditions:
        return "requirements_not_met"
    return "continue"

Models can evaluate the configured tasks, while deterministic code decides how the workflow advances. Separating inference from control logic makes routing behavior easier to explain, test, and audit.

Memory is also treated as an explicit resource. Workloads are windowed according to the estimated size of their embedding matrices, limiting the volume of vector data held by a worker at one time. Why do this? This matters in cohort-scale processing, where an implementation that performs well for one healthcare record can fail abruptly when it encounters unusually large or complex healthcare records.

At every stage, Matcha validates key invariants: batch inputs and outputs must align; text snippets and offsets must remain synchronized; embeddings must match chunk counts; referenced content addresses must exist; and failed outcomes must contain structured failure information. These checks convert subtle data drift and pipeline inconsistencies into immediate, diagnosable failures.

The resulting pattern is broadly applicable across enterprise level AI: models interpret complex information, deterministic software governs execution, and traceable metadata connects every output to its source.

Scaling with Ray: next time

In our next post, we will share more about how we have scaled computation efficiently with Ray, enabling us to migrate off of Databricks, both reducing our cost and increasing throughput over 90%.