llm-bench — Privacy Policy

KAPUA Labs LLC 30 N Gould St Ste R, Sheridan, WY 82801, United States

This policy explains what the llm-bench service stores, where, for how long, who else sees it, and what a deletion request can and cannot reach. It is written to be checkable. Where the honest answer is uncomfortable — §5 in particular — it is stated plainly rather than softened.

Every factual statement below describes the service as built on 2026-07-27, not as planned.

1. Scope

This policy covers the llm-bench inference API service — the OpenAI-compatible gateway, its management console, and the evaluation and benchmark machinery behind it — and the public llm-bench benchmark site and journal.

Other properties operated by KAPUA Labs LLC have their own notices. The API service and the public site are separated by tenancy, not by infrastructure: they share the same database, cache, object storage and application deployment. We say so because a claim of “separate systems” would be false.

2. Who we are

3. Our two roles

The service handles two materially different kinds of data, and our role differs between them.

If you are an individual whose personal data reached us inside a customer’s prompt, that customer is the controller and you should direct your request to them. We will assist them, and we will forward a request we receive directly. Note the practical limit: prompt content is unstructured and we apply no schema to it, so we cannot search a customer’s tenant for the records relating to a particular individual. Our assistance is limited to tenant-wide erasure, erasure or amendment of a record the customer identifies to us by identifier, and the information described in DPA §13.

We do not require, expect or ask for personal data in prompt content, and we have no means of detecting it. Terms of Service §4 prohibits any personal data on the free tier, personal data in a Template on any tier, and — on any tier, absent a signed amendment — HIPAA-regulated health information, PCI cardholder data, GLBA-regulated information, government identification numbers, biometric identifiers, and the personal data of children under 13.

4. What the service stores

Everything below is a real store in the running system.

4.1 Account and identity

4.2 Customer provider credentials

The service supports two access modes, and this section is about the first. In bring-your-own-key mode a customer supplies their own API key for a model provider and we call that provider as them. In operator-supplied mode we supply the model access on our own provider credentials and bill the customer for the usage — it is the mode the service, the portal and the documentation call managed; the two names describe one arrangement, and this policy uses “operator-supplied” throughout. §7.2 sets out what that changes, and the bullet at the end of this list explains why it is not a fallback.

• Stored under envelope encryption: a fresh 256-bit data key per credential, AES-256-GCM over the secret, and the data key wrapped by that account’s own key-encryption key in Azure Key Vault using RSA-OAEP-256. Ciphertext and wrapped data key live in our database; the key-encryption key never leaves Key Vault.
• Plaintext exists only in worker memory, cached for a configurable period, currently 300 seconds — which is also the bound on how long a revocation at the provider takes to take effect in a warm worker.
• Decryption fails closed: a Key Vault outage or a failed unwrap raises an error. It never falls back to a stored or operator value.
• We store the last four characters for display only.
• Every materialization of plaintext writes an append-only credential-access record (purpose, provider, capability — never key material).
• A customer account never falls back to our provider keys. Where a capability is served in the operator-supplied access mode, our keys are used by design and disclosed on the capability (§7.2) — not as a fallback when the customer’s own credential is missing, which always fails the call instead.
• Where the service accepts a key supplied on a per-request header rather than from the vault, that key is used for that request only and is never persisted.

4.3 Call content

Every API call writes one append-only call record. What that record holds depends on the account’s tier.

Prompt content. In an OpenAI-compatible request, the messages are folded into the record’s prompt-variable field — so that field is the customer’s prompt text. Whether it is written at all is decided at a single write chokepoint:

The gate fails closed: an unknown, missing or unreadable tier is treated as private, the most restrictive setting. It governs every write path — synchronous calls, batch enrolment, image calls, judge calls, fan-out sibling records and internal repair calls — because there is exactly one call-record creation site.

What “not written” does and does not mean. It means the prompt is not persisted on the call record. It does not mean the prompt never exists in our systems: it is necessarily present in worker memory while your call is being served, including when we retry or fail over to a second model within the same request. And for batched calls it is persisted, on every tier, for the operational window — see Batch working copies below. That is one of two places this guarantee is qualified; the other is the routing embedding retained on shared_anon, which we treat as content because it is partially invertible (§6, §8.1).

Model output. The parsed output — the validated result on success, or the raw response on a parse failure — is written on every tier, including private, because the delivery paths rebuild your result from it. How long it then lives depends on the tier: on shared_anon and private it is deleted once delivered and evaluated. See §5.

Everything else on the record is metadata rather than content: capability, model, provider, token counts, latency, cost, error category and message, service tier, and a pointer to the exact prompt-template snapshot used.

Batch working copies. A batched call cannot be served without keeping its rendered request: the provider submission reads it back, and a retry or a resubmission needs it again. So for batch traffic, on every tier including private, the rendered messages are held on the call record’s operational metadata for the operational window. They are stripped when the batch finalizes, and again by a daily sweep that catches anything left behind and strips any record still in flight after 72 hours. A batch also carries a hard 24-hour expiry backstop. In practice the window is hours to a couple of days.

4.4 Batch API files

A batch itself expires after a hard 24-hour backstop.

4.5 Webhooks

• A customer registers a destination URL, validated at creation as HTTPS-only and public-IP-resolving, plus a signing secret shown exactly once and stored in the vault; the endpoint row holds a pointer, never the secret.
• Delivery records are deliberately content-free — the payload is rebuilt from the call record at send time. This is precisely why §5’s output retention exists.
• Undelivered records dead-letter after 24 hours.

4.6 Billing

• The service is prepaid: a customer buys credit and usage draws it down. Metering, the credit balance and invoicing run through Metronome (§7.3); the card is held by Stripe.
• We store Stripe and Metronome customer identifiers, subscription identifiers and plan state. Card and bank details never reach our systems — they are entered in a Stripe-hosted form, and we do not hold so much as the last four digits.
• Metronome receives an account identifier, per-request usage records and invoice amounts. It does not receive prompt content, model output or an email address.
• Usage and cost aggregates derived from call records are retained for billing and legal-retention purposes and survive erasure (§8).

4.7 Operational and security data

5. Model output — the honest statement

We write the model’s output on every tier, including shared_anon and private, and on those two tiers we delete it once you have it. Both halves of that are deliberate, documented decisions, and we will not describe them any other way.

Writing it is delivery-driven: three paths rebuild the customer’s own deliverable from that field after the call has finalized.

1. Webhook delivery. Delivery records are content-free by design, so the payload is rebuilt from the stored output at send time. Without it, the delivery is lost.
2. Batch API output files. The OpenAI-compatible output file is rendered from stored output at finalization.
3. Retrieval by the customer. We serve the result back on request, for as long as we still hold it — which is also the recovery path when a webhook is lost.

Beyond those three, stored output is used for the evaluation and judging operations the customer instructs, and — on the free tier only — as a comparison baseline in model-introduction replay (§7.2A), which we perform for our own purposes as the consideration for that tier. It is used for nothing else.

On shared_anon and private, retention is bound to delivery. We delete the output of a call promptly — typically within minutes — once it has been delivered to the customer and any evaluation of it has finished. An output we could not deliver is held for retrieval for at most 72 hours, then deleted. There is no retrieval of a past output beyond that. The one exception is the evaluation evidence a customer elects to keep for review of their own criteria (§6). This is a schedule we enforce with a continuous sweep and monitor; if it lengthens, that is a new version of these documents.

On free, output is retained until erasure or account closure, because it is the replay baseline. Once delivery is complete we may also delete it at our own discretion to manage storage.

We do not say, because it would not be true:

• “We do not retain the content of your requests or responses." — on free we do, verbatim, and on every tier we hold the output for the time it takes to deliver it.
• “On the shared and private tiers no raw content is retained." — a batch working copy, the routing embedding on shared_anon until a capability’s one-time evaluation is done, and the output until delivered and evaluated, are all raw content held for a bounded time.
• “Outputs are discarded immediately after delivery." — “promptly, once delivered and evaluated, backstopped at 72 hours” is the true sentence, and the difference is the evaluation and the undelivered case.

Erasure is a separate matter. It is real, it covers model output on every tier, and it is not governed by the retention matrix below — see §8, including §8.2 on what it does not reach.

6. Retention by tier

This is the authoritative statement of what we keep. It is enforced in code at a single write chokepoint, and it fails closed.

Two things this table does not cover. First, a capability’s own definition — its slug, display name and description — is configuration a customer authors to declare what a capability is, and it is indexed for routing on every tier, including private; the routing index cannot function without it. Second, this table applies by tier from the moment a tier takes effect: an account that moves to free limits is handled under the free column from the time of the move, whether or not it has since accepted the free-tier data grant (Terms §6.1 and §14).

Two points that matter:

• Two categories we use for our own purposes, not just to run the service. free-tier prompt text (re-executed against newly introduced models, and used for taxonomy) and template content on free and shared_anon (used for taxonomy). For those two categories we are not acting as a “service provider” under the CCPA. We still do not sell or share them. A customer that needs service-provider treatment for everything it sends should use the private tier. See DPA §6.4.
• free and shared_anon pool their statistics globally. Their verdicts and cost/quality signals feed a shared pool used to rank models. private keeps its own statistics. This pooling is the economic basis of the free tier, and it is disclosed at signup — which is what the version-stamped consent record exists to evidence.
• Tier changes are forward-only. Moving from free to private does not retroactively un-pool verdicts already contributed; it changes what happens from that point on. Verdicts already pooled will not be un-pooled. We word that as a commitment rather than as an impossibility: re-tagging and recomputing is technically possible, we reserve it for repairs and for honoring erasure requests where we determine it is required, and we do not offer it as a customer-triggered right. Saying “cannot” would be false.

7. Who else sees the data

7.1 Model providers under customer keys — not our subprocessors

For ordinary serving traffic, a customer’s prompt is sent to the model provider using that customer’s own vaulted API key, under that customer’s own contract with that provider. We never substitute our key. The provider is therefore the customer’s processor, not our subprocessor, and the provider’s own terms govern what it may do with that content.

This has a consequence customers must act on: at least one provider (Google’s Gemini API) applies materially different data-use terms to unpaid projects, under which it uses submitted content and generated responses to improve its products and human reviewers may read, annotate and process API input and output. The service does not detect the billing status of a supplied credential. Choosing a paid provider project is the customer’s responsibility, and no confidentiality commitment we make can override the terms of a provider the customer chose.

The current registered provider set is Anthropic, OpenAI, Google (Gemini), Groq, Meta, xAI, Perplexity, OpenRouter, DeepSeek, Moonshot AI, Z.AI, MiniMax and Alibaba Cloud (DashScope). Several are established outside the United States, the EU and the UK, including in the People’s Republic of China. Which providers are enabled is the customer’s choice.

7.2 Operator-supplied serving — model providers are our subprocessors

Operator-supplied is the access mode the service, the portal and the documentation call managed (§4.2). The two names describe one arrangement, and nothing else answers to either.

Where a capability is served in the operator-supplied access mode, we supply the model access on our own provider credentials and bill the customer for the usage. On that path the model providers are our subprocessors, because the content travels to them under our account and on our instructions.

• It serves the customer’s own request and returns them the result, unlike the replay in §7.2A.
• It is available on every tier, and the tier retention rules in §6 apply unchanged — the access mode decides who contracts with the provider, not what we keep.
• The customer cannot constrain the provider set by their choice of credentials, because the roster is ours. The 30-day notice and objection rights below apply to it in full.

7.2A The other path on our credentials: free-tier model-introduction replay

One further operation runs on our provider credentials rather than the customer’s, and is distinct from operator-supplied serving: model-introduction replay, in which we re-execute free-tier prompt text against a newly introduced model to measure it. On that path the model providers are our subprocessors, because the content travels under our own account.

• It applies to the free tier only. No shared_anon or private content travels this path — on those tiers prompt text is not retained, so there is nothing to replay.
• The customer cannot constrain which providers are reached on this path, because the roster is ours. A customer that does not want its content to travel this path should not use the free tier.
• The work returns the customer nothing and is not billed to them. It is the consideration for the free tier.
• The replay corpus is strictly the account’s own calls — one customer’s prompts are never replayed for another customer’s benchmark. Likewise, judge reference examples are panel-scoped and therefore account-scoped, so one customer’s outputs do not today appear inside another customer’s judge prompt. The Terms of Service (§11) and DPA §4.2(c) permit a narrow shared-judging exception on the free and shared_anon tiers; no code path does this today, and if we begin, we will say so here.

Benchmark runs are not on this path. When a customer uploads a benchmark (golden) set and requests a run, execution uses that customer’s own provider credentials and is billed to them as ordinary inference. The model providers are therefore not our subprocessors for benchmark execution, on any tier.

Benchmark-set inputs are, however, exempt from tier retention limits: an uploaded set is retained on every tier including private, so that it can be re-run. The service records the uploader’s acknowledgement of that retention — who, and when — and refuses to execute a set for which no acknowledgement exists. Benchmark sets are covered by erasure (§8.1).

7.3 Infrastructure and business subprocessors

Stripe is not our subprocessor. Billing contact and payment data are collected by Stripe through a Stripe-hosted checkout; cardholder data does not reach our systems. For that data Stripe acts as an independent controller, determining its own purposes and means, and its processing is governed by Stripe’s own terms rather than by our DPA.

We will give at least 30 days’ notice before adding or replacing a subprocessor, by updating the register and by email to the account address. A customer may object on reasonable data protection grounds within that period; if we cannot resolve the objection, the customer may terminate the affected service without penalty.

8. Your rights, and the honest limit on erasure

8.1 What we delete

On a verified erasure request, or on account closure, erasure completes within 30 days and produces a certificate listing exactly what was erased and what survived. It is a real, implemented, idempotent operation with a dry-run mode, and it refuses to run against our own operator account.

It blanks or deletes, for that account:

• every call record’s prompt text, content-bearing metadata, and model output — the last of these on every tier, because the retention matrix governs what we keep by default while erasure is the customer overriding it;
• evaluation dispatch input and output snippets, and the evaluation evidence kept for a customer’s review, which is deleted outright;
• fan-out judge candidate and winner payloads (the same model output, multiplied across the roster);
• the text of every template and every stored version of it — the record of which version ran a call survives, without the text;
• routing embeddings — hard-deleted, not blanked, because an embedding is partially invertible and an embedding is the content;
• Batch API content at rest: every uploaded input blob and every rendered output and error file is deleted and its row tombstoned, and every per-line request body is blanked;
• uploaded benchmark sets: their items and uploaded files are deleted, and the set’s own row is retained, emptied of your content and deactivated, because operator records of benchmark runs reference it;
• vaulted provider credentials, which are destroyed, and service API keys, which are disabled;
• support data, other than the records §8.3 identifies as surviving. Your account and its API keys are deactivated rather than deleted: key rows hold only a one-way hash and never key material, and all API access stops immediately at the authentication layer.

Because several of these tables are append-only by design, erasure overwrites fields rather than deleting rows. A blanked row no longer matches the erasure query, which is what makes the operation safe to re-run. The procedure defaults to a dry run, cannot be executed against our own operator account, and is followed by a verification query whose report is available to the customer.

Before we erase, you get your data out. Following termination we give at least 7 days’ written notice before erasure begins. On written request made before then, we will provide within 14 days a copy of retained prompt text, model output, Batch API input files, benchmark sets and templates in a structured, machine-readable format; while such a request is outstanding erasure does not begin, and the 30-day period is extended. Provider credentials and webhook signing secrets are not exported.

8.2 What erasure does not reach

We list these because the alternative is a promise we would break. We make no warranty that customer content is absent from them.

If you require any of the retained items above to be erased, contact us and we will treat it as a specific request rather than declining it. We do not represent it as automatic.

8.3 What survives, and why — the asymmetry

Capitalized terms in this section have the meaning given in Terms of Service §1; this section is worded identically to Terms of Service §12.3 by design.

We can delete your content. We cannot un-derive an Aggregate. This asymmetry is stated identically here and in the Terms of Service; DPA §8.4 states it subject to the qualification below. It is not a hedge; it is a structural property.

These survive an erasure request by design:

• Verdicts, and the Aggregates computed from them. A Verdict is a fact about a model — “model X scored 0.72 on Capability Y” — computed across many contributors. Once a free or shared_anon Account’s Verdicts have been folded into the global pool, the pooled Aggregate cannot be un-derived: we can recompute the average without them, and will where erasure requires it, but the recomputed number is a new number. It does not undo routing decisions already made, models already retired, or rankings already published. Our erasure report counts these rows explicitly rather than silently omitting them. One qualification, and it now runs in your favour: a Verdict also carries the judge model’s written rationale, which is free text produced by a model that saw your Output and can therefore restate or quote part of it (“the answer omitted the revenue figure…”). Erasure blanks that rationale, on every tier. What survives a Verdict is the score, the failure mode and the counts — the numbers the comparison rests on — and no free text.
• Aggregate usage and billing records, retained for the period required by applicable tax and accounting law.
• Access and audit records, including the append-only record of every time one of your Provider Credentials was decrypted and by which subsystem.
• Consent records. The fact that you accepted a specific version of our terms on a specific date is itself the basis on which we processed your data. These records are append-only and survive Account closure; withdrawal is recorded as a new superseding record, never as an edit or a deletion. On erasure we purge the incidental metadata attached to them (IP address, user agent) and retain the fact of acceptance.

8.4 Exercising your rights

Under the California Consumer Privacy Act and the comprehensive consumer privacy statutes of other US states, you may have rights to know what personal information we hold about you, to have it corrected or deleted, to opt out of sale or sharing, and to not be discriminated against for exercising those rights.

We do not sell personal information and we do not share it for cross-context behavioral advertising.

Write to the privacy contact in §2. We respond within 30 days for requests about data we hold as controller, extendable by a further 45 days where a request is complex; we will tell you if we extend. We will ask you to verify your identity — typically by confirming control of the account email address — and we may decline a request we cannot verify. An authorized agent may act for you on written proof of authority.

Two practical limits, stated plainly. First, for call content we act as a processor on our customer’s instructions; if your data reached us inside a customer’s prompt, that customer is the controller and the request belongs to them (§3). Second, prompt content is unstructured and we apply no schema to it, so we cannot search a tenant for the records relating to a particular individual. What we can do is erase a whole tenant, erase or amend a record identified to us by identifier, and provide the compliance information in DPA §13.

Customers can retrieve their own call records through the API, which covers much of access and portability in practice, and can export their content before erasure under §8.1.

9. Security

Each item is a description of the running system:

• Tenant isolation is enforced at the data-access layer. Models are account-scoped by default, and every deliberate cross-account read must appear in a reviewed allowlist that an automated test enforces — both new entries and stale ones fail the test suite.
• Credentials at rest: per-credential data key, AES-256-GCM, data key wrapped by a per-account key-encryption key in Azure Key Vault, fail-closed decryption, no plaintext at rest.
• API keys at rest: SHA-256 digest only; the raw token is unrecoverable.
• Webhook URLs validated HTTPS-only with public-IP resolution as a server-side request forgery guard; signing secrets vaulted and shown once.
• Error reporting scrubbed at two composed layers with a fail-closed fallback (§4.7).
• Retention decisions enforced at one write chokepoint that fails closed on an unknown or unreadable tier.
• Traffic in transit is carried over HTTPS.

Security incidents. We notify affected customers at their registered account email address within 72 hours of becoming aware of a breach leading to accidental or unlawful destruction, loss, alteration or unauthorized disclosure of or access to customer content or to the data we hold as controller. Our detection is alert-driven rather than a continuously monitored security operations function, so the 72 hours run from our actual awareness.

What we do not have, and do not claim. No SOC 2 Type II or ISO 27001 certification. We do not represent that we conduct third-party penetration testing, formal business-continuity or disaster-recovery testing, or that we maintain documented recovery point or recovery time objectives. We have not verified whether database-volume encryption at rest is enabled by our hosting provider — the encryption described above protects provider credentials specifically, not the datastore as a whole. We will say so here when any of this changes.

10. International transfers

All of our own infrastructure, and all of our subprocessors, are in the United States. Compute, the self-hosted database and task broker, object storage, key management and alerting are in Azure Central US; error monitoring, where enabled, is in the United States.

We do not currently offer the service for personal data subject to the EU or UK GDPR. The DPA does not incorporate the European Commission’s Standard Contractual Clauses or the UK International Data Transfer Addendum, and processing of such data requires the applicable instrument to be executed as an addendum first (DPA §12.5). A customer that needs this should contact us before sending such data through the service.

Model providers: establishment and serving location are different questions.

• Establishment is the domicile of the contracting entity, and determines which legal system can compel it to disclose data regardless of where the data sits. The providers established in the People’s Republic of China are DeepSeek, Moonshot AI, Z.AI, MiniMax and Alibaba Cloud. The remainder are established in the United States or the European Union.
• Serving location is where inference actually runs. It does not follow from establishment: several providers, including PRC-established ones, operate region-scoped endpoints elsewhere. Alibaba Cloud, for example, serves from Singapore, the United States, Germany, Hong Kong and mainland China through separate endpoints with separate credentials.
• A provider established in one jurisdiction and serving from another may remain subject to the laws of the jurisdiction where it is established. Choosing a non-PRC endpoint of a PRC-established provider addresses where the data sits; it does not by itself address the reach of PRC law over the provider.

Who chooses. Where a call runs on a customer’s own provider credential — every benchmark run on any tier, and every call on a capability in bring-your-own-key mode — the credential is bound to a specific regional endpoint, so the customer selects both the provider and the serving region, and the transfer is made under the customer’s own agreement with that provider. Where free-tier prompt text is replayed on our credentials (§7.2A), or a call is served in the operator-supplied access mode (§7.2), we are the exporter and we choose the endpoint; a customer that does not want us making that determination should not use the free tier.

We surface the serving region for each provider and model where the provider makes it determinable, and indicate where it does not. We take that from the provider’s own documentation and endpoint configuration; we do not independently verify it.

11. Children

The service is not directed to children. You must be at least 18 to open an account, and the Terms of Service prohibit submitting the personal data of children under 13 through the service. We do not knowingly collect personal data from children, and we will delete it if we learn we have.

12. The public benchmark site

The public llm-bench site publishes per-model quality bands, cost figures and rankings, generated from an immutable snapshot of the aggregate statistics described in §6.

The published artifact contains no customer prompt or output text — judge-derived scores are numbers. Published rankings name the providers’ models; that is comparative commentary about third-party products, not a disclosure about any customer.

Visitors to the public site. Requests to the public site and to the management console are recorded in our application logs (§4.7), which include the requesting IP address, the user agent and the URL requested. We use those records to operate and secure the service. They are subject to the log retention position in §4.7.

Cookies. The management console uses strictly necessary session cookies, and we do not use advertising or cross-site tracking cookies on either surface.

13. Changes to this policy

This policy is versioned by date (this version: 2026-09-08). Where consent is the basis for processing, consent is recorded against a specific version: accepting one version does not carry to the next, and we re-prompt and record a fresh acceptance on a version change. Exactly one version of each document kind is current at any time, and we record the URL of the published text — and, where we do not control the bytes of the published page, a SHA-256 digest of it — alongside the version.

We publish a new version in the console, which shows at all times which version is current and whether your acceptance is up to date. We do not undertake to email you about it — the same position the Terms of Service §22 takes, and for the same reason: a commitment to deliver mail turns an undeliverable mailbox into a breach, for something the console states plainly and at any time.

14. Contact

KAPUA Labs LLC 30 N Gould St Ste R Sheridan, WY 82801 United States

Privacy and data requests: privacy@kapualabs.com

Version 2026-09-14. Prepared by KAPUA Labs LLC. Verified against the running system on 2026-09-09; the retention statements in §3, §5, §6 and §8 (delivery-bound output retention, the customer’s template delete, the evaluation review window, and the erasure of template text) were verified on 2026-09-14.