Short · AI security · 18 min read

Should You Let an AI Vendor Train on Your Company's Data?

Approve training only for a defined dataset, purpose, model boundary and recipient group that your company is authorized to support. Require evidence of a useful result, credible handling controls and a workable response when the data must be withdrawn. If those conditions remain unresolved, permit only the narrower processing you can justify or withhold the data. Permission to answer today's question should not silently become permission to improve a model used by other customers.

I would ask the vendor to describe the proposed transformation before asking for its security certifications. Will the data remain in a retrieval collection, become examples for a dedicated fine-tuned model, or enter a shared training pipeline? Will reviewers label it? Will outputs become another training set? Those choices change who may benefit from the information, where it persists and what a later exit can accomplish.

This paper develops an admission method for that specific training decision. It synthesizes twelve original research, government and provider sources; it does not report a vendor audit or establish a company's legal authority. The decision framework and numerical example are proposed analysis, not validated scoring instruments or observed commercial results. Existing service approval is relevant evidence, but the additional training purpose needs its own decision.

Make the training request specific enough to approve or refuse

“Improve our services” is too broad to evaluate. I would require a proposal that names the originating dataset, fields, time period, selection rule, preprocessing, training objective, intended model versions and permitted recipients. Include what will happen to prompts, attachments, corrections and employee feedback. If the vendor cannot distinguish these objects, it cannot explain which promise covers them.

A useful boundary is the learning artifact rather than the product brand. The company may allow inference over selected records without allowing parameter updates. It may authorize a model dedicated to its own use without authorizing a shared model. It may permit a public test dataset while excluding operational records. These are different grants of use, not different descriptions of the same permission.

Microsoft's Foundry documentation illustrates the distinction. For the covered Models sold by Azure, it states that customer prompts, completions and training data are not used to train foundation models without permission or instruction, and that customer fine-tuned models are exclusive to that customer's use. The same document separately describes uploaded training data, retrieval and stateful features (Microsoft, 2026). Those representations apply to the specified service arrangement; they are not evidence about an unrelated reseller's processing pipeline.

My approval record would therefore specify both prohibited destinations and the authorized operation. “Dedicated adaptation for this support classifier, without incorporation into shared models” is more inspectable than “AI use approved.” The vendor should explain how that boundary survives subcontracting, support access, future model updates and a change in account settings. A promise that cannot be mapped to a processing path is difficult to test and easy to misunderstand.

Keep no-training promises separate from retention promises

A service can refrain from training on a request while still retaining information for another purpose. Conversely, a vendor can delete an uploaded file after using it in a learning process. The terms describe different events. I would give training, storage, human access, onward transfer and deletion separate entries in the evidence record, even when the contract discusses them in one paragraph.

Google Cloud's documentation states that covered managed models will not use customer data for training or fine-tuning without prior permission or instruction. Its separate retention discussion identifies feature-specific conditions, including abuse monitoring, grounding and optional logging or stateful features (Google Cloud, 2026). The relevant conclusion is not that retention is inherently unacceptable. It is that a training restriction does not answer the retention question.

For a reseller or application built on another model provider, inspect the complete arrangement. The underlying provider's restriction may be useful, but the application operator could maintain its own logs, feedback collection or training process. Ask which party receives which object and which terms bind that party. Do not infer that every subcontractor receives the full dataset, and do not assume that a downstream provider's controls automatically govern upstream copies.

This is also why a settings screenshot is incomplete evidence. It can show a selected option for a particular account at a particular time. It does not establish that every employee used that account, that a separate feedback route was covered or that the recorded setting applied before an earlier upload. The employee generative-AI approval paper addresses that broader use-control problem. Here, the additional question is what changes when training becomes an authorized purpose.

Establish the right to reuse before measuring the benefit

The company may possess a record without having unrestricted permission to repurpose it. A customer supplied a support attachment to resolve a fault; an employee wrote an internal assessment; a partner provided a specification under agreed restrictions. I would ask the responsible information and legal owners to identify the applicable constraints before the engineering team selects a training set. A model's potential usefulness cannot supply missing authority.

The Australian privacy regulator's guidance expressly covers organizations supplying personal information to a developer for model development or fine-tuning. It warns that public availability does not itself establish permission to train, and addresses using already-held information for a secondary training purpose (Office of the Australian Information Commissioner [OAIC], 2024). These are Australian regulatory considerations, not a universal rule that a particular consent mechanism settles every jurisdiction's requirements.

NIST's generative-AI profile identifies privacy risks concerning training-data provenance, transparency, purpose specification, memorization and sensitive inferences (Autio et al., 2024, §2.4). That wider view matters because the assessment should include both personal information and company confidentiality. Removing a person's name does not necessarily remove a sensitive commercial fact, and a permission to process personal information does not automatically authorize disclosure of a partner's confidential design.

I would divide the candidate data into approved, excluded and unresolved categories, with a reason and accountable owner for each. The unresolved category is a hold, not an invitation to upload first and clarify later. If a narrower extract would meet the technical objective, assess that extract on its own merits. Do not call a dataset anonymous merely because the easiest identifiers were removed or because the vendor describes the transformation as aggregation.

Require a training-data manifest that survives transformation

The evidence needs to follow the dataset after extraction. Identify the source snapshot, selection criteria, preprocessing version, excluded categories and any additions made during labeling. Give the released dataset an identifier and an accountable owner. The purpose is to connect the approved input with the material actually used, rather than rely on a file name such as “final-clean-training-data.”

Datasheets for Datasets proposes structured documentation of motivation, composition, collection, processing, uses, distribution and maintenance. Its questions include restrictions and updates or deletions (Gebru et al., 2021, §§3.1–3.7). I use that approach to make the vendor's input evidence legible. Completing a datasheet is documentation, not proof that every record satisfies its declared restrictions.

My additional requirement would be traceability between the dataset release and the resulting training run. Record which preprocessing and labeling steps created derived examples, which model or adapter received them, and which evaluation sets were kept separate. If records are combined, deduplicated or converted into synthetic examples, document the relationship sufficiently to investigate a later exclusion request. A transformation can make the relationship harder to observe without eliminating the underlying responsibility to assess it.

Avoid retaining sensitive source material everywhere merely to prove lineage. A restricted manifest can reference approved record identifiers and transformations while operational tickets contain only the release identifier and decision. The organization still needs an access and retention decision for the lineage store itself. An audit trail that becomes an uncontrolled copy of the training corpus creates another disclosure path rather than resolving the first one.

Distinguish memorization, membership and useful generalization

An AI model's legitimate purpose is to learn patterns. The assessment therefore needs to distinguish useful generalization from disclosure of particular training records and from evidence that a record participated in training. Those are different outcomes. A system can avoid repeating a document verbatim while still exposing a sensitive relationship, and a successful extraction test can demonstrate a failure without measuring its prevalence across every user.

Carlini et al. (2021) demonstrated extraction of verbatim training sequences from GPT-2, including publicly available identifying information. Their study establishes that a trained language model can disclose particular training material under the examined conditions. It does not establish that every model memorizes every record, or provide a current leakage rate for the vendor you are assessing.

Membership inference asks whether a particular example was in the training dataset. Carlini et al. (2022) argue that evaluation should examine true-positive rates at low false-positive rates rather than rely on average attack accuracy. This matters when a vendor presents a privacy test with an attractive aggregate score but no explanation of which records, attack capabilities or thresholds the test covered.

I would define the concern before choosing the test: disclosure of a confidential passage, evidence of participation in a sensitive dataset, or inappropriate inference about a person. Then require a suitable threat model, authorized testing scope and evidence limits. Do not use genuine customer secrets as convenient public test prompts. A synthetic probe can exercise a mechanism; it cannot by itself prove how historical customer data behaved during training.

Read a privacy-test result through its base rate

Consider a deliberately fictional membership test over 100,000 candidate records. Exactly 1,000 were training members. Suppose the test identifies 80% of members and falsely identifies 1% of nonmembers. These are invented rates for illustrating interpretation, not measurements of a real attack, model or vendor.

The test produces 800 true positives and 200 false negatives. Among 99,000 nonmembers, it produces 990 false positives and 98,010 true negatives. Overall accuracy is 98.81%, but only about 44.69% of the 1,790 positive findings are genuine members. Reporting the accuracy alone would conceal how many individual findings are wrong. Conversely, the imperfect precision does not make the 800 true membership disclosures harmless.

Fictional resultCountDecision significance
True member identified800Demonstrated membership disclosure within the invented test
Member missed200A negative result does not reliably exclude participation
Nonmember flagged990Individual allegations need careful interpretation
Nonmember correctly rejected98,010Drives high aggregate accuracy because nonmembers dominate

The arithmetic is an illustration of why the evaluation population matters. In practice, a buyer may not know the membership base rate and may have access only to selected challenge records. The test report should describe that limitation rather than present a precision estimate without a justified population. Threshold selection, independent holdout data and uncertainty also belong in the report; one point estimate does not establish a stable operating characteristic.

My acceptance question would be whether the tested disclosure mechanism is compatible with the proposed data and access model. A high aggregate utility score cannot offset a demonstrated release of protected information. Nor should an unsuccessful limited attack become a universal claim of non-memorization. The outcome should identify the version, conditions and remaining uncertainty, with a decision about whether that uncertainty is acceptable for this dataset.

Ask what differential privacy actually protects

Differential privacy can provide a formal bound on how a mechanism's output distribution changes when neighboring datasets differ. NIST emphasizes that the parameters and the unit of privacy both matter, and that protecting individual events is not the same as protecting all of one person's contributions (Near et al., 2025, §§2.1–2.4). “We use differential privacy” is therefore the beginning of an evidence request.

Ask the vendor to specify the protected unit, the guarantee's parameters, the mechanism and the accounting across releases. For example, one customer might contribute many support records. A guarantee defined over one record should not be casually described as protection for everything associated with that customer. Require the reviewer to connect the formal neighboring-dataset definition to the business entity the organization intends to protect.

NIST also separates differential privacy from security controls protecting the original data and its processing environment (Near et al., 2025, §4.4). A mathematical output guarantee does not grant permission to collect a dataset, protect an unrestricted source export or prove that an implementation follows its claimed mechanism. The assessment needs both a scoped technical guarantee and evidence about the surrounding system.

I would request an intelligible explanation from a qualified reviewer rather than invent an acceptable privacy parameter in procurement. The vendor should describe what was included in the analysis, what was excluded and how a changed release plan affects it. If that evidence is unavailable, record the claim as unsubstantiated for the proposed decision. A complex term should not receive more credit merely because the buyer finds it difficult to challenge.

Evaluate the model's value without contaminating the test

The training request should name the improvement that would justify the additional processing. Faster drafting, better classification and more accurate retrieval are different objectives. I would compare the proposal with a baseline that does not use the candidate training data, then ask whether a narrower adaptation can deliver the required benefit. The comparison is about this task and its consequences, not a general leaderboard position.

Model Cards for Model Reporting proposes documenting intended uses, metrics, evaluation data, training data and limitations. It emphasizes context-dependent metrics and evaluation across relevant conditions (Mitchell et al., 2019, §4). A model card can structure the evidence, but the buyer still needs to establish that the card describes the delivered model version and the intended deployment.

Hold evaluation records apart from training and tuning decisions. If a vendor repeatedly changes a model in response to the same test set, the final result no longer provides the same evidence of performance on unseen cases. My proposed review would preserve an independent acceptance set and record its provenance, exclusions and access. A vendor may need representative examples to develop the model; that does not require handing over every record reserved for final assessment.

Report consequential failure categories separately. A support model that improves common low-risk answers but invents warranty commitments may be a poor trade for the business. A useful evaluation can show where the gain occurs, where the model becomes less dependable and how much review the workflow still requires. The privacy and authority decisions remain separate gates: a genuine performance gain does not make unauthorized reuse acceptable.

Decide whether the proposed isolation matches the promised use

A dedicated model is not automatically private simply because it has one customer's name. I would ask who can query it, download it, change its deployment, inspect its training artifacts or use its outputs to train something else. The relevant boundary includes administration and subsequent reuse, not just the customer-facing endpoint.

Conversely, a shared model is not automatically prohibited for every dataset. A company may deliberately license appropriate information for a broader purpose. The question is whether the permission, technical evidence and business decision support that broader dissemination. Treat that as a separate transaction with explicit recipients and consequences, rather than a hidden condition of buying an otherwise useful service.

The European Data Protection Board's Opinion 28/2024 rejects a blanket assumption that models trained on personal data are anonymous. Its assessment considers extraction and disclosure through queries, reasonable means of identification, and direct model access (European Data Protection Board [EDPB], 2024, paras. 34–45). This is a European regulatory analysis, not a test result for a particular model or a determination that every model always contains identifiable information.

For the proposed arrangement, I would compare the evidence under the access actually offered. A constrained endpoint and a downloadable model present different opportunities for examination and reuse. If the vendor's assessment covers only the endpoint while the agreement permits distributing model weights, the evidence and permission are misaligned. Either narrow the release or obtain an assessment that covers the broader access; do not silently carry the earlier assurance across the change.

Make withdrawal cover the learning artifacts, not just the upload

Before approving training, ask what happens when an input must no longer be used. Possible actions include deleting source files, stopping future training, replacing a model, applying an unlearning process or retiring an affected release. These actions support different claims. A file-deletion receipt does not establish that a trained model's behavior is equivalent to a model that never received those records.

Chourasia and Shah (2023) examine limitations of deletion guarantees based on resemblance to retraining. Their analysis considers adaptive deletion requests and information revealed across model releases, including retained computational state. It shows why a single post-deletion model comparison can be an incomplete privacy argument. It does not establish that every unlearning method fails in every setting or that a particular vendor cannot meet a narrower obligation.

I would require the vendor to define the claimed removal property, the artifacts it covers and the evidence supporting it. Does the process include deployed versions, retained checkpoints, adapters, training snapshots and subsequent derived models? Which artifacts remain available to another party? If a released copy cannot be recalled, the proposed exit statement must say so. Withholding future data is useful, but it is not the same as withdrawing information already incorporated elsewhere.

The FTC's 2021 Everalbum order supplies a concrete, bounded example of derivative-artifact obligations. It defined affected work product to include specified models or algorithms and required its deletion or destruction, with stated retention exceptions (Federal Trade Commission [FTC], 2021, Definitions A; §III). This was a particular consent order, not a universal technical specification or automatic remedy in every dispute. It demonstrates why an assessment that inventories only uploaded files can miss consequential artifacts.

The data-deletion evidence paper addresses ordinary stores and vendor receipts. For training approval, add the learning artifact and release history to that discussion. A proposed deletion method should be supportable before the organization promises an outcome to customers that the vendor's architecture cannot deliver.

Use an approval rule with explicit non-negotiable conditions

I would treat training approval as a conjunction of conditions rather than an average score. The organization needs authority for the proposed use, a bounded processing and release scope, an adequate privacy and security case, demonstrated usefulness for the task, and an exit arrangement consistent with its obligations. A failure in one condition cannot be erased by excellent evidence in another.

The practical outcomes are approval, narrower conditional approval, a hold for missing evidence, or refusal of the proposed training use. These outcomes should describe what the vendor may do now. A hold on shared-model training need not prevent an independently acceptable inference-only workflow. A limited pilot should identify its permitted data and stopping conditions, rather than become a quiet exception for unrestricted production copying.

ConditionEvidence that would support a decisionReason to narrow or hold
AuthorityDocumented use constraints and accountable approval for the datasetRights, purpose or affected-party obligations unresolved
ScopeDataset-to-model lineage and permitted recipient/deployment boundaryShared reuse or derived artifacts remain unspecified
ProtectionRelevant privacy tests, justified guarantees and operating controlsUnsupported anonymity claim or mismatched threat model
ValueIndependent task evaluation against a suitable baselineBenefit asserted without representative evidence
ExitDefined removal/retirement claims and evidence for retained artifactsRequired withdrawal outcome cannot be supported

This is my proposed decision structure, not a certification scheme. Its value is that every approval condition can be challenged independently. The decision record should include who accepted residual uncertainty and which change would reopen the review: a new data category, a shared release, different retention, another training recipient or a materially changed model pipeline.

Apply the rule to a fictional support-data proposal

Imagine the fictional Meridian Instruments wants a vendor to improve a support assistant. The proposed corpus contains ordinary troubleshooting text, customer attachments and internal escalation notes. The vendor offers either a dedicated adaptation or permission to improve a shared model. No facts in this example describe a real customer, contract or deployment.

I would first separate the routine examples from attachments and notes whose reuse authority has not been established. The approved dataset would contain only records that pass the stated selection and review conditions. The vendor would identify the particular training run and resulting model, while an independent task set would remain outside tuning. Neither a polished demonstration nor a general security report would substitute for those decisions.

Suppose the dedicated adaptation improves the target task on the agreed evaluation, and its access and exit evidence are adequate. That may support a narrow approval. It does not support the shared-model option automatically. The shared option changes recipients and the consequences of withdrawal; it needs its own authority and evidence case even though the training algorithm and initial corpus might be identical.

Now change the premise: the task can be performed acceptably using approved retrieval over a controlled collection, without adapting model parameters. Training may still offer benefits, but the company can compare those benefits against the additional artifact and withdrawal complexity. This does not establish that retrieval is universally safer or sufficient. It identifies a narrower alternative that should be evaluated rather than excluded by the vendor's initial framing.

Finally, suppose the vendor cannot trace which model versions received the data. That is a concrete evidence gap. The company can withhold the training corpus while requesting a narrower proposal, rather than accepting a vague promise to remove it later. The decision remains useful because it says what capability is missing and which evidence would change the answer.

Account for the limits and the commercial tradeoff

The strongest argument for training is that representative operational data may capture patterns unavailable in public or synthetic examples. That benefit deserves a fair test. A blanket refusal can sacrifice useful capability, while a blanket grant can commit information to a future use the company never intended. I would compare bounded alternatives using the actual task, required confidentiality and realistic exit conditions.

There is also a cost to assurance. A small, isolated adaptation may not justify the same review as broad shared-model training on sensitive records. Scale the evidence request to consequence and uncertainty, while keeping the authority requirement intact. Where the organization cannot independently inspect the provider, distinguish contractual commitments, independent assurance and direct observations. The absence of direct inspection is a limit to disclose, not permission to relabel a vendor statement as a tested fact.

This synthesis has limits. The studies examine specific models, attacks or formal settings; their results cannot be transferred as current vendor failure rates. Documentation proposals improve visibility without guaranteeing accurate disclosures. Provider representations are dated snapshots, and legal materials are jurisdiction-specific. The framework has not been empirically validated, and no finite test proves universal non-disclosure or complete removal of every historical influence.

If a vendor wants permission to train on your production data and the proposal stops at “secure and anonymized,” request an AI data-use assessment. I would help turn the request into explicit dataset, model, recipient, evaluation and exit decisions. The useful result is a defensible scope the business can approve, narrow or reject before the data becomes difficult to retrieve.

References