AI Gave Your Incident Archive a Second Job
Security records are increasingly being used by vendors to train their models, creating new risks for enterprises.
OPINION
Security records are increasingly being used by vendors to train their models, creating new risks for enterprises.
Traditionally, security incident records have served a singular, very important purpose: to document what the organization observed, decided, and did. Closure notes, rule text, false-positive dispositions, and the reason why an alert was dropped are treated by security teams as records contained in the ticket queue. For organizations to reconstruct their decisions and, crucially, be accountable for them later, all of this evidence must be retained.
To be sure, security automation is anything but new. For years, SOAR platforms have executed playbooks, closed alerts, and recorded dispositions. They run a rule that customers wrote. The change is what happens to the record afterward, as a portion of the evidence has an alternate destination for a different purpose. Once incident records are ingested into a security AI system with permission to capture data for model building or validation, this evidence can become data the vendor uses for model development and validation.
The Archive’s Wealth of Information
A sufficiently large collection of prompts and related records can offer more than a glimpse into which signals are treated by the organization as meaningful, which detections are suppressed, what analysts mark as benign, and where it misses something.
A single closure is a comparatively harmless note. A series, however, presents a map; suppressing the same class of alerts delineates boundaries of concern. The rationales for closures indicate acceptable levels of PowerShell noise or failed logins that warrant a ticket. Tuning decisions encode which signals the team stops trusting and when. The contents of the ticket do not need to be accessed by an external party for it to understand the pattern, which is the sensitive portion.
This isn’t about reconstructing an AI incident; that’s a logging decision. Nor is it about prompt injection, or the familiar case of a vendor training on customer chat logs. The problem is what happens when ordinary security records acquire a second use.
Copilot Options
Microsoft Security Copilot provides an option for the owner to allow Microsoft to capture and human-review Security Copilot data to enhance and validate Microsoft’s security-specific model, which is built on the Azure OpenAI foundation model. The data captured for model improvement can include whatever Microsoft plug-ins and Security Copilot agents have used while interacting with the product. The documentation states that this data is not used to train the foundation model and is not shared with OpenAI. A second setting controls the capture of data for the purpose of product-performance reviews.
The default for each setting depends on the method of provisioning the workspace. With the exception of Microsoft 365 E5 and E7 auto-provisioning, data sharing is enabled by default. The customer that enabled the setting has made an intentional decision, but the customer that has not viewed the settings page may not know which default applies. Using the service and allowing a vendor to capture data for model development or validation are different decisions, and Microsoft has already made them separable.
The Incident Archive’s New Duty
The archive had one job: namely, to help the organization reconstruct and defend its own decisions. Now the same records may be held by a vendor for model development or validation, which in turn creates new risks. A vendor breach puts the records outside the customer’s network, while a legal demand can expose them without the customer’s control. The retention rules and access controls were built for an archive the organization controlled, but they were not built for a second copy elsewhere.
An auditable record has to preserve more than the final decision. It has to distinguish what the organization knew, what the analyst decided, what the system was given, and what the system recommended. Otherwise, the model’s observation effectively becomes the decision. Counsel asking what the organization knew at the time of the incident will not be satisfied by this. An auditor asking why the detections were tuned down should not be met with the closure rationale that also appears in the evidence stream.
This belongs in the procurement contract. A vendor’s use of customer security records to provide a service does not by itself authorize their use for model improvement, and evidence retention and model development are different things that should require separate permission. Rule content, closure notes, and ticket text should not be covered by a generic AI clause. If the vendor wants the data for model improvement, then this should be an explicit, revocable choice.
Until these uses are formally separated, the archive has two functions. One of them continues, as expected, to be operated by the organization. The other turns all the evidence into the vendor’s resource.
Read more about:
