Communicating Uncertainties in CTI Reporting

Overview

Threat intelligence information is often imperfect. It can include intelligence gaps, assumptions, assessments, or judgments. The commonly used language that attempts to overcome these information imperfections - words such as “may”, “we believe”, and “we assess” - can be misleading and often depend on interpretation. To avoid inconsistency and miscommunications in Cyber Threat Intelligence (CTI) reporting, statements that convey judgments and assessments need to be qualified and use standardized language. Intelligence analysis utilizes Words of Estimative Probability (WEP) and Levels of Confidence in Assessment (LCA) to address information imperfections and uncertainties. Each organization is unique, and the language concerning probability and confidence levels can be tailored to suit individual requirements. The FIRST Standard provides a streamlined approach for conveying uncertainties, as outlined in the section 'Consider How To Communicate The Uncertainty' and specifically in '3. Use only LCA'. This particularly benefits CTI teams aiming to provide intelligence analysis without overwhelming consumers with complex standards.

FIRST WEP Standard

The FIRST CTI SIG recommends the use of the NATO Words of Estimative Probability (WEP) terms describing the probability of the assessment in CTI reports:

The percentages express the relationship of some of these terms to each other, not quantitative probability.

Note: For machine interpretation and automatic report reading and production the upper boundary of Unlikely should be considered to include 40%, and the lower boundary of Likely to include 60%. In mathematics, those ranges would be expressed as unlikely [10%-40%], even chance (40%-60%), likely [60%- 90%].

Framework for assessing the likelihood of an event or outcome to a specific probability level:

Very Likely Likely Even chance Unlikely Highly Unlikely
Characteristic Event or outcome is almost guaranteed to happen Event or outcome is likely to happen, but with some degree of uncertainty Event or outcome not as certain as "Likely" nor as doubtful as "Unlikely" - moderate level of possibility with some degree of confidence. Event or outcome is not likely to happen, but with some degree of uncertainty Event or outcome is almost guaranteed not to happen
Evidence Strong and consistent evidence supporting the outcome Moderate to strong evidence supporting the outcome Moderate evidence, indicating a balance between supporting and opposing evidence. It overall suggests a plausible scenario. Limited or weak evidence supporting the outcome Strong and consistent evidence supporting the opposite outcome
Expert opinions Expert opinions that strongly support the outcome Expert opinions that lean towards the outcome, but not strongly Expert opinions are mixed. There isn't a strong consensus either for or against the outcome among experts. Expert opinions that lean against the outcome, but not strongly Expert opinions that strongly oppose the outcome
History A long history of similar events with the same outcome A history of similar events with some uncertainty or variability of outcome Some precedent or past occurrences supporting the outcome - instances where similar events have varying outcomes. A history of similar events with different outcomes A long history of similar events with different outcomes

FIRST LCA Standard

Confidence in assessment addresses the fact that judgments and assessments are based on evidence that varies in quality. FIRST recommends the use of the NATO levels of confidence in assessment1:

Consider How To Communicate The Uncertainty

1. Use WEP and LCA for each analytical judgment or assessment, but not in the same sentence to avoid confusion.
Example: Based on the analysis of the malware and intrusion activity, it is likely that the [name of the threat actor] was responsible for the unauthorized access and exfiltration of sensitive data. We have moderate confidence in this conclusion.

2. Use only WEP if probability is a more important aspect of uncertainties than communicating the quality of information and confidence in your assessment.
Example: Based on the analysis of the malware and intrusion activity, it is likely that the [name of the threat actor] was responsible for the unauthorized access and exfiltration of sensitive data.

3. Use only LCA if the most important aspect of the information that you want to communicate to the consumer is how sure you are about your conclusions and/or what is the level of quality of your information. Using only LCA is also the least complex way to communicate uncertainties aiming to provide intelligence analysis without overwhelming consumers with complex standards.
Example: Based on the analysis of the malware and intrusion activity, we assess with moderate confidence that the [name of the threat actor] was responsible for the unauthorized access and exfiltration of sensitive data.

References

1. ICD-203, ICD-203_TA_Analytic_Standards_21_Dec_2022.pdf^_