Incident Reporting Guidelines

Change History

Incident Reporting

Systems, processes, and people aren’t perfect. Omissions and deviations from expected outcomes can sometimes happen. However, when omissions or deviations are discovered, the underlying issues (i.e., root causes) need to be identified and remediated to discourage future recurrence. Formally documenting these cases, in the form of an Incident Report, promotes a thorough understanding of contributing factors and facilitates clear communication of remediation plans, while also fostering a culture of continuous improvement within the Web PKI ecosystem.

This page describes the CCADB Incident Reporting Framework and corresponding guidelines. For questions, contact support [at] ccadb [dot] org.

Definitions

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” on this page are to be interpreted as described in RFC 2119.

Unless otherwise stated, “certificate” on this page refers to a final certificate, distinct from a precertificate (as described in RFC 6962).

Table of Contents

Useful background

• What is considered an incident?
• What is considered an audit finding?
• Why is public reporting important?
• What are the key characteristics of these reports?
• What does root cause analysis consider?

Community participation in the reporting process

• Who can submit an Incident Report?
• Are there other ways to become involved in the reporting process?
• What Bugzilla account can I use?

Report lifecycle management

• How do I submit a report?
• How are reports scoped?
• What format is used?
• When are reports expected?
• When are reports updated?
• How are reports closed?

Report templates

• Preliminary Incident Report Template
• Full Incident Report Template
• Report Closure Template

Report field definitions and expectations

• Incident Reports
• Incident Closure Summary

Illustrative practices

• Are there examples of “good” practices?
• Are there examples of “bad” practices?
• Are there examples of “good” reports?

Useful background

What is considered an incident?

Minimally, a failure to meet the commitments described in any of the following policies is considered an incident:

• a CA Owner’s own policies (e.g., CP, CPS, or combined CP/CPS);
• applicable requirements promulgated by the CA/Browser Forum;
• the CCADB Policy; or
• any applicable Root Store Operator policy.

Root Store Operator policies may further define what those individual programs consider incidents and/or outline additional incident reporting expectations.

It’s important to note that the existence of an Incident Report does not generally indicate serious problems with a CA.

For publicly-trusted CA Owners, the number of incident reports filed in Bugzilla is less important than the content and quality of those reports.

What is considered an audit finding?

Qualifications, non-conformities, and other deviations or omissions identified during audits are considered “findings.”

These items are commonly, but not exclusively, presented as either:

• major non-conformities,
• minor non-conformities,
• qualifications,
• qualified opinions, or
• other matters.

Why is public reporting important?

Incident Reports provide lessons learned and transparency about the steps the CA Owner takes to address the immediate issue and prevent future issues. If the underlying problem goes unfixed, then other issues that share the same root cause will subsequently surface.

The public reporting process is important because it promotes continuous improvement, information sharing, and highlights opportunities to define and adopt improved practices, policies, and controls. Further, public reporting helps convey the implications and impact of an event so that affected stakeholders have an opportunity to assess risk and determine if it warrants further action. Together, these activities help build a more secure web.

What are the key characteristics of these reports?

Reports are expected to:

• be candid (i.e., focusing on honesty and forthrightness, even when revealing uncomfortable truths), timely (i.e., released without undue delay), and transparent (i.e., emphasizing open accessibility of information);
• clearly (i.e., avoiding jargon, defining technical terms, adding useful background information) and objectively (i.e., supporting claims with data) communicate the scope, impact, and severity of an incident;
• help community members understand the relevant architectures, implementations, operations, and external dependencies surrounding an incident;
• demonstrate a detailed understanding of the root cause(s) of an incident;
• unambiguously explain how systems, processes, and/or policies failed;
• describe the circumstances that prevented the problem(s) from being detected earlier;
• describe a clear timeline of a CA Owner’s actions while responding to and remediating an incident;
• include a detailed and measurable explanation of actions taken or planned by the CA Owner that demonstrate a substantive commitment of how the CA Owner’s systems, policies, and/or processes will be made more robust (i.e., demonstrate continuous improvement); and
• share lessons learned that could be helpful to all CA Owners in building better systems policies, and/or processes.

What does Root Cause Analysis consider?

Effective Root Cause Analysis (RCA) minimally considers the following points:

1. Focus on the “why,” not just the “what”. RCAs go beyond identifying what went wrong and delve deeper into why it happened. This often involves looking past the immediate technical or policy failure and considers contributing factors like human error, process deficiencies, or system design flaws.

2. Use a systematic approach. Employ structured methodologies like the following approaches to help ensure a thorough and organized investigation:
   • Chesterton’s Fence: Before changing anything, understand why it’s there. This helps avoid unintended consequences and ensures solutions address the root cause, not just symptoms.
   • 5 Whys: Repeatedly ask “why” as many times as necessary to uncover the underlying cause of a problem. This simple technique helps drill down to the core issue and prevents superficial solutions.
   • Multi/Second Order Thinking (1, 2, 3): Think beyond immediate consequences. Consider the ripple effects of actions and potential downstream impacts to make more informed decisions.
   • CATWOE: Analyze a problem from different perspectives (customers, actors, transformation, world view, owner, environmental constraints). This ensures a holistic understanding of the issue and its context.
   • Cause and Effect / Fishbone or Ishikawa: Visually map potential causes of a problem, categorizing them to identify root causes and relationships. This promotes systematic exploration and prevents overlooking factors.
   • Drilling Down: Break complex problems into smaller, more manageable parts. This allows for focused investigation and helps identify specific areas for improvement.
   • SRE Handbook Guidance: These chapters provide practical advice on managing incidents effectively, including communication, coordination, and postmortem analysis. They emphasize a blameless culture focused on learning and improvement.
     • Chapter 14
     • Chapter 15

3. Consider all potential contributing factors. RCAs consider a broad range of potential causes, including technical issues, policy issues, human factors, process breakdowns, and external influences. It’s crucial to avoid jumping to conclusions or focusing solely on the most obvious cause(s).

4. Collect and analyze data. Data is critical for supporting RCA conclusions and may include reviewing logs, monitoring metrics, internal incident reports, and other relevant information to identify patterns and anomalies.

5. Involve relevant stakeholders. RCAs are a collaborative effort involving engineers, operators, support teams, and other relevant stakeholders to ensure a diverse range of perspectives and expertise is considered.

6. Prioritize action items. The ultimate goal of an RCA is to prevent future incidents caused by the same failures; therefore, it’s important to identify and prioritize actionable recommendations that address the identified root causes.

7. Focus on blameless postmortems. Emphasize a blameless culture when conducting postmortems. The focus is on learning from the incident and improving systems, not on assigning blame or punishing individuals.

8. Continuously improve the process. RCA is an iterative process where an organization continuously refines its approach based on lessons learned from previous incidents and evolving best practices.

Community participation in the reporting process

Who can submit an Incident Report?

Anyone should feel encouraged to submit an Incident Report that’s founded upon credible and well-substantiated evidence.

Some Root Store Operator policies require CA Owners to submit an Incident Report as described on this page after self-discovering or being made aware of an Incident (e.g., receiving and corroborating an issue described in a Certificate Problem Report).

Are there other ways to become involved in the reporting process?

Absolutely! There are many ways to participate in the incident reporting process beyond submitting new reports. Everyone is encouraged to actively contribute by commenting on existing reports and engaging in constructive discussions. This can include, but is not limited to:

• providing additional information,
• asking clarifying questions,
• discussing technical aspects of the incident,
• suggesting corrective actions,
• highlighting opportunities for improvement,
• contributing lints to open source linting projects to promote improved future issue detection/prevention, and
• sharing lessons learned from past experiences.

Individuals representing CA Owners are especially encouraged to participate broadly in the reporting processes, extending their contributions beyond incidents involving only their own organization. Sharing insights and perspectives across organizational boundaries fosters a collaborative learning environment and strengthens the overall security posture of the Web PKI ecosystem.

Please keep all comments constructive, relevant, and in line with the CCADB Code of Conduct to ensure productive dialogue.

What Bugzilla account can I use?

For individuals affiliated with a Publicly-Trusted CA Owner:

• To better encourage blamelessness, when posting incident reports or responding to comments on reports for which they are affiliated, participants MAY respond from a Bugzilla account associated with one of the CA e-mail aliases disclosed to the CCADB, rather than an individual contributor’s account.
• To better respect a desire for individual privacy and potential risk of retaliation, individuals participating in the reporting process MAY participate responsibly from an account that does not identify the individual posting or their organizational affiliation.

For everyone else: Create a new Bugzilla account following these instructions.

Remember, please keep all comments constructive, relevant to the corresponding report, and in line with the CCADB Code of Conduct.

Report lifecycle management

How do I submit a report?

Create a new Bugzilla issue by filling out this form.

• The “Summary” field in Bugzilla (i.e., “Subject line”) MUST begin with the CA Owner’s name, followed by a colon, and a brief title that highlights the type of incident being reported (e.g., “EXAMPLE CA OWNER: Incorrect Subject RDN Encoding”). The CA Owner’s name SHOULD match exactly with the CA Owner value in the CCADB.
• The “Description” field MAY contain a Preliminary or Full Incident Report (copied and pasted from the corresponding Markdown template, as explained below).

How are reports scoped?

There SHOULD be a single Incident Report for each distinct matter, and CA Owners MUST submit an additional, separate Incident Report when:

• policy requires the revocation of one or more certificates by a certain deadline, such as those in BR Section 4.9, but that deadline will not be or has not been met by the CA Owner (i.e., a delayed revocation Incident Report).
• in the process of researching one incident, another incident with a distinct root cause and/or remediation is discovered.
• after an incident is marked resolved in Bugzilla, the incident reoccurs.

All reports MUST be free-standing and not rely on the contents of other reports. While reports MAY repeat information from discussions or Bugzilla comments, they SHOULD summarize previous findings. CA Owners are responsible for compiling a complete report according to these guidelines, even if information exists elsewhere.

What format is used?

Use one of the templates below, depending on the type of report being disclosed:

• Preliminary Incident Report (for third party reporters and CA Owners submitting Preliminary Incident Reports)
• Full Incident Report (for CA Owners submitting Full Incident Reports)

Report content MUST be provided in Markdown (i.e., not in the form of an attachment) and SHOULD utilize Markdown’s formatting features (headers, lists, emphasis, etc.) to ensure clarity and readability. Individuals submitting reports SHOULD use Bugzilla’s “preview” feature to confirm rendering appears as expected before posting.

Learn more about expected report content and a description of the fields included in the reporting templates here.

When are reports expected?

Within 72 hours of a CA Owner becoming aware of an incident (i.e., the “initial incident disclosure”) or an audit finding not previously disclosed in an Incident Report, the CA Owner MUST either:

• disclose a Preliminary or Full Incident Report; or
• respond to a Preliminary Incident Report previously created for the incident by a third party reporter.

In its initial report (i.e, Preliminary or Full Incident Report) or reply to a third-party report, the CA Owner MUST:

• accurately disclose the impact of the incident (e.g., the corpus of then-known mis-issued certificates); and
• describe whether the incident should be considered contained (e.g., because certificate issuance was stopped) or ongoing.

If the described impact of the incident is later found to be inaccurate, the CA Owner MUST clearly communicate a correction, identifying when each following change was detected, the circumstances that led to the inaccuracy, and how this will be avoided in the future.

While Full Incident Reports SHOULD be posted as soon as possible, they MUST be posted within 14 days of the incident’s initial disclosure.

When are reports updated?

CA Owners should respond promptly to comments and questions, and MUST respond within 7 days, even if only to acknowledge the request and provide a timeline for a full response.

Open reports MUST be updated:

• on or before the “Next update” date in the “Whiteboard” field of the bug (note: CA Owners MAY request the “Next update” Whiteboard field be set by a Root Store Operator to align with a specific date related to an open Action Item.);
• within 7 days, if a “Next update” date is not recorded;
• in response to community questions or comments as described above; or
• when Action Items are changed, completed, or delayed.

In the case of Incident Reports with a Whiteboard field of “revocation-delay”, reports SHOULD be updated every 3 days and MUST be updated no less frequently than every 7 days to describe a summary of:

• the number of certificates that have been revoked;
• the number of certificates that have not yet been revoked;
• the number of certificates planned for revocation that have expired; and
• an estimate for when all remaining revocations will be completed.

How are reports closed?

When all Action Items are complete and no outstanding comments or questions remain, CA Owners MUST request closure in a Bugzilla comment using the template below. Upon doing so, a final call for comments will be made by a Bugzilla moderator, and the report will be closed accordingly.

Report Templates

The following templates MUST be used when submitting incident reports or requesting report closures.

• Preliminary Incident Report Template
• Full Incident Report Template
• Report Closure Template

CA Owners submitting reports MUST complete all applicable fields in the relevant template. Fields that are not applicable MUST still be included and marked ‘N/A’.

Learn more about expected report content and a description of the fields included in the templates below.

Preliminary Incident Report

## Preliminary Incident Report

### Summary
- **Incident description:**
- **Relevant policies:** 
- **Source of incident disclosure:** 

Full Incident Report

## Full Incident Report

### Summary

- **CA Owner CCADB unique ID:** 
- **Incident description:** 
- **Timeline summary:**
   - **Non-compliance start date:** 
   - **Non-compliance identified date:** 
   - **Non-compliance end date:** 
- **Relevant policies:** 
- **Source of incident disclosure:** 

### Impact

- **Total number of certificates:**
- **Total number of "remaining valid" certificates:** 
- **Affected certificate types:** 
- **Incident heuristic:** 
- **Was issuance stopped in response to this incident, and why or why not?:** 
- **Analysis:** 
- **Additional considerations:** 

### Timeline

### Related Incidents

| Bug                                | Date                        | Description                                                            |
|------------------------------------|-----------------------------|------------------------------------------------------------------------|
| [Related Bug ID](Related Bug URL)  | Date Related Bug was opened | A description of how the subject Bug is related to the Bug referenced. |

### Root Cause Analysis

** Contributing Factor #: title**
- **Description:** 
- **Timeline:** 
- **Detection:** 
- **Interaction with other factors:** 
- **Root Cause Analysis methodology used:**

### Lessons Learned

- **What went well:** 
- **What didn’t go well:** 
- **Where we got lucky:** 
- **Additional:** 

### Action Items

| Action Item | Kind    | Corresponding Root Cause(s) | Evaluation Criteria | Due Date   | Status |
| ----------- | ----    | --------------------------- | ------------------- | -----------| ------ |
| Example     | Prevent | Root Cause # 1              | Criteria            | 2025-01-19 | Value  |

### Appendix

Closure Report

### Report Closure Summary

- **Incident description:** 
- **Incident Root Cause(s):**  
- **Remediation description:**
- **Commitment summary:**  

All Action Items disclosed in this report have been completed as described, and we request its closure.

Report field definitions and expectations

The following sections are intended to describe expected Incident Report content. All fields are mandatory for the corresponding report type, except when described below.

If submitted by the CA Owner corresponding with the report, all fields included in the relevant template MUST be completed. Fields that are not applicable MUST be included in the report and identified as ‘N/A’.

Incident Reports

Summary: The Summary section contains a short description of the nature of the issue and useful background information. This provides just enough context for readers to understand the details in the rest of the report.

Note: If notified of an incident by an external, third party reporter, please respect their privacy by only disclosing their name if affirmatively approved to do so (e.g., say “we received a report from a community member” instead of explicitly naming individuals). Fields marked with an asterisk (*) are not required in Preliminary Incident Reports if submitted by a third party reporter.

Impact: The Impact section contains a description of the size and nature of the incident. For example: how many certificates, OCSP responses, or CRLs were affected; whether the affected objects share features (such as issuance time, signature algorithm, or validation type); and whether the CA Owner had to cease issuance during the incident.

If certificates are impacted, the Impact section MUST include the following information:

Timeline: The Timeline section includes a detailed timeline of all events and actions leading up to and taken during and after the incident. The timeline MUST include not just the actual discovery of the incident and subsequent events, but also relevant events occurring beforehand (e.g., something changed or was introduced). All times MUST be in UTC or UTC+local offset, and SHOULD have at least minute-level granularity.

Expected Timeline elements:

• All policy, process, and software changes that contributed to the Root Cause(s)
• The time at which the incident began
• The time at which the CA Owner became aware of the incident
• The time at which the CA Owner received a Certificate Problem Report (if applicable)
• The time at which the CA Owner provided a preliminary report on its findings to the entity who filed the Certificate Problem Report (if applicable)
• The time at which the CA Owner provided a preliminary report on its findings to the affected Subscriber(s) (if applicable)
• The time at which the CA Owner concluded and disclosed the scope and impact of the incident
• The time at which the CA Owner updated the scope and impact of the incident (if applicable)
• The time(s) at which the CA Owner is expected to complete revocation of affected certificates (if applicable, but required when Whiteboard field contains ‘revocation-delay’)
• The time(s) at which the CA Owner actually completed revocation of affected certificates (if applicable, but required when Whiteboard field contains ‘revocation-delay’)
• The time at which the incident ended
• The times at which issuance ceased and resumed (if applicable)

Related Incidents: The Related Incidents section includes a list of all incidents disclosed to the ‘CA Certificate Compliance’ Bugzilla Component related to this incident that have taken place minimally in the last two (2) years. “Related Incidents” MUST consider incidents beyond those corresponding to the CA Owner subject of this report.

Root Causes: The Root Causes section contains a detailed analysis of the conditions which combined to give rise to the issue. It is unusual for an incident to have a single root cause; often there is a confluence of several issues such as a software bug, insufficient checks, and a malformed request. Make sure that all contributing factors are identified and described, including noting when they first arose and how they avoided detection until they were discovered or identified.

Lessons Learned: The Lessons Learned section describes what the organization learned from the incident, including what they did well and what they need to improve.

Action Items: The Action Items section contains a list of remediation items that will be undertaken to ensure that similar incidents do not reoccur in the future. For example, if the Whiteboard field contains ‘revocation-delay’, the Action Items list MUST include steps reasonably calculated to prevent or reduce future revocation delays. Note that it is not sufficient for these action items to simply stop this incident, they MUST create additional protections to prevent future incidents.

Appendix: The Appendix section is for all supporting data: log files, graphs and charts, etc. In the case of incidents that directly impact certificates (i.e., not only precertificates), the Appendix MUST disclose details related to the time-valid, including revoked, certificates affected by the incident.

For incidents affecting less than 10,000 certificates, a CA Owner MUST attach a comma separated listing of certificate details including the following fields for each:

For incidents affecting 10,000 or more certificates, a CA Owner MAY instead attach a text file where each line is of the form https://crt.sh/?sha256=[sha256 fingerprint of the certificate].

When the incident being reported involves an S/MIME certificate, if disclosure of personally identifiable information in the certificate MAY be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate.

Incident Closure Summary

The Incident Closure Summary allows a CA Owner to signal they believe an incident is ready for closure.

Illustrative practices

Are there examples of “good” practices?

1. Be detailed and precise:
   • Document, in detail, all findings and steps taken from discovery or initial notification through to remediation.
   • Include precise timestamps, technical data, and decision rationales.
   • Share relevant configurations, code snippets, or settings that were implicated, in compliance with security norms (i.e., do not disclose confidential or sensitive information).
2. Respond promptly:
   • Acknowledge receipt of the incident report quickly.
   • Provide initial assessments and expected timelines for resolution to keep stakeholders informed.
   • Respond to comments or questions within one business day.
3. Be candid, transparent, and objective:
   • Doing so allows community members to better understand report content, while also promoting integrity and transparency.
4. Rely on comprehensive Root Cause Analysis frameworks:
   • Use structured methodologies like the “Five Whys” or fault tree analysis to trace back to underlying issues.
   • Involve diverse teams (security, operations, engineering) to ensure a multi-faceted evaluation.
5. Be proactive and thorough:
   • Implement immediate fixes to mitigate any ongoing risks.
   • Develop long-term solutions that might involve architectural changes or the integration of new security tools and practices.
6. Engage with the community:
   • Participate in Web PKI forums and email distributions (e.g., public@ccadb.org) to both learn from and contribute to community knowledge.
   • Share incident learnings in a way that respects confidentiality but helps other organizations prevent similar issues.
   • Read and adopt best practices found in the Bugzilla Incident Reports filed by other CA Owners.
7. Share iterative updates:
   • Provide an initial response, followed by regular updates aligned with “When should Incident Reports be updated?” as new information becomes available or as fixes are deployed..
   • Close the loop with a final summary once the issue is fully resolved, detailing the lessons learned and future prevention strategies.
8. Respect external reporter privacy:
   • If notified of an incident by an external, third party reporter, please respect their privacy by only disclosing their name if affirmatively approved to do so (e.g. say “we received a report from a community member” instead of explicitly naming individuals).
9. Use Markdown formatting:
   • Markdown formatting (e.g., headers, lists, bolding, italics, etc.) can promote clarity, readability, and accessability. Learn more about Markdown formatting here.

Are there examples of “bad” practices?

1. Generic or evasive responses:
   • Avoid generic statements that do not address specific issues raised in the report or in response to community questions or comments.
   • Refrain from providing non-committal or ambiguous answers that might be interpreted as evasive.
2. Posturing:
   • Avoid blaming others or external factors.
   • Resist downplaying the severity or implications of the incident.
3. Claims that are subjective, unqualified opinions, speculative, or impossible to substantiate:
   • Avoid making claims that are speculative or that cannot be corroborated (e.g. “there is no security impact due to this issue.”)
4. Non-acknowledgment of responsibility:
   • Clearly accept responsibility where and when due, which helps in rebuilding trust and demonstrating accountability.
5. Superficial Root Cause Analysis:
   • Avoid superficial analyses that do not thoroughly explore all contributing factors.
   • Ensure that the analysis is not prematurely concluded, missing deeper systemic issues.
6. Committing to opaque actions:
   • Ensure actions and follow-up work items are detailed and specific.
   • Promote accountability by describing how the success of an action item can be evaluated as having been successful in addressing the root cause.
7. Inadequate monitoring post-report:
   • Establish mechanisms to monitor the long-term effectiveness of implemented changes.
   • Be ready to revisit and revise solutions if subsequent issues indicate that the initial response was not entirely effective.

Are there examples of “good” reports?

Here are some examples of good practice, where a CA Owner did most or all of the things recommended above:

• Serving invalid or incomplete CRLs
  • Clear Summary and Impact statements allow readers to quickly understand the scope of the incident
  • Detailed and thorough evaluation of existing safeguards that failed, allowing for an understanding of why the the incident was allowed to take place
  • Comments and questions from the community were responded to promptly.
• keyCompromise key blocking deviation from CP/CPS
  • Clear indication of Preliminary and Full Incident Reports.
  • Detailed timeline that identifies all policy, process, and software changes that contributed to the root cause, and an indication of when the incident began and ended.
  • Detailed Root Cause Analysis that offers background on the various conditions that gave rise to the issue.
  • Timely updates in response to questions posed, continued analysis, and changes to Action Items.
• Failure to properly validate IP address
  • Significant amount of background information that informs the timeline of the incident.
  • Clear identification of the contributing factors that contributed to the incident that notes how many of them avoided detection in the Root Cause Analysis.
  • Action Items that prevent, mitigate, and detect what didn’t go well.
  • Timely and detailed updates conveying Action Item status.

Upon adoption of the updated report templates described on this page, examples of good practice will be updated.