Sabre2025h1 MMD violations

Joe DeBlasio

unread,

Feb 3, 2025, 4:37:11 PMFeb 3

to #CTOps, Certificate Transparency Policy, chrome-certificate-transparency

(Resending to include ct-policy@)

Hi Sectigo folks,

Yesterday morning (US time), we started encountering a large spike in 404 errors coming from sabre2025h1. This has resulted in not just significantly dropped log availability, but many certificates violating MMD. I've attached two arbitrary examples, but we've collected several hundred, both from certificates generated by our monitoring infrastructure and in-the-wild certificates.

Please stop this log from accepting additional submissions ASAP until such time as you have confidence that the log will be able to fully recover and include all outstanding and subsequently submitted certificates in a timely manner.
Please acknowledge your investigation on ct-policy@ as soon as possible.
As you learn more, please provide regular updates, info on whether the log will be able to fully recover, etc., to ct-policy@

Thank you,

Joe, on behalf of the Chrome CT team

1.pem

2.pem

Rob Stradling

unread,

Feb 3, 2025, 11:06:32 PMFeb 3

to Joe DeBlasio, #CTOps, Certificate Transparency Policy, chrome-certificate-transparency

Hi Joe. Thanks for reporting this.

Our Ops team just implemented a temporary block on add-chain and add-pre-chain requests to Sabre2025h1.

We're investigating, and we'll follow up with more details when we can.

From: Joe DeBlasio <jdeb...@chromium.org>
Sent: 03 February 2025 16:36
To: #CTOps <ct...@sectigo.com>
Cc: Certificate Transparency Policy <ct-p...@chromium.org>; chrome-certificate-transparency <chrome-certific...@google.com>
Subject: Sabre2025h1 MMD violations

Rob Stradling

unread,

Feb 5, 2025, 5:47:26 PMFeb 5

to Joe DeBlasio, #CTOps, Certificate Transparency Policy, chrome-certificate-transparency

Sabre2025h1's sequencing backlog dropped to zero a little while ago, and so we believe that all outstanding certificates have now been included.

Consequently, we've unblocked add-chain and add-pre-chain requests to Sabre2025h1.

We continue to do our best to ensure inclusion of "subsequently submitted certificates in a timely manner", both for Sabre2025h1 and for our other log shards. For example, we are currently monitoring an increase in Mammoth2025h1's sequencing backlog, and we'll take action to block submissions if it becomes necessary. But to actually address the root cause, which is the relatively poor sequencing performance of Trillian's MySQL/MariaDB backend, the only real solution we've identified is to migrate away from that backend.

Our Ops team has very nearly finished load-testing Trillian's new PostgreSQL backend. This has taken longer than expected due to the holiday period, other projects, and several iterations of configuration before we correctly understood how to configure the CTFE Storage Saving feature in a performant fashion. As soon as this load-testing is complete (hopefully today or tomorrow), I'll be pushing for our team to stand up new logs for which we'll submit inclusion requests to Chrome and Apple.

From: Rob Stradling <r...@sectigo.com>
Sent: 03 February 2025 23:06
To: Joe DeBlasio <jdeb...@chromium.org>; #CTOps <ct...@sectigo.com>

Cc: Certificate Transparency Policy <ct-p...@chromium.org>; chrome-certificate-transparency <chrome-certific...@google.com>

Subject: Re: Sabre2025h1 MMD violations

Hi Joe. Thanks for reporting this.

Our Ops team just implemented a temporary block on add-chain and add-pre-chain requests to Sabre2025h1.

We're investigating, and we'll follow up with more details when we can.

From: Joe DeBlasio <jdeb...@chromium.org>
Sent: 03 February 2025 16:36
To: #CTOps <ct...@sectigo.com>
Cc: Certificate Transparency Policy <ct-p...@chromium.org>; chrome-certificate-transparency <chrome-certific...@google.com>
Subject: Sabre2025h1 MMD violations

This Message Is From an External Sender

This message came from outside your organization.

Report Suspicious

Joe DeBlasio

unread,

Feb 5, 2025, 6:02:32 PMFeb 5

to Rob Stradling, #CTOps, Certificate Transparency Policy, chrome-certificate-transparency

Thank you for the update and the work, Rob and other Sectigo folks. It's very much appreciated. (And I know, in particular, that the mysql/mariadb backend for Trillian is an ongoing source of pain for many log operators, and very much appreciate your work to come up with a better long-term solution.)

Joe

Martijn Katerbarg

unread,

Apr 4, 2025, 8:06:38 AMApr 4

to Certificate Transparency Policy

Based on our monitoring, and a notification from the Chrome CT team, Sabre2025h1's MMD was exceeded between 18:17 UTC on the 26th of March and 22:29 UTC on the 27th of March.

In the last few days, we've seen a large decrease in usage of sabre2025h1, with a similar increase in usage on sabre2025h2 as 90 day certificates now fall within that log shard's window. So far, sabre2025h2 is coping.

We keep monitoring this, but as we mentioned previously, we view the best path forward for the long term resolution on these issues to be the usage of our Postgres-backed CT Logs.

Meanwhile, we may further tweak rate limits, especially for submission of requests. While we're not particularly fond of having to restrict submissions, between risking an MMD violation or stricter rate limits, the latter seems the lesser of two evils.