Ability to limit which CT log to listen to

When you're running certstream on cheap VPS, you'll notice very high CPU usage because it connects to lots of CT logs.

By limiting where it can connect to, I was able to force certstream use much less CPU usage.

For example, if I block (return "0.0.0.0" for "*ct.sectigo.com" in DNS server):

ct-watcher.go:410: Could not get STH for 'https://tiger2026h2.ct.sectigo.com': Get "https://tiger2026h2.ct.sectigo.com/ct/v1/get-sth?": tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match tiger2026h2.ct.sectigo.com
ct-watcher.go:359: Worker for 'https://tiger2026h2.ct.sectigo.com' failed - could not fetch STH
ct-watcher.go:360: Stopping worker for CT log: https://tiger2026h2.ct.sectigo.com
ct-watcher.go:216: Removing worker for CT log: https://tiger2026h2.ct.sectigo.com
ct-watcher.go:410: Could not get STH for 'https://tiger2026h1.ct.sectigo.com': Get "https://tiger2026h1.ct.sectigo.com/ct/v1/get-sth?": tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match tiger2026h1.ct.sectigo.com
ct-watcher.go:359: Worker for 'https://tiger2026h1.ct.sectigo.com' failed - could not fetch STH
ct-watcher.go:360: Stopping worker for CT log: https://tiger2026h1.ct.sectigo.com
ct-watcher.go:216: Removing worker for CT log: https://tiger2026h1.ct.sectigo.com

Please consider adding an option to control where it can connect to.

Benefits:

• Much less CPU usage
• Internet bandwidth decreases: great for people who have internet montly cap

---

e.g., Config example

• tue/false switch

limit_ct_server: true <--- default: false, false allows all CT logs
    ct_server_allowed: (googleapis|letsencrypt.org).com$ <---  allows connection only to those CT logs

• When using null (or no specification at all), allow all CTs

limit_ct_servers: null

• Here, allowing those 2 domains and its subdomains only

limit_ct_servers:
  - letsencrypt.org
  - googleapis.com

[d-Rickyy-b]

certstream-server-go is already as lightweight as possible while at the same time trying to monitor global certificates at full scale (aka. processing all issues certificates). Only using certificates provided by e.g. the letsencrypt CT logs will leave out many certificates.

There is already an option to disable the default logs provided by the Google log list. Check the disable_default_logs option in the config file.

Custom logs can be added via the additional_logs and additional_tiled_logs options. That requires you to regularly update the logs which requires a restart.

Currently, there are no plans to add filtering mechanisms for the default log list.

For more config options, please refer to https://github.com/d-Rickyy-b/certstream-server-go/blob/master/config.sample.yaml

Only using certificates provided by e.g. the letsencrypt CT logs will leave out many certificates.

I do know the disadvantage. I just want to listen to some certs only (such as letsencrypt/google/cloudflare/...) so my VPS can handle nicely.
Before disabling some CTs with DNS, I experienced massive CPU spike & bandwidth impact.

There is already an option to disable the default logs provided by the Google log list. Check the disable_default_logs option in the config file.
Custom logs can be added via the additional_logs and additional_tiled_logs options. That requires you to regularly update the logs which requires a restart.

Yes, I know. But I don't want to disable default CT subscription because I want to make sure certstream listen on latest CT endpoint.

In other issue you said "Reducing bandwidth and processing cost definitely is something I support.", so I thought this limit_ct_servers fits your goal.

So there are 2 unuseful workarounds currently:

---

Solution 1. [My current workaround] Prevent Certstream from connecting to unnecesary CTs using DNS server. (e.g. configure dns to return "0.0.0.0")
Solution 2. Make a script to do: "Everyday, read https://www.gstatic.com/ct/log_list/v3/log_list.json, update a config file with "additional_logs", kill certstream, start certstream.", then run this script daily

[eernam]

Where can I find a list of operator or url?

Also can we have a Domain match instead of full URL bc the URI part may change?

excluded_logs:
  - operator: "Google"
  - operator: "xyz"
  - fqdn: ct2.googleapis.com <--- exact match (not useful than domains, hostname(ct2) may change)
  - domain: googleapis.com <----- means *.googleapis.com + googleapis.com

[d-Rickyy-b]

The feature to exclude logs is still only in dev status, so expect changes. Would you mind explaining the "domain match" a bit more? Usually one operator always operates their logs from the same domains. So excluding Google from the logs should be sufficient.

Is there another use case I am missing here?