I love CT logs! I use them to discover subdomains for my hosts file block list [1]. While it can't expand on domains that use wildcard certs - or no certs at all - it's better then nothing since hosts files don't support wildcard blocking.
Some things I've learned while working with them:
* CertSpotter [2] is a fantastic CT client written in Go that supports pattern matching. I've been running it locally with `.` match pattern and so far have a 4 gig file of unique domain names. I'm excited to see the end result once it catches up to current time.
* https://crt.sh/ is a great website to search CT logs and supports wildcards. It's currently the workhorse behind my hosts project, but I hope to remove them as a dependency once my own domain list is caught up to present day
* It looks like OP's tool is just a thin client for entrust API [3] and is not actually downloading logs directly - which isn't clear in the article. It made more since once I figured that out because these logs are huge and go back years.
I write security reports for websites, and I use CT to inform the website owner if there are unused certificates for the given domain. Usually the customer is quite surprised that this information publicly available.
But in 99% of the cases it's not so much a security problem. For bug hunters it may be usable as unlisted subdomains have less exposure, so they may be the first to scan it for bugs. It is still a concern for the website owner though, because they don't want the world to know about a new product or experiment they are running.
General advice: don't obtain certificates for a subdomain until you are ready to tell the world about it.
Or even better don’t register CNAMES or A records for your sub domain until you’re ready to tell the world. The cert is meaningless if there’s nowhere for the traffic to route.
That’s fair point for giving intent if there’s a human facing name for the DNS entry. I was referring to the security implications of having a public endpoint exposed, or more accurately not being exposed because there’s no way to route traffic to it.
The minor note that his ISP rate limited DNS to 36 or 50 response batches stood out to me. I don't understand why they'd want to do that or how that benefits them.
Is this the sort of targeted traffic shaping net neutrality would prohibit?
DNS (used to be?) is a great amplifier for DDOS attacks. Because the 'source' address is user supplied, and there are DNS responses that are much larger than the requests.
So you can use 1Mb/s of data to send DNS queries with your targets IP as the source address to get e.g. a 10Mb/s stream of data to your target.
Maybe the rate-limiting is an attempt to subvert being used in this kind of DDOS.
He was [likely] using his ISP's or Other providers resolvers, Large bursts of queries are very annoying to deal with, and are very rarely legit ( and more software going wrong )
It's very common to rate limit querying down to some low number to protect the rest of the customers using it
Some things I've learned while working with them:
* CertSpotter [2] is a fantastic CT client written in Go that supports pattern matching. I've been running it locally with `.` match pattern and so far have a 4 gig file of unique domain names. I'm excited to see the end result once it catches up to current time.
* https://crt.sh/ is a great website to search CT logs and supports wildcards. It's currently the workhorse behind my hosts project, but I hope to remove them as a dependency once my own domain list is caught up to present day
* It looks like OP's tool is just a thin client for entrust API [3] and is not actually downloading logs directly - which isn't clear in the article. It made more since once I figured that out because these logs are huge and go back years.
[1] https://github.com/lightswitch05/hosts
[2] https://github.com/SSLMate/certspotter
[3] https://www.entrust.com/ct-search/