Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixed crt.sh parsing issue, -subs-only quirk and added support for riskiq #42

Open
wants to merge 25 commits into
base: master
Choose a base branch
from

Conversation

cybercdh
Copy link

Fix parsing of return values from crt.sh
Sometimes the results from crt.sh will produce multiple results in the name_value field, separated by a newline character. The whole returned string which may include multiple subdomains is treated as one subdomain and then passed to cleanDomain which removes any erroneous prefixes.

However, consider searching for subdomains of yhs.search.yahoo.com and this will yield, as an example,
*.qa-release.yhs.search.yahoo.com\n*.qa-trunk.yhs.search.yahoo.com which is of course two subdomains.

Assetfinder will then output

qa-release.yhs.search.yahoo.com
*.qa-trunk.yhs.search.yahoo.com

thereby giving the appearance that *.qa-trunk.yhs.search.yahoo.com hasn't been passed through cleanDomain.

I've updated the crtsh.go file to parse this field more effectively which ensures a "clean" output.

-subs-only flag still producing non-subdomain results

I found that by moving the following substring comparison

if subsOnly && !strings.HasSuffix(res.n, res.domain) {
	continue
}

outside of the main goroutine and into the for loop over the out range removed any erroneous results.

…ich caused issues with printing clean domains, also fixed issue where -subs-only didnt appear to work by moving the check outside of the main goroutine.
@cybercdh cybercdh changed the title fixed issue where multiple hits returned in name_value from crt.sh wh… fixed issue in crt.sh and ensure that -subs-only flag works as intended Jul 21, 2021
@cybercdh cybercdh changed the title fixed issue in crt.sh and ensure that -subs-only flag works as intended fixed issue in parsing results. from crt.sh and ensuring that -subs-only flag works as intended Jul 21, 2021
@cybercdh cybercdh changed the title fixed issue in parsing results. from crt.sh and ensuring that -subs-only flag works as intended fixed issue in parsing results from crt.sh and ensuring that -subs-only flag works as intended Jul 21, 2021
@cybercdh cybercdh changed the title fixed issue in parsing results from crt.sh and ensuring that -subs-only flag works as intended fixed issue crt.sh parsing issue, -subs-only quirk and added support for riskiq Nov 23, 2022
@cybercdh cybercdh changed the title fixed issue crt.sh parsing issue, -subs-only quirk and added support for riskiq fixed crt.sh parsing issue, -subs-only quirk and added support for riskiq Nov 23, 2022
@cybercdh cybercdh force-pushed the cybercdh branch 2 times, most recently from b0156f1 to a8bbc28 Compare January 2, 2023 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant