introduce pagination

[dogancanbakir]

Closes #1104.

before:

$ go run . -d salesforce.com -s securitytrails

               __    _____           __         
   _______  __/ /_  / __(_)___  ____/ /__  _____
  / ___/ / / / __ \/ /_/ / __ \/ __  / _ \/ ___/
 (__  ) /_/ / /_/ / __/ / / / / /_/ /  __/ /    
/____/\__,_/_.___/_/ /_/_/ /_/\__,_/\___/_/

                projectdiscovery.io

[INF] Current subfinder version v2.6.4-dev (development)
[INF] Loading provider config from /Users/fortytwo/Library/Application Support/subfinder/provider-config.yaml
[INF] Enumerating subdomains for salesforce.com
...
[INF] Found 10000 subdomains for salesforce.com in 2 seconds 254 milliseconds

after:

$ go run . -d salesforce.com -s securitytrails

               __    _____           __         
   _______  __/ /_  / __(_)___  ____/ /__  _____
  / ___/ / / / __ \/ /_/ / __ \/ __  / _ \/ ___/
 (__  ) /_/ / /_/ / __/ / / / / /_/ /  __/ /    
/____/\__,_/_.___/_/ /_/_/ /_/\__,_/\___/_/

                projectdiscovery.io

[INF] Current subfinder version v2.6.4-dev (development)
[INF] Loading provider config from /Users/fortytwo/Library/Application Support/subfinder/provider-config.yaml
[INF] Enumerating subdomains for salesforce.com
...
[INF] Found 46100 subdomains for salesforce.com in 10 minutes 251 milliseconds

[DhiyaneshGeek]

i tested with the following command

subfinder -d apple.com -rl 1 -s securitytrails -v -o apple.txt -proxy=http://127.0.0.1:8080

gsp80-ssl.ls.apple.com
nwk-downloads-origin-old.apple.com
sg03p00ls-gsp82-sg03.ls.apple.com
linear-ak-b-har3.tv.apple.com
[INF] Found 39895 subdomains for apple.com in 6 minutes 39 seconds

Issues Found are

• Securitytrails has rate limit of 2 API request per second, so we need to set rate limit while running this particular source using -rl 1 rate limit flag.
• Scan gets stops if there is 502 Bad Gateway error and shows the results without completion.
• If the subdomains are more than 50K it takes more than 10 mins of the subfinder default time, users need to provide -max-time flag.

[DhiyaneshGeek]

Hi @dogancanbakir

These changes looks good to me !

This will solve the problem between free or paid users

[INF] Enumerating subdomains for dell.com
[DBG] Response for failed request against https://api.securitytrails.com/v1/domains/list?include_ips=false&scroll=true:
{
  "endpoint": "/v1/domains/list",
  "message": "This feature is not available for your subscription package. Consider upgrading your package or contact [email protected]"
}
[securitytrails] en.community.dell.com
[securitytrails] i.dell.com
[securitytrails] content.dell.com
[securitytrails] linux.dell.com
[securitytrails] software.dell.com

LGTM 😄