Create a custom wordlist using /robots.txt

ffuf everything

samidu@sudoroot:~$ cat robot-words.txt
admin
dev
sys
cgi-bin
logs
*.php$
*.txt$
*.gz$

samidu@sudoroot:~$ ffuf -u $domain/FUZZ -w robot-words.txt -ac -c -mc all

admin             [Status: 403,Size: 0, Words: 1,..]
logs              [Status: 404,Size: 50, Words: 10,..]
cgi-bin           [Status: 301,Size: 0, Words: 1,..]
dev               [Status: 500,Size: 20, Words: 16,..]
sys               [Status: 403,Size: 0, Words: 1,..]



samidu@sudoroot:~$ ffuf -u $domain/admin/FUZZ  -w robot-words.txt -ac -c -D -e php,gz,txt

 admin/logs.txt     [Status: 200, Size: 117276, Words: 1719,..] $*****BOUNTY