Html Spiders são programas para baixar sites inteiros, uteis para ler depois etc. Dois programas são bem populares o comando wget do linux e o httrack ambos podem ser rodados em diversos sistemas operacionais (LINUX, WIN, MAC)

Wget is a classic command-line tool for this kind of task, and HTTrack is a free (GPL, libre/free software) and easy-to-use offline browser utility.

Httrak

https://www.httrack.com/

http://packages.debian.org/stable/web/webhttrack

sudo apt install webhttrack -y

Comando Wget

Wget is a classic command-line tool for this kind of task

wget -r --no-parent http://site.com/dir/

wget -P /path/directory/ -mpck --user-agent="" -e robots=off --wait 1 -E https://www.site.com/

HELP AND EXAMPLES