0 x00 preface

When we learn the front end, we always want to learn how other people’s websites are made, or when we come across some useful knowledge point, we may choose to use Ctrl + C/Ctrl + V to rip down the content. However, I am not encouraging this kind of ripping, but as long as it is open source, Not for commercial use, I think we can learn from each other, after all, reinventing the wheel is a waste of time. And through Ctrl + C/Ctrl + V this way is too troublesome, and now the picture basically has anti-theft chain, or the path from the graph bed changed to the local relative path, simple copy and paste is difficult to clean the relevant content of the website, so we have the following thinking: How to clone the contents of a website completely?

Beginners, we through the view – source: https://xxx.xxx.xxx view the web page source code this way, a new index. The HTML file, and then copy and paste content to index. The HTML content, or wget down can also be directly. But as I said above, there is no way to copy the entire content of a web page. As I learned more about python crawlers, there were instances of this, but it wasn’t implementation-friendly.

With further learning, there are special software for website cloning, such as WebZip, awwwb.com and so on, it is said to be very good, I have not tried, I am not sure (years of not using Windows). Today I would like to introduce an open source and super easy to use website cloning machine HTTrack.

0x01 What is HTTrack?

HTTrack is a free and easy to use offline browser tool, short for HTTrack Website Copier for Windows, it enables you to download specific websites from the Internet for offline browsing (offline browsing). It can also be used to gather information (some sites even use hidden password files), and some of the most fake sites (to trick users into using passwords) use similar tools. Browsing offline sites is no different than browsing online.

HTTrack can also be offline and online site synchronization, support breakpoint continuation. HTTrack is a fully configurable tool that includes a comprehensive help system. For traditional sites like the presence of robots.txt, if the program is run without restrictions, in the default environment, the program will not mirror the site, simply HTTPrack to follow the basic JavaScript or APPLet, flash links, Complex links (those created using functions and expressions) or ImageMap on the server side cannot be mirrored. You don’t have to dig too far to get target information such as the site’s physical address, phone number, email address, hours of operation, business relationships, employee names, social connections, and public tidbits. News is also important when doing penetration testing. Companies often disclose things they are proud of. These reports may reveal useful information about how the company’s acquisition servers are functioning, etc.

Httrack can be used to copy a website for offline detection, thus reducing direct interaction with the target website. Httrack is easy to use, just follow the steps of its wizard.

HTTrack has been pre-installed on the following Linux systems:

  • BackBox Linux
  • Kali Linux
  • Pentoo
  • SamuraiWTF
  • BlackArch

If no, you can download and install it by referring to the following installation method.

0x02 Installation and use of HTTrack

HTTrack supports mainstream operating systems such as Windows, Linux and MacOS, and for Windows, HTTrack has the support of a visual interface, with the following effects:

The latest version of HTTrack can be downloaded and used in the following ways.

Windows installation:

Through www.httrack.com/page/2/en/i. Download the corresponding version.

Install under Linux:

#Under the Debian/Ubuntu installation
sudo apt install httrack
#Install CentOS/Fedora
sudo yum install httrack
#The Gentoo, the installation
sudo emerge httrack
Copy the code

Mac OSX installation:

sudo port install httrack
#or
brew install httrack
Copy the code

Install directly from source code compilation:

git clone https://github.com/xroche/httrack.git --recurse
cd httrack
./configure --prefix=$HOME/usr && make -j8 && make install
Copy the code

Specific reference: www.httrack.com/page/2/en/i…

Common parameter options can be viewed using httrack –help.

0x03 Clone site demo

Environment to prepare

  • Mac OS x 10.11
  • Httrack tools

Now I’m in a test site, https://progit.bootcss.com/, for example, to demonstrate its operation process.

The cloning process

1. Httrack enters the operation interface

On a Mac OSX 10.11 with httrack installed, open the shell terminal and type httrack directly.

2. Enter the necessary information

Next we need to fill in the following information:

Welcome to HTTrack Website Copier (Offline Browser) 3.49-2 Copyright (C) 1998-2017 Xavier Roche and other Ficolin-3 To see the option list, enter a blank line or try httrack --help#1. Enter a project name to be generated
Enter project name :progit
#2. Enter the path of the project to be saved
Base path (return=/Users/apple/websites/) :/Users/apple/Desktop
#3. Enter the URL of the website to be cloned
Enter URLs (separated by commas or blank spaces) :https://progit.bootcss.com/

Action:
(enter)	1	Mirror Web Site(s)
	2	Mirror Web Site(s) with Wizard
	3	Just Get Files Indicated
	4	Mirror ALL links in URLs (Multiple Mirror)
	5	Test Links In URLs (Bookmark Test)
	0	Quit
:
#4. Press Enter if there is no special request
Proxy (return=none) :

You can define wildcards, like: -*.gif +www.*.com/*.zip -*img_*.zip
#5. Press Enter if there is no special request
Wildcards (return=none) :

You can define additional options, such as recurse level (-r<number>), separated by blank spaces
To see the option list, type help
#6. Press Enter if there is no special request
Additional options (return=none) :

---> Wizard command line: httrack https://progit.bootcss.com/  -O "/Users/apple/Desktop/progit"  -%vReady to launch the mirror? (Y/n) :Y Mirror launched on Thu, 15 Aug 2019 11:54:40 by HTTrack Website Copier / 3.49 2 [XR&CO '2014] mirroring https://progit.bootcss.com/ with the wizard  help.. Done. Thanks for using HTTrack! *Copy the code
3. Complete cloning

The cloning effect is as follows:

The cloned web page almost has no difference with the original site, the effect is super nice.

You can also clone with different options by looking at the Help option.

Mirror down web pages can be in offline mode, seepage analysis, the traditional on site host to browse the web, the more time you browse and fumble, activity may be the site tracking, even the casual browsing website, is recorded, as long as belonging to the target resource, at any time to direct interaction, may have traces of Numbers.

However, do not use this software to mirror pages on a website without authorization, as professional software with security dogs or other firewalls may record this as an attack.