download:Python’s three-year-old distributed crawler, Scrapy, builds a search engine
What era is the future? It’s the data age! Data analysis services, Internet finance, data modeling, natural language processing, medical case analysis… More and more work is done based on data, and crawlers are the most important way to get data quickly, compared to other languages, Python crawler is more simple and efficient for the crowd Suitable for interested in reptiles, want to do is to develop large data but couldn’t find the data Don’t know how to set up a set of stable and reliable distributed crawler classmates Want to set up the search engine technology reserve requirements but don’t know how to proceed with classmates have a certain native crawler based Understanding the front page, object-oriented concepts, Computer network protocol and database knowledge SQL injection
SQL injection is the behavior that the server side does not strictly verify the data sent by the client, so the SERVER side SQL statement is maliciously corrected and successfully executed.
Essence: User input data is executed as code. Any central interaction with the database is likely to have injection.
SQL injection Type Data transmission: GET POST COOKIE
Data type: Integer character type
Injection form: combined with the query error report Boolean blind injection time blind injection push query
The common steps of SQL injection determine whether changes in injection-controlled parameters can affect the results of a page. —-> If an error is reported in the database, you can see the statement trace of the database. Whether the entered SQL statement does not report errors —-> the statement can be successfully closed. Check whether the injection-type statement can be corrected by malicious intent and whether it can be executed to get the data we want
MySQL allows an “information_schema” database to be stored in the database with three important table names schemata,tables, and columns.
The schemata table stores the database names of all databases created by the user. The field name is schema_name.
The tables table stores the database names and table names of all databases created by the user. The database database name and table name fields are table_SCHEMA and table_name respectively.
Columns Specifies the name of the database where all columns are stored. The columns are table_schema,table_name, and column_name respectively.
Select the field name from the database name. Table name select the name of the field to be queried from the database name. Table name WHERE known field name = ‘known field value’ SELECT the field name from the database name. Name of the table where the known conditions field name 1 = ‘the value of the known conditions 1 and 2 the condition of known field name =’ 2 the value of the known conditions limit usage limit m, n m record the position of the beginning, from zero beginning said first documented; N means take n records.
The important function database() is the database used by the current site. Version () Indicates the current MySQL version. User () Indicates the current MySQL user. @@datadir Database approach. @@version_compile_OS OS version concat(STR1, STR2…) Concatenate string concat_ws(separator,str1,str2…) without separator. Concatenation string with delimiters group_concat(str1,str2…) Concatenate all strings in a group and separate each data comment with a comma. Common comments are: # — space /**/ expressed in the URL as: %23 –+
A conditional field cannot be indexed by a function.
Select * from t1 where date© = ‘2019-05-21’; Optimization: Change to range query
Select * from t1 where c>= ‘2019-05-21 00:00:00’ and c<= ‘2019-05-21 23:59:59’; Implicit conversion operators, when used with different types of operation objects, perform type conversions to make operations compatible.
select user_name,tele_phone from user_info where tele_phone =11111111111; / tele_phone varchar/practice will do function operations:
select user_name,tele_phone from user_info where cast(tele_phone as singed int) =11111111111; Optimization: Type unification
Select user_name,tele_phone from user_info where tele_phone = ‘11111111111’; The ambiguous query wildcard comes first
Select * from t1 where a like ‘%1111%’; Optimization: Vague queries must contain the value in front of the condition field
Select * from t1 where a like ‘1111%’; Scope query Scope query data volume is too large, need to return to the table, so do not go to the index.
select * from t1 where b>=1 and b <=2000; Optimization: reduce the range of a single query, divided into repeated queries. (Practice may not be too fast, take the index)
select from t1 where b>=1 and b <=1000; show profiles; + — — — — — — — — — – — — — — — — — — — — — – — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — + | Query_ID | Duration | Query | + — — — — — — — — — – — — — — — — — — — — — – — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — + | 1 | | 0.00534775 select the from t1 where b and b > = 1 < = 1000 | | 2 | | 0.00605625 select * from t1 where b > = 1 and b < = 2000 | + — — — — — — — — — – — — — — — — — — — — — – — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — – + 2 rows in the set, 1 warning (0.00 SEC) operation Even a simple calculation
explain select * from t1 where b-1 =1000; Optimization: Put the calculation operation after the equal sign