Preface:

  • What is performance tuning? (what)

  • Why do you need performance tuning? (why)

  • When do you need performance tuning? (when)

  • Where do you need performance tuning? (where)

  • Who does the performance tuning? (who)

  • How do you tune performance? (How)

** Hardware configuration: **CUP Xeon E5620 x 28 Core, 16GB memory, RAID 10 hard disk

Operating system: CentOS 6.4x86_64 (64-bit)

Note: 5W + 1H is a scientific analysis of the work done. Based on the investigation and research of a certain work:

  • In terms of What they do

  • Responsible Person (Who)

  • Jobs (Where)

  • When do you work

  • How to operate (How

  • And Why they do it

That is, write a written description of “5W” and “1H”, and operate according to the description to achieve the goal of completing the job task.

What is performance tuning?

(what)

Before we talk about performance tuning, let’s talk about the architecture of the computer.

As shown in the figure above, it simply consists of three pieces: hardware, operating system, and applications. Performance tuning is all about tuning these things, including hardware, operating systems, and applications. Among them, these three aspects contain a number of content.

Hardware includes cpus, memory, disks, nics, and others. .

The operating system includes processes, virtual memory, file systems, networks, and others. .

Applications I don’t need to say we all know, common Apache, MySQL, Nginx, Memcahed, etc.

So what is performance tuning?

Performance tuning is to have an in-depth understanding of computer hardware, operating systems, and applications, and adjust the relationship among the three to maximize the performance of the entire system (including hardware, operating systems, and applications) and continuously meet existing service requirements.

This is what we call performance tuning, guest officer, do you understand?

Why do you need performance tuning?

(why)

There are two reasons why you need performance tuning: one is to get better system performance (your existing system is working fine, but tuning it can work even better). The second is to meet increasing business requirements through performance tuning. To help you intuitively understand why performance tuning is necessary?

Let’s talk about it from three aspects:

  • Hardware selection (Select servers based on server application types)

  • Operating system Distribution (Select distribution)

  • Applications (Nginx, MySQL, etc.)

1. Hardware selection

Whether you rent a server or buy your own server or you will encounter a problem, we choose what kind of hardware configuration of the server.

Generally, we choose servers based on application type, because you can’t fit all applications with one hardware configuration, because the specific needs of each application are different. Let’s take a look at the types of applications in project implementation:

  • ** Load balancing: ** performance requirements are relatively low, because only responsible for forwarding data, but ensure that you choose a network card with outstanding performance. (Recommended configuration: CPU E5620 x 1 Memory 8 GB Hard disk 500 GB (RAID5))

  • ** Web server: ** generally only processing some static pages or images, so the requirements are not very high, the mainstream server can. (Recommended configuration: CPU E5620 x 1 Memory 16 GB Hard disk 500 GB (RAID5))

  • ** Application server: ** General application server, he undertakes the implementation of website functions, occupies a relatively heavy position in the architecture, especially in the website architecture only one application server, CPU, memory, disk requirements are relatively high. (Recommended configuration: CPU E5620 x 2 Memory 32 GB Hard disk 500 GB (RAID10))

  • ** Cache server: ** is divided into front-end page cache and back-end data cache. Their typical applications are Varnish and Memcached respectively, which require a large amount of memory. Generally, we use a large amount of memory when configuring the server. (Recommended configuration: CPU E5620 x 1 Memory 32 GB Hard disk 500 GB (RAID10))

  • ** Database server: ** Data server on CPU, memory, disk requirements are very high, but a hardware is a short board will bring performance problems. (Recommended configuration: CPU E5620 x 2 Memory 64 GB SSD 500 GB (RAID10))

  • ** Backup server: ** Backup servers generally have no requirements, but one thing is certain: they must have a large enough hard disk space. (recommended configuration: CPU E5620 x 1 memory 4 gb hard disk 2TB (RAID5))

  • ** Monitoring server: ** generally there is no need, ordinary PC server can. (Recommended configuration: CPU E5620 x 1 Memory 4 GB Hard disk 500 (RAID5))

  • ** Other servers: ** As for other servers, it depends on your specific needs for specific analysis.

Now you know what is the hardware performance tuning bar, according to your specific application, specific analysis, especially like MySQL server, CPU, memory, disk requirements are relatively high.

Therefore, to tune the performance of hardware, we must choose the appropriate hardware configuration. This is the site frame architecture or project implementation to solve the first problem!

2. Operating system

There’s a book called Linux Performance Tuning, which was written by Fernando Apesteguia.

Why do we need performance tuning? He came to this conclusion:

“When a distribution is packaged and sent to the customer, it is designed to be fully compatible with the majority of computers in the market. It’s a fairly jumbled collection of hardware (hard drives, graphics cards, network cards, etc.).

So Red Hat, SUSE, Mandriva, Ubuntu, and other distributions opt for conservative Settings to ensure success.”

Simply put, your operating system already works fine, but you can adjust it to achieve higher performance. For example, if you have a high-performance disk, some of the operating system parameters are not enabled by default, so you cannot implement these advanced features to improve the performance of the disk.

RedHat or CentOS operating systems are mostly used in project implementation or website architecture. They are mainly developed for enterprise applications.

Operating systems like Ubuntu support the desktop better, so be careful when choosing a distribution. (CentOS is commonly used in enterprises.) Another is that we generally do not choose the latest release, because the release of the relatively more bugs, do not be a “guinea pig”.

For example, CentOS 7 can be used after it is stable for a period of time. At present, we can choose CentOS 6.4 or 6.5.

(But the new version also has a lot of benefits, the new version has a lot of new features, remove known bugs, for some unimportant applications, you can try to use the new operating system)

3. Apps

Finally, let’s talk about applications. Let’s take a quick look at the Apache MPM configuration file:

** Prefork model: **

As you can see from the above configuration file, apache starts with 8 processes. The minimum number of processes is 5, and the maximum number of processes is 20. Each process is limited to 256 requests and can accept a maximum of 4000 requests.

The worker model:

The configuration file of worker model starts 2 processes by default, and each process can accept 150 requests. The minimum number of threads in each process is 25, the maximum number of threads is 75, and the default number of threads is 25. The number of requests that each thread can accept is not limited to 0.

If you want to get a high performance Apache server, you must also perform performance tuning, including Apache compilation options, configuration file optimization, etc., the specific tuning we will not go into details here.

Through our above explanation, we respectively from the hardware, operating system, application, these three aspects and we talk about why you need performance tuning, I believe you already know and understand, I believe we can’t wait it.

Hey hey, before we rush, there are still many problems that are not clear, let’s talk to you about when we need to tune performance?

When do you need performance tuning?

(when)

Generally divided into two periods:

  • Pre-launch (basic optimization)

  • After launch (continuous optimization)

Why do you say so, normally we this period of time in the project implementation to the project launch, not only to prepare the server hardware, operating system installation, environment building, there is a very important problem is the performance optimization, including the operating system and application environment optimization, etc., before I call online optimization of the basic optimization, also known as experience optimization.

Perform basic performance optimizations on pre-launch servers or architectures to meet business requirements based on your projects and experience in your work.

Then there is the optimization after the project goes online. Before the project goes online, we have gone through basic performance optimization and solved most of the performance problems. But after all, all the tests before the project goes online are simulation tests and related performance optimization, and there is still a considerable difference from the real environment after the project goes online.

The first thing we do is to monitor the performance of the project after it goes live, including server performance monitoring and service performance monitoring.

Server performance monitoring includes:

CPU usage, CPU load, memory usage, disk I/O, disk space usage, network traffic, and system processes. Service performance monitoring includes Apache, Nginx, and mysql.

All of the services in the above architecture need to be monitored for performance. Once a problem is found, we have to perform performance optimization. In this process, I call continuous optimization or monitoring optimization. Now let’s talk more specifically about where performance tuning is needed.

Where do you need performance tuning?

(where)

When we say performance tuning above, we only talk about the big aspects, including hardware, operating system, and application.

In fact, there is also the optimization of the program itself. The program developed by developers according to their needs needs performance optimization, but for our operation and maintenance personnel, it is less contact.

Here’s a look at the three:

  • Hardware (CPU, memory, disk, nic)

  • Operating system (process, file system, kernel…)

  • Applications (Nginx, MySQL…)

1. The hardware

Hardware optimization generally includes two parts:

  • Before going online (Hardware selection)

  • After launch (hardware expansion)

General project construction needs to be based on the specific application of hardware configuration selection, in this aspect requires some experience in operation and maintenance of friends can be a little lack in this aspect, but nothing generally done after one or two projects, the selection of hardware configuration will be, hey hey.

But as an unwritten rule of thumb, the higher the hardware, the better (not to mention me).

Why do we say that we need to select the type according to the specific application? What kind of application needs what kind of hardware configuration? There is also a point that is very important to save cost.

So we’re talking about the beginning of the project, and you were lucky to be there at the beginning of the project.

Generally, experienced operation and maintenance engineers have no problems in hardware selection, so we do not consider the hardware in performance optimization. Theoretically speaking, our server hardware configuration generally does not have such performance problems.

However, as our business is getting better and better, we did not consider such a large performance requirement (traffic) at the beginning of the project. Now some hardware cannot meet the business needs, so we need to replace better CPU, larger memory and faster disk.

We won’t go into details here about how hardware is a performance bottleneck, but we will in a later article. Finally, let’s look at a hardware architecture diagram to help you better understand hardware optimization, as shown below (Dell R 710 architecture) :

2. Operating system

Let’s talk about the operating system, in fact, most of the optimization in the operating system and application optimization, in addition to the online hardware selection and online hardware expansion, let’s take a look at the operating system optimization includes what:

  • Optimized OS installation

  • System initialization

  • Process tuning

  • Memory tuning

  • IO tuning

  • File system tuning

  • Network tuning,

Let’s take a look at a picture, which can help us understand more intuitively, as shown below:

3. Apps

Finally, let’s talk about application optimization. Here we’ll take a look at MySQL optimization examples to give you a more intuitive understanding.

  • MySQL compilation and installation optimization

  • MySQL configuration file optimization

  • The index optimization

  • MySQL Engine Optimization

  • Query cache optimization

  • SQL statement optimization

  • Optimize table types (MyISAM or InnoDB)

  • Lock mechanism optimization

  • MySQL server optimization (SSD)

Access to the above hardware, operating system, application specific description BELIEVE that you have a deeper understanding of performance optimization, let’s talk about an important question, who to carry out performance optimization?

Who does the performance tuning?

(who)

When we think about performance optimization, the first thing that comes to mind is operations engineers, who do the optimization.

In fact, I would say that one-sided performance optimization is not just for operations engineers.

Actually, performance tuning is a team thing. Why do I say that?

Let’s talk about it, we want to ah a company need to do a project, we take the most common e-commerce mall project to say, the company confirmed that because of business needs we need to do a building materials mall online, the specific process of the project is what? Maybe not in great detail, but the general process is the same:

  • Operation request

  • Product finishing requirement

  • Develop specific business applications

  • Operation and maintenance to build the development environment

  • QA conducts project testing

  • Operation and maintenance project on-line

  • Monitoring Monitors projects

Developing a specific application requires the participation of operations, Product, Development, Operations, QA (testing), monitoring and so on.

When the same project (business) has a performance problem, it is not only the operations department that needs performance tuning, but all departments work together to solve the performance problem, which is indispensable.

It may be in the product, it may be in the program (*.php), it may be in the business needs itself, it may be in the operation of the environment set up problems. But performance tuning is more about development, operations, testing, and monitoring.

How do you tune performance?

(How)

Let’s talk about how to perform performance tuning. The steps are as follows:

Performance Indicators – > Confirm the measurement criteria

Performance Test – > Verify performance indicators

Performance analysis – > Find performance bottlenecks

Performance tuning – > Resolve performance issues

Performance Monitoring – > Verify the tuning effect

1. Performance indicators

As we said above, the purpose of optimization is to get better performance, so what are the performance indicators? How do we measure it? Generally, there are three indicators to measure a project (the website here) :

Throughput – > is the number of user or system requests completed per unit of time.

Number of concurrent requests – > How many users can receive access requests at the same time

Response time – > The time between the user sending a request and receiving a response.

2. Performance test

When we build products or projects (more frankly, websites) for users to use, we have to analyze from the perspective of users what performance they need to pay attention to.

For the user, the time spent clicking a button, link or instruction until the request is processed and presented to the user in the form of a web page is an intuitive impression of the site’s performance.

This is what we call the response time. When the response time is small, the user experience is relatively good. Of course, the response time of user experience includes both subjective factors and objective response time.

In website development and construction, we need to consider how to better combine these two parts to achieve the best user experience. The user is concerned with the corresponding time of the user action.

Secondly, we consider the performance points that need attention from the perspective of operations and maintenance. Third, we need to put ourselves in the developer’s (designer’s) shoes and think about site performance. Finally, QA tests and feedback our website performance.

After the above description, we will test the system performance, we need to collect the system throughput, concurrency, response time these three important indicators.

The specific steps are:

  • Verify throughput, concurrency, and response time

  • Find or develop performance testing tools

  • Perform performance tests

  • Feedback results and submit test report

As a result, there are two kinds of results: one is to achieve our expected performance target, so we do not need to complete the performance optimization task and hand it over to operation and maintenance online, but only need to conduct related performance monitoring, which is convenient for performance optimization after online.

The other is not where we want to be, where we look for performance bottlenecks and optimize performance.

3. Performance analysis

Through the above performance test, we found that the website did not meet the performance target defined by our expectations. At this time, what we need to do is to monitor the existing system (server), including hardware and software monitoring, to provide effective performance monitoring data for performance tuning.

Let’s focus on what tools can be used to identify performance bottlenecks:

Hardware:

  • Check for CPU bottlenecks using vmstat, SAR, and iostat

  • Check for memory bottlenecks using free and vmstat

  • Check for disk I/O bottlenecks using iostat

  • Use netstat to check whether it is a network bandwidth bottle

Operating system:

  • process

  • The file system

  • SWAP partition

  • Kernel parameter adjustment

Applications (MySQL, etc.) :

  • Mysqlreport Performance analysis report

  • Mysqlsla slow query log analysis

4. Performance tuning

Identify tuning goals

Our goal of performance optimization is to improve the performance of the website by 10% or 20%. We can’t say that if you optimize the performance of the website today, you can double the performance of the website.

First, you need to ask him what goal we need to achieve.

Then, we need to take a look at the entire environment (architecture), including the code (of course you need to understand the business logic, but it can’t hurt to have a general understanding), and talk to the developer when you have time to ask how many holes need to be filled in the code.

Often they optimize the SQL query in the code, than you can optimize the system for days, haha.

Specific tuning steps

  • If you do not understand the parameters of the system, you must not arbitrarily change the parameters of the system, or you will regret.

  • Debug only one system resource at a time, such as CPU, memory, or disk.

  • You are advised to change as few parameter Settings as possible at a time.

  • When analyzing a system resource, using multiple tools often produces unexpected results. Less than better than too much (would rather do a little less, do not do too much, performance has reached the requirements do not arbitrarily move, do your monitoring).

Check the tuning results

The performance process must be detected after each performance tuning. For example, the AB tool of the Web server is a good detection tool. After each tuning, specific changes can be seen.

5. Monitor performance

Performance monitoring is important, including server performance monitoring and service-specific performance monitoring. Here are the specific performance monitoring indicators:

Server performance monitoring

  • CPU utilization

  • CPU load

  • Memory usage

  • Disk I/O

  • The network traffic

  • Disk space

  • System processes

Service Performance Monitoring (MySQL)

  • MySQL query throughput, including Change DB, Select, Insert, Update, and Delete

  • MySQL persistent connection usage

  • MySQL queries the cache space usage

  • MySQL query cache hit ratio

  • Number of MySQL cache queries

  • MySQL index cache hit ratio

  • MySQL index read statistics

  • MySQL connection throughput rate

  • MySQL connection cache hit ratio

  • Number of concurrent MySQL connections, including the maximum number of allowed connections, the actual maximum number of connections, the current number of connections, the number of active connections, and the number of cached connections

  • MySQL Traffic Statistics

  • The statistics of MySQL tables are locked

Conclusion:

In this article on the summary of performance optimization is to explain the specific optimization idea, help people understand performance optimization, so that people are easier to understand, to let everyone know that performance optimization is not the legendary so difficult, difficult to doing it not, as long as we have good method, what problem can be solved.

Believe in yourself, there is nothing impossible, only unexpected

If you feel that this article is helpful to you, welcome to join the group QQ communication group: 644196190 wechat public number: Terminal R&D Department