“This article has participated in the call for good writing activities, click to view: the back end, the big front end double track submission, 20,000 yuan prize pool waiting for you to challenge!”

Remember before and after fixing Struts2’s S2-046 vulnerability

This is an interesting development accident in 2017, here to share with you. (PS: I’ve been writing technical articles in Jane’s book, but I recently decided to move my old articles to Denver and make them better. This is the first one.)

Struts2 vulnerability:

Because the exception handler of the upload function does not correctly process the error information entered by the user, remote attackers can use this vulnerability to execute arbitrary commands on the affected server by sending malicious data packets. Attackers can make use of this vulnerability by sending malicious HTTP packets to execute system commands on the affected server, and further control the server completely, resulting in denial of service, data leakage, website tampering and other impacts. Because this vulnerability exploits without any preconditions (such as enabling DMI, DEBUG and other functions) or enabling any plug-in, the vulnerability is relatively serious. (Vulnerability Introduction: CVE-2017-5638)

The bug came out in early March of this year, and people started to pay attention to it, but at that time, no one was working on a project that was using Struts2, and we didn’t hear of any company project that was using Struts2, so people didn’t pay much attention. To 3 months (estimated popped the hole is 3 days), someone attack into the company’s electric business platform (project is a bit old, and no one use) estimation is white hat hackers, just sent a warning to us, at this time of the operations were combined together we repair, also not too seriously and other projects were cast, is procrastination. And then at the end of March, someone else comes in and this time it’s not as easy to talk about, and they just get access and wipe the data out. The operation and maintenance probably panicked and reinstalled the system without complete data backup (in fact, the version we repaired at this time was ready, I don’t know why it was not mounted). Here comes the bitter day.

Fixing this vulnerability is not too complicated, and changing jar packages is a snap. The most serious problem, however, comes from system reinstallation.

Background:

I was not included in the list of repair personnel (I was not invited to attend the meeting, and of course I was not included in the responsibility list). Later, I was really interested in watching the meeting and gave some suggestions, and the group leader invited more people into the group for discussion (I guess I wanted to invite more people so that we would not be scolded miserably. At that time, a new colleague was also temporarily assigned to the team), and later experience makes me feel that the help was well worth it!

Here are the details before and after each problem is solved:

Summary of problems:

  1. After the system was reinstalled, no pictures or files could be uploaded online (PS: requests were not submitted to the server at all), but the test server and development machine performed normally;
  2. The user cannot register: after submitting information, the prompt page times out;
  3. Users can’t use the platform’s gold coins to pay for purchases;
  4. After problem 1 is fixed, some pictures are uploaded (uploading pictures as head pictures), reporting the following exception:
java.lang.ClassNotFoundException: com.sun.image.codec.jpeg.JPEGCodec
Copy the code
  1. After all the above problems were solved, our search server: Solr server wouldn’t start (if it didn’t start here, it wouldn’t use search at all);
  2. When Solr started, it still didn’t work. Then a colleague said Solr wanted to synchronize indexes;
  3. Solr failed to synchronize index:400 Bad Request: unknown field 'type'

Details before and after each problem is solved:

Question 1: The first question, which seems to be the easiest one, actually took us the most time. The leaders of the morning meeting (fortunately, I was not called to go there) had a scolding, which started at about 10:30 and was finished at around 2:00 in the afternoon. At the beginning of this problem, we had the following positions:

  • A. The upload path separator of the project is dead.
  • B. Struts2 has interceptors/filters that block submitted forms;
  • C. Jar replaced to solve s2-046 vulnerability does not work properly under Linux;
  1. About A: After looking at the source code, the delimiters are standardFile.separatorTo get, the path is usedSystem.getProperty("user.dir")There is no problem with the code. Back afraid of online files have been replaced and a copy for comparison, or no problem ——— exclusion
  2. About B: A very serious problem was found when locating B. Due to system reinstallation, the server time is 3 hours behind the standard time. By the way, problem 2,3 is solved perfectly (because all money transactions are on a different server, and each transaction is verified by time. Registration and login work much the same on another server). We were just about to fix the problem when we realized that the original code didn’t have time to validate. We did not give up, in the development environment simulation line such a time error, a test can still upload (that is to say, the failure to upload the problem has nothing to do with the server time error). When it turns out that the code has an interceptor with an administrator IP limit, I think I finally found the problem, the break point discovery request didn’t even get in here, and the test server would have a problem if it did.
  3. About C: Direct Pass (normal in Linux Test environment)

The final result of question 1: The more I thought about it, the more I wondered why the test server was ok and the developer was ok but not online. Later, someone asked if it was possible that the online application server did not have upload permission due to the reinstallation. Is really a wake up dreamers, how to think how like, and the page jump is also reported 404 rather than 500. Write a simple file upload page and test it online. All the problems are clear, it is this permission issue, operation and maintenance we beat up.

Problem 2: The server time is not standard (3 hours difference) due to the reinstallation. The registered login server is deployed on a different machine from the project. Failed to pass the time verification during registration, so a timeout exception is reported.

Question 3: Similar to Question 2

Question4: Actually, this problem is also online. We get to the Internet search, and some people say that com. Sun.. Image codec. Jpeg. JPEGCodec family is no longer recommended, then changed the ImageIO. We tested it online and offline.

Problem 5: This is not really a problem, but the previous Solr server backup cannot be found. I found a backup from a long time ago and deployed it on a Tomcat according to solr’s deployment rules. Problem solved.

Question 6: I thought that after solr was deployed, search would come back. Tested it – no thread!! Tried to hit someone, but look at yunwei Ren already lying dead in the corner. A key —- “Solr is indexed synchronously” came from my hitherto nonexistent new colleague.

Question 7:Listen to my new colleague said that Solr index synchronization can be done in the background of Solr, after logging in to the background, I found that this is nothing like baidu and Google. It took a long time… Later, the operation and maintenance did something that made me deeply ashamed. The operation and maintenance actually found a way to connect with the new Solr index library in the background. I blame myself! Should not be so rude to operation and maintenance, so many big men to him a little cruel… Just when we thought we could walk away, the method went wrong. Ha ha ha!

Fortunately, the Solr server has an obvious error400 Bad Request: unknown field 'hnihh'. At this point, we solved the problem completely by guessing without Baidu or Google. Solr is like a database, so the index is probably configured somewhere (like MySQL’s table structure statement). Solr does not have the field ‘hnihh’ configured. Solr does not have the field ‘hnihh’ configured. Then you pull down all XML under solr and lock it in scheme.xml. Sure enough, the index fields of the project are in there except for “HNihh”. Add the < filed… > < span style = “max-width: 100%; clear: both; Finally found the result of that moment, that kind of mood, estimated that only programmers can understand.

Follow-up questions:

After all the code, system-level issues are resolved. The page display is still buggy – many images are missing because they are not backed up. Among them to find a variety of backup restore, which hard enough for humanitarian.

This vulnerability repair experience:

  1. For the vast majority of people, there is no problem that code can’t solve. Many times a small problem in the code can leave you exhausted and helpless, but the underlying problem is really very small (see questions 1, 2, 3). Most of the time, technology that seems complicated is something you probably already know. Take an hour or two to learn, without getting anxious or panicking (see questions 5, 6, 7).
  2. The 999 that many Internet companies advocate is unnecessary; It took 12 hours to solve the bug, and about 9 hours to remove the non-problem (Problem 1). Participants: 3 people, which shows what the problem is estimated that we all understand;
  3. Encounter loopholes must pay attention to;
  4. When there is a major problem or incident to be solved in the company or group, participate actively if possible. Even if you don’t ask, you can be cheeky enough to ask and look at him a few times. It’s something we don’t want to happen, but it’s a really good chance;
  5. Assume boldly where the problem lies and make associations and verification through other aspects;
  6. Try not to choose Struts2.