Introduction:

Some time ago, we needed to use HUDi + Flink for development, but a cloud department of our company customized a set of big data dependency packages (reasons unknown). The environment is provided by them and can only be used), and these dependencies are published to their repository, and a proxy is required to connect to their repository. In addition to unity, a group of have the same prior to the scaffolding, a project thought easily import the scaffold project will soon development, younger sister think of his colleague said he opened a company in the “transparent” agent (this is what east east, I don’t know), but I don’t want to apply for alone make the transparent proxy, so decided to change the configuration, It took a lot of work to get it right. And I helped my colleagues with the scaffolding later.

During the process, I encountered some maven configuration items that I did not notice at ordinary times. For information about some companies, only the bullet points are recorded here.

Background:

Before, a colleague in the group was responsible for connecting with the brother department (referred to as c Department) in our group, and then he made a “framework”, and did some HDFS configuration, Hadoop configuration, dependency configuration, etc., so that other people with the scaffolding did not have to bother with various configurations and dependencies. However, I later found that the scaffolding project of my colleague released some JAR packages to the Maven warehouse of our own department, while those of Hadoop, HDFS and HUDi rely on the warehouse of C department, and this warehouse cannot be directly connected, so it needs to use proxy.

Question:

Problem phenomenon:

The proxy dependency has been reported as an error, and the IDEA Maven external libraries have no dependency.

And then there’s the weird thing: you can’t just add an external dependency, like Druid. No response at all. No dependency.

Solution:

Output the dependency tree with MVN

The MVN uses -x to output debugging information.

mvn dependency:tree -s D:\work_code\xxxx\xxx\settings.xml -PJDK18.,Myprofile -X
Copy the code

Then various errors are reported:

1. The first warehouse you notice it goes to isDg.xxx.com/artifactory…So guess is their installation of the MVN setting file. Turns out it is.

The local MVN setting file needs to be replaced

2. After the first step above, execute again, there is an obvious error: XXXX transfer file XXX “WAREHOUSE B” XXX 502

So I searched 502 and found that it was bad gateway. Anyway, it was related to gateway and proxy.

Check the proxy configuration in the setting file and find the following one.

<proxy> <id>optional</id> <active>true</active> <protocol> HTTP </protocol> <username> My ID </username> <password>xxx</password> <host>proxy.xxx.com</host> <port>8080</port> <nonProxyHosts>local.net|some.host.com</nonProxyHosts> </proxy>Copy the code

Then look at the output of MVN Dependency: Tree and you can see a lot of information, for example, from dg.xxx.com/artifactory… When this gets the package it’s going to print the username and the password.

That reminds me that the setting file now has two important configurations:

1.

2.

The functions of these two labels can be referred to:

1. www.cnblogs.com/jingmoxukon…

2. Juejin. Cn/post / 684490…

There is an important explanation for comparing mirror and proxy:

Proxy configuration relates to a standard HTTP proxy you might need reaching Internet, such as a proxy in your web browser. It does not relate to maven functionalities as such, but to reaching Internet connectivity. If you can use a browser without a proxy, you probably don't need one for Maven either, and vice versa. Mirror configuration, on the other hand, is meant to replace a maven repository with another. They are not related to connectivity - they change the target which  your maven tries to reach.Copy the code

In other words, when we connect to Repository, we sometimes need a proxy to connect to the network. This proxy acts as a “middleman” for downloading dependencies from Repository

  1. Focus on the server label and proxy label

    The two tags are similar in that they both have username and password

The server id refers to the repository ID. So I guess: server username and password are used for the jar package

<server>
    <id>xxx</id>
    <username>xxx</username>
    <password>password</password>
</server>
Copy the code
  1. To download jar packages for your department, configure the following repositories
<repositories>
    <repository>
        <id>xxxlocal</id>
        <url>http://dg.artifact.xxx.com/artifactory/maven-local-group/</url>
        <releases>
            <enabled>true</enabled>
        </releases>
        <snapshots>
            <enabled>true</enabled>
        </snapshots>
    </repository>
</repositories>
Copy the code

I configured the above configuration in the POM file. Then the corresponding user name and password are also configured in the setting file

<server>
    <id>xxxlocal</id>
    <username>xxx</username>
    <password>xxx#</password>
</server>
Copy the code

The complete effective poM is as follows:

<repositories>
  <repository>
    <releases>
      <enabled>true</enabled>
    </releases>
    <snapshots>
      <enabled>false</enabled>
    </snapshots>
    <id>xxxCloudSDK</id>
    <url>https://mirrors.xxxcloud.com/repository/maven/xxxcloudsdk/</url>
  </repository>
  <repository>
    <releases>
      <enabled>true</enabled>
    </releases>
    <snapshots>
      <enabled>true</enabled>
    </snapshots>
    <id>xxxlocal</id>
    <url>http://dg.artifact.xxx.com/artifactory/maven-local-group/</url>
  </repository>
  <repository>
    <snapshots>
      <enabled>false</enabled>
    </snapshots>
    <id>central</id>
    <name>Central Repository</name>
    <url>https://repo.maven.apache.org/maven2</url>
  </repository>
</repositories>
Copy the code
5. Combine the proxy502 question above. Modify proxy as follows
<proxy> <id>optional</id> <active>true</active> <protocol> HTTP </protocol> <username> My ID </username> <password>xxx</password> <host>proxy.xxx.com</host> <port>8080</port> <nonProxyHosts>dg.artifact.xxx.com</nonProxyHosts>  </proxy>Copy the code

Important: don’t delegate the repository address dg.artifacxxxei.com.

MVN depencency: tree

Conclusion:

  1. Just look at the Effective POM file. See which repositories there are and whether the Mirror of setting intercepts any
  2. Repository is tried in turn. Last but not least, central storage

Supplement: maven.apache.org/guides/mini…

The official website says that if you define a repository, you will add the Super POM repository as well

The website also looks at effective POM

Repository Order
Remote repository URLs are queried in the following order for artifacts until one returns a valid result:

effective settings:
Global settings.xml
User settings.xml
local effective build POM:
Local pom.xml
Parent POMs, recursively
Super POM
effective POMs from dependency path to the artifact.
For each of these locations, the repositories within the profiles are queried first in the order outlined at Introduction to build profiles.

Before downloading from a repository, mirrors configuration is applied.

Effective settings and local build POM, with profile taken into account, can easily be reviewed to see their repositories order with mvn help:effective-settings and mvn help:effective-pom -Dverbose.
Copy the code

3. On the basis of effective poM:

3.1 Repository first. Maven attempts to download jars sequentially from each repository until the last central repository (if available).

3.2 The ID of the server tag corresponds to a Repository (repository ID). Enter the username and password in the Server TAB. Use this username and password to authenticate with repository.

3.2 Proxy Serves as a proxy for network connectivity. By default, proxy proxies for all repository urls. If you don’t need proxies for some repositories, configure nonProxyHosts in the proxy tag