dependency conflicts between spark and springboot

[bebee4java]

Spark 2.4.3 integrates with springboot 2.2.1.RELEASE. When the spring application starts, it will throw java.lang.NoSuchMethodError:

or java.lang.AbstractMethodError:

or other errors.
These error must be because of conflicting Jars between spring-boot and apache spark.
My brief pom.xml look like this:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.2.1.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <properties>
        <java.version>1.8</java.version>
        <scala.version>2.11.12</scala.version>
        <scala.compat.version>2.11</scala.compat.version>
        <scope>provided</scope>
        <spark.version>2.4.3</spark.version>
        <spark.bigversion>2.4</spark.bigversion>
        <hadoop-client-version>2.7.6</hadoop-client-version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
            <scope>${scope}</scope>
        </dependency>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-compiler</artifactId>
            <version>${scala.version}</version>
            <scope>${scope}</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.compat.version}</artifactId>
            <version>${spark.version}</version>
            <scope>${scope}</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_${scala.compat.version}</artifactId>
            <version>${spark.version}</version>
            <scope>${scope}</scope>
        </dependency>
      <dependencies>
</project>

Mark note:
I found the following jars will be in conflict:

In spark	In SpringBoot
snakeyaml-1.15.jar	snakeyaml-1.25.jar
validation-api-1.1.0.Final.jar	validation-api-2.0.1.jar
gson-2.2.4.jar	gson-2.8.6.jar
guava-14.0.1.jar	guava-20.0.jar

In my solution：

1. exclusion the jar if the application is not affected(Such as logback jar).
2. the application remains the same version jar with spark, if spark cannot be adjusted and the application is not affected.
3. If neither spark and the application can adjust the jar version, you can use the shade plug-in relocation the conflicting classes or package.

According to the third point, I adjusted the pom.xml file, look like this:

......
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
    <exclusions>
        <exclusion>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-classic</artifactId>
        </exclusion>
    </exclusions>
</dependency>
......
<plugins>
  <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.2.1</version>
        <configuration>
            <filters>
                <filter>
                    <artifact>*:*</artifact>
                    <excludes>
                        <exclude>META-INF/*.SF</exclude>
                        <exclude>META-INF/*.DSA</exclude>
                        <exclude>META-INF/*.RSA</exclude>
                    </excludes>
                </filter>
            </filters>
            <createDependencyReducedPom>false</createDependencyReducedPom>
            <!--<minimizeJar>true</minimizeJar>-->
        </configuration>
        <executions>
            <execution>
                <phase>package</phase>
                <goals>
                    <goal>shade</goal>
                </goals>
                <configuration>
                    <relocations>
                        <relocation>
                            <pattern>org.yaml.snakeyaml</pattern>
                            <shadedPattern>my.yaml.snakeyaml</shadedPattern>
                        </relocation>
                        <relocation>
                            <pattern>javax.validation</pattern>
                            <shadedPattern>shade.javax.validation</shadedPattern>
                        </relocation>
                        <relocation>
                            <pattern>com.google.gson</pattern>
                            <shadedPattern>shade.google.gson</shadedPattern>
                        </relocation>
                        <relocation>
                            <pattern>com.google.common</pattern>
                            <shadedPattern>shade.com.google.common</shadedPattern>
                        </relocation>
                    </relocations>
                </configuration>
            </execution>
        </executions>
    </plugin>
</plugins>
......

Build the fat jar(with dependences) is like:

[yyi]

thx u,it works