一次spark 使用protobuf3过程记录

JamesB · 发表于 2022-5-28 10:32

背景

因业务需要使用protobuf3对数据进行压缩，spark自带的protobuf-java-2.5.0.jar。需要在spark环境上跑代码，所以项目引入spark的jar包都是provided，没办法屏蔽掉spark的protobuf，又需要同时存在protobuf3.
protobuf使用

proto 文件

syntax = "proto3";option java_package = "com.xxx.protobuf";//生成的java名option java_outer_classname = "Foo";message FooItem { string bar = 1;}maven依赖

<properties> <protobuf-maven-plugin-version>0.6.1</protobuf-maven-plugin-version> <protoc.version>3.18.1</protoc.version> <os-maven-plugin-version>1.6.2</os-maven-plugin-version> <maven-clean-plugin.version>3.1.0</maven-clean-plugin.version> </properties> <dependency> <groupId>com.google.protobuf</groupId> <artifactId>protobuf-java</artifactId> <version>3.18.1</version> </dependency> <dependency> <groupId>com.googlecode.protobuf-java-format</groupId> <artifactId>protobuf-java-format</artifactId> <version>1.2</version> </dependency> <build> <extensions> <extension> <groupId>kr.motd.maven</groupId> <artifactId>os-maven-plugin</artifactId> <version>${os-maven-plugin-version}</version> </extension> </extensions> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>2.4.3</version> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <relocations> <relocation> <pattern>com.google.protobuf</pattern> <shadedPattern>shaded.com.google.protobuf</shadedPattern> </relocation> </relocations> <minimizeJar>false</minimizeJar> <shadedArtifactAttached>true</shadedArtifactAttached> <artifactSet> <includes>  <include>*:*</include> </includes> </artifactSet> <filters> <filter> <artifact>*:*</artifact> <excludes> <exclude>META-INF/*.SF</exclude> <exclude>META-INF/*.DSA</exclude> <exclude>META-INF/*.RSA</exclude> <exclude>**/log4j.properties</exclude> </excludes> </filter> </filters> <transformers> <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"> <resource>reference.conf</resource> </transformer> <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"> <resource> META-INF/services/org.apache.spark.sql.sources.DataSourceRegister </resource> </transformer> </transformers> </configuration> </execution> </executions> </plugin> <plugin> <groupId>net.alchim31.maven</groupId> <artifactId>scala-maven-plugin</artifactId> <version>3.3.2</version> <executions> <execution> <id>scala-compile-first</id> <phase>process-resources</phase> <goals> <goal>compile</goal> </goals> </execution> <execution> <id>scala-test-compile-first</id> <phase>process-test-resources</phase> <goals> <goal>testCompile</goal> </goals> </execution> </executions> </plugin> <plugin> <groupId>org.xolstice.maven.plugins</groupId> <artifactId>protobuf-maven-plugin</artifactId> <version>${protobuf-maven-plugin-version}</version> <configuration> <includes> <include>**/*.proto</include> </includes> <protoSourceRoot>${basedir}/src/main/scala/com/foo/protobuf</protoSourceRoot> <outputDirectory>${basedir}/src/main/scala</outputDirectory>  <clearOutputDirectory>false</clearOutputDirectory> <protocArtifact> com.google.protobuf:protoc:${protoc.version}:exe:${os.detected.classifier} </protocArtifact> </configuration> <executions> <execution> <id>compile</id> <goals> <goal>compile</goal>  </goals> </execution> <execution> <id>test-compile</id> <goals> <goal>test-compile</goal>  </goals> </execution> </executions> </plugin> </plugins> </build>
build 中protobuf-maven-plugin 会在打包时编译proto文件，会覆盖原来的文件，os-maven-plugin获取系统信息
问题排查过程

尝试一
拉下来protobuf java部分源码，自行编译，需要先对整个protobuf进行c++编译，然后对java部分进行maven编译，目的是为了直接把protobuf3的源码复制到项目里，改掉包路径，这样就不用引protobuf的依赖。
存在问题

<parent> <groupId>com.google.protobuf</groupId> <artifactId>protobuf-parent</artifactId> <version>3.18.1</version> </parent>

放弃
依赖冲突解决方案一

使用jarjar.jar 更改protobuf的包名然后打包到本地仓库，可正常打包
问题1: 因为打包时会自动编译protobuf文件，但是编译出来的proto文件包名还是com.google.protobuf，打进来会报错
问题2:所以需要手动把改过的proto文件压进jar包里（这个操作比较复杂，可以网上搜索相关方法）

这个方法虽然复杂，但是这个方法的确可以解决依赖冲突
依赖冲突解决方案二

推荐使用
使用maven-shade-plugin插件，编译时重命名protobuf3的包，反编译查看生效，包括proto生成的文件也生效
在本页面搜索关键词 shaded.com.google.protobuf
百度关键词 protobuf3 shade 打包

image.png

完美解决问题 !!!!
参考

https://w3sun.com/1287.html protobuf使用
https://blog.csdn.net/weixin_38336546/article/details/121686301 Idea中使用Maven插件配置Protobuf方法总结

https://www.jianshu.com/p/3b7f386c5c4a

https://www.freesion.com/article/78311237323/  jarjar.jar使用

https://www.itxm.cn/post/33379.html    maven shade使用
https://blog.csdn.net/Edgar_LQS/article/details/123101560

https://stackoverflow.com/questions/64224830/illegalaccesserror-tried-to-access-field-com-google-protobuf-abstractmessage-me  sbt打包参考这个

		自动登录	找回密码
密码			立即注册

一次spark 使用protobuf3过程记录

本帖子中包含更多资源