EMR-flinksql运行失败问题

芒果4个月前技术文章128

运行flinksql
sql-client.sh
报错:
[root@emr1 bin]# ./sql-client.sh
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/dtstack/Flink/flink_pkg/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/dtstack/Tez/tez_pkg/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
10:55:25.408 [main] ERROR org.apache.flink.table.factories.FactoryUtil - Could not load service provider for factories.
java.util.ServiceConfigurationError: org.apache.flink.table.factories.Factory: Provider org.apache.flink.table.formats.raw.RawFormatFactory could not be instantiated
    at java.util.ServiceLoader.fail(ServiceLoader.java:232) ~[?:1.8.0_144]
    at java.util.ServiceLoader.access$100(ServiceLoader.java:185) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) ~[?:1.8.0_144]
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[?:1.8.0_144]
    at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_144]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:623) [flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:378) [flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.createTableEnvironment(ExecutionContext.java:114) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.<init>(ExecutionContext.java:66) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.SessionContext.create(SessionContext.java:246) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.local.LocalContextUtils.buildSessionContext(LocalContextUtils.java:87) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:87) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.SqlClient.start(SqlClient.java:88) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:187) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava18/com/google/common/collect/Sets
    at org.apache.flink.table.formats.raw.RawFormatFactory.<clinit>(RawFormatFactory.java:144) ~[flink-table-blink_2.11-1.12.5.jar:1.12.5]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_144]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_144]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_144]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
    at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ~[?:1.8.0_144]
    ... 14 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.shaded.guava18.com.google.common.collect.Sets
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_144]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_144]
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) ~[?:1.8.0_144]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_144]
    at org.apache.flink.table.formats.raw.RawFormatFactory.<clinit>(RawFormatFactory.java:144) ~[flink-table-blink_2.11-1.12.5.jar:1.12.5]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_144]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_144]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_144]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
    at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ~[?:1.8.0_144]
    ... 14 more
10:55:25.420 [main] ERROR org.apache.flink.table.client.SqlClient - SQL Client must stop. Unexpected exception. This is a bug. Please consider filing an issue.
org.apache.flink.table.api.TableException: Could not instantiate the executor. Make sure a planner module is on the classpath
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:165) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.createTableEnvironment(ExecutionContext.java:114) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.<init>(ExecutionContext.java:66) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.SessionContext.create(SessionContext.java:246) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.local.LocalContextUtils.buildSessionContext(LocalContextUtils.java:87) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:87) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.SqlClient.start(SqlClient.java:88) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:187) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
Caused by: org.apache.flink.table.api.TableException: Could not load service provider for factories.
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:627) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:378) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    ... 8 more
Caused by: java.util.ServiceConfigurationError: org.apache.flink.table.factories.Factory: Provider org.apache.flink.table.formats.raw.RawFormatFactory could not be instantiated
    at java.util.ServiceLoader.fail(ServiceLoader.java:232) ~[?:1.8.0_144]
    at java.util.ServiceLoader.access$100(ServiceLoader.java:185) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) ~[?:1.8.0_144]
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[?:1.8.0_144]
    at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_144]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:623) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:378) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    ... 8 more
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava18/com/google/common/collect/Sets
    at org.apache.flink.table.formats.raw.RawFormatFactory.<clinit>(RawFormatFactory.java:144) ~[flink-table-blink_2.11-1.12.5.jar:1.12.5]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_144]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_144]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_144]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
    at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) ~[?:1.8.0_144]
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[?:1.8.0_144]
    at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_144]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:623) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:378) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    ... 8 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.shaded.guava18.com.google.common.collect.Sets
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_144]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_144]
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) ~[?:1.8.0_144]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_144]
    at org.apache.flink.table.formats.raw.RawFormatFactory.<clinit>(RawFormatFactory.java:144) ~[flink-table-blink_2.11-1.12.5.jar:1.12.5]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_144]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_144]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_144]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
    at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) ~[?:1.8.0_144]
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[?:1.8.0_144]
    at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_144]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:623) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:378) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    ... 8 more
Exception in thread "main" org.apache.flink.table.client.SqlClientException: Unexpected exception. This is a bug. Please consider filing an issue.
    at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:201)
    at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161)
Caused by: org.apache.flink.table.api.TableException: Could not instantiate the executor. Make sure a planner module is on the classpath
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:165)
    at org.apache.flink.table.client.gateway.context.ExecutionContext.createTableEnvironment(ExecutionContext.java:114)
    at org.apache.flink.table.client.gateway.context.ExecutionContext.<init>(ExecutionContext.java:66)
    at org.apache.flink.table.client.gateway.context.SessionContext.create(SessionContext.java:246)
    at org.apache.flink.table.client.gateway.local.LocalContextUtils.buildSessionContext(LocalContextUtils.java:87)
    at org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:87)
    at org.apache.flink.table.client.SqlClient.start(SqlClient.java:88)
    at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:187)
    ... 1 more
Caused by: org.apache.flink.table.api.TableException: Could not load service provider for factories.
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:627)
    at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:378)
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156)
    ... 8 more
Caused by: java.util.ServiceConfigurationError: org.apache.flink.table.factories.Factory: Provider org.apache.flink.table.formats.raw.RawFormatFactory could not be instantiated
    at java.util.ServiceLoader.fail(ServiceLoader.java:232)
    at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
    at java.util.Iterator.forEachRemaining(Iterator.java:116)
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:623)
    ... 10 more
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava18/com/google/common/collect/Sets
    at org.apache.flink.table.formats.raw.RawFormatFactory.<clinit>(RawFormatFactory.java:144)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.lang.Class.newInstance(Class.java:442)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
    ... 14 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.shaded.guava18.com.google.common.collect.Sets
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 21 more
检查发现有个lib包的版本后缀存在问题

F382BD31-0482-46D8-9ED0-25D7BD554D39.png
删除后重新运行解决
6EDA7728-7489-442D-8149-A1692489AD10.png
放入kafka的时候需要驱动 不然会报错
094EFAC2-A0D7-44C8-8FC0-55CBF68D4C45.png
maven地址
https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-kafka_2.12
当前环境的flink是1.14.5的所以我们选择
https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-kafka_2.12/1.14.5
放到flink/lib下
2D0D1835-5F71-4BC0-92C7-0C34CDE82874.png


相关文章

HDP部署Tez UI

HDP部署Tez UI

首先部署tomcat在官网下载apache-tomcat-9.0.22.tar.gz[root@hdp02 opt]# wget https://archive.apache.org/dist/tom...

CDP实操--集群扩容

CDP实操--集群扩容

一、前提准备工作1.确保OS的yum源可以正常使用,通过yum repolist命令可以查看到匹配的OS的所有包2.确保Cloudera Manager的yum源运行正常3.hosts文件配置,需要将...

Trino配置yanagishima-23.0(包含编译)

Trino配置yanagishima-23.0(包含编译)

1 环境介绍1.1 本文采用trino 359yanagishima v23.02 编译yanagishima2.1 安装编译yanagishima需要的工具安装编译yanagishima需要的工具w...

hive 报 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster

hive 报 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster

解决办法:关键需要配置两个配置:mapred-site.xml 和 yarn-site.xml下面配置hadoop classpath。先运行shell命令:hadoop classpath添加一个配...

nginx配置反向代理某个url

nginx配置反向代理某个url

本文讲的这个漏洞 主要是为了解决漏洞扫描的问题我先介绍下这个漏洞主要是因为访问https://172.16.120.17:18090/ws/v1/cluster/info这个 yarn rest的一个...

正式发布 | 《云运维服务白皮书》开放下载!

正式发布 | 《云运维服务白皮书》开放下载!

在全球数字化变革的背景下,为适应数字经济环境下企业生存发展和市场变化的需要,企业进行主动的、系统性、整体性的数字化转型升级。大数据、云计算、人工智能、区块链等新一代信息通信技术为企业的数字化转型提供了...

发表评论    

◎欢迎参与讨论,请在这里发表您的看法、交流您的观点。