EMR-flinksql运行失败问题

芒果11个月前技术文章669

运行flinksql
sql-client.sh
报错：
[root@emr1 bin]# ./sql-client.sh
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/dtstack/Flink/flink_pkg/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/dtstack/Tez/tez_pkg/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
10:55:25.408 [main] ERROR org.apache.flink.table.factories.FactoryUtil - Could not load service provider for factories.
java.util.ServiceConfigurationError: org.apache.flink.table.factories.Factory: Provider org.apache.flink.table.formats.raw.RawFormatFactory could not be instantiated
    at java.util.ServiceLoader.fail(ServiceLoader.java:232) ~[?:1.8.0_144]
    at java.util.ServiceLoader.access$100(ServiceLoader.java:185) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) ~[?:1.8.0_144]
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[?:1.8.0_144]
    at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_144]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:623) [flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:378) [flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.createTableEnvironment(ExecutionContext.java:114) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.<init>(ExecutionContext.java:66) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.SessionContext.create(SessionContext.java:246) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.local.LocalContextUtils.buildSessionContext(LocalContextUtils.java:87) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:87) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.SqlClient.start(SqlClient.java:88) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:187) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava18/com/google/common/collect/Sets
    at org.apache.flink.table.formats.raw.RawFormatFactory.<clinit>(RawFormatFactory.java:144) ~[flink-table-blink_2.11-1.12.5.jar:1.12.5]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_144]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_144]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_144]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
    at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ~[?:1.8.0_144]
    ... 14 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.shaded.guava18.com.google.common.collect.Sets
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_144]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_144]
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) ~[?:1.8.0_144]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_144]
    at org.apache.flink.table.formats.raw.RawFormatFactory.<clinit>(RawFormatFactory.java:144) ~[flink-table-blink_2.11-1.12.5.jar:1.12.5]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_144]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_144]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_144]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
    at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ~[?:1.8.0_144]
    ... 14 more
10:55:25.420 [main] ERROR org.apache.flink.table.client.SqlClient - SQL Client must stop. Unexpected exception. This is a bug. Please consider filing an issue.
org.apache.flink.table.api.TableException: Could not instantiate the executor. Make sure a planner module is on the classpath
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:165) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.createTableEnvironment(ExecutionContext.java:114) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.<init>(ExecutionContext.java:66) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.SessionContext.create(SessionContext.java:246) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.local.LocalContextUtils.buildSessionContext(LocalContextUtils.java:87) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:87) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.SqlClient.start(SqlClient.java:88) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:187) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
Caused by: org.apache.flink.table.api.TableException: Could not load service provider for factories.
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:627) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:378) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    ... 8 more
Caused by: java.util.ServiceConfigurationError: org.apache.flink.table.factories.Factory: Provider org.apache.flink.table.formats.raw.RawFormatFactory could not be instantiated
    at java.util.ServiceLoader.fail(ServiceLoader.java:232) ~[?:1.8.0_144]
    at java.util.ServiceLoader.access$100(ServiceLoader.java:185) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) ~[?:1.8.0_144]
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[?:1.8.0_144]
    at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_144]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:623) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:378) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    ... 8 more
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava18/com/google/common/collect/Sets
    at org.apache.flink.table.formats.raw.RawFormatFactory.<clinit>(RawFormatFactory.java:144) ~[flink-table-blink_2.11-1.12.5.jar:1.12.5]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_144]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_144]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_144]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
    at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) ~[?:1.8.0_144]
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[?:1.8.0_144]
    at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_144]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:623) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:378) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    ... 8 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.shaded.guava18.com.google.common.collect.Sets
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_144]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_144]
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) ~[?:1.8.0_144]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_144]
    at org.apache.flink.table.formats.raw.RawFormatFactory.<clinit>(RawFormatFactory.java:144) ~[flink-table-blink_2.11-1.12.5.jar:1.12.5]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_144]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_144]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_144]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
    at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) ~[?:1.8.0_144]
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[?:1.8.0_144]
    at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_144]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:623) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:378) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    ... 8 more
Exception in thread "main" org.apache.flink.table.client.SqlClientException: Unexpected exception. This is a bug. Please consider filing an issue.
    at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:201)
    at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161)
Caused by: org.apache.flink.table.api.TableException: Could not instantiate the executor. Make sure a planner module is on the classpath
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:165)
    at org.apache.flink.table.client.gateway.context.ExecutionContext.createTableEnvironment(ExecutionContext.java:114)
    at org.apache.flink.table.client.gateway.context.ExecutionContext.<init>(ExecutionContext.java:66)
    at org.apache.flink.table.client.gateway.context.SessionContext.create(SessionContext.java:246)
    at org.apache.flink.table.client.gateway.local.LocalContextUtils.buildSessionContext(LocalContextUtils.java:87)
    at org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:87)
    at org.apache.flink.table.client.SqlClient.start(SqlClient.java:88)
    at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:187)
    ... 1 more
Caused by: org.apache.flink.table.api.TableException: Could not load service provider for factories.
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:627)
    at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:378)
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156)
    ... 8 more
Caused by: java.util.ServiceConfigurationError: org.apache.flink.table.factories.Factory: Provider org.apache.flink.table.formats.raw.RawFormatFactory could not be instantiated
    at java.util.ServiceLoader.fail(ServiceLoader.java:232)
    at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
    at java.util.Iterator.forEachRemaining(Iterator.java:116)
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:623)
    ... 10 more
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava18/com/google/common/collect/Sets
    at org.apache.flink.table.formats.raw.RawFormatFactory.<clinit>(RawFormatFactory.java:144)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.lang.Class.newInstance(Class.java:442)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
    ... 14 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.shaded.guava18.com.google.common.collect.Sets
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 21 more
检查发现有个lib包的版本后缀存在问题

F382BD31-0482-46D8-9ED0-25D7BD554D39.png
删除后重新运行解决
6EDA7728-7489-442D-8149-A1692489AD10.png
放入kafka的时候需要驱动 不然会报错
094EFAC2-A0D7-44C8-8FC0-55CBF68D4C45.png
maven地址
https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-kafka_2.12
当前环境的flink是1.14.5的所以我们选择
https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-kafka_2.12/1.14.5
放到flink/lib下
2D0D1835-5F71-4BC0-92C7-0C34CDE82874.png


标签: 大数据运维运维
返回列表

上一篇：单节点Kafka部署并开启Kerberos配置

下一篇：ORC、Parquet 等列式存储的优点

相关文章

使用Sqoop将数据从Hive导入MySQL（一）

使用Sqoop将数据从Hive导入MySQL（一）

使用Sqoop将数据从Hive导入MySQL首先查看csv数据类型创建类似的hive表并导入数据CREATE TABLE data (    province STRING,    code INT,...

中国式IT运维，趟出自己的“长期主义”河流

2018年，Gartner曾做过一次长达6个月的调查问卷。在这个主题为AIOps的问卷中，Gartner选取了500家各行业头部的企业，针对其当下的IT架构来判断调研企业未来是否会加大对于AIOps的...

CDH时钟同步问题

CDH时钟同步问题

首先执行如下命令：查看chronyd是否启动systemctl status chronyd如果没启动执行如下命令启动systemctl restart chronyd每台检查时间状态chronyc...

Gartner权威报告解读｜应用可观测性列为2023年重要战略技术趋势！

Gartner于今日发布企业机构在2023年需要探索的十大战略技术趋势。Gartner杰出研究副总裁Frances Karamouzis表示：“为了在经济动荡时期增加企业机构的盈利，首席信息官和IT高...

在经济低迷时管理云服务的策略！

近几年全球经济在疫情等各方面影响之下持续低迷，Wanclouds公司发布的一份研究报告指出，81%的美国IT领导者表示，他们的首席执行官要求他们减少或者不增加云计算支出。事实上，在那些被要求削减成本的...

kubernetes HPA

kubernetes HPA

Horizontal Pod Autoscaling 可以根据 CPU 利用率自动伸缩一个 ReplicaSet、Deployment 或者中的 Pod 数量cat hpa-deploy.yaml a...

发表评论    

◎欢迎参与讨论，请在这里发表您的看法、交流您的观点。

©Copyrights 2016-2022 YUNCHE 浙ICP备2021017017号