EMR-flinksql运行失败问题

芒果6个月前技术文章224

运行flinksql
sql-client.sh
报错:
[root@emr1 bin]# ./sql-client.sh
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/dtstack/Flink/flink_pkg/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/dtstack/Tez/tez_pkg/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
10:55:25.408 [main] ERROR org.apache.flink.table.factories.FactoryUtil - Could not load service provider for factories.
java.util.ServiceConfigurationError: org.apache.flink.table.factories.Factory: Provider org.apache.flink.table.formats.raw.RawFormatFactory could not be instantiated
    at java.util.ServiceLoader.fail(ServiceLoader.java:232) ~[?:1.8.0_144]
    at java.util.ServiceLoader.access$100(ServiceLoader.java:185) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) ~[?:1.8.0_144]
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[?:1.8.0_144]
    at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_144]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:623) [flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:378) [flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.createTableEnvironment(ExecutionContext.java:114) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.<init>(ExecutionContext.java:66) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.SessionContext.create(SessionContext.java:246) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.local.LocalContextUtils.buildSessionContext(LocalContextUtils.java:87) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:87) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.SqlClient.start(SqlClient.java:88) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:187) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava18/com/google/common/collect/Sets
    at org.apache.flink.table.formats.raw.RawFormatFactory.<clinit>(RawFormatFactory.java:144) ~[flink-table-blink_2.11-1.12.5.jar:1.12.5]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_144]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_144]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_144]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
    at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ~[?:1.8.0_144]
    ... 14 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.shaded.guava18.com.google.common.collect.Sets
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_144]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_144]
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) ~[?:1.8.0_144]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_144]
    at org.apache.flink.table.formats.raw.RawFormatFactory.<clinit>(RawFormatFactory.java:144) ~[flink-table-blink_2.11-1.12.5.jar:1.12.5]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_144]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_144]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_144]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
    at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ~[?:1.8.0_144]
    ... 14 more
10:55:25.420 [main] ERROR org.apache.flink.table.client.SqlClient - SQL Client must stop. Unexpected exception. This is a bug. Please consider filing an issue.
org.apache.flink.table.api.TableException: Could not instantiate the executor. Make sure a planner module is on the classpath
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:165) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.createTableEnvironment(ExecutionContext.java:114) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.<init>(ExecutionContext.java:66) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.SessionContext.create(SessionContext.java:246) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.local.LocalContextUtils.buildSessionContext(LocalContextUtils.java:87) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:87) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.SqlClient.start(SqlClient.java:88) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:187) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161) [flink-sql-client_2.11-1.14.5.jar:1.14.5]
Caused by: org.apache.flink.table.api.TableException: Could not load service provider for factories.
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:627) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:378) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    ... 8 more
Caused by: java.util.ServiceConfigurationError: org.apache.flink.table.factories.Factory: Provider org.apache.flink.table.formats.raw.RawFormatFactory could not be instantiated
    at java.util.ServiceLoader.fail(ServiceLoader.java:232) ~[?:1.8.0_144]
    at java.util.ServiceLoader.access$100(ServiceLoader.java:185) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) ~[?:1.8.0_144]
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[?:1.8.0_144]
    at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_144]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:623) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:378) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    ... 8 more
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava18/com/google/common/collect/Sets
    at org.apache.flink.table.formats.raw.RawFormatFactory.<clinit>(RawFormatFactory.java:144) ~[flink-table-blink_2.11-1.12.5.jar:1.12.5]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_144]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_144]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_144]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
    at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) ~[?:1.8.0_144]
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[?:1.8.0_144]
    at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_144]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:623) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:378) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    ... 8 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.shaded.guava18.com.google.common.collect.Sets
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_144]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_144]
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) ~[?:1.8.0_144]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_144]
    at org.apache.flink.table.formats.raw.RawFormatFactory.<clinit>(RawFormatFactory.java:144) ~[flink-table-blink_2.11-1.12.5.jar:1.12.5]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_144]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_144]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_144]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
    at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ~[?:1.8.0_144]
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) ~[?:1.8.0_144]
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480) ~[?:1.8.0_144]
    at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_144]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:623) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:378) ~[flink-table_2.11-1.14.5.jar:1.14.5]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156) ~[flink-sql-client_2.11-1.14.5.jar:1.14.5]
    ... 8 more
Exception in thread "main" org.apache.flink.table.client.SqlClientException: Unexpected exception. This is a bug. Please consider filing an issue.
    at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:201)
    at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161)
Caused by: org.apache.flink.table.api.TableException: Could not instantiate the executor. Make sure a planner module is on the classpath
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:165)
    at org.apache.flink.table.client.gateway.context.ExecutionContext.createTableEnvironment(ExecutionContext.java:114)
    at org.apache.flink.table.client.gateway.context.ExecutionContext.<init>(ExecutionContext.java:66)
    at org.apache.flink.table.client.gateway.context.SessionContext.create(SessionContext.java:246)
    at org.apache.flink.table.client.gateway.local.LocalContextUtils.buildSessionContext(LocalContextUtils.java:87)
    at org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:87)
    at org.apache.flink.table.client.SqlClient.start(SqlClient.java:88)
    at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:187)
    ... 1 more
Caused by: org.apache.flink.table.api.TableException: Could not load service provider for factories.
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:627)
    at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:378)
    at org.apache.flink.table.client.gateway.context.ExecutionContext.lookupExecutor(ExecutionContext.java:156)
    ... 8 more
Caused by: java.util.ServiceConfigurationError: org.apache.flink.table.factories.Factory: Provider org.apache.flink.table.formats.raw.RawFormatFactory could not be instantiated
    at java.util.ServiceLoader.fail(ServiceLoader.java:232)
    at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
    at java.util.Iterator.forEachRemaining(Iterator.java:116)
    at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:623)
    ... 10 more
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava18/com/google/common/collect/Sets
    at org.apache.flink.table.formats.raw.RawFormatFactory.<clinit>(RawFormatFactory.java:144)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.lang.Class.newInstance(Class.java:442)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
    ... 14 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.shaded.guava18.com.google.common.collect.Sets
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 21 more
检查发现有个lib包的版本后缀存在问题

F382BD31-0482-46D8-9ED0-25D7BD554D39.png
删除后重新运行解决
6EDA7728-7489-442D-8149-A1692489AD10.png
放入kafka的时候需要驱动 不然会报错
094EFAC2-A0D7-44C8-8FC0-55CBF68D4C45.png
maven地址
https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-kafka_2.12
当前环境的flink是1.14.5的所以我们选择
https://mvnrepository.com/artifact/org.apache.flink/flink-sql-connector-kafka_2.12/1.14.5
放到flink/lib下
2D0D1835-5F71-4BC0-92C7-0C34CDE82874.png


相关文章

Java-API-MapReduce的操作WordCount篇

Java-API-MapReduce的操作WordCount篇

首先就是pom文件<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/...

hive 报 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster

hive 报 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster

解决办法:关键需要配置两个配置:mapred-site.xml 和 yarn-site.xml下面配置hadoop classpath。先运行shell命令:hadoop classpath添加一个配...

nginx配置反向代理某个url

nginx配置反向代理某个url

本文讲的这个漏洞 主要是为了解决漏洞扫描的问题我先介绍下这个漏洞主要是因为访问https://172.16.120.17:18090/ws/v1/cluster/info这个 yarn rest的一个...

磁盘分盘脚本分享

磁盘分区脚本名称:mg_fdisk.sh#!/bin/bashif [ "$#" -ne 1 ]; then  echo "请传入磁盘参数"  exit 1fidisk=$1# 检查磁盘是否存在if...

kubernetes HPA

kubernetes HPA

Horizontal Pod Autoscaling 可以根据 CPU 利用率自动伸缩一个 ReplicaSet、Deployment 或者中的 Pod 数量cat hpa-deploy.yaml a...

中国式IT运维,趟出自己的“长期主义”河流

2018年,Gartner曾做过一次长达6个月的调查问卷。在这个主题为AIOps的问卷中,Gartner选取了500家各行业头部的企业,针对其当下的IT架构来判断调研企业未来是否会加大对于AIOps的...

发表评论    

◎欢迎参与讨论,请在这里发表您的看法、交流您的观点。