Flink sql 集成hive metastore对接s3存储

櫰木2年前技术文章1465

1、hive metastore使用s3如何适配

1、在flink conf目录中添加core-site.xml和hive-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>fs.s3a.connection.ssl.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>s3a://spark1125</value>
    </property>
    <property>
        <name>fs.s3a.path.style.access</name>
        <value>true</value>
    </property>
    <property>
        <name>fs.s3a.access.key</name>
        <value>admin</value>
    </property>
    <property>
        <name>fs.s3a.secret.key</name>
        <value>Admin@123</value>
    </property>
    <property>
        <name>fs.s3a.impl</name>
        <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
    </property>
    <property>
        <name>fs.s3a.endpoint</name>
        <value>http://172.16.104.165:30153</value>
    </property>
    <property>
        <name>fs.s3a.fast.upload</name>
        <value>true</value>
    </property>
    <property>
        <name>fs.s3a.multipart.threshold</name>
        <value>512M</value>
    </property>
    <property>
        <name>fs.s3a.socket.recv.buffer</name>
        <value>65536</value>
    </property>
    <property>
        <name>fs.s3a.socket.send.buffer</name>
        <value>65536</value>
    </property>
    <property>
        <name>fs.s3a.fast.upload.active.blocks</name>
        <value>2048</value>
    </property>
    <property>
        <name>fs.s3a.fast.upload.buffer</name>
        <value>disk</value>
    </property>
    <property>
        <name>fs.s3a.max.total.tasks</name>
        <value>2048</value>
    </property>
    <property>
        <name>fs.s3a.threads.max</name>
        <value>2048</value>
    </property>
    <property>
        <name>fs.s3a.committer.staging.conflict-mode</name>
        <value>append</value>
    </property>
    <property>
        <name>fs.s3a.connection.establish.timeout</name>
        <value>5000</value>
    </property>
    <property>
        <name>fs.s3a.multipart.size</name>
        <value>512M</value>
    </property>
    <property>
        <name>fs.s3a.committer.magic.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>fs.s3a.committer.abort.pending.uploads</name>
        <value>true</value>
    </property>
    <property>
        <name>fs.s3a.committer.threads</name>
        <value>8</value>
    </property>
    <property>
        <name>fs.s3a.committer.generate.uuid</name>
        <value>false</value>
    </property>
    <property>
        <name>fs.s3a.committer.require.uuid</name>
        <value>false</value>
    </property>
    <property>
        <name>fs.s3a.committer.name</name>
        <value>magic</value>
    </property>
</configuration>
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>hive.metastore.event.db.notification.api.auth</name>
        <value>false</value>
    </property>
    <property>
        <name>hive.metastore.uris</name>
        <value>thrift://172.16.104.165:31228</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://172.16.104.165:30066/spark1125?useSSL=false</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.cj.jdbc.Driver</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>root</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>123456</value>
    </property>
    <property>
        <name>hive.metastore.warehouse.dir</name>
        <value>s3a://spark1125/hive/warehouse</value>
    </property>
</configuration>


相关文章

PromQL语法

PromQL语法

一、PromQL语法1.1、什么是PromQLPromQL(Prometheus Query Language)是 Prometheus 自己开发的表达式语言,语言表现力很丰富,内置函数也很多。使用它...

数仓主流架构简介之二

数仓主流架构简介之二

一、流批一体数据批流一体是一种云计算架构模式,它结合了批处理和流处理的特点,以实现更高效、灵活和可扩展的数据处理能力。在这种模式下,数据可以同时进行批处理和流处理,以满足不同场景下的需求流批一体:是指...

K8s Pod无法访问SLB地址

K8s Pod无法访问SLB地址

一、问题现象客户反馈172.16.11.220 telnet 47.99.87.129 80 不通,另一台172.16.11.219可以通。二、排查思路172.16.11.220,172.16.11....

ES运维(七)添加sql插件

ES运维(七)添加sql插件

一、概述ElasticSearch安装SQL插件下载地址(中国大佬开发)二、集成sql插件1、下载es-sql插件下载地址: https://github.com/NLPchina/elastics...

Elasticsearch数据生命周期如何规划

Elasticsearch中的open状态的索引都会占用堆内存来存储倒排索引,过多的索引会导致集群整体内存使用率多大,甚至引起内存溢出。所以需要根据自身业务管理历史数据的生命周期,如近3个月的数据op...

sparksql集成ranger权限测试

sparksql集成ranger权限测试

##启动thriftserver cd /opt/dtstack/spark/spark_pkg/sbin ./start-thriftserver.sh \ --master yarn \...

发表评论    

◎欢迎参与讨论,请在这里发表您的看法、交流您的观点。