Flink sql 集成hive metastore对接s3存储

櫰木2年前技术文章1997

1、hive metastore使用s3如何适配

1、在flink conf目录中添加core-site.xml和hive-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>fs.s3a.connection.ssl.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>s3a://spark1125</value>
    </property>
    <property>
        <name>fs.s3a.path.style.access</name>
        <value>true</value>
    </property>
    <property>
        <name>fs.s3a.access.key</name>
        <value>admin</value>
    </property>
    <property>
        <name>fs.s3a.secret.key</name>
        <value>Admin@123</value>
    </property>
    <property>
        <name>fs.s3a.impl</name>
        <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
    </property>
    <property>
        <name>fs.s3a.endpoint</name>
        <value>http://172.16.104.165:30153</value>
    </property>
    <property>
        <name>fs.s3a.fast.upload</name>
        <value>true</value>
    </property>
    <property>
        <name>fs.s3a.multipart.threshold</name>
        <value>512M</value>
    </property>
    <property>
        <name>fs.s3a.socket.recv.buffer</name>
        <value>65536</value>
    </property>
    <property>
        <name>fs.s3a.socket.send.buffer</name>
        <value>65536</value>
    </property>
    <property>
        <name>fs.s3a.fast.upload.active.blocks</name>
        <value>2048</value>
    </property>
    <property>
        <name>fs.s3a.fast.upload.buffer</name>
        <value>disk</value>
    </property>
    <property>
        <name>fs.s3a.max.total.tasks</name>
        <value>2048</value>
    </property>
    <property>
        <name>fs.s3a.threads.max</name>
        <value>2048</value>
    </property>
    <property>
        <name>fs.s3a.committer.staging.conflict-mode</name>
        <value>append</value>
    </property>
    <property>
        <name>fs.s3a.connection.establish.timeout</name>
        <value>5000</value>
    </property>
    <property>
        <name>fs.s3a.multipart.size</name>
        <value>512M</value>
    </property>
    <property>
        <name>fs.s3a.committer.magic.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>fs.s3a.committer.abort.pending.uploads</name>
        <value>true</value>
    </property>
    <property>
        <name>fs.s3a.committer.threads</name>
        <value>8</value>
    </property>
    <property>
        <name>fs.s3a.committer.generate.uuid</name>
        <value>false</value>
    </property>
    <property>
        <name>fs.s3a.committer.require.uuid</name>
        <value>false</value>
    </property>
    <property>
        <name>fs.s3a.committer.name</name>
        <value>magic</value>
    </property>
</configuration>
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>hive.metastore.event.db.notification.api.auth</name>
        <value>false</value>
    </property>
    <property>
        <name>hive.metastore.uris</name>
        <value>thrift://172.16.104.165:31228</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://172.16.104.165:30066/spark1125?useSSL=false</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.cj.jdbc.Driver</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>root</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>123456</value>
    </property>
    <property>
        <name>hive.metastore.warehouse.dir</name>
        <value>s3a://spark1125/hive/warehouse</value>
    </property>
</configuration>


返回列表

上一篇：Flink sql 集成hive metastore-测试

下一篇：Elasticsearch数据生命周期如何规划

相关文章

Redis Sentinel与Cluster安装部署(三)

4.3redis-cluster-resharding重新分配槽1、查看当前集群节点状态信息 # redis-cli --cluster check 172.32.1.59:7000 -a dt20...

HPA控制器

HPA控制器

HPA （动态扩缩容）kubectl 有一个 scale 命令 kubectl scale deployment/nginx --replicas=4 它可以帮助 Pod 进行扩缩容，这个过程完全手动...

Trino开启ldap认证

Trino开启ldap认证

1、背景由于对于安全监管的要求，需要对trino服务开启安全认证体系。2、基于私有证书的httpskeytool -genkeypair -validity 36500 -ext SAN=IP:172...

MySQL运维实战之备份和恢复（8.9）xtrabackup备份指定表

备份部分表如果实例设置了参数innodb_file_per_table，xtrabackup可以备份部分表。通过--tables，--tables-file，--databases，--databas...

shell脚本--sed

sed后面接的动作，务必以 '........'两个单引号包住1.擅长对文件进行操作处理2.对文件信息进行内容的信息修改-i 参数代表直接修改原文件a ：往后新增一行， a 的后面可以接字串，而这些字...

MongoDB的索引(一)

一、TTL索引1、语法db.eventlog.createIndex( { "lastModifiedDate": 1 }, { expireAfterSeconds: 3600 } )2、TTL索引...

发表评论    

◎欢迎参与讨论，请在这里发表您的看法、交流您的观点。

©Copyrights 2016-2022 YUNCHE 浙ICP备2021017017号