Flink sql 集成hive metastore对接s3存储

櫰木1年前技术文章1050

1、hive metastore使用s3如何适配

1、在flink conf目录中添加core-site.xml和hive-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>fs.s3a.connection.ssl.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>s3a://spark1125</value>
    </property>
    <property>
        <name>fs.s3a.path.style.access</name>
        <value>true</value>
    </property>
    <property>
        <name>fs.s3a.access.key</name>
        <value>admin</value>
    </property>
    <property>
        <name>fs.s3a.secret.key</name>
        <value>Admin@123</value>
    </property>
    <property>
        <name>fs.s3a.impl</name>
        <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
    </property>
    <property>
        <name>fs.s3a.endpoint</name>
        <value>http://172.16.104.165:30153</value>
    </property>
    <property>
        <name>fs.s3a.fast.upload</name>
        <value>true</value>
    </property>
    <property>
        <name>fs.s3a.multipart.threshold</name>
        <value>512M</value>
    </property>
    <property>
        <name>fs.s3a.socket.recv.buffer</name>
        <value>65536</value>
    </property>
    <property>
        <name>fs.s3a.socket.send.buffer</name>
        <value>65536</value>
    </property>
    <property>
        <name>fs.s3a.fast.upload.active.blocks</name>
        <value>2048</value>
    </property>
    <property>
        <name>fs.s3a.fast.upload.buffer</name>
        <value>disk</value>
    </property>
    <property>
        <name>fs.s3a.max.total.tasks</name>
        <value>2048</value>
    </property>
    <property>
        <name>fs.s3a.threads.max</name>
        <value>2048</value>
    </property>
    <property>
        <name>fs.s3a.committer.staging.conflict-mode</name>
        <value>append</value>
    </property>
    <property>
        <name>fs.s3a.connection.establish.timeout</name>
        <value>5000</value>
    </property>
    <property>
        <name>fs.s3a.multipart.size</name>
        <value>512M</value>
    </property>
    <property>
        <name>fs.s3a.committer.magic.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>fs.s3a.committer.abort.pending.uploads</name>
        <value>true</value>
    </property>
    <property>
        <name>fs.s3a.committer.threads</name>
        <value>8</value>
    </property>
    <property>
        <name>fs.s3a.committer.generate.uuid</name>
        <value>false</value>
    </property>
    <property>
        <name>fs.s3a.committer.require.uuid</name>
        <value>false</value>
    </property>
    <property>
        <name>fs.s3a.committer.name</name>
        <value>magic</value>
    </property>
</configuration>
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>hive.metastore.event.db.notification.api.auth</name>
        <value>false</value>
    </property>
    <property>
        <name>hive.metastore.uris</name>
        <value>thrift://172.16.104.165:31228</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://172.16.104.165:30066/spark1125?useSSL=false</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.cj.jdbc.Driver</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>root</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>123456</value>
    </property>
    <property>
        <name>hive.metastore.warehouse.dir</name>
        <value>s3a://spark1125/hive/warehouse</value>
    </property>
</configuration>


相关文章

系统RDSCPU打满问题分析报告

系统RDSCPU打满问题分析报告

1. 问题概述在2023年9月01日09点13分,玳数运维组侧接收到业务侧反馈系统响应缓慢,与此同时运维群内新系统RDS 发出CPU打满的告警,告警通知如下: 2. 问题分析a. 数据库会话管理核查玳...

开源大数据集群部署(二十一)Spark on yarn 部署

开源大数据集群部署(二十一)Spark on yarn 部署

3.0.1 spark on yarn安装(每个节点) cd /root/bigdata/ tar -xzvf spark-3.3.1-bin-hadoop3.tgz -C /opt/ ln -s...

linux下xfs文件系统类型/目录扩容

1、查看分区信息[root@172-16-121-112 ~]# fdisk -lDisk /dev/vda: 107.4 GB, 107374182400 bytes, 209715200 sect...

CDH-Kafka节点迁移

CDH-Kafka节点迁移

1、节点迁移先添加节点,将需要换掉的节点关闭,等待选举leader启动新添加的节点关闭旧的节点等待选举leader出现离线分区。需要将分区分配到新的节点2、重分配分区kafak kerberos环境下...

MySQL运维实战之Clone插件(10.2)Clone插件原理

MySQL运维实战之Clone插件(10.2)Clone插件原理

clone插件实现clone操作主要分为几个阶段:1、初始阶段。初始阶段,会开启页面跟踪(Page Tracking)。开启页面跟踪后,修改过的页面的编号会被记录下来。页面的修改可分为两个阶段:首先在...

docker日志管理

docker日志管理

docker的日志分类 一、Docker 引擎日志(也就是 dockerd 运行时的日志)Ubuntu14.04: /var/log/upstart/docker.logCentos 6/7或ubun...

发表评论    

◎欢迎参与讨论,请在这里发表您的看法、交流您的观点。