ranger对接metastore

九月4个月前 (12-24)技术文章146

前提：本文前提是基于集群中已经安装部署了ranger组件、hive组件的情况下，增加ranger metastore插件的对接。

安装部署

1、ranger metastore插件编译

插件下载

https://git.lrting.top/xiaozhch5/ranger-metastore-plugin.git

下载后进行安装部署

tar -zxvf ranger-metastore-plugin-master.tar.gz

修改pom文件为指定版本,本文部署的版本是2.4

agents-audit/pom.xml
<parent>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-metastore-plugin</artifactId>
<version>2.4.0-SNAPSHOT</version>
<version>2.4.0</version>
<relativePath>..</relativePath>
</parent>
<dependencies>
 
agents-common/pom.xml
<parent>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-metastore-plugin</artifactId>
<version>2.4.0-SNAPSHOT</version>
<version>2.4.0</version>
<relativePath>..</relativePath>
</parent>
<dependencies>

agents-cred/pom.xml
    <parent>
        <groupId>org.apache.ranger</groupId>
        <artifactId>ranger-metastore-plugin</artifactId>
        <version>2.4.0</version>
        <relativePath>..</relativePath>
    </parent>

agents-installer/pom.xml
<parent>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-metastore-plugin</artifactId>
<version>2.4.0-SNAPSHOT</version>
<version>2.4.0</version>
<relativePath>..</relativePath>
</parent>
<dependencies>

credentialbuilder/pom.xml
    <parent>
        <groupId>org.apache.ranger</groupId>
        <artifactId>ranger-metastore-plugin</artifactId>
        <version>2.4.0</version>
        <relativePath>..</relativePath>
    </parent>

metastore-agent/pom.xml
   <parent>
        <groupId>org.apache.ranger</groupId>
        <artifactId>ranger-metastore-plugin</artifactId>
        <version>2.4.0</version>
        <relativePath>..</relativePath>
    </parent>

ranger-plugin-classloader/pom.xml
  <parent>
        <groupId>org.apache.ranger</groupId>
        <artifactId>ranger-metastore-plugin</artifactId>
        <version>2.4.0</version>
    </parent>

ranger-util/pom.xml
    <parent>
        <groupId>org.apache.ranger</groupId>
        <artifactId>ranger-metastore-plugin</artifactId>
        <version>2.4.0</version>
    </parent>

pom.xml
    <modelVersion>4.0.0</modelVersion>
    <artifactId>ranger-metastore-plugin</artifactId>
    <groupId>org.apache.ranger</groupId>
    <version>2.4.0</version>
    <name>Hive Metastore Security Plugin</name>
    <description>Hive Metastore Security Plugins</description>

进行编译

mvn clean compile package install assembly:assembly -DskipTests

2、插件包解压安装

#进行解压
tar -zxvf ranger-metastore-plugin-2.4.0-metastore-plugin.tar.gz -C ranger-metastore-plugin

3、修改配置进行部署

vim install.properties

#进行配置
POLICY_MGR_URL=http://172.16.121.0:6080
REPOSITORY_NAME=metastore
COMPONENT_INSTALL_DIR_NAME=/opt/hive

#audit相关配置
XAAUDIT.SOLR.ENABLE=true             
XAAUDIT.SOLR.URL=http://172.16.121.0:8983/solr/ranger_audits
XAAUDIT.SOLR.USER=NONE
XAAUDIT.SOLR.PASSWORD=NONE
XAAUDIT.SOLR.ZOOKEEPER=hadoop001:2181,hadoop002:2181,hadoop003:2181/ranger_audits
XAAUDIT.SOLR.FILE_SPOOL_DIR=/var/log/hive/audit/solr/spool

#Solr Audit Provder
XAAUDIT.SOLR.IS_ENABLED=true
XAAUDIT.SOLR.MAX_QUEUE_SIZE=1
XAAUDIT.SOLR.MAX_FLUSH_INTERVAL_MS=1000
XAAUDIT.SOLR.SOLR_URL=http://172.16.121.0:8983/solr/ranger_audits

./enable-metastore-plugin.sh

4、ranger页面设置metastore权限

测试链接连通性

冒烟测试

以test用户为例，目前test用户对hive的test库、hudi_hive库中的表有所有权限，以test测试hudi_hive库。

 /root/jiuyue/image/trino/trino-cli-416-executable.jar --server https://172.16.121.114:31443 --keystore-path /root/jiuyue/helm/conf/trino/ldap_conf/trino.jks --keystore-password admin@123  --catalog hudi --user test --password --debug
trino> use hudi_hive;
USE
trino:hudi_hive> show tables;
      Table       
------------------
 fare_adjustment  
 host_hudi        
 hudi_table       
 hudi_table2      
 hudi_table_keyed 
 student          
 tbl_customer     
(7 rows)

Query 20231124_072453_00016_mizze, FINISHED, 3 nodes
https://172.16.121.114:31443/ui/query.html?20231124_072453_00016_mizze
Splits: 36 total, 36 done (100.00%)
CPU Time: 0.0s total,   304 rows/s, 9.04KB/s, 35% active
Per Node: 0.0 parallelism,     9 rows/s,   276B/s
Parallelism: 0.1
Peak Memory: 2.35KB
0.26 [7 rows, 213B] [27 rows/s, 829B/s]

trino:hudi_hive> select * from hudi_table2 limit 1;
 _hoodie_commit_time | _hoodie_commit_seqno  |          _hoodie_record_key          | _hoodie_partition_path |                            _hoodie_file_>
---------------------+-----------------------+--------------------------------------+------------------------+----------------------------------------->
 20231123161346626   | 20231123161346626_2_0 | 3eeb61f7-c2b0-4636-99bd-5d7a5a1d2c04 | city=chennai           | b0b19f4e-7530-4df6-ab29-638afcea17ef-0_2>
(1 row)

能正常查询。

将test用户权限进行调整。

将库的权限由hudi_hive库改为test库

看权限已经更新，重新登录查询,无法进行查看，没有权限

/root/jiuyue/image/trino/trino-cli-416-executable.jar --server https://172.16.121.114:31443 --keystore-path /root/jiuyue/helm/conf/trino/ldap_conf/trino.jks --keystore-password admin@123  --catalog hudi --user test --password --debug