Doris审计插件的安装

恩慈2年前技术文章1034

插件下载地址:https://doris.apache.org/download

1.安装Doris审计插件

(1)解压 Audit Loader 插件

步骤1. 复制插件文件

cp  /opt/dtstack/Doris/extensions/audit_loader/auditloader.zip

/opt/dtstack/Doris/fe/plugins/

 

步骤2. 进入目录并解压插件包

cd /opt/dtstack/Doris/fe/plugins/

unzip auditloader.zip

 

(2)修改配置

vi plugin.conf

 

配置说明

frontend_host_portFE 节点 IP 地址和 HTTP 端口,格式为 <fe_ip>:<fe_http_port>。 默认值为 127.0.0.1:8030

database:审计日志库名。

audit_log_table:审计日志表名。

slow_log_table:慢查询日志表名。

enable_slow_log:是否开启慢查询日志导入功能。默认值为 false

user:集群用户名。该用户必须具有对应表的 INSERT 权限。

password:集群用户密码。

image.png 

 

(3)重新打包 Audit Loader 插件

zip -r -q -m auditloader.zip auditloader.jar plugin.properties plugin.conf

(4)创建库表

若需开启慢查询日志导入功能,还需要额外创建慢表 doris_slow_log_tbl__,其表结构与 doris_audit_log_tbl__ 一致。其中 dynamic_partition 属性根据自己的需要,选择审计日志保留的天数。

create database doris_audit_db__;

 

create table doris_audit_db__.doris_audit_log_tbl__

(

    query_id varchar(48) comment "Unique query id",

    `time` datetime not null comment "Query start time",

    client_ip varchar(32) comment "Client IP",

    user varchar(64) comment "User name",

    db varchar(96) comment "Database of this query",

    state varchar(8) comment "Query result state. EOF, ERR, OK",

    error_code int comment "Error code of failing query.",

    error_message string comment "Error message of failing query.",

    query_time bigint comment "Query execution time in millisecond",

    scan_bytes bigint comment "Total scan bytes of this query",

    scan_rows bigint comment "Total scan rows of this query",

    return_rows bigint comment "Returned rows of this query",

    stmt_id int comment "An incremental id of statement",

    is_query tinyint comment "Is this statemt a query. 1 or 0",

    frontend_ip varchar(32) comment "Frontend ip of executing this statement",

    cpu_time_ms bigint comment "Total scan cpu time in millisecond of this query",

    sql_hash varchar(48) comment "Hash value for this query",

    sql_digest varchar(48) comment "Sql digest for this query",

    peak_memory_bytes bigint comment "Peak memory bytes used on all backends of this query",

    stmt string comment "The original statement, trimed if longer than 2G"

) engine=OLAP

duplicate key(query_id, `time`, client_ip)

partition by range(`time`) ()

distributed by hash(query_id) buckets 1

properties(

    "dynamic_partition.time_unit" = "DAY",

    "dynamic_partition.start" = "-30",

    "dynamic_partition.end" = "3",

    "dynamic_partition.prefix" = "p",

    "dynamic_partition.buckets" = "1",

    "dynamic_partition.enable" = "true",

    "replication_num" = "3"

);

 

create table doris_audit_db__.doris_slow_log_tbl__

(

    query_id varchar(48) comment "Unique query id",

    `time` datetime not null comment "Query start time",

    client_ip varchar(32) comment "Client IP",

    user varchar(64) comment "User name",

    db varchar(96) comment "Database of this query",

    state varchar(8) comment "Query result state. EOF, ERR, OK",

    error_code int comment "Error code of failing query.",

    error_message string comment "Error message of failing query.",

    query_time bigint comment "Query execution time in millisecond",

    scan_bytes bigint comment "Total scan bytes of this query",

    scan_rows bigint comment "Total scan rows of this query",

    return_rows bigint comment "Returned rows of this query",

    stmt_id int comment "An incremental id of statement",

    is_query tinyint comment "Is this statemt a query. 1 or 0",

    frontend_ip varchar(32) comment "Frontend ip of executing this statement",

    cpu_time_ms bigint comment "Total scan cpu time in millisecond of this query",

    sql_hash varchar(48) comment "Hash value for this query",

    sql_digest varchar(48) comment "Sql digest for this query",

    peak_memory_bytes bigint comment "Peak memory bytes used on all backends of this query",

    stmt string comment "The original statement, trimed if longer than 2G "

) engine=OLAP

duplicate key(query_id, `time`, client_ip)

partition by range(`time`) ()

distributed by hash(query_id) buckets 1

properties(

    "dynamic_partition.time_unit" = "DAY",

    "dynamic_partition.start" = "-30",

    "dynamic_partition.end" = "3",

    "dynamic_partition.prefix" = "p",

    "dynamic_partition.buckets" = "1",

    "dynamic_partition.enable" = "true",

    "replication_num" = "3"

);

(5)部署

步骤1. 拷贝auditloader.zip

将重新打包的zip拷贝到所有 FE /opt/dtstack/Doris/fe/plugins/目录

 

步骤2. 连接FE

mysql -uroot -P9030 -h127.0.0.1

 

步骤3. 执行安装命令

INSTALL PLUGIN FROM /opt/dtstack/Doris/fe/plugins/auditloader.zip


相关文章

MySQL索引与事务的透析——(超详解)

MySQL索引与事务的透析——(超详解)

索引2.1索引存在意义在数据库使用select查询数据的时候有以下几个步骤:1.先遍历表;2.在把当前的行带入条件看条件是否成立;3.若成立则保留,反之就不保留,跳过;那么在一个表的数据非常大的时候,...

Flink sql 集成hive metastore

Flink sql 集成hive metastore

1、前置条件**确认hive metastore版本(本次版本3.1.2) **hive metastore 部署成功hive依赖jar包分发到每个flink节点:/flink-1.13.6    ...

Kafka副本策略

Kafka副本策略

Kafka的高可靠性的保障来源于其健壮的副本(replication)策略。1. 数据同步kafka在0.8版本前没有提供Partition的Replication机制,一旦Broker宕机,其上的所...

开源大数据集群部署(一)集群实施规划

1、集群规划1.1 本次集群规划信息本次实际生产业务体量存在巨大差异,但集群规划内容相同,因此建议实际生产环境按照按照一定比例扩展即可。主机操作系统要求软硬件信息参数配置8C16G操作系统版本Cent...

基于Gitlab和Kubernetes的CI/CD

基于Gitlab和Kubernetes的CI/CD

此套CI/CD流程仅依赖gitlab。runner等组件安装在kubernetes集群中,尽量减少其他依赖,便于维护。依赖介绍gitlab runnergitlab runner用来运行我们的作业并将...

SQL隐式转换导致索引失效_字符集不一致

3.字符集不一致导致索引失效示例 SQL 如下,通过查看执行计划发现 XXX 和 XXXX 表在进行表关联的时候没有走索引,导致 SQL 扫描数量较大。核实表结构发现表关联对应列都存在索引,最终查看字...

发表评论    

◎欢迎参与讨论,请在这里发表您的看法、交流您的观点。