Flinksql Kafka 接收流数据并打印到控制台

芒果7个月前技术文章620

本文目的
使用Flink SQL创建一个流处理作业,将来自Kafka主题"dahua_picrecord"的数据写入到另一个表”print_table”控制台中。
使用sql-client前 需要启动yarn-session哦
首先需要在CREATE TABLE
CREATE TABLE test_source (
  objId STRING,
  data STRING,
  capTime STRING,
  dataType STRING,
  channelCode STRING
) WITH (
  'connector' = 'kafka',
  'topic' = 'test',
  'properties.bootstrap.servers' = '172.16.121.194:9092',
  'properties.group.id' = 'test-dataq-01',
  'format' = 'json',
  'scan.startup.mode' = 'earliest-offset'
);

CFC9E3F3-A2FA-43C5-A99C-C765F1A0ACAB.png
创建”print_table"
CREATE TABLE print_table (
  objId STRING,
  data STRING,
  capTime STRING,
  dataType STRING,
  channelCode STRING
) WITH (
  'connector' = 'print'
);
ACF330CB-6770-4C05-8D86-23C622FAD014.png
将数据从test_source 插入到 print_table 中
INSERT INTO print_table
SELECT objId, data, capTime, dataType, channelCode
FROM test_source;

接下来我们去查看yarn任务
2735D3C5-74DB-43DB-BB91-82DA077CACEB.png
点进去看看
开始向test写一些json数据
/opt/kafka/bin/kafka-console-producer.sh --bootstrap-server 172.16.121.194:9092 --topic test
{"objId":"12345","data":"example data 1","capTime":"2023-11-07T08:00:00","dataType":"exampleType","channelCode":"ABCDEF"}
{"objId":"54321","data":"example data 2","capTime":"2023-11-07T08:15:00","dataType":"anotherType","channelCode":"GHIJKL"}
{"objId":"99999","data":"more example data","capTime":"2023-11-07T08:30:00","dataType":"additionalType","channelCode":"ZYXWVU"}
{"objId":"11111","data":"extra data","capTime":"2023-11-07T08:45:00","dataType":"extraType","channelCode":"QRSTUV"}
{"objId":"77777","data":"additional example data","capTime":"2023-11-07T09:00:00","dataType":"moreType","channelCode":"MNBVCX"}
{"objId":"88888","data":"more and more data","capTime":"2023-11-07T09:15:00","dataType":"typeX","channelCode":"POIUYT"}
{"objId":"22222","data":"different data","capTime":"2023-11-07T09:30:00","dataType":"typeY","channelCode":"LAKSDJ"}
{"objId":"66666","data":"sample data","capTime":"2023-11-07T09:45:00","dataType":"testType","channelCode":"QWERTY"}
{"objId":"44444","data":"new data","capTime":"2023-11-07T10:00:00","dataType":"newType","channelCode":"ZXCVBN"}
{"objId":"55555","data":"fresh data","capTime":"2023-11-07T10:15:00","dataType":"freshType","channelCode":"EDCRFV"}
7FBE0A68-D3AD-433C-8672-FB49C7C81FAA.png
查看flinkweb看数据过来了
2B6440A3-1BD4-4721-B9C4-3C4233BC02FF.png
输出到了控制台
AB8DC330-4763-4CE7-84D3-F09B86863507.png
完成


标签: 大数据运维

相关文章

CDP实操--HDFS角色迁移

CDP实操--HDFS角色迁移

    hdfs角色迁移功能在cdp页面中就可以实现该功能,迁移的时间与namenode元数据大小,以及block数量多少有关,注意迁移过程中集群需要关闭,要预留出操作时间窗口。1、页面选择迁移角色2...

ElasticSearch开启xpack

ElasticSearch开启xpack

ES开启xpack1、生成ca证书(用户名和密码不用设置,一路回车,生成证书文件elastic-stack-ca.p12,生成kibana证书的时候也需要该ca证书)/opt/dtstack/es-6...

CDH实操--kudumaster迁移

CDH实操--kudumaster迁移

1 概述本次kudumaster迁移,中间不需要停kudu集群(会涉及滚动重启kudu角色); 注:若因为任务持续运行导致kudu停止超时可手动一台台停止-启动2 master迁移将cdh2中的ma...

Hive 重新编译-解决Tez JobName的问题

Hive 重新编译-解决Tez JobName的问题

本文采用linux编译首先下载源码https://dlcdn.apache.org/hive/hive-3.1.2/apache-hive-3.1.2-src.tar.gz源码位置ql/src/jav...

Trino配置yanagishima-23.0(包含编译)

Trino配置yanagishima-23.0(包含编译)

1 环境介绍1.1 本文采用trino 359yanagishima v23.02 编译yanagishima2.1 安装编译yanagishima需要的工具安装编译yanagishima需要的工具w...

HDP实操--NameNode开启高可用

HDP实操--NameNode开启高可用

为了确定在namenode组件失败后集群中有其他的namenode可以工作,需要对hdp集群配置高可用,当前我们配置的非安全集群的高可用。前置条件:(1)确保你的集群至少有3个节点并且至少有3个Apa...

发表评论    

◎欢迎参与讨论,请在这里发表您的看法、交流您的观点。