Skip to content
/ cdc Public

通过flink cdc实时同步数据,支持mysql到mysql(数据库迁移场景),mysql到iceberg、starrocks(报表场景)

Notifications You must be signed in to change notification settings

luckyQing/cdc

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

27 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

背景:
1、因有上百的数据库实例(上万个库,一百多万个表)需要从阿里云迁移到华为云,故有了mysql-cdc-mysql,从通过flink cdc同步mysql数据库到mysql(支持全量、增量);
2、因部分表数据量比较大,报表数据查询很慢,故有了mysql-cdc-iceberg,通过flink cdc同步mysql数据到iceberg(ods层),再通过flink sql对iceberg表数据进一步清理成为dwd层等;用户界面层通过trino查询iceberg表数据供使用。

一、模块介绍

  • cdc-common
    公共代码封装(常量、工具类等)
  • mysql-cdc-common
    mysql cdc公共代码封装
  • mysql-cdc-ods
    通过flink cdc实时同步mysql数据到ods,sink端支持iceberg、starrocks,支持多实例整库同步
  • mysql-cdc-mysql
    通过flink cdc实时同步mysql到mysql,源数据库实例共用数据库连接(节约资源)

二、使用的组件

名称 版本
flink 1.17.1
flink cdc 2.4.0
iceberg 1.3.0
flink-connector-jdbc 3.1.1-1.17

About

通过flink cdc实时同步数据,支持mysql到mysql(数据库迁移场景),mysql到iceberg、starrocks(报表场景)

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy