Elastic Sink Connector Demo

This project demonstrates Change Data Capture (CDC) architecture using Debezium to synchronize data between MySQL and Elasticsearch in real-time through Kafka. The Spring Boot application sends data to MySQL, Debezium captures changes via CDC and streams them to Kafka, and a sink connector synchronizes the data to Elasticsearch.

🏗️ Architecture Overview

CDC Data Pipeline

Spring Boot App → MySQL Database → Debezium CDC → Kafka Topics → Elasticsearch Sink → Elasticsearch Index

Service Ports

Service	Port	URL	Description
Spring Boot App	8080	http://localhost:8080	Main application API
Swagger UI	8080	http://localhost:8080/swagger-ui.html	API Documentation
MySQL Database	3306	localhost:3306	Primary database
Elasticsearch	9200	http://localhost:9200	Search engine
Kibana	5601	http://localhost:5601	Elasticsearch UI
Kafka Broker	9092	localhost:9092	Message broker
Kafka Connect	8083	http://localhost:8083	Connector management
Kafka UI	8090	http://localhost:8090	Kafka monitoring

Kafka Connect Configuration

The project includes pre-configured connector settings in the kafka-connect/config/ directory:

MySQL Source Connector

• File: mysql-source-connector.json
• Purpose: Captures changes from MySQL database
• Target Topic: elastic-sink.{table_name} (Automatically mapped to each table in the database)
• Configuration: POST to http://localhost:8083/connectors/

Elasticsearch Sink Connector

• File: elasticsearch-sink-connector.json
• Purpose: Streams data from Kafka to Elasticsearch
• Source Topics: elastic-sink.* (regex pattern)
• Configuration: POST to http://localhost:8083/connectors/

Quick Start

1. Start Infrastructure Services

   docker-compose up -d

2. Configure Kafka Connect Connectors

   Post the MySQL source connector configuration:

   curl -X POST http://localhost:8083/connectors/ \
     -H "Content-Type: application/json" \
     -d @kafka-connect/config/mysql-source-connector.json

   Post the Elasticsearch sink connector configuration:

   curl -X POST http://localhost:8083/connectors/ \
     -H "Content-Type: application/json" \
     -d @kafka-connect/config/elasticsearch-sink-connector.json

3. Start Spring Boot Application

   mvn spring-boot:run

4. Verify Setup

   • Verify connectors: GET http://localhost:8083/connectors/
   • Check Kafka topics: Visit http://localhost:8090
   • Check data sync: GET http://localhost:8080/api/products/compare

API Endpoints

📊 CRUD Operations

Method	Endpoint	Description
POST	/api/products	Create new product
GET	/api/products/{id}	Get product by ID
GET	/api/products	Get all products
PUT	/api/products/{id}	Update product
DELETE	/api/products/{id}	Delete product

🔍 Search Operations

Method	Endpoint	Description
GET	/api/products/elastic/search/text?searchText={query}	Text search in Elasticsearch
GET	/api/products/elastic/search/fuzzy?searchText={query}	Fuzzy search
GET	/api/products/elastic/search/name?name={name}	Search by product name
GET	/api/products/elastic/search/price?minPrice={min}&maxPrice={max}	Price range search
GET	/api/products/elastic/category/{category}	Search by category
GET	/api/products/elastic/search/advanced	Advanced search with pagination

🔄 Data Synchronization

Method	Endpoint	Description
GET	/api/products/compare	Compare data count between MySQL and Elasticsearch
POST	/api/products/sync	Manual sync from MySQL to Elasticsearch

📈 Elasticsearch Direct Access

Method	Endpoint	Description
GET	/api/products/elastic/{id}	Get product from Elasticsearch by ID
GET	/api/products/elastic	Get all products from Elasticsearch

Example API Calls

Create New Product

curl -X POST http://localhost:8080/api/products \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Product Name",
    "description": "Product description goes here",
    "price": 100.00,
    "stock": 100,
    "category": "Product Category Enum"
  }'

Response:

{
  "id": 1,
  "name": "Product Name",
  "description": "Product description goes here",
  "price": 100.00,
  "stock": 100,
  "category": "Product Category Enum",
  "isActive": true,
  "createdAt": "2025-01-01T00:00:00",
  "updatedAt": "2025-01-01T00:00:00"
}

Technology Stack

• Java 21
• Spring Boot 3.x
• MySQL 8.4
• Elasticsearch 8.11
• Apache Kafka
• Kafka Connect (Debezium, CDC)