A CDC pipeline from MySQL into ClickHouse took p90 query latency from fifteen seconds to two. What it cost to run, and the reasons you might not want one.
The Architecture
The read path runs alongside MySQL rather than replacing it. Nothing in the application writes to ClickHouse directly.
MySQL stays the system of record. ClickHouse answers the statistical queries it is actually built for, and CDC keeps the two in step.
The split was necessary because MySQL was not capable of handling the dynamic predicate space. Customers choose their own custom filters and their own date ranges, so there is no fixed set of queries to optimize for. Covering indexes are infeasible when you can't know the predicates in advance, which makes row scans inevitable, and row scans over millions of rows are not something you can tune your way out of in MySQL. We had the wrong storage engine for the question being asked.
The Problem
Our largest customers were the ones staring at a blank page.
They were also the ones paying us the most. The use case was straightforward enough: let customers build visualizations across millions of datapoints using arbitrary filters and date ranges. For a small tenant it worked fine. For the enterprise accounts we had sold it to — the ones importing millions of rows — p90 query latency was around fifteen seconds, and the largest of them would stall for thirty seconds and then time out. This was a churn risk for the highest-paying customers and a blocker for closing larger enterprise deals.
Performance
The pipeline ingests up to 100,000 rows per second, which was sufficient for our use case. We could have pushed it roughly an order of magnitude further by tuning MSK to egress more. Max replication lag was five seconds, comfortably inside our data freshness SLA. Most importantly, application reads against ClickHouse came back with a p90 of about two seconds, against MySQL's fifteen. The pages that had been timing out were now rendering in a few seconds.
The CDC infrastructure — not including ClickHouse — cost around $1K per month. ClickHouse costs fluctuated between $2K and $3K per month based on the CPU demands of queries.
What it changed
The platform was demonstrable at enterprise scale, and several seven-figure contracts followed. The engineering work was a latency improvement; the business result was that we could sell to companies whose data volume had previously been a reason to say no. The data platform also served as a foundation for future features that relied on that statistical data, such as detecting trends in customer feedback.
Support
In the wild, this system was relatively easy to support ops-wise. Most failure scenarios were accounted for, with one exception: Debezium would occasionally drop writes to Kafka for no apparent reason.
Debezium's logs are sparse and gave no indication as to why writes to Kafka would suddenly fail. At the time we couldn't get Kafka logs either. AWS addressed this in February 2026, but it wasn't available to us then. So we were debugging a component that had stopped working, with no signal from either side of the boundary. Luckily, restarting Debezium recovered it. I eventually wired up a Lambda that triggered a restart whenever the alarm tripped.
Handling divergence
The most concerning problem with the analytics plane was the corruption that could ensue from outages, bugs, or transient network failures. To detect divergence, I wrote a suite of tests that reported deltas between data cohorts in both MySQL and ClickHouse. Since the system was eventually consistent, there were always deltas between both data stores. However, those deltas would eventually correct and go back to 0. A CloudWatch alarm with a 5 minute evaluation period and 5 out of 6 data points to alarm on was sufficient for catching real issues.
I was able to automate correcting divergence from failures. The integrity check could detect the offending cohort and then trigger an incremental snapshot of that cohort by writing to the Debezium signal table with the appropriate additional-conditions.
I was not able to automate correcting divergence from corruption. Occasionally, data engineers would issue custom SQL statements to clean and format third-party data. Those SQL statements triggered an update to a single record in MySQL. If the update touched a column in the ClickHouse primary key, two versions of the record would exist. This was difficult to solve programmatically. It was also a rare occurrence. So I accepted it as operational burden.
Reasons not to build this
If your platform has known query patterns that are indexable, this solution is not for you. It would have been far easier — and cheaper — to define covering indexes or create materialized views for data visualizations ahead of time.
If you do find yourself in the same bucket and your system can batch writes to 100k rows per second, then I would strongly consider just using ClickHouse as the primary database. ClickHouse was able to handle all kinds of queries in production, not just the statistical queries typical of an OLAP store.
If you're running something shaped like this, a few questions worth being able to answer:
If your warehouse quietly diverged from your source of truth tonight, how would you find out, and how long would it take?
When a connector stops working and won't say why, what happens next? Does someone get paged, or does the data just get quieter?
Can you repair one divergent slice without re-snapshotting everything?
Do you know what the pipeline costs, and whether it still earns it?
I had to answer all four of these the hard way, and none of them were the part I expected to spend my time on. Most of what's hard about running distributed systems lives in questions like these rather than in the architecture diagram — the caveats nobody puts in the docs.
If you're working through any of them, or you just want to compare notes, I'd enjoy the conversation. Send me an email — I'm happy to talk through what I got right and what I'd do differently.
I'm an independent consultant. If you're weighing a decision like the one above — or you're already living with one and it isn't going well — I'm happy to compare notes.
