Building a Telecom Data Lake: Lessons from 20 Years
Lessons from twenty years of building telecom data platforms: start from the business questions, check the plumbing before the numbers, maintain your tables and certify KPIs before anyone uses them.

Telecom analytics started with revenue assurance: take the call records from the switch, take what billing charged, and explain every difference. The tools have changed many times since then. The problem hasn't. Operators still have billions of events spread across network elements, mediation, billing, CRM, interconnect and mobile money. Someone still has to bring them together, keep them correct and answer questions from finance, fraud and network teams while the answers are still useful.
These are the lessons that have held up across every change of technology.
Start from the questions
A data lake that nobody queries is an expensive archive. The most common failure we see is a project that starts with "let's put everything in one place" and spends a year on ingestion before anyone asks what the data is for.
Begin by listing the questions the business asks every week. Which revenue streams leak between the switch and the bill? Which subscribers behave like SIM boxes? Where does interconnect traffic disagree with the partner's statement? Each question tells you which feeds you need, how fresh they must be and how long to keep them.
Keep raw data replayable
Sooner or later someone asks to run a new rule over the last six months. A new fraud pattern appears, a tariff dispute arrives, or the regulator asks how a figure was calculated. If you only kept aggregates, the answer is no.
Keep raw CDRs, xDRs and billing records in open table formats such as Apache Iceberg, and treat everything downstream as something you can rebuild. Table snapshots also let you see the data as it was on the day a report was produced, which helps when an audit or a partner dispute comes in.
Check the plumbing before the numbers
Much of what first looks like revenue leakage or fraud turns out to be missing or duplicated data: a file that never arrived, a job that failed over a weekend, a feed that resent its history.
Most network elements number their output files, so track those sequences and report the gaps. Count records at every stage, from file to table. Log every failed, partial or duplicate file with when it happened and how it was resolved. These controls are cheap to build, and without them every result further down is open to argument.
Respect how telecom data behaves
Generic platforms treat a CDR like any other row. A long call can arrive as several partial records, the same event can appear in two files, and subscribers are identified by MSISDN on one feed and IMSI on another. Feeds also arrive at very different speeds: some within minutes, roaming records often hours later.
Use the event time in the record, not the time it arrived, and handle these quirks once in the platform rather than in every report that touches the data.
Maintain your tables from day one
Open table formats are not maintenance-free. Without regular compaction and snapshot expiry, high-volume tables collect small files and history, and loading slows down month by month until it falls behind. Schedule maintenance for every table before the first load, run it after each day's load rather than before, and don't wait for the first slowdown.
Certify KPIs before people use them
Finance does not want schema-on-read. They want one agreed number for last month's revenue, the same number every time they ask.
Keep raw events in the lake and reconciled facts in a curated layer built from it, on the same platform with the same security and catalogue. Run data quality checks on every table, and trace each KPI back through its dataset to the tables behind it. A figure should only be marked complete for a day when all of its data is.
Stream only what needs to be fast
SIM-box routes and fraudulent top-ups need to be caught within minutes, so they belong on streaming and event processing. Monthly revenue assurance, churn models and capacity planning don't. Batch is simpler to run and cheaper to fix, so decide per use case.
Keep it inside the network
Subscriber records, call detail and location data are among the most sensitive data a country holds, and many regulators require them to stay in-country. The platform, including its AI, should be able to run fully on-premise.
Build it to be handed over
The project team leaves. The operator's own staff run the platform for the next five or ten years. Give them a searchable data catalogue with lineage, a business glossary, configuration they can change without code, and open formats they are never locked into. Size storage with room to spare as well, because data always stays a little longer than the retention policy says.
We will cover several of these topics in more depth in future posts. Meanwhile, our telecom data platform page shows how Prompto® puts them into practice, or get in touch to see it on your own data.