DMML -- Formula Sheet -- BITS Subject Vault -- cvam.sight

DMML is mostly architecture, not calculus. These are the operational formulas that make architecture answers precise: quality rates, freshness, lag, throughput, storage cost, availability, retries, sampling and ML-data checks. Use them to support design answers and scenario diagnostics.

Symbols

Symbol	Meaning
\(N\)	total records/events/files in scope
\(N_{bad}\)	records failing quality checks
\(t_{event}\)	time event happened at source
\(t_{ingest}\)	time event entered pipeline/storage
\(t_{serve}\)	time event became available to consumers
\(R\)	record/event rate per second
\(S\)	average record/file size
\(C\)	capacity or cost, depending on formula context

Data quality

Completeness rate

\[ \text{Completeness} = 1 - \frac{N_{missing}}{N} \]

Use for required fields, mandatory source feeds or partition arrival. Low completeness can make an ML model fail even when the pipeline technically ran.

Validity rate

\[ \text{Validity} = \frac{N_{valid}}{N} \]

Records pass schema/type/range/domain checks. Example: date format valid, country code in accepted list, amount not negative.

Duplicate rate

\[ \text{DuplicateRate} = \frac{N_{duplicates}}{N} \]

Important for ingestion, CDC and event processing. Duplicates inflate counts, leak repeated examples into training and break analytics.

Null rate by column

\[ \text{NullRate}(c) = \frac{\text{null values in column }c}{N} \]

Track per critical feature. A sudden null-rate spike is often a source schema or upstream deployment issue.

Error budget for data checks

\[ \text{AllowedBadRecords} = N \times \text{ErrorBudget} \]

Converts a quality SLO into an operational threshold. Example: if budget is 0.1% over 10M records, allow at most 10,000 bad records before blocking/reviewing.

Freshness and lag

Ingestion lag

\[ \text{IngestionLag} = t_{ingest} - t_{event} \]

Delay from source event to pipeline entry. Use for logs, Kafka events, CDC and source API pulls.

Serving lag / data freshness

\[ \text{ServingLag} = t_{serve} - t_{event} \]

Delay until consumers can query/use the data. Dashboards, feature stores and reverse ETL care about serving lag, not only ingestion lag.

Staleness

\[ \text{StalenessNow} = t_{now} - t_{latest\_event\_served} \]

How old the newest visible data is. Useful for "latest sales," operational analytics and online feature checks.

CDC backlog time

\[ \text{BacklogSeconds} = \frac{\text{UnprocessedEvents}}{\text{ProcessingRate} - \text{ArrivalRate}} \]

Only meaningful if processing rate is greater than arrival rate. If arrival rate exceeds processing rate, backlog grows without bound.

Throughput, latency and capacity

Throughput

\[ \text{Throughput} = \frac{\text{records processed}}{\text{time}} \]

Core batch/stream metric. Pair with latency because high throughput can still hide slow individual events.

Pipeline latency

\[ \text{Latency}_{end-to-end} = \sum_i \text{Latency}_i + \sum_j \text{QueueWait}_j \]

End-to-end time includes processing stages and queue waits. Message brokers improve reliability but can add queue delay under load.

Required workers

\[ \text{Workers} \ge \left\lceil \frac{\text{ArrivalRate}}{\text{PerWorkerRate}} \right\rceil \]

First-order capacity sizing for ingestion/transformation. Add headroom for bursts, retries and maintenance.

Queue drain time

\[ \text{DrainTime} = \frac{\text{QueuedItems}}{\text{ProcessingRate} - \text{ArrivalRate}} \]

How long it takes to clear a queue once processing catches up.

Storage and compression

Raw storage volume

\[ \text{RawBytesPerDay} = R \times S \times 86400 \]

Event rate times average size times seconds/day. Base calculation for object storage, lake partitions and streaming retention.

Retained storage

\[ \text{RetainedBytes} = \text{BytesPerDay} \times \text{RetentionDays} \times \text{ReplicationFactor} \]

Remember replicas/copies. A "cheap lake" gets expensive if retention and replication are ignored.

Compression ratio

\[ \text{CompressionRatio} = \frac{\text{UncompressedSize}}{\text{CompressedSize}} \]

Columnar data often compresses better because similar values sit together.

Storage cost

\[ \text{MonthlyCost} = \text{StoredGB} \times \text{CostPerGBMonth} + \text{RequestCost} + \text{EgressCost} \]

Cloud storage cost is not only bytes. Read/write requests and data egress can matter for pipeline-heavy workloads.

Reliability and availability

Availability

\[ \text{Availability} = \frac{\text{Uptime}}{\text{Uptime} + \text{Downtime}} \]

Used for data platforms, orchestrators, warehouses, feature stores and serving systems.

Serial dependency availability

\[ A_{serial} = \prod_i A_i \]

If every component must be up, total availability drops as dependencies increase.

Parallel redundancy availability

\[ A_{parallel} = 1 - \prod_i (1 - A_i) \]

If any replica can serve, redundancy raises availability.

MTBF/MTTR availability

\[ A = \frac{MTBF}{MTBF + MTTR} \]

Availability improves when failures are rarer (higher MTBF) or repair is faster (lower MTTR).

Retry with exponential backoff

\[ \text{Delay}_n = \min(\text{BaseDelay} \times 2^n, \text{MaxDelay}) \]

Useful for ingestion APIs and transient failures. Add jitter in real systems to avoid synchronized retry storms.

Sampling and validation

Sampling rate

\[ \text{SampleRate} = \frac{N_{sample}}{N_{total}} \]

Sampling saves cost but can lose rare classes/events. Track what quality or minority-segment signal is lost.

Population estimate from uniform sample

\[ \widehat{Total} = \frac{\text{SampleTotal}}{\text{SampleRate}} \]

Only valid for uniform/random sampling and stable distributions. Not safe for biased sampling.

Schema change impact

\[ \text{ImpactedConsumers} = |\{c: c \text{ reads changed field/table}\}| \]

Not a math-heavy formula, but the right operational question: who reads this field, and can older consumers survive it?

Decision tables

Requirement	Prefer	Why
Many small current-state writes	OLTP/RDBMS	Transactions, constraints, consistency.
Wide analytical scans over few columns	Columnar warehouse/lakehouse	Column pruning, compression, aggregate performance.
Raw mixed-format low-cost storage	Data lake/object storage	Flexible schema-on-read and cheap retention.
Real-time events with replay	Kafka/Pulsar/Kinesis	Durable log, consumer decoupling, stream processing.
Historical correctness + realtime view	Lambda	Batch path for truth, speed path for latency.
One event-first processing path	Kappa	Less duplicate logic, replayable stream backbone.
Send model outputs back to CRM	Reverse ETL	Operational users work in their source tool.
Thousands of scheduled dependencies	Orchestrator	Retries, dependencies, backfills, observability.

Exam use. Do not dump formulas alone. State the requirement, choose the metric/pattern, calculate or compare, then explain the tradeoff.

DMML formula sheet -- metrics, checks and decision tables.