This unpacks the five uploaded decks in Slides/: CS1.pptx, L2-Data Management Fundamentals.pptx, L3-Data Architectures.pptx, L4-Data Pipelines.pptx, and L5- Modern Data Infrastructure and DataOps.pptx. The big idea: ML is only as reliable as the data system feeding it. DMML teaches how data is represented, stored, governed, moved, transformed, served, automated and monitored.
CS1 -- Data Representations
CS1 · formats, models, layouts, serialization, processing
Why data management starts before ML
The first deck begins with the information hierarchy: raw numbers or symbols are data; organized data becomes information; actionable information becomes knowledge; and integrated judgement becomes wisdom. This matters because machine learning does not begin at the model. It begins when raw events, records, logs, images or transactions are turned into data that a model can trust.
The practical definition in the deck is direct: data management is the practice of ingesting, processing, securing and storing an organization's data so it can support business decisions. For ML, this means data management is part of the model system. A recommender, fraud detector or demand forecaster depends on source reliability, freshness, transformations, access control and storage layout before the first training run begins.
Formats: structured, semi-structured and unstructured
Structured data follows a predefined schema: database tables, rows, columns, types and constraints. It is easy to validate and query, but less flexible. Unstructured data has no strict schema: images, text, video, email, PDFs and audio. It can hold rich signals, but the system must extract structure before analysis. Semi-structured data sits between them: JSON, XML, logs and nested records have keys or tags but not always a rigid relational schema.
The exam trap is to treat this as a vocabulary list. The real engineering question is: what can the system safely assume? If the schema is fixed, validation and query planning are easier. If the data is loose, ingestion can be faster and more flexible, but downstream transformations carry more responsibility.
Data models: choose by access pattern
A data model is the way data is structured, stored and organized. The deck walks through relational, hierarchical, graph, document and key-value models.
| Model | Core idea | Use when |
|---|---|---|
| Relational | Tables/relations, rows/tuples, constraints and joins. | Data is structured and integrity matters. |
| Hierarchical | Parent-child tree. | Data naturally nests, like org charts, registries or indexes. |
| Graph | Nodes and edges; relationships are first-class. | Queries are about paths, neighbors and connections. |
| Document | JSON/XML-like documents with flexible shape. | Records vary and are usually fetched as whole documents. |
| Key-value | Lookup by key to retrieve value. | Caches, sessions, feature lookups and simple fast retrieval. |
The slide comparing normalized and messy relational tables points at a core database lesson: one wide table can be easy to create but hard to trust. Repeated values create redundancy; updates can become inconsistent; expansion becomes painful. Normalization reduces redundancy and improves integrity, but analytics sometimes denormalizes later for read performance. The right design depends on workload.
Query languages: declarative vs imperative
With data available, the next question is how to retrieve and combine it. Imperative style tells the computer every step: open, check, loop, update, fetch. Declarative style states the desired result: "give me all matching records." SQL is declarative: the query says what, and the database optimizer decides how.
Declarative systems can optimize execution, parallelize work and change plans without changing user logic. Imperative logic gives more explicit control but usually forces the programmer to handle ordering, memory and optimization details.
Storage layouts: rows for transactions, columns for analytics
Row-major storage keeps all columns of a row together. It is efficient when the workload reads or writes full records, which is common in OLTP systems. Column-major storage keeps values of one column together. It is efficient when analytics scans a few columns across many rows, which is common in OLAP and ML feature analysis. Columnar storage also compresses well because adjacent values are often similar.
The practical memory hook: row = record workflow; column = analytical scan workflow.
Serialization: moving data safely between systems
Serialization converts data from one form into another for storage or transfer. A producer may hold an object in memory, serialize it into JSON, BSON, Protobuf or Avro, send or store it, and the consumer deserializes it back. The serializer and deserializer must agree on format and schema. Otherwise fields can be lost, misread or silently corrupted.
Text formats such as CSV/JSON/XML are easy to inspect and integrate. Binary formats such as Protobuf and Avro are compact and faster for machines, and they handle schema more formally. For data pipelines, schema evolution is as important as the format itself: today's consumer may be older than today's producer.
OLTP, OLAP and ACID/BASE
OLTP systems handle live transactions: orders, payments, profile edits, inventory updates. They need correctness, low-latency writes and often ACID guarantees. OLAP systems support analysis: roll-ups, drill-downs, margins, shares, moving averages, region/month/product slices and trend analysis. They need fast reads across historical data.
The extra slides on ACID and BASE introduce a distributed-systems tradeoff. ACID prioritizes transaction correctness: atomicity, consistency, isolation and durability. BASE accepts softer state and eventual consistency to improve availability and scale. A good answer does not say one is "better"; it says which tradeoff the workload needs.
CS2 -- Data Management Fundamentals
CS2 · data as asset/liability, platform, governance, lifecycle
Data as asset and liability
The second deck changes the framing from "what is data?" to "what responsibility comes with data?" Data is an asset when it improves decisions, enables AI/ML, supports personalization or reveals operational problems. But data is a liability when it is sensitive, legally restricted, stale, inaccurate, expensive to retain or hard to delete.
The deck explicitly challenges the myth: more data == better. For ML, more data helps only if it is relevant, lawful, representative, fresh and correctly labeled or transformed. Extra PII without consent is not an asset. Duplicates are not an asset. A region missing from the feed can make a model fail even if every processing job reports success.
Data sensitivity in ML pipelines
The ecommerce scenario is important: a company predicts New Year's Eve sales by country, region or language, but data from one segment disappears. A normal data pipeline may still run because it only moves and transforms what exists. The ML pipeline can behave differently because model quality depends on distribution coverage. The missing segment changes feature distribution, training signal and evaluation truth.
This is why ML data checks must look beyond job success. They need row counts by segment, null rates, freshness, schema checks, distribution drift, label availability and feature coverage.
Data reliability
The deck asks simple reliability questions: will data be lost? Will copies agree? Can the team time-travel when data changes? How fast is data available? Is it available when needed? These map to durability, consistency, versioning/time travel, freshness and availability.
For ML systems, versioning is especially important. If an experiment used training data from last Monday, the team must be able to reproduce that dataset or at least know exactly what changed.
Data management strategy
The deck names four major components: data integration and processing, data storage, data governance and data security. Integration brings data from APIs, apps, IoT devices and forms. Processing filters, validates, cleans and transforms it. Storage preserves it in warehouses, lakes, operational databases or other systems. Governance defines ownership, metadata, quality and policy. Security decides who can access what, under which controls.
These are not independent boxes. A governance policy that says salary data is restricted must be enforced by storage permissions, pipeline transformations, access logs and dashboard permissions.
Data platform and lifecycle
A data platform is bigger than a database. It is the repository, processing house and delivery system for organizational data. It ingests, stores, normalizes, transforms and serves data to different consumers: analysts, dashboards, ML training jobs, applications and operational teams.
The life cycle in the slides is: generation or creation, ingestion, processing, post-processing/storage and use. Generation means data appears at a source: logs, images, medical diagnostic output, transactions. Ingestion means receiving it and writing it into the data system, often with selection or sampling. Processing validates, cleans and enriches. Post-processing stores outputs in a form useful for analytics, ML or visualization.
Roles and the connecting dots
The deck lists analyst, DBA, data engineer and related roles to show that data management is cross-functional. Data engineering is the bridge: design, build and maintain systems that move and prepare data. Data architecture is the blueprint: standards, models and integration flows. Data management is the larger discipline that includes governance, storage, security, quality and lifecycle ownership.
L3 -- Data Architectures
L3 · architecture categories, storage patterns, governance, Lambda/Kappa
What data architecture means
Data architecture describes the structure and interaction of data assets, sources, storage, platforms and management resources. The reason it matters is change: data requirements, tools and business needs move quickly. Without architecture, every team creates local fixes until the organization has silos, duplicate truths and brittle integrations.
The deck separates what from how. Operational architecture asks what business capabilities and processes data must support. Technical architecture asks how systems store, move, process and serve that data.
Centralized, decentralized and hybrid architecture
A centralized data architecture provides a single point of control for governance, auditing and enterprise reporting. It is strong for consistency, but can become slow if one central team must satisfy every domain. A decentralized architecture lets business units manage their own data front-to-back. It can move faster locally, but creates silos and inconsistent definitions. A hybrid architecture organizes by domain with golden sources and shared standards, trying to combine domain ownership with enterprise consistency.
Most mature organizations move toward hybrid thinking: domain teams know their data best, but shared governance prevents five contradictory definitions of "active customer."
Warehouse, marts, lake and lakehouse
A data warehouse stores structured, curated data for analytics and business intelligence. It usually expects a defined schema, high-performance analytical queries and dimensional modeling. A star schema places a fact table in the center and dimension tables around it; it is easy for analysts and BI tools. A snowflake schema normalizes dimensions into sub-dimensions; it saves space and organizes detail but adds joins.
A data mart is a focused subset of the warehouse for one department or line of business. It reduces load on the central warehouse and gives a team data shaped for its own questions.
A data lake stores raw or varied data, often in cheap object storage. It is useful for high-volume, semi-structured or unstructured data and exploration. The danger is a data swamp: files dumped without metadata, ownership or quality. A lakehouse tries to bring warehouse-like governance, transactions and query performance to lake storage.
Data mesh and data fabric
Data mesh decentralizes ownership. Each business domain treats data as a product: owned, documented, discoverable and quality-controlled. A self-serve platform provides common infrastructure, monitoring and catalogs so domain teams can focus on domain logic.
Data fabric creates a unified data layer across warehouses, lakes, databases and SaaS tools using APIs, CDC, metadata and virtualization. A 360-degree customer view is a good example: combine Salesforce, transaction logs, support tickets and social sentiment without pretending all sources are the same database.
Mesh is more about operating model and ownership; fabric is more about integration layer and access architecture. They can coexist.
Big data architecture
Big data means data too massive, fast or varied for traditional database systems to handle comfortably. The common "5 Vs" are volume, velocity, variety, veracity and value. Architecture must handle sources, ingestion, storage, processing, analysis and serving. It may include databases, files, IoT devices, streams, warehouses, lakes and ML workloads.
The benefits are flexibility and scale; the challenges are complexity, testing, troubleshooting, governance and cost. Big data architecture should not be a diagram of tools only. It should explain how data becomes useful and trustworthy.
Lambda and Kappa
Lambda architecture splits data into two paths. The batch layer processes complete historical data for correctness. The speed layer handles recent events for low latency. The serving layer combines views. The fraud example fits: historical transaction analysis plus real-time alerting.
The drawback is duplicated logic: batch and speed paths may implement similar business rules in different frameworks. Kappa architecture responds by using one stream-processing backbone for all data. Everything flows through a durable, replayable event log. Historical rebuilds happen by replaying the log. Kappa is simpler if the event log is complete and stream processing can cover the workload.
For ML, Lambda can support both comprehensive historical training and real-time inference. Kappa can simplify continuous learning and online features if the organization is ready for event-first design.
L4 -- Data Pipelines
L4 · data flow modes, modern stack, ETL/ELT, batch/stream/CDC
Why pipelines exist
The weather-data example makes the point: humans cannot manually feed thousands of sensors or stations into downstream systems reliably. A data pipeline automates movement from source to destination and applies rules for extraction, validation, transformation, loading and delivery.
A pipeline is not just a file copy. It is a repeatable path that turns source data into usable data for analytics, applications, ML and AI systems.
Flow through databases and schema evolution
One integration mode is data exchange through databases. Multiple applications read and write shared state. This is familiar but risky: rolling upgrades mean some processes run new code and others run old code. If new code writes a new field and old code reads the record, updates it and writes it back, the old code should preserve the unknown field rather than delete it.
This is the idea behind schema evolution and compatibility. Rewriting huge datasets to new schemas can be expensive, so systems prefer additive changes, defaults and compatibility rules.
Flow through services
Services expose APIs over a network. Client-server architecture has clients requesting data or behavior from servers. SOA decomposes a large application into services by business function. Microservices evolved from SOA with lighter, independently deployable services.
REST uses HTTP principles and resource-oriented URLs. SOAP uses a more formal XML-based web service stack. RPC tries to make remote calls feel like local function calls, which is convenient but dangerous: networks fail, time out, duplicate calls and behave with latency. A remote service is not a local method.
Flow through message passing
Message brokers sit between databases and direct RPC. A producer sends messages; consumers process them asynchronously. Brokers can buffer when recipients are down or overloaded, redeliver after crashes and decouple producers from consumers. This is why message passing is common in pipelines, event-driven systems and streaming architectures.
Modern data stack
The traditional data stack relied heavily on on-prem infrastructure, long setup cycles, high maintenance cost and slow response to changing needs. The modern data stack is cloud-oriented and integrated: ingestion, storage, transformation, analysis and governance tools that are easier to set up, scale and connect.
Its key promise is self-service: analysts, data scientists and business teams can discover and use data with less dependency on central IT. The risk is tool sprawl and weak governance if metadata, lineage and access controls do not keep up.
Pipeline architectures: ETL, ELT, batch, stream, CDC
ETL extracts, transforms and loads. It is useful when data quality and governance must be enforced before the target system, or when the target system has limited transformation power. ELT extracts, loads and then transforms inside the target platform, usually a scalable warehouse or lakehouse.
Batch processing runs bounded chunks on a schedule. It is reliable for daily reports, monthly accounting, snapshots and training data creation. Stream processing continuously processes events from sensors, apps or user interactions. It supports low-latency monitoring, fraud, personalization and operational analytics.
Change data capture reads changes from a source database -- inserts, updates and deletes -- and sends them downstream. It keeps replicas, warehouses, caches or feature stores current without full reloads.
Implementation and tooling
A simple pipeline might be CSV -> database -> dashboard. A complex pipeline includes source contracts, validation, transformation, orchestration, observability and consumer-specific delivery. The slides introduce development pipelines and execution pipelines, then recommend micro-pipelines: split a large flow into smaller stages so each one can be changed and tested independently.
Tool choice depends on scale and complexity. Basic ingestion tools copy data well but may not handle complex transformations. ETL platforms can be powerful but rigid. Data engineering platforms increasingly abstract the "how" so teams can focus on what data moves, who owns it and where it must go.
L5 -- Modern Data Infrastructure and DataOps
L5 · ingestion, transformation, orchestration, storage, serving, DataOps
Infrastructure follows the data pipeline
L5 starts by revisiting the architecture path: generation/source -> ingestion -> transformation -> serving. Infrastructure exists to make each stage repeatable, scalable and observable. The last two decades changed the default: public cloud, managed databases, object storage, SaaS APIs, serverless compute and open-source data tools made large data platforms easier to assemble.
Ingestion and transformation infrastructure
Sources include application databases such as Postgres/MySQL, REST APIs, stream systems such as Kafka and flat files in shared storage or cloud buckets. Ingestion infrastructure must handle connector reliability, authentication, schema changes, rate limits and retries.
Transformation is the "T" in ETL/ELT, but it includes more than formatting. It can hash PII, normalize types, deduplicate records, join sources, compute aggregates, validate schemas and model data into analytical or ML-ready tables.
Workflow orchestration
As pipelines multiply, ad hoc scripts fail. A workflow orchestration platform schedules tasks, tracks dependencies, retries failures, supports backfills and exposes operational state. The orchestrator is the control plane for pipeline execution. If it is down or blind, the team loses confidence in all downstream data.
Storage infrastructure
The deck splits storage into raw ingredients and systems. Raw ingredients include HDD, SSD, RAM, networking, CPU, serialization, compression and caching. Systems include file storage, block storage, object storage, memory caches, HDFS and streaming storage.
File storage exposes files and directories. Block storage exposes raw blocks and is common for databases and virtual disks. Object storage stores immutable objects by key; it is cheap and scalable, ideal for data lakes, but does not behave like a local appendable file. Cache/memory systems serve low-latency data but need durable backing storage. HDFS couples distributed storage with compute locality and remains important in many big-data engines. Streaming storage adds retention and replay semantics for event streams.
Storage abstractions and serving
Warehouses, lakes, lakehouses and data platforms are abstractions built on top of storage systems. They define how consumers organize, query and trust data.
Serving is the last stage: delivering data to consumers. The deck covers file exchange, databases, streaming systems, query federation and notebooks. File exchange is universal but loose. Databases impose schema and enable SQL. Streaming systems serve fresh operational data. Query federation lets users query across many systems without centralizing all data first. Notebooks let data scientists explore, engineer features and train models interactively.
Reverse ETL and analytics serving
Reverse ETL sends processed data back to source or operational systems. If a lead-scoring model writes scores to the warehouse, the sales team may still work inside CRM. Instead of emailing spreadsheets, reverse ETL puts the score back where the salesperson works.
Analytics serving appears in three forms. Business analytics supports strategic decisions with dashboards, reports and ad hoc analysis. Operational analytics supports immediate action with fresher data. Embedded analytics exposes analytics inside customer-facing applications.
Data science and ML infrastructure stack
The ML stack in the deck has layers: data warehouse/storage at the bottom, compute resources above it, job scheduler/orchestrator, application architecture, versioning, model operations, feature engineering and model development at the top. Model development is visible, but it depends on every layer below it.
This is the core DMML lesson for ML engineers: freedom to experiment comes with responsibility to make experiments reproducible, schedulable, deployable and monitorable. A model is not production-ready until its data, features, code, environment, training run and serving path are controlled.
Automation, cloud, security and DataOps
Automation covers infrastructure provisioning, pipeline execution, retries, schema-change detection and downstream updates. IaC tools such as Terraform and cloud-native equivalents make infrastructure repeatable. GitOps applies Git-based change control to operations.
Cloud infrastructure gives managed storage, databases, pipeline builders, streaming systems and APIs. Security must cover data at rest, data in transit and data in use. The deck then connects DevOps to DataOps: DevOps works on code delivery; DataOps applies agile and operational discipline to data pipelines, governance and analytics delivery.
CT/CD and observability
Continuous training and deployment extends CI/CD for ML. In the experimental phase, an engineer changes features or code and trains a candidate. Continuous integration tests code and can trigger automated training. Validation checks model and data behavior. Deployment moves approved models into production. Observability watches the system after deployment: data freshness, schema drift, feature distribution, prediction behavior, latency and failures.
The DoorDash-style case study shows why this matters. Fragmented analytics, batch ML and real-time ML workflows create slow deployment and low observability. A feature store, real-time aggregators, historical aggregators, prediction service and monitoring create a more coherent platform.
Final mental model
DMML is easiest to remember as a chain:
- Represent data correctly: format, model, schema, layout and serialization.
- Manage data responsibly: asset/liability, governance, security, reliability and lifecycle.
- Architect for organizational reality: centralized/decentralized/hybrid, warehouse/lake/lakehouse, mesh/fabric, Lambda/Kappa.
- Move data reliably: databases, services, messages, ETL/ELT, batch, stream and CDC.
- Operate the platform: ingestion, transformation, orchestration, storage, serving, automation, DataOps, observability and ML CT/CD.
More in this vault
- Cheatsheet -- one-card revision.
- Question Bank -- deck-derived questions, no answers.
- Formula Sheet -- operational formulas and decision tables.
- Book Guide -- how the books map to lectures.
- References -- books, tools and external readings.