One-card revision for DSE/AIML ZG529. The course is not "databases only." It is the full path from raw data to reliable ML: representation, storage, governance, architecture, movement, transformation, serving, automation, monitoring and feedback into production systems.
Course map
| Deck | Theme | Must remember |
|---|---|---|
| CS1 | Data representations | Data -> information -> knowledge -> wisdom; formats, models, layouts, serialization, OLTP/OLAP, ACID/BASE. |
| CS2 | Data management fundamentals | Data as asset/liability, data sensitivity, reliability, governance/security, platform components and data life cycle. |
| L3 | Data architectures | Centralized/decentralized/hybrid; warehouse, lake, lakehouse, marts, mesh, fabric, Lambda and Kappa. |
| L4 | Data pipelines | Flow through databases, services and message brokers; modern data stack; ETL/ELT, batch, stream, CDC, Beam/Dataflow. |
| L5 | Infrastructure and DataOps | Ingestion, transformation, orchestration, storage systems, serving, automation, cloud/security, CT/CD and observability. |
CS1: data representation basics
CS1
- Data is raw material. Information is organized data. Knowledge is actionable information. Wisdom is integrated understanding used for judgement.
- Data management = ingesting, processing, securing and storing organizational data so it can improve business outcomes and support ML systems.
- ML system as pipeline: source data -> transformation -> features/training data -> model -> predictions -> feedback. Bad data can break ML while ordinary data movement still "looks green."
- Structured data: fixed schema, tables, rows, constraints. Semi-structured: partial schema, JSON/XML/logs. Unstructured: text, images, audio, video, emails.
- Data models: relational for strong structure; hierarchical for tree-like parent-child data; graph for relationship-first queries; document for flexible JSON/XML objects; key-value for lookup/cache patterns.
- Declarative query: say what result you want; optimizer decides how. Imperative/procedural: specify step-by-step instructions; easier to reason locally, harder for systems to optimize globally.
- Row layout: faster when reading/writing full records. Column layout: faster for analytical scans over selected columns, compression and feature-heavy tables.
- Serialization: convert in-memory data into a storable/transmittable format. CSV/JSON/XML are text-friendly; Protobuf/Avro are compact binary formats with schema support.
- OLTP: many small transactions, current state, strict consistency. OLAP: analytical queries, aggregations, drill-down, roll-up, slice-and-dice and trend analysis.
- ACID: atomicity, consistency, isolation, durability. BASE: basically available, soft state, eventual consistency; common in highly available distributed systems.
CS2: data as asset, liability and platform
CS2
- Data as asset: valuable when accessible, accurate, governed and reusable for decisions or models.
- Data as liability: dangerous when it contains PII, violates consent boundaries, is stale, unneeded, poorly secured or expensive to retain/delete.
- More data != better ML. More wrong, biased, missing, delayed, duplicated or legally risky data makes systems worse.
- Data reliability minimums: durability (not lost), consistency (same truth across copies), version/time travel where needed, availability and freshness.
- Data management components: integration/processing, storage, governance and security.
- Data governance: policies, stewardship, taxonomy, metadata, standards, quality rules, lineage and ownership.
- Data security: roles, least privilege, access control, PII controls, encryption, audit and separation of duties.
- Data platform: central repository + processing house + pipeline system that ingests, normalizes, transforms and delivers data to consumers.
- Life cycle: generation -> ingestion -> validation -> cleaning -> enrichment -> storage -> transformation -> serving/visualization -> feedback/retention.
- Data engineering role: design, build and maintain systems that collect, clean, move, store and serve data reliably.
L3: architecture patterns
L3
| Pattern | Best when | Risk |
|---|---|---|
| Centralized | Need one control plane for governance, auditing and enterprise reporting. | Can bottleneck domain teams and become slow to change. |
| Decentralized | Business units move fast with their own data and reporting. | Silos, duplication and inconsistent definitions. |
| Hybrid | Domains need ownership but enterprise still needs golden sources and shared standards. | Requires clear domain boundaries and governance. |
| Warehouse | Structured analytics, BI, curated schema, dimensional models. | Less flexible for raw/semi-structured high-velocity data. |
| Lake | Raw/semi-structured/unstructured data, cheap storage, ELT and data science exploration. | Can become a data swamp without metadata and governance. |
| Lakehouse | Need lake flexibility plus warehouse governance/performance. | Operational complexity and table-format discipline. |
| Data mesh | Large org with domain-owned data products and self-serve platform. | Needs mature ownership, standards and product thinking. |
| Data fabric | Need integrated access across lakes, warehouses, databases and SaaS systems. | Can hide complexity rather than solve data-quality problems. |
- Star schema: central fact table + denormalized dimensions. Simple and fast for BI.
- Snowflake schema: normalized dimensions split into sub-dimensions. Saves storage and improves organization but adds joins.
- Data mart: department/line-of-business subset of the warehouse, optimized for a specific consumer group.
- Lambda architecture: batch path for historical correctness + speed path for low-latency results + serving layer. Powerful but duplicates logic.
- Kappa architecture: one streaming backbone for ingestion, processing and serving. Simpler logic; depends on robust replayable logs.
L4: data pipelines
L4
- Data pipeline: sequence of steps that moves data from source to destination while applying ingestion, validation, transformation and delivery rules.
- Through databases: shared DB acts as integration point. Watch schema evolution, rolling upgrades, old code reading new fields, and expensive migrations.
- Through services: client-server, SOA, microservices, REST/SOAP/RPC. APIs reduce direct DB coupling but network calls are not local method calls.
- Through message brokers: asynchronous buffer between producers and consumers. Improves resilience, decouples components and supports redelivery.
- Modern data stack: cloud-based ingestion, storage, transformation, governance, analysis and serving tools designed for fast setup and self-service.
- ETL: transform before loading; useful when target warehouse needs clean structured data upfront. ELT: load raw data first, transform inside scalable warehouse/lakehouse.
- Batch pipeline: scheduled, bounded data. Good for monthly reports, daily aggregates and training snapshots.
- Stream pipeline: continuous events. Good for monitoring, fraud, recommendation signals, operational analytics and low-latency ML features.
- CDC: change data capture streams inserts/updates/deletes from source databases to downstream systems.
- Micro-pipeline: break complex pipeline into small stages so requirements, ingestion, validation, transformation and delivery can evolve independently.
L5: infrastructure, DataOps and serving
L5
- Ingestion infrastructure: connectors for databases, APIs, Kafka/streams, file shares and cloud buckets.
- Transformation infrastructure: code or SQL that filters, joins, cleans, masks PII, aggregates and models data into usable shapes.
- Orchestration: schedules and coordinates pipeline tasks, retries failures, manages dependencies, backfills data and tracks execution state.
- Storage raw ingredients: HDD/SSD/RAM, network, CPU, serialization, compression and caching.
- Storage systems: local file systems, NAS, block storage, object storage, memory caches, HDFS and streaming storage.
- Object storage: immutable key-value object store for files/blobs; cheap and scalable but no random writes/append.
- Serving: file exchange, databases, streaming systems, query federation, notebooks, dashboards, reverse ETL, embedded analytics and ML feature/model serving.
- Reverse ETL: send processed data back into operational tools, e.g. lead scores from warehouse into CRM.
- DataOps: DevOps-style discipline for data pipelines: versioning, testing, monitoring, observability, automation, governance and continuous improvement.
- CT/CD for ML: code changes trigger tests and training; approved models move through deployment; data/model drift and data quality feed back into retraining.
Fast comparisons
| Pair | Difference |
|---|---|
| OLTP vs OLAP | OLTP optimizes transactional writes and current state; OLAP optimizes analytical reads, aggregates and historical slicing. |
| ETL vs ELT | ETL transforms before load; ELT loads raw data first and transforms inside the target platform. |
| Warehouse vs lake | Warehouse is curated/schema-first for analytics; lake is flexible/raw storage for varied data. |
| Lake vs lakehouse | Lakehouse adds table formats, governance and query performance controls on top of lake storage. |
| REST vs RPC | REST models resources over HTTP; RPC models remote calls like functions. RPC can hide network failure semantics too much. |
| Batch vs stream | Batch handles bounded chunks on a schedule; stream handles unbounded events continuously. |
| DataOps vs DevOps | DevOps optimizes code delivery; DataOps applies similar discipline to data pipelines, data quality and analytical delivery. |
Exam gotchas
- If a question says relationship-first query, think graph model.
- If it says high-volume analytics over few columns, think columnar storage.
- If it says strict transactions, think OLTP and ACID.
- If it says enterprise BI and dashboards, think warehouse, dimensional schema, OLAP.
- If it says raw mixed-format data for data science, think data lake/lakehouse.
- If it says two separate paths for historical + realtime, think Lambda.
- If it says one replayable stream path, think Kappa.
- If it says CRM needs model scores back, think reverse ETL.
- If it says pipeline green but model bad, think data sensitivity, drift, quality, missing segment or feature breakage.
One-line memory hook. DMML = make data reliable enough that ML can trust it: collect it legally, model it correctly, move it predictably, transform it reproducibly, serve it fast enough, and observe it continuously.
More in this vault
- Slides Explained -- full concepts behind the card.
- Question Bank -- practice prompts, no answers.
- Formula Sheet -- metrics and operational formulas.
- DMML vault -- all resources.