openskills.info
← Data Engineer career guide

Data and AI

Data Engineer timeline

How the field of a Data Engineer got here: the sourced milestones from every published course of this career, on one axis.

1956 to 2025

260 milestones across 28 courses

  • Core
  • Supporting
  • Specialization

The axis stretches toward recent years, where most milestones fall. Bars count milestones per year; dots are single milestones.

Before 1990

13 milestones

  1. 1956 Sep

    IBM 350 RAMAC disk file introduced

    The RAMAC introduced random-access disk storage for large computer data volumes. Data processing could move beyond sequential punched cards and magnetic tape for selected workloads.

  2. 1964 Apr

    IBM announces System/360

    System/360 established a compatible computer family across a broad performance range. Organizations could scale data-processing workloads without moving to an unrelated software architecture.

  3. 1968

    IBM releases Information Management System

    IBM released IMS for System/360 mainframes. It made database management a distinct platform concern for large operational data.

  4. 1970

    Codd publishes the relational model

    Edgar F. Codd proposed organizing shared data as relations and separating logical structure from physical storage concerns. The work made declarative relationships and constraints central design ideas.

  5. 1970 Jan 1

    POSIX epoch establishes a common software reference point

    The IANA time-zone database uses 1970-01-01 00:00:00 UTC as its main scope boundary for locations whose clocks agree. Software later adopted this instant as the POSIX epoch for many timestamp representations.

  6. 1970 Jun

    E. F. Codd publishes the relational model

    Codd described relations and a data sublanguage for large shared data banks. The model separated a user's data view from internal data representation, which became central to later analytical processing.

  7. 1971

    CODASYL publishes the DBTG report

    The Database Task Group documented the network database approach, which made record and set relationships explicit in navigational designs. It became an important contrast for later relational modeling.

  8. 1976

    Chen formalizes entity-relationship modeling

    Peter Chen published the entity-relationship model as a way to represent entities and their relationships at a conceptual level. ER notation gave modeling a domain-facing design language before physical implementation.

  9. 1979

    Oracle V2 commercializes SQL

    Relational Software, Inc. released Oracle V2 as a commercial SQL-based relational database. SQL moved from research systems into a product category that competing database vendors adopted.

  10. 1986

    ANSI publishes SQL-86

    ANSI published the first SQL standard. A standard core gave database products a common reference point even as their dialects continued to differ.

  11. 1986

    ANSI publishes the first SQL standard

    SQL became a published standard for relational database language. Modeling decisions about tables, keys, and constraints could increasingly be expressed across vendor implementations.

  12. 1987

    ISO publishes ISO/IEC 9075

    ISO adopted SQL as ISO/IEC 9075. The paired ANSI and ISO work made SQL an international database-language standard.

  13. 1988

    ISO 8601 standardizes numeric date and time representations

    ISO 8601 established an international date-and-time representation standard. Its year-month-day ordering became the basis for unambiguous machine-readable dates.

1990s

7 milestones

  1. 1990

    Codd publishes The Relational Model for Database Management

    Codd's later book brought relational concepts and their interpretation together for database management practice. It reinforced the distinction between the model's meaning and a particular storage implementation.

  2. 1991 Feb

    Python is posted to Usenet

    Python's first public release established the language that later became the host environment for numerical and tabular libraries.

  3. 1992

    Building the Data Warehouse frames the discipline

    Bill Inmon's book established a named body of warehouse architecture and historical decision-support practice.

  4. 1992

    SQL-92 expands the standard

    ANSI and ISO published the revision commonly called SQL-92. It became a long-lived baseline for the portable syntax developers still recognize.

  5. 1996

    The Data Warehouse Toolkit popularizes dimensional modeling

    Ralph Kimball's toolkit put fact tables, dimensions, and star schemas at the center of a practical modeling approach.

  6. 1997

    Kimball describes dimensional modeling for warehouses

    Ralph Kimball contrasted entity-relationship models for transaction capture with dimensional models for end-user analysis. Facts, dimensions, and grain became a practical vocabulary for analytical design.

  7. 1999

    SQL:1999 adds object and procedural features

    SQL:1999 added features including recursive queries and standardized procedural extensions. It widened SQL beyond basic table queries while retaining the relational core.

2000s

21 milestones

  1. 2000

    Data Vault modeling is released publicly

    Dan Linstedt released Data Vault as another approach to historical, auditable integration from changing source systems.

  2. 2002 Jul

    RFC 3339 defines an Internet timestamp profile

    RFC 3339 specified a constrained Internet profile of ISO 8601 using UTC or numeric offsets. It gave protocols a common syntax for representing an instant.

  3. 2003

    Google File System paper published

    Google documented a distributed file system designed for large data-intensive applications. The paper made storage behavior a first-class concern in large-scale data processing designs.

  4. 2003

    Matplotlib enters its early public era

    Matplotlib's original 2003-2008 project identity marks the emergence of a Python plotting layer for array-based analysis.

  5. 2003

    SQL:2003 standardizes window functions

    SQL:2003 added window functions and SQL/XML work. Window functions made ranking and calculations across related result rows part of standard SQL practice.

  6. 2004

    Google publishes MapReduce

    MapReduce described a programming model for processing large data sets on clusters. The runtime handled input partitioning, scheduling, failures, and inter-machine communication.

  7. 2004

    MapReduce paper published

    Google described a programming model and implementation for processing large datasets on clusters. It separated map and reduce work into a reusable distributed processing pattern.

  8. 2005

    NumPy is created

    NumPy combined earlier Numeric and numarray work into the array library that provides the vectorized numerical base used by pandas.

  9. 2006

    Bigtable paper published

    Google described a distributed storage system for structured data. The design connected large-scale storage with tables, schemas, and application-facing access patterns.

  10. 2006

    Google publishes Bigtable

    Bigtable described a distributed system for structured data organized around a sparse, multidimensional sorted map. It showed how large-scale storage designs could use a data model unlike conventional relational tables.

  11. 2006 Mar 14

    Amazon S3 launches

    Amazon S3 made web-scale object storage available through an API. Object storage later became a common durable layer beneath cloud data lakes.

  12. 2006 Apr

    Apache Hadoop 0.1.0 released

    Hadoop made its first release after the Nutch MapReduce and distributed-file-system code became a separate project. It provided an open-source path to cluster batch processing.

  13. 2007

    Amazon publishes Dynamo

    Dynamo described a highly available key-value store built around partitioning and replication. Its design highlighted how access patterns and failure behavior shape data structure choices in distributed systems.

  14. 2008

    pandas development begins

    Development started at AQR Capital Management, introducing the project that made labeled DataFrames central to Python data work.

  15. 2008 Jan

    Hadoop became an Apache top-level project

    Apache created the Hadoop Project Management Committee when Hadoop moved out of Lucene. Hadoop became an independently governed open-source platform for distributed storage and processing.

  16. 2008 Jan

    Hadoop becomes an Apache top-level project

    Apache Hadoop moved from Lucene to become an Apache top-level project. Its distributed storage and batch-processing ecosystem shaped early large-scale data-lake practice.

  17. 2009

    Development starts at Yandex

    Alexey Milovidov and team begin an experimental project to generate analytical reports in real time from non-aggregated data, initially as a filter and aggregation operator for web-scale log data.

  20. 2009

    Spark begins at UC Berkeley AMPLab

    Spark begins as a UC Berkeley AMPLab research project. Its initial direction is distributed data processing with a programming model that can reuse work.

  21. 2009 Sep

    RFC 5545 defines iCalendar time-zone components

    RFC 5545 standardized iCalendar objects, including components that carry time-zone definitions for calendar data. Calendar interchange therefore had an explicit mechanism for local-time rules rather than only fixed offsets.

2010s

109 milestones

  1. 2010

    Apache Spark open sourced

    Spark moved from the UC Berkeley AMPLab research project to an open-source project. It became another distributed execution engine for data transformations beyond the original MapReduce stack.

  2. 2010

    Dremel demonstrates interactive analysis at web scale

    Google's Dremel paper described a distributed system for interactive analysis of nested data, influencing later large-scale SQL services.

  3. 2010

    Dremel paper published at VLDB

    The Dremel paper described interactive analysis of web-scale datasets through a columnar, multi-level execution tree, publishing the architecture that BigQuery's query engine follows.

  6. 2011

    BigQuery becomes generally available

    BigQuery made a serverless cloud warehouse available, moving capacity management away from the team submitting analytical SQL.

  7. 2011

    SQL:2011 adds temporal features

    SQL:2011 added temporal database features. The standard gained a common way to express time-aware data and queries.

  8. 2011

    Storm is released as open source

    Storm made distributed, tuple-at-a-time stream processing broadly available as an open-source system. Its keyed topologies established an early operational model for continuously updated computations.

  9. 2011 Jul

    Kafka enters the Apache Incubator

    Kafka entered incubation as a distributed publish and subscribe project. Apache stewardship established the open project that later became a common event-log implementation.

  11. 2011 Nov

    IANA publishes continuing tz database releases

    The IANA release archive records 2011 time-zone-data releases. Distributing updated rule data became an operational concern for systems that schedule future civil times.

  12. 2012

    Launch powering Yandex.Metrica

    After three years of development ClickHouse launches in production to power Yandex.Metrica, the second-largest web analytics platform, whose reports it serves from non-aggregated data.

  13. 2012

    Presto development begins at Facebook

    Engineers began Presto to support low-latency interactive analytics over a large Hadoop warehouse. The project established the distributed SQL model inherited by Trino.

  14. 2012 Jun

    pandas 0.8 improves time-series handling

    pandas 0.8 adopted NumPy datetime64 for time-series representation, improving joins, memory use, and serialization around dated data.

  15. 2012 Oct

    Kafka graduates as an Apache project

    Kafka graduated from the Apache Incubator. The project then continued as an Apache top-level project.

  16. 2013

    Spark moved to Apache

    Spark moved to the Apache Software Foundation after its early open-source development. This placed the processing engine in the same open-source governance ecosystem as many data infrastructure projects.

  18. 2013 Feb

    Amazon Redshift becomes broadly available

    Redshift made a managed, petabyte-scale AWS warehouse broadly available for SQL analysis through existing BI tools.

  19. 2013 Apr

    PROV-O becomes a W3C Recommendation

    PROV-O standardized a vocabulary for representing provenance. It gave governance implementations a common model for entities, activities, and agents behind data.

  20. 2013 Sep

    Streaming inserts arrive

    Release notes added streaming inserts and time-based table decorators, letting rows enter BigQuery continuously instead of through load jobs alone.

  21. 2013 Nov

    Presto is released as open source

    Facebook announced the open-source release of Presto. The project could then be adopted and extended outside its original environment.

  24. 2014

    DCAT is standardized by W3C

    The first Data Catalog Vocabulary recommendation established a shared vocabulary for dataset and data-service metadata. Catalog interoperability became a standards concern rather than a local schema choice.

  25. 2014

    Project Jupyter begins

    Jupyter grew from IPython to support interactive data science and scientific computing across programming languages.

  26. 2014

    Spark 1.0 introduces Spark SQL

    Spark 1.0 introduces Spark SQL and a common spark-submit workflow. Structured-data processing and deployment become first-class parts of the release line.

  27. 2014 Mar

    Kafka 0.8.1 releases

    The 0.8.1 release continued the replicated-log generation and provided a stable base for production adoption of partition leaders and in-sync replicas.

  28. 2014 Mar

    Pricing overhaul lowers rates and adds reserved capacity

    BigQuery changed its pricing model, lowering rates and adding reserved capacity pricing, the first step toward the capacity model that later editions formalize.

  29. 2014 May

    Spark 1.0 released

    Apache training material records Spark 1.0 in May 2014. A production-ready Spark release broadened the practical choice of engines for batch and stream-oriented workloads.

  30. 2014 Aug

    Apache Flink 0.6 becomes the first Flink release in the Incubator

    Flink moved from the Stratosphere name into Apache incubation. This established a distributed data-processing engine that would become a major stateful-streaming runtime.

  31. 2014 Aug

    Flink 0.6 enters Apache incubation

    The project published its first incubating release. Stream processing began its path as an Apache community project with downloadable runtime artifacts.

  32. 2014 Aug

    Flink 0.6 is released in the Apache Incubator

    This was the first Apache Incubator release under the Flink name. The project later centered event time, state, and recovery as core stream-processing concepts.

  33. 2014 Sep

    Apache Storm becomes a top-level project

    Storm graduated from the Apache Incubator, placing distributed real-time computation under Apache project governance. Stream processing became a maintained part of the wider data platform ecosystem.

  34. 2014 Oct

    Apache Airflow starts at Airbnb

    Airflow began as an open-source project for defining and scheduling directed acyclic graphs of tasks. It made orchestration a separately visible concern around data-processing work.

  35. 2014 Oct

    Snowflake describes a warehouse built for the cloud

    Snowflake documented a cloud-built warehouse architecture with independently sized virtual warehouses for different workloads.

  36. 2014 Dec

    Apache Flink becomes a top-level Apache project

    Flink graduated from the Apache Incubator. The project then continued toward a stable runtime and API for production use.

  38. 2015

    Spark 1.4 expands the DataFrame API

    Spark 1.4 expands the DataFrame API and packages SparkR. More structured interfaces become available across language ecosystems.

  39. 2015

    Timely Dataflow formalizes virtual time progress

    The timely dataflow model associated communication events with virtual time and formalized progress relationships. It supplied a foundation for stateful incremental systems that reason about when work is complete.

  40. 2015 Mar

    LinkedIn documents Kafka at scale

    LinkedIn described tiered clusters and audit counts that checked delivery from producers through downstream consumers. The operational model made end-to-end completeness a concern beyond broker health.

  41. 2015 Apr

    The 0.9 milestone introduces the Table API

    The 0.9 milestone added a SQL-like Table API alongside other APIs. This established a relational path that later became central to Flink SQL practice.

  42. 2015 Aug

    The Dataflow Model paper is published

    The paper formalized choices around event time, windows, watermarks, triggers, and accumulation for unbounded out-of-order data. These abstractions now shape how streaming feature windows are specified.

  43. 2015 Nov

    Flink 0.10 matures the DataStream API

    Flink 0.10 moved the DataStream API out of beta and added event-time, out-of-order-stream, exactly-once, windowing, state, and high-availability capabilities. These features made operational stream processing a first-class use case.

  44. 2015 Nov

    Kafka 0.9.0 releases

    Kafka 0.9.0 added Kafka Connect, creating a framework for reusable source and sink integrations instead of application-specific movement for every system.

  45. 2016

    Apache Pulsar is committed to open source

    Pulsar was committed to open source. It expanded the set of distributed messaging and streaming implementations available to practitioners.

  46. 2016

    Released under the Apache 2.0 license

    ClickHouse is released as an open-source project under the Apache 2.0 license, turning an internal analytics backend into a database other companies can run.

  47. 2016

    Spark 2.0 introduces Structured Streaming

    Spark 2.0 starts the 2.x line with Structured Streaming and SQL improvements. Batch and streaming work move toward the same structured programming model.

  48. 2016

    SQL:2016 adds JSON support

    SQL:2016 incorporated JSON-related features. Relational databases could standardize some work with semi-structured values without abandoning tables, constraints, and joins.

  49. 2016

    Uber develops Hudi

    Uber developed Hudi to address inefficient snapshot-and-reload processing and to support incremental updates in its data lake.

  50. 2016 Feb

    Apache Beam entered incubation

    Google and partners moved Cloud Dataflow SDKs and runners into the Apache Beam Incubator. The project centered one programming model across batch and streaming pipelines.

  52. 2016 Mar

    Apache Flink 1.0 releases

    The 1.0 release marked a stable project milestone and brought Flink to a broader production audience for distributed stream processing.

  53. 2016 Mar

    Flink 1.0 begins stable 1.x APIs

    Flink 1.0 formally guaranteed compatibility for APIs marked public across the 1.x line. It also added a RocksDB state backend and checkpoint statistics for operating larger state.

  54. 2016 Mar

    RFC 7808 specifies a time-zone data distribution service

    RFC 7808 defined an HTTP service for clients to discover and retrieve time-zone data. The protocol recognized that applications may need current rule data beyond their local installation.

  55. 2016 Apr

    GDPR is adopted

    Regulation EU 2016/679 set a common European framework for personal-data processing. Governance programs needed to connect processing decisions to accountable roles and controls.

  56. 2016 May

    Kafka 0.10.0.0 releases

    Kafka 0.10.0.0 introduced Kafka Streams, making a Java client library part of the path for transformations, joins, and aggregations over topic data.

  57. 2016 Jun

    Apache Beam 0.1.0-incubating released

    Beam released a pipeline SDK with runners for Flink, Spark, and Google Cloud Dataflow. It formalized a portable model that can express batch and streaming pipelines across execution environments.

  58. 2016 Jun

    Apache Beam 0.1.0-incubating releases

    Beam first released a unified programming model with Flink, Spark, and Google Cloud Dataflow runners. Pipeline logic could be expressed separately from an execution engine.

  59. 2016 Jun

    Kafka Streams ships with Kafka 0.10

    Kafka 0.10 introduced an embedded stream-processing library and message timestamps. Application teams could maintain keyed transformations without operating a separate processing cluster.

  60. 2016 Jun

    Standard SQL enters beta with IAM and time partitioning

    An ANSI-compliant Standard SQL dialect entered beta alongside fine-grained IAM roles and native time partitioning, removing the need for manually date-suffixed tables.

  61. 2016 Jul

    dbt development begins

    dbt development began in 2016, bringing versioned SQL modeling, testing, and documentation practices into warehouse transformation.

  63. 2016 Sep

    Standard SQL reaches general availability with DML and flat-rate pricing

    Standard SQL became generally available with INSERT, UPDATE, and DELETE support, and flat-rate pricing offered unlimited querying for a fixed monthly price.

  64. 2016 Dec

    Apache Beam graduated as a top-level project

    Apache Beam graduated from incubation in December 2016. The project established a community-governed model for engine-independent pipeline definitions.

  66. 2017

    Uber open-sources Hudi

    Uber released Hudi as open source after developing it for incremental data processing. The project introduced a table-oriented approach to lake data before the lakehouse label became widespread.

  67. 2017 Mar

    Apache Beam adds a Python SDK

    Beam released its Python SDK in version 0.6.0. This expanded access to the same pipeline model beyond Java.

  68. 2017 May

    Apache Beam 2.0 became the first stable release

    Apache Beam published version 2.0.0 as its first stable release. The release signaled API stability for teams building portable batch and streaming pipelines.

  69. 2017 May

    Apache Beam 2.0.0 becomes the first stable release

    Beam declared API stability with version 2.0.0. That made the portable model more suitable for long-lived production pipelines.

  70. 2017 May

    DVC is publicly introduced

    Dmitry Petrov published the first DVC blog post on May 4, 2017. The project began with the goal of bringing Git-linked versioning and reproducibility to data science artifacts.

  71. 2017 Jun

    Flink 1.3 adds operational visibility for time

    Flink 1.3 added Web UI watermark monitoring and completed-job history through the HistoryServer. Practitioners gained direct signals for diagnosing stalled event-time progress and reviewing finished jobs.

  72. 2017 Jun

    Kafka 0.11.0.0 adds idempotent and transactional producers

    Kafka 0.11 made idempotent and transactional producer modes available, which enabled atomic Kafka output and consumed-offset commits within a defined boundary.

  73. 2017 Sep

    First public architecture writing

    Yandex publishes the account of Metrica's storage evolution from MyISAM through Metrage and OLAPServer to ClickHouse, the narrative this course uses for the problem the engine exists to solve.

  75. 2017 Sep

    Uber describes Michelangelo and its feature store

    Uber documented a centralized feature store whose canonical features could be consumed online and offline. The design made feature reuse and training-serving consistency visible platform concerns.

  76. 2017 Nov

    Apache Kafka 1.0.0 ships

    Kafka 1.0.0 included improvements for exactly-once semantics. The release made transactional processing a documented part of the platform's delivery toolkit.

  77. 2017 Nov

    Kafka 1.0.0 released

    Kafka 1.0.0 was released in November 2017. The 1.0 release became a long-lived upgrade starting point for clusters whose operators later planned multi-version paths.

  78. 2018

    DuckDB development begins at CWI

    Mark Raasveldt and Hannes Mühleisen start DuckDB at Centrum Wiskunde & Informatica in Amsterdam as an embeddable analytical database project.

  79. 2018

    First Dagster code written

    Nick Schrock began the project that became Dagster. The early work established the software-engineering approach that preceded the stable 1.0 APIs.

  80. 2018

    Original creators leave Facebook

    The original creators left Facebook and formed an independent community around PrestoSQL. This created the project line that later became Trino.

  82. 2018 May

    GDPR becomes applicable

    The regulation began applying across EU Member States. Organizations needed operational evidence for privacy decisions, not only written policies.

  84. 2018 Jul

    Kafka 2.0.0 released

    Kafka 2.0.0 was released in July 2018. Operators gained another major-version boundary where compatibility and staged rolling upgrades became explicit maintenance work.

  85. 2018 Aug

    Pulsar 2.1.0-incubating adds Pulsar IO and tiered storage

    The release added the connector framework and storage offload capabilities that widened the platform beyond brokered delivery.

  86. 2018 Sep

    Apache Pulsar becomes a top-level project

    Pulsar graduated as an Apache top-level project. Its governance milestone reflected the growth of a separate streaming ecosystem.

  87. 2018 Sep

    Presto SQL on Everything documents the engine

    The paper described the distributed coordinator and worker architecture, connectors, query planning, resource management, and fault-tolerance tradeoffs in production Presto.

  88. 2018 Nov 16

    Iceberg enters the Apache Incubator

    The project began incubation at the ASF after originating at Netflix. Public community development of the open table format started under incubator governance.

  89. 2018 Dec

    Hopsworks publishes its open-source feature store

    Hopsworks introduced an open-source feature-store layer for sharing, documenting, and governing feature data. This moved the pattern beyond internal platforms.

  90. 2019

    SIGMOD demo paper published

    The peer-reviewed demo paper "DuckDB: An Embeddable Analytical Database" documents the design goals behind the early public release.

  91. 2019 Jan

    Airflow becomes an Apache top-level project

    Apache Airflow graduated from the Incubator and became a top-level project. The project then operated with its own Apache project governance and release process.

  92. 2019 Jan

    Feast is released as an open-source feature store

    GO-JEK and Google Cloud announced Feast for defining, storing, discovering, and reusing features across training and production. It established a vendor-neutral integration layer around existing data systems.

  93. 2019 Jan

    Hudi enters the Apache Incubator

    Uber submitted Hudi to the Apache Incubator. The move placed a transactional data-lake project under Apache community governance.

  94. 2019 Feb

    Delta Lake becomes generally available

    The February platform release recorded Delta Lake general availability, making reliable table behavior a standard platform capability.

  95. 2019 Apr

    Delta Lake is announced as open source

    Databricks announced the Delta Lake open-source project. The table layer became available outside its original managed platform context.

  97. 2019 Apr

    Flink 1.8 continues the compatible 1.x line

    Flink 1.8 remained API-compatible for public APIs in the 1.x series. Compatibility became a practical consideration for maintaining long-running applications through upgrades.

  99. 2019 May

    BigQuery ML reaches general availability

    BigQuery ML exited beta, making SQL-native model creation and prediction a supported part of warehouse practice.

  100. 2019 May 20

    Data mesh article asks data products to state their guarantees

    Zhamak Dehghani's article on moving beyond a monolithic data lake proposed domain-owned data products, each defining its integrity as a set of service level objectives and publishing schemas. It framed data as a product with explicit promises to consumers.

  101. 2019 Jun

    DuckDB v0.1 announced at SIGMOD

    The first preview release and conference announcement present an analytical SQL engine that runs as a binary or inside host processes such as Python and R.

  102. 2019 Jun

    Federal Data Strategy framework is issued

    The United States issued a government-wide framework for managing federal data. It placed ethical governance, accountability, and implementation practices in a long-term strategy.

  103. 2019 Jun 23

    Table specification moves onto ASF infrastructure

    The Iceberg specification was migrated to git and the ASF site. The format contract became an openly versioned community artifact rather than a company-only document.

  104. 2019 Aug

    Dictionary blocks improve UNNEST processing

    A Trino engineering change used dictionary blocks for UNNEST output. It reduced CPU and allocation work for the tested operator workload.

  105. 2019 Sep

    Federal agencies establish Data Governance Bodies

    The Federal Data Strategy required inclusive and empowered governance bodies. Authority, chartering, and senior participation became concrete operating requirements.

  107. 2019 Oct

    RFC 8536 documents the TZif binary format

    RFC 8536 documented the Time Zone Information Format used to distribute compiled time-zone data. It made the boundary between IANA source rules and runtime-consumable zone files explicit.

  108. 2019 Oct 25

    First incubating binary release 0.7.0

    The project shipped 0.7.0-incubating with source and convenience binaries. Downstream engines could depend on a numbered Apache release artifact.

  109. 2019 Nov

    Azure Synapse unifies warehouse and big-data analytics

    Microsoft introduced Azure Synapse as the next evolution of Azure SQL Data Warehouse, combining warehouse and broader analytics workloads.

2020s

110 milestones

  1. 2020 Jan

    NIST publishes Privacy Framework version 1.0

    NIST released a privacy-risk framework for enterprise risk management. It connected data-processing decisions with privacy risk.

  2. 2020 Jan

    pandas 1.0 establishes its major-version policy

    pandas 1.0 formalized a deprecation and major-release policy, making API evolution more legible for long-lived analysis code.

  3. 2020 Feb

    DCAT 2 becomes a W3C Recommendation

    DCAT 2 revised the catalog vocabulary after implementation experience and new use cases. It extended the standards path for metadata exchange.

  4. 2020 Mar

    Storage Read API becomes generally available

    The high-throughput, parallelizable read API over BigQuery storage reached GA, letting engines such as Spark and TensorFlow read warehouse data directly.

  5. 2020 Mar

    Version 20.3 releases

    The 20.3 LTS release continues the versioned release cadence that ClickHouse still follows, anchoring the era when the open-source project's adoption broadened beyond Yandex.

  6. 2020 May

    DVC 1.0 enters prerelease

    The 1.0 prerelease consolidated pipeline stages into one metadata file and added run-cache and remote-operation improvements. These changes prepared a stable project structure for automated data and model workflows.

  7. 2020 May

    Next-generation streaming becomes generally available

    A rewritten streaming path delivered seconds-level availability of streamed rows and removed the warm-up delay of the original streaming implementation.

  8. 2020 May

    PEP 615 adds zoneinfo to Python's standard library

    PEP 615 introduced the zoneinfo module for IANA time-zone support in Python. It brought named-zone handling and transition behavior into a standard runtime API.

  9. 2020 May 20

    Iceberg graduates as a top-level project

    The project left the Incubator. Governance and releases continued as an Apache top-level project for the table format.

  10. 2020 Jun

    DVC 1.0 is released

    DVC 1.0 formalized text metadata pointing to artifact content in remote storage and improved pull, push, remote status, and garbage-collection behavior. The release marked the versioning workflow as a mature foundation for MLOps.

  11. 2020 Jun

    Spark 3.0 adds adaptive query execution

    Spark 3.0 adds adaptive query execution and dynamic partition pruning. Runtime statistics can now influence selected query-execution strategies.

  12. 2020 Jun 4

    Apache Hudi becomes a top-level project

    The Apache Software Foundation announced Hudi as a top-level project. Its maturation reflected sustained adoption of transactional capabilities over data-lake storage.

  13. 2020 Jul

    Airbyte started

    Michel Tricot and John Lafleur began building Airbyte as an open-source data integration project. The goal was to replace hand-written per-source extraction code with reusable connectors.

  14. 2020 Jul

    BigQuery Omni announced

    Omni extended BigQuery queries to data in Amazon S3 and Azure Blob Storage by running the engine in the other cloud, with no cross-cloud data movement.

  16. 2020 Sep

    First public release on GitHub

    Airbyte published version 0.1 with three source connectors and two destination connectors. It established the connector-plus-platform split the product still uses.

  17. 2020 Oct

    Unaligned checkpoints address backpressure delays

    The project documented unaligned checkpoints as a response to checkpoint alignment delays under backpressure. Checkpoint behavior became a more explicit latency and recovery design choice.

  19. 2020 Dec

    Airflow 2.0 releases

    Airflow 2.0 introduced the TaskFlow API, scheduler high-availability work, and separately released provider packages. Dag authors could use decorated Python functions while deployments selected providers for external systems independently of core.

  20. 2020 Dec

    Apache Flink 1.12 advances unified batch and streaming APIs

    The release continued Flink's work toward unified DataStream and Table APIs. Practitioners could use a closer programming surface for bounded and unbounded workloads.

  21. 2020 Dec

    PrestoSQL is renamed Trino

    The PrestoSQL project adopted the Trino name. The rename retained the project and community while separating its identity from Presto.

  22. 2021

    DuckDB Foundation incorporated

    An independent foundation takes custody of most project intellectual property so DuckDB remains MIT-licensed with long-term continuity separate from any single company.

  23. 2021 Jan

    The lakehouse architecture is published at CIDR

    The CIDR paper defined a lakehouse as data-lake files plus a metadata, caching, and indexing layer. It connected transaction and versioning mechanisms with SQL, data-science, and machine-learning workloads.

  24. 2021 Jan 3

    Release 351 is published

    Release 351 was published as an early Trino release. Its release notes record the continuing release stream after the rename.

  25. 2021 Mar

    DVC 2.0 adds lightweight experiments

    DVC 2.0 introduced Git-reference-based experiments, model checkpoint versioning, pipeline templating, and DVCLive metrics logging. Artifact versions could now remain connected to rapid experiment states without ordinary branches for every run.

  26. 2021 Mar

    Seed funding

    Airbyte raised a 5.2 million dollar seed round led by Accel. The funding supported building the core team and connector catalog.

  27. 2021 Mar 12

    Shopify documents CDC coupling consumers to internal models

    Shopify's engineering blog described its move to log-based change data capture and named its main drawback: external consumers become tightly coupled to the source's internal data model, so breaking changes spread into every consumer.

  28. 2021 Apr

    Kafka 2.8.0 introduces early KRaft access

    Kafka 2.8.0 introduced early access to replacing ZooKeeper with a self-managed quorum. This began the shift toward operating a controller quorum as part of Kafka itself.

  29. 2021 Apr 8

    Andrew Jones proposes data contracts

    Writing about GoCardless, Andrew Jones proposed that producers emit events designed for consumers, with a strongly defined schema that is documented, versioned, and protected from breaking changes. He called that schema a data contract.

  30. 2021 May

    Apache Flink 1.13 releases

    Flink 1.13 extended the runtime and Table API ecosystem for streaming applications. It continued the platform's progression after the earlier API-unification work.

  31. 2021 May

    Unity Catalog is introduced

    Databricks introduced Unity Catalog for fine-grained governance of data and artificial intelligence assets across workspaces.

  32. 2021 May 25

    Series A funding

    Airbyte announced a 26 million dollar Series A led by Benchmark. The round funded development toward a hosted service.

  33. 2021 Jun

    DVC Studio launches

    DVC Studio added a web interface for experiment tables, metrics, parameters, and collaboration based on Git and DVC project history. Teams gained a shared view without placing large data or models in Git.

  35. 2021 Jun

    Table snapshots enter preview

    Table snapshots offered low-cost, read-only point-in-time copies of tables, turning audit and restore patterns into a built-in feature; they reached GA the same October.

  36. 2021 Jul 14

    DuckDB Labs announced

    The core engineering team forms DuckDB Labs as a CWI spin-off for full-time development and commercial support around the open-source engine.

  37. 2021 Sep

    ClickHouse, Inc. incorporates and raises

    ClickHouse, Inc. incorporates in Delaware with an initial $50M investment from Index Ventures and Benchmark, followed in October by a $250M Series B at a $2B valuation.

  38. 2021 Sep

    Flink 1.14 advances mixed bounded and unbounded jobs

    Flink 1.14 supported checkpoints after tasks finish and a final checkpoint for bounded inputs, enabling mixed bounded and unbounded applications to commit sink results correctly.

  40. 2021 Sep

    Kafka 3.0 advances KRaft metadata management

    Kafka 3.0 introduced KRaft metadata and API improvements while the mode was still not recommended for production. Operators began planning for a controller model without ZooKeeper.

  41. 2021 Sep

    Kafka 3.0.0 adds KRaft metadata snapshots

    Kafka 3.0.0 added metadata-topic snapshots and other self-managed quorum improvements. KRaft operation became a more concrete path for clusters evaluating their metadata plane.

  42. 2021 Oct

    Airflow 2.2 releases

    Airflow 2.2.0 released as the project continued the 2.x series. The release history records it as the 2021 major release before the later data-aware scheduling work.

  43. 2021 Oct

    Comcast publishes JVM production lessons

    A production report connected uneven task CPU time and JVM compilation behavior to slow queries. It demonstrated why worker-level observations matter alongside SQL plans.

  44. 2021 Oct

    Storage Write API becomes generally available

    The stream-based write API with exactly-once delivery semantics reached GA, replacing the legacy streaming path for high-throughput ingestion.

  45. 2021 Nov 16

    Elementl announced its Series A

    The company behind Dagster announced a 14 million dollar Series A. The funding supported development of the open-source project and its commercial service.

  47. 2021 Dec 17

    Series B funding

    Airbyte closed a 150 million dollar Series B led by Altimeter Capital and Coatue Management, at a reported 1.5 billion dollar valuation.

  48. 2022

    Cloud service early access opens

    The company launches the early access program for its managed cloud service, opening the storage-and-compute management layer that ClickHouse Cloud automates today.

  49. 2022 Apr

    BigLake unifies lakes and warehouses

    BigLake launched with tables over GCS, S3, and Azure storage, enforcing fine-grained security consistently across BigQuery and open-source engines.

  51. 2022 Apr 7

    Airbyte Cloud and the Grouparoo acquisition

    Airbyte launched its hosted Cloud service and acquired Grouparoo, an open-source reverse-ETL company, adding a path to send warehouse data back out to operational tools.

  52. 2022 Jun

    The DVC extension for Visual Studio Code launches

    The extension brought dataset and model management, experiment execution, metrics, and plots into the editor. DVC workflows gained a graphical path that still used project metadata and local artifacts.

  53. 2022 Jul 16

    0.14.0 adds REST catalog client and branching metadata

    Release 0.14.0 introduced a common REST catalog client with change-based commits and metadata APIs for table branches and tags. Catalog interoperability and named refs became first-class project features.

  54. 2022 Aug

    Unity Catalog reaches general availability

    Unity Catalog became generally available, making its governed namespace and access model a stable platform foundation.

  55. 2022 Aug 5

    Dagster 1.0 released

    Dagster marked its core abstractions as stable. Assets, ops, graphs, jobs, schedules, and sensors could then be adopted without the pre-1.0 breaking-change expectation.

  56. 2022 Aug 9

    Dagster Cloud reached general availability

    The managed service became generally available alongside the 1.0 era. Teams could use a hosted control plane instead of operating every deployment component themselves.

  57. 2022 Aug 22

    The Rise of Data Contracts

    Chad Sanderson described production database schemas used by ELT and CDC tools as non-consensual APIs and presented data contracts as API-like agreements between service engineers and data consumers.

  58. 2022 Sep

    Airflow 2.4 introduces data-aware scheduling

    Airflow 2.4.0 released as the project's data-aware release. It made dataset-based scheduling a first-class direction alongside time-based schedules.

  59. 2022 Sep

    DuckDB 0.5.0 release line

    The 0.5 series continues the named-release cadence as the engine moves through wider production adoption ahead of a stability-focused 1.0.

  60. 2022 Oct

    Kafka 3.3 marks KRaft ready for new clusters

    Kafka 3.3 marked KRaft production ready for new clusters. The guidance still distinguished new deployments from migration of existing ZooKeeper-based clusters.

  61. 2022 Oct

    KRaft becomes production ready for new clusters

    Kafka 3.3.1 documented KRaft mode as production ready for new clusters, making controller and broker role design a standard operational concern.

  62. 2022 Nov

    Long-term partnership with MotherDuck

    DuckDB Labs partners with MotherDuck to provide a managed cloud path for shared storage, collaboration, and operations on top of DuckDB.

  63. 2022 Nov 3

    1.0.0 guarantees public API stability

    The 1.0.0 release committed to semantic versioning for the public iceberg-api module. Integrators could treat the Java API as stable across 1.x.

  64. 2022 Nov 28

    1.1.0 exposes branches and tags to engines

    Release 1.1.0 let Spark and related paths read and write named branches and tags. Write-audit-publish style workflows moved from metadata API only into engine-facing operations.

  65. 2023

    ISO publishes the 2023 SQL framework edition

    ISO/IEC 9075-1:2023 published the sixth framework edition and references SQL property graph queries as part of the standard family. SQL continues to grow through coordinated parts rather than one vendor's dialect alone.

  66. 2023

    Spark 3.4 expands Spark Connect

    Spark 3.4 adds Spark Connect client API coverage in Python. Client applications can interact with a remote Spark service through a decoupled interface.

  67. 2023 Feb

    Kafka 3.4 introduces early-access migration

    Kafka 3.4 added early-access migration from ZooKeeper to KRaft. The feature was intended for testing rather than production use at that point.

  68. 2023 Feb

    Kafka 3.4 supports ZooKeeper-to-KRaft migration

    Kafka 3.4 added the migration path from ZooKeeper mode to KRaft mode, turning the architectural transition into an upgrade-planning concern for existing clusters.

  69. 2023 Feb

    Version-aware cloud remotes are documented

    DVC added a workflow that records provider version identifiers for data already protected by cloud object versioning. This created an alternative to the traditional hash-shaped remote layout for supported storage.

  70. 2023 Mar

    BigQuery editions replace flat-rate pricing

    Three compute tiers, Standard, Enterprise, and Enterprise Plus, introduced capacity-based pricing with autoscaling and per-second slot billing, closing flat-rate commitments to new purchases.

  73. 2023 Apr 27

    dbt Core 1.5 adds model contracts

    dbt Core 1.5 introduced model governance features, including access, contracts, and versions. An enforced contract makes dbt check column names and data types before building a model.

  74. 2023 May 1

    PayPal open-sources its data contract template

    PayPal released its data contract template, used in its data mesh implementation, under the Apache 2.0 license. Version 2.1.1 covered schema, data quality, pricing, stakeholders, roles, and service levels, not only structure.

  75. 2023 May 18

    No-code Connector Builder

    Airbyte released the Connector Builder, an in-UI tool for creating API source connectors without a local development environment. It also underpins the low-code declarative connector format.

  76. 2023 May 24

    Elementl announced its Series B

    The company announced a 33 million dollar Series B. The announcement reported total funding of 48.8 million dollars.

  77. 2023 Jun

    DVC 3.0 is released

    DVC 3.0 expanded partial dataset modification, cloud-versioned imports, the DVC filesystem API, and transfer performance. The release kept data versioning at the core while broadening command-line, Python, editor, and web access.

  78. 2023 Jun

    Kafka 3.5.0 deprecates ZooKeeper mode

    Kafka 3.5.0 marked ZooKeeper as deprecated and described ZooKeeper-to-KRaft migration as early access. Upgrade planning now needed a migration decision, not only a broker rollout.

  79. 2023 Jul 27

    The template becomes the Open Data Contract Standard

    Version 2.2.0 renamed the template to the Open Data Contract Standard and finalized its fork under the AIDA User Group, turning a company template into a community specification.

  80. 2023 Aug 21

    Elementl became Dagster Labs

    The company renamed itself Dagster Labs. The change aligned the company identity with the open-source project and managed product.

  81. 2023 Aug 29

    Destinations V2 and Typing and Deduping

    Airbyte replaced dbt-based basic normalization with destination-native Typing and Deduping. Streams now map one-to-one to final tables, and a bad value is nulled and logged in _airbyte_meta rather than failing the sync.

  82. 2023 Sep 12

    First public release of the Data Contract Specification

    The datacontract.com project published version 0.9.0 of its own YAML format for data contracts, following OpenAPI and AsyncAPI conventions.

  83. 2023 Oct 4

    1.4.0 lands and v2 becomes the usual default

    Iceberg 1.4.0 shipped a large feature set. From 1.4.0 onward, new tables default to format-version 2, making delete-file capable tables the common baseline for fresh deployments.

  84. 2023 Oct 11

    Dagster University launched

    Dagster Labs introduced a free self-guided learning platform. Dagster Essentials was its first course.

  85. 2023 Nov 30

    Bitol joins the LF AI & Data Foundation

    The Bitol project, home of ODCS, became a sandbox project of the Linux Foundation AI & Data Foundation, placing the standard under vendor-neutral governance.

  86. 2024

    MotherDuck contributes to Node.js client work

    Ecosystem collaboration continues as MotherDuck contributes substantially to the DuckDB Node.js client later in 2024.

  87. 2024 Feb 27

    PyAirbyte released

    Airbyte published PyAirbyte, a Python library that runs connectors locally with a DuckDB cache and no Airbyte server. It made connectors usable directly from scripts and notebooks.

  89. 2024 Apr 17

    Dagster+ launch event held

    Dagster Labs presented Dagster+ as the managed product identity. The name now distinguishes the managed offering from the open-source engine used in this course.

  90. 2024 May

    RFC 9557 adds named time zones to Internet timestamps

    RFC 9557 extended RFC 3339 timestamps with additional information, including a bracketed IANA time-zone name. It lets a representation carry both an observed offset and the rules intended for local-time interpretation.

  91. 2024 Jun

    Docker Compose deployment deprecated

    Airbyte announced it would end support for Docker Compose deployments in favor of Kubernetes, with the new abctl tool running a local kind cluster for single-machine installs.

  93. 2024 Jun

    NumPy 2.0 is released

    NumPy 2.0 marked a major release of the numerical layer beneath many Python analysis workflows.

  94. 2024 Jun 3

    DuckDB 1.0.0 released

    Version 1.0 emphasizes storage backward compatibility and a more cautious SQL and C API change policy so applications can depend on clearer compatibility expectations.

  95. 2024 Jul 25

    Release 453 is published

    Release 453 appears in the official release archive. The release cadence shows the project continuing to evolve after the Trino rename.

  96. 2024 Aug

    Airflow 2.10 releases

    Airflow 2.10.0 released in August 2024 with dataset improvements and the Hybrid Executor. The release showed continued investment in data-aware workflows and execution choices in the 2.x line.

  97. 2024 Aug

    DCAT 3 becomes a W3C Recommendation

    DCAT 3 updated the catalog vocabulary for additional use cases. It remains a current standards reference for interoperable catalog metadata.

  98. 2024 Aug

    VLDB 2024 paper published

    The first ClickHouse research paper appears at VLDB, documenting the MergeTree storage layer, merge-time transformation, pruning techniques, and vectorized query execution; the docs' architecture overview is its web version.

  99. 2024 Sep 24

    Airbyte 1.0

    Airbyte marked its platform 1.0 and made Self-Managed Enterprise generally available. The release signaled a stable API and upgrade path for self-hosted users.

  100. 2024 Oct 21

    ODCS v3.0.0 generalizes the schema model

    ODCS v3 replaced the table-and-column schema with objects and properties, so one format can describe relational, hierarchical, and streaming data, and added the servers and support sections. A technical steering committee from 14 companies backed the release.

  101. 2024 Oct 31

    Declarative Automation became stable

    Dagster 1.9 marked Declarative Automation as stable. AutomationCondition became the mature asset-state mechanism alongside schedules and sensors.

  102. 2024 Nov

    Kafka 3.9.0 adds dynamic KRaft quorums

    Kafka 3.9.0 introduced dynamic KRaft controller membership and was the final major release with ZooKeeper mode. Controller membership became an administrative operation.

  103. 2025

    Series C at the first OpenHouse conference

    ClickHouse announces $350M Series C financing at its inaugural user conference, OpenHouse, in San Francisco, marking the company's second major funding round after incorporation.

  104. 2025

    Spark 4.0 broadens Spark Connect and SQL

    Spark 4.0 begins the 4.x line with expanded Spark Connect and SQL capabilities. Client-server usage and structured query features continue to evolve together.

  105. 2025 Mar

    DuckDB Local UI introduced

    DuckLabs and MotherDuck ship a browser-based local UI via the ui extension, with an explicit path to connect to MotherDuck when cloud features are needed.

  107. 2025 Mar

    Kafka 4.0.0 removes ZooKeeper mode

    Kafka 4.0.0 became the first major release to operate entirely without ZooKeeper. KRaft mode became the default operational model and upgrade paths needed to account for it.

  108. 2025 Apr

    Airflow 3.0 becomes generally available

    Airflow 3.0 introduced the Task Execution Interface and API-server architecture, Dag versioning, scheduler-managed backfills, and data assets. These changes separated task execution more clearly from the metadata-database control plane.

  109. 2025 May

    DuckLake format introduced

    DuckLabs introduces DuckLake, keeping lakehouse catalog metadata in a SQL database while storing table data as open Parquet.

  110. 2025 Sep

    Airflow 3.1 releases

    Airflow 3.1.0 added human-in-the-loop workflows, deadline alerts, internationalization, and a React plugin system. It extended the Airflow 3 platform after the 3.0 architecture transition.