Advanced SQL for Analytics
Advanced SQL for analytics is the set of query techniques - window functions, multi-level grouping, recursive queries, and set operations - that compute rankings, running totals, and period comparisons directly inside a relational database, instead of in application code.
itData engineering and analytics | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Intro
Advanced SQL for Analytics
Basic SQL retrieves and filters rows. Advanced SQL for analytics computes across
rows: rankings, running totals, period-over-period comparisons, subtotal
hierarchies, and reshaped result sets, all inside the query itself instead of
in application code or a spreadsheet. The techniques in this course - window
functions, QUALIFY, GROUPING SETS/ROLLUP/CUBE, common table expressions
including recursive ones, set operations, and PIVOT/UNPIVOT - are what
separate a query that lists facts from one that answers an analytical
question directly.
Window functions: calculating across related rows without collapsing them
A GROUP BY aggregate collapses many rows into one row per group. A window
function keeps every row and adds a value computed from a set of related
rows. The OVER clause is what makes a function a window function; it is
required, and it is the syntax that tells the engine to keep rows individual
while still calculating across them.
SELECT depname, empno, salary,
avg(salary) OVER (PARTITION BY depname) AS dept_avg
FROM empsalary;
PARTITION BY divides the rows into groups that share the same values,
similar to GROUP BY, but each row keeps its own columns alongside the
computed value. ORDER BY inside OVER sets the logical order the function
processes rows in; it does not change the order of the query's output.
Adding ORDER BY also changes which rows the function sees by default: with
an order, the default window is every row from the start of the partition
through the current row - useful for running totals - while omitting ORDER BY uses the entire partition for every row.
Window functions fall into three groups:
Continue the course
This section is part of the paid course.
See pricing to subscribe, or log in if you already have access.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://www.postgresql.org/docs/current/tutorial-window.html
Supports
- Window function definition
- OVER clause requirement
- PARTITION BY
- ORDER BY inside OVER
- default frame behavior with and without ORDER BY
- WINDOW clause for naming a shared specification
- https://www.postgresql.org/docs/current/functions-window.html
Supports
- Full built-in window function catalog
- which functions require ORDER BY
- ranking versus navigation versus aggregate-as-window categorization
- https://www.postgresql.org/docs/current/queries-table-expressions.html
Supports
- GROUPING SETS syntax and semantics
- ROLLUP and CUBE as GROUPING SETS shorthand
- empty grouping set
- NULL substitution for absent columns
- composite grouping elements
- GROUP BY DISTINCT
- https://www.postgresql.org/docs/current/queries-with.html
Supports
- CTE basic syntax
- WITH RECURSIVE structure
- iterative (not true-recursive) evaluation algorithm
- SEARCH DEPTH/BREADTH FIRST
- CYCLE clause
- hierarchical/tree traversal examples
- discard-duplicates-on-UNION behavior of the non-recursive term
- https://www.postgresql.org/docs/current/sql-select.html
Supports
- UNION
- INTERSECT
- EXCEPT syntax
- DISTINCT-by-default and ALL behavior
- INTERSECT ALL and EXCEPT ALL multiplicity formulas
- column-count and type-compatibility requirement
- operator precedence
- https://www.postgresql.org/docs/8.4/release.html
Supports
- PostgreSQL 8.4 release notes confirming window functions and WITH RECURSIVE were added in that release
- July 2009
- https://dev.mysql.com/doc/refman/8.4/en/window-functions.html
Supports
- MySQL window function support
- OVER/PARTITION BY/frame syntax
- aggregate versus non-aggregate window functions
- named windows
- https://dev.mysql.com/doc/refman/8.4/en/with.html
Supports
- MySQL WITH RECURSIVE syntax
- recursive-term restrictions (no aggregates
- window functions
- GROUP BY
- ORDER BY
- DISTINCT; single FROM-clause reference; no right side of LEFT JOIN)
- cte_max_recursion_depth default of 1000 and its effect
- https://dev.mysql.com/blog-archive/mysql-8-0-ga-is-here/
Supports
- MySQL 8.0 general availability date
- April 19 2018
- https://learn.microsoft.com/en-us/sql/t-sql/queries/select-over-clause-transact-sql
Supports
- OVER clause syntax
- PARTITION BY
- ORDER BY
- ROWS/RANGE frame clause syntax and defaults
- ROWS versus RANGE distinction
- applicability of ROWS/RANGE starting SQL Server 2012
- supporting-index and batch-mode performance guidance
- https://learn.microsoft.com/en-us/sql/t-sql/queries/with-common-table-expression-transact-sql
Supports
- SQL Server CTE syntax without a RECURSIVE keyword
- automatic detection of self-reference
- anchor and recursive member rules
- UNION ALL requirement between last anchor and first recursive member
- MAXRECURSION hint default of 100 and range 0-32767
- analytic-function-in-recursion pitfall
- https://learn.microsoft.com/en-us/sql/t-sql/queries/from-using-pivot-and-unpivot
Supports
- SQL Server native PIVOT/UNPIVOT syntax and examples
- PIVOT replacing a SELECT-CASE pattern
- UNPIVOT not being an exact inverse of PIVOT
- NULL values disappearing in UNPIVOT output
- performance note on repeated PIVOT/UNPIVOT use
- https://learn.microsoft.com/en-us/archive/blogs/craigfr/ranking-functions-row_number
Supports
- SQL Server 2005 introduction of ROW_NUMBER
- RANK
- DENSE_RANK
- and NTILE ranking functions
- https://www.sqlservercentral.com/articles/common-table-expressions-in-sql-server-2005
Supports
- SQL Server 2005 introduction of common table expressions and the WITH clause
- https://docs.snowflake.com/en/sql-reference/constructs/qualify
Supports
- QUALIFY clause semantics
- its HAVING-for-window-functions framing
- execution order relative to WINDOW and DISTINCT
- requirement of at least one window function
- https://docs.snowflake.com/en/sql-reference/constructs/pivot
Supports
- Snowflake native PIVOT/UNPIVOT syntax
- the five supported aggregate functions (AVG
- COUNT
- MAX
- MIN
- SUM)
- static versus dynamic pivot
- https://docs.snowflake.com/en/sql-reference/functions-analytic
Supports
- Snowflake's categorization of analytic/window functions into general
- ranking
- and specialized aggregation groups
- https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax
Supports
- BigQuery QUALIFY clause purpose and its relationship to WHERE and HAVING
- https://cloud.google.com/blog/products/gcp/bigquery-111-now-with-standard-sql-iam-and-partitioned-tables
Supports
- BigQuery Standard SQL beta announcement
- June 2 2016
- https://docs.cloud.google.com/bigquery/docs/recursive-ctes
Supports
- BigQuery WITH RECURSIVE syntax
- anchor and recursive query structure
- fixed 500-iteration limit and abort-on-error behavior
- single self-reference restriction
- https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-qry-select-qualify.html
Supports
- Databricks SQL QUALIFY clause syntax
- requirement of a window function in SELECT or QUALIFY
- example filtering a RANK() result
- https://docs.aws.amazon.com/redshift/latest/dg/c_Window_functions.html
Supports
- Amazon Redshift window function syntax
- restriction to SELECT list and ORDER BY
- ranking versus aggregate window functions
- parallel-versus-serial execution note tied to PARTITION BY
- nondeterministic row order without a unique ORDER BY
- https://clickhouse.com/docs/sql-reference/window-functions
Supports
- ClickHouse window function support
- ROWS/RANGE frame types
- ranking and navigation function support
- lagInFrame/leadInFrame extensions
- real-time analytics use cases
- https://duckdb.org/docs/current/sql/query_syntax/qualify
Supports
- DuckDB QUALIFY clause semantics
- its position after WINDOW and before ORDER BY
- its purpose of avoiding a wrapping subquery
- https://docs.oracle.com/en/database/oracle/oracle-database/19/dwhsg/sql-analysis-reporting-data-warehouses.html
Supports
- Oracle Database analytic function categories (ranking
- windowing
- reporting
- LAG/LEAD
- FIRST/LAST
- linear regression
- inverse percentile
- hypothetical rank)
- three-stage query processing order
- RANK versus DENSE_RANK example
- IGNORE NULLS option
- https://docs.oracle.com/cd/A87860_01/doc/server.817/a76994/analysis.htm
Supports
- Oracle8i introduction of the analytic function family: ranking
- reporting
- windowing
- and LAG/LEAD functions
- https://github.com/sindresorhus/awesome
Supports
- Required discovery starting point and link to the Awesome DB Tools list
- https://github.com/mgramin/awesome-db-tools
Supports
- Discovery of SQLFluff
- pgFormatter
- DBeaver
- Beekeeper Studio
- Querybook
- and DataGrip
- https://sqlfluff.com/
Supports
- SQLFluff product identity
- dialect-flexible linting/fixing/parsing description
- dbt/Jinja templating support
- https://github.com/darold/pgFormatter
Supports
- pgFormatter product identity
- supported SQL standard versions
- formatting behavior
- web version availability
- https://dbeaver.com/
Supports
- DBeaver product identity and destination
- https://www.beekeeperstudio.io/
Supports
- Beekeeper Studio product identity
- open-source licensing
- list of 25+ supported databases including Snowflake
- BigQuery
- Redshift
- and DuckDB
- https://www.querybook.org/
Supports
- Querybook product identity
- Pinterest origin
- notebook-style DataDoc workflow
- table metadata and collaboration features
- https://www.jetbrains.com/datagrip
Supports
- DataGrip product identity and destination
- https://www.postgresql.org/
Supports
- PostgreSQL product identity and destination
- https://www.mysql.com/
Supports
- MySQL product identity and destination
- https://www.microsoft.com/en-us/sql-server
Supports
- Microsoft SQL Server product identity and destination
- https://www.oracle.com/database/
Supports
- Oracle Database product identity and destination
- https://www.snowflake.com/
Supports
- Snowflake product identity and destination
- https://cloud.google.com/bigquery
Supports
- Google BigQuery product identity and destination
- https://aws.amazon.com/redshift/
Supports
- Amazon Redshift product identity and destination
- https://www.databricks.com/product/databricks-sql
Supports
- Databricks SQL product identity and destination
- https://clickhouse.com/
Supports
- ClickHouse product identity and destination
- https://duckdb.org/
Supports
- DuckDB product identity and destination
