Advanced SQL for Analytics
Advanced SQL for analytics is the set of query techniques - window functions, multi-level grouping, recursive queries, and set operations - that compute rankings, running totals, and period comparisons directly inside a relational database, instead of in application code.
itData engineering and analytics | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Don't Panic
Don't Panic: Advanced SQL for Analytics
Advanced SQL for analytics is what happens when a query stops listing rows and starts answering a question. Rankings, running totals, subtotals, hierarchy walks, set comparisons, and reshaped reports can all happen inside the database. This is excellent news, provided you remember that the database is very literal and has never been impressed by your spreadsheet habits.
The load-bearing idea is the window function, which calculates across related rows while keeping each row in the result. A GROUP BY aggregate collapses rows. A window function adds context beside them. The OVER clause tells the engine how to partition rows, how to order them, and which frame of nearby rows the function can read.
The frame is where many surprises live. A running total wants rows from the start of the partition through the current row. LAST_VALUE often needs a wider frame, because the default frame may end at the current row. The name sounds confident. The frame has the receipts.
Filtering also has timing. Window functions are computed after WHERE, so a database cannot filter on a value that does not exist yet. Some engines provide QUALIFY, which filters window results directly. Others need the old reliable wrapper: compute in a subquery or CTE, then filter outside.
The rest of the course is a set of controlled superpowers. GROUPING SETS produce several subtotal levels in one result. Recursive CTEs walk trees without knowing their depth first. UNION, INTERSECT, and EXCEPT combine result sets, sometimes with deduplication you did not mean to ask for. PIVOT turns values into columns, which is useful and occasionally how reports become furniture.
Start with the Practice Reference when writing queries. Use the Exercise to build a layered report and check where each clause runs. Keep the Cheatsheet open for dialect differences, especially QUALIFY, frames, recursion guards, and set operators. The point is not fancy SQL. The point is a query whose analytical answer is still correct after every convenient shortcut has been interrogated.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://www.postgresql.org/docs/current/tutorial-window.html
Supports
- Window function definition
- OVER clause requirement
- PARTITION BY
- ORDER BY inside OVER
- default frame behavior with and without ORDER BY
- WINDOW clause for naming a shared specification
- https://www.postgresql.org/docs/current/functions-window.html
Supports
- Full built-in window function catalog
- which functions require ORDER BY
- ranking versus navigation versus aggregate-as-window categorization
- https://www.postgresql.org/docs/current/queries-table-expressions.html
Supports
- GROUPING SETS syntax and semantics
- ROLLUP and CUBE as GROUPING SETS shorthand
- empty grouping set
- NULL substitution for absent columns
- composite grouping elements
- GROUP BY DISTINCT
- https://www.postgresql.org/docs/current/queries-with.html
Supports
- CTE basic syntax
- WITH RECURSIVE structure
- iterative (not true-recursive) evaluation algorithm
- SEARCH DEPTH/BREADTH FIRST
- CYCLE clause
- hierarchical/tree traversal examples
- discard-duplicates-on-UNION behavior of the non-recursive term
- https://www.postgresql.org/docs/current/sql-select.html
Supports
- UNION
- INTERSECT
- EXCEPT syntax
- DISTINCT-by-default and ALL behavior
- INTERSECT ALL and EXCEPT ALL multiplicity formulas
- column-count and type-compatibility requirement
- operator precedence
- https://www.postgresql.org/docs/8.4/release.html
Supports
- PostgreSQL 8.4 release notes confirming window functions and WITH RECURSIVE were added in that release
- July 2009
- https://dev.mysql.com/doc/refman/8.4/en/window-functions.html
Supports
- MySQL window function support
- OVER/PARTITION BY/frame syntax
- aggregate versus non-aggregate window functions
- named windows
- https://dev.mysql.com/doc/refman/8.4/en/with.html
Supports
- MySQL WITH RECURSIVE syntax
- recursive-term restrictions (no aggregates
- window functions
- GROUP BY
- ORDER BY
- DISTINCT; single FROM-clause reference; no right side of LEFT JOIN)
- cte_max_recursion_depth default of 1000 and its effect
- https://dev.mysql.com/blog-archive/mysql-8-0-ga-is-here/
Supports
- MySQL 8.0 general availability date
- April 19 2018
- https://learn.microsoft.com/en-us/sql/t-sql/queries/select-over-clause-transact-sql
Supports
- OVER clause syntax
- PARTITION BY
- ORDER BY
- ROWS/RANGE frame clause syntax and defaults
- ROWS versus RANGE distinction
- applicability of ROWS/RANGE starting SQL Server 2012
- supporting-index and batch-mode performance guidance
- https://learn.microsoft.com/en-us/sql/t-sql/queries/with-common-table-expression-transact-sql
Supports
- SQL Server CTE syntax without a RECURSIVE keyword
- automatic detection of self-reference
- anchor and recursive member rules
- UNION ALL requirement between last anchor and first recursive member
- MAXRECURSION hint default of 100 and range 0-32767
- analytic-function-in-recursion pitfall
- https://learn.microsoft.com/en-us/sql/t-sql/queries/from-using-pivot-and-unpivot
Supports
- SQL Server native PIVOT/UNPIVOT syntax and examples
- PIVOT replacing a SELECT-CASE pattern
- UNPIVOT not being an exact inverse of PIVOT
- NULL values disappearing in UNPIVOT output
- performance note on repeated PIVOT/UNPIVOT use
- https://learn.microsoft.com/en-us/archive/blogs/craigfr/ranking-functions-row_number
Supports
- SQL Server 2005 introduction of ROW_NUMBER
- RANK
- DENSE_RANK
- and NTILE ranking functions
- https://www.sqlservercentral.com/articles/common-table-expressions-in-sql-server-2005
Supports
- SQL Server 2005 introduction of common table expressions and the WITH clause
- https://docs.snowflake.com/en/sql-reference/constructs/qualify
Supports
- QUALIFY clause semantics
- its HAVING-for-window-functions framing
- execution order relative to WINDOW and DISTINCT
- requirement of at least one window function
- https://docs.snowflake.com/en/sql-reference/constructs/pivot
Supports
- Snowflake native PIVOT/UNPIVOT syntax
- the five supported aggregate functions (AVG
- COUNT
- MAX
- MIN
- SUM)
- static versus dynamic pivot
- https://docs.snowflake.com/en/sql-reference/functions-analytic
Supports
- Snowflake's categorization of analytic/window functions into general
- ranking
- and specialized aggregation groups
- https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax
Supports
- BigQuery QUALIFY clause purpose and its relationship to WHERE and HAVING
- https://cloud.google.com/blog/products/gcp/bigquery-111-now-with-standard-sql-iam-and-partitioned-tables
Supports
- BigQuery Standard SQL beta announcement
- June 2 2016
- https://docs.cloud.google.com/bigquery/docs/recursive-ctes
Supports
- BigQuery WITH RECURSIVE syntax
- anchor and recursive query structure
- fixed 500-iteration limit and abort-on-error behavior
- single self-reference restriction
- https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-qry-select-qualify.html
Supports
- Databricks SQL QUALIFY clause syntax
- requirement of a window function in SELECT or QUALIFY
- example filtering a RANK() result
- https://docs.aws.amazon.com/redshift/latest/dg/c_Window_functions.html
Supports
- Amazon Redshift window function syntax
- restriction to SELECT list and ORDER BY
- ranking versus aggregate window functions
- parallel-versus-serial execution note tied to PARTITION BY
- nondeterministic row order without a unique ORDER BY
- https://clickhouse.com/docs/sql-reference/window-functions
Supports
- ClickHouse window function support
- ROWS/RANGE frame types
- ranking and navigation function support
- lagInFrame/leadInFrame extensions
- real-time analytics use cases
- https://duckdb.org/docs/current/sql/query_syntax/qualify
Supports
- DuckDB QUALIFY clause semantics
- its position after WINDOW and before ORDER BY
- its purpose of avoiding a wrapping subquery
- https://docs.oracle.com/en/database/oracle/oracle-database/19/dwhsg/sql-analysis-reporting-data-warehouses.html
Supports
- Oracle Database analytic function categories (ranking
- windowing
- reporting
- LAG/LEAD
- FIRST/LAST
- linear regression
- inverse percentile
- hypothetical rank)
- three-stage query processing order
- RANK versus DENSE_RANK example
- IGNORE NULLS option
- https://docs.oracle.com/cd/A87860_01/doc/server.817/a76994/analysis.htm
Supports
- Oracle8i introduction of the analytic function family: ranking
- reporting
- windowing
- and LAG/LEAD functions
- https://github.com/sindresorhus/awesome
Supports
- Required discovery starting point and link to the Awesome DB Tools list
- https://github.com/mgramin/awesome-db-tools
Supports
- Discovery of SQLFluff
- pgFormatter
- DBeaver
- Beekeeper Studio
- Querybook
- and DataGrip
- https://sqlfluff.com/
Supports
- SQLFluff product identity
- dialect-flexible linting/fixing/parsing description
- dbt/Jinja templating support
- https://github.com/darold/pgFormatter
Supports
- pgFormatter product identity
- supported SQL standard versions
- formatting behavior
- web version availability
- https://dbeaver.com/
Supports
- DBeaver product identity and destination
- https://www.beekeeperstudio.io/
Supports
- Beekeeper Studio product identity
- open-source licensing
- list of 25+ supported databases including Snowflake
- BigQuery
- Redshift
- and DuckDB
- https://www.querybook.org/
Supports
- Querybook product identity
- Pinterest origin
- notebook-style DataDoc workflow
- table metadata and collaboration features
- https://www.jetbrains.com/datagrip
Supports
- DataGrip product identity and destination
- https://www.postgresql.org/
Supports
- PostgreSQL product identity and destination
- https://www.mysql.com/
Supports
- MySQL product identity and destination
- https://www.microsoft.com/en-us/sql-server
Supports
- Microsoft SQL Server product identity and destination
- https://www.oracle.com/database/
Supports
- Oracle Database product identity and destination
- https://www.snowflake.com/
Supports
- Snowflake product identity and destination
- https://cloud.google.com/bigquery
Supports
- Google BigQuery product identity and destination
- https://aws.amazon.com/redshift/
Supports
- Amazon Redshift product identity and destination
- https://www.databricks.com/product/databricks-sql
Supports
- Databricks SQL product identity and destination
- https://clickhouse.com/
Supports
- ClickHouse product identity and destination
- https://duckdb.org/
Supports
- DuckDB product identity and destination
