feat(expr-ir): Support `DataFrame.pivot` #3373

dangotbanned · 2025-12-27T23:10:11Z

Description

Chapter 6 gets us closer to the recently-introduced DslPlan::Pivot.

But you're probably reading this because the PR finally adds support for pivot on pyarrow>=20 😄

Important

This was a pretty complicated one

After I figured out how pyarrow.compute.pivot_wider worked, I discovered it didn't natively support:

Any kind of aggregate_function
Multiple on columns

So ... after quite a journey, I've got those working too 🥳

Related issues

Can be adapted to close [Enh]: Add pivot to Arrow #2179
Child of feat(RFC): A richer Expr IR #2572
Follows feat(expr-ir): Add concat #3371
Towards oh-nodes...expr-ir/logical-plan
GH-45269: [C++][Compute] Add "pivot_wider" and "hash_pivot_wider" functions apache/arrow#45562
[C++][Python] Incorrect results from hash_pivot_wider apache/arrow#48679

`pivot` will need to check from `options` and either `dataframe`, `functions`, or `group_by`

Towards #2179

Still needs a lot of work, but it is definitely possible

narwhals/_plan/compliant/dataframe.py

narwhals/_plan/dataframe.py

narwhals/_plan/arrow/dataframe.py

narwhals/_plan/arrow/acero.py

Fell out of #3373 (comment)

Found some more errors on the rust-side Probably gonna rewrite this to avoid checking all the paths this exhaustively

Working on a repro, have found this doesn't trigger any exotic paths

- Leaving `Selector`s out for now - but `list[str]` was much narrower than what is already supported

`mypy` wasn't happy with the typing after I switched to `chain` - so no need point in #3373 (comment)

Opened apache/arrow#48679

dangotbanned added 30 commits December 24, 2025 19:13

refactor: Move compatibility flags

9542273

`pivot` will need to check from `options` and either `dataframe`, `functions`, or `group_by`

feat(DRAFT): Start adding ArrowDataFrame.pivot

f4b5778

Towards #2179

feat(DRAFT): Handle unnest-ing the struct column

b96f9b2

feat(DRAFT): Add most of DataFrame.pivot

42e76db

add the bits that work so far

366e951

test: start adding pivot_test

efea337

test: Add example from docstring

0dde622

fix: Support multiple index columns

2c69502

feat: Support ArrowDataFrame.pivot(values: list[str])

4079b74

to-done

3bd4e50

test: Add most of the remaining tests

2091c91

test: Add all remaining tests

86469e4

test: Remove aggregation from sort_columns tests

c1f86e4

feat: Support DataFrame.pivot(sort_columns=True)

8761c7c

test: Use fixtures instead of globals

aac49f8

test: Split up test_pivot_on_multiple_names_out

cac30bc

test: Use more distinct names/values

e298c06

test: Make .columns-only tests more visible

1981613

test: 🧹🧹🧹

2e54da3

feat(DRAFT): Support pivot(on: list[str], values: str)

8194b49

Still needs a lot of work, but it is definitely possible

oops, didn't mean to commit that

7d20578

"fix" concat_str typing

9795db8

fix: Short-circuit correctly in sort fastpath

46d6a1d

cov

7c77975

feat: Support DataFrame.pivot(on: list[str], values: list[str])

30a118b

docs: Explain the funky stuff

76fbe74

refactor: Move cast->collect into options.pivot_wider

4941455

todos

8e59131

feat: Support ArrowDataFrame.pivot_agg(on: str)

25c0b40

refactor: Add a dedicated acero wrapper

b3d8c73