Skip to content

GH-37219: [C++][Dataset] Preserve field nullability through casts - #55720

Open
lll-hhh wants to merge 1 commit into
apache:mainfrom
lll-hhh:fix/cast-projection-nullability-37219
Open

lll-hhh wants to merge 1 commit into
apache:mainfrom
lll-hhh:fix/cast-projection-nullability-37219

Conversation

@lll-hhh

@lll-hhh lll-hhh commented Nov 22, 2026

Copy link
Copy Markdown

Rationale for this change

Fixes #37219.

ProjectionDescr::FromExpressions copies field nullability and metadata only for a bare field_ref. A projection such as dataset.field("id").cast(int64()) therefore produced a nested field with the default nullable=true and dropped the source field's metadata, even though a safe cast preserves the source values and cannot introduce nulls when the source is required.

What changes are included in this PR?

  • When building a projection, follow a chain of single-argument cast calls back to its field reference.
  • Preserve the source field's nullability and metadata while still using the cast's output type.
  • Add a regression covering required int32 -> int64 (including metadata) and optional int16 -> float64.

Are these changes tested?

Yes, against an editable Arrow C++ 27.0.0-SNAPSHOT build in WSL.

  • A standalone C++ harness against the unmodified base printed nullable=1 metadata_preserved=0.
  • The same harness against the changed library printed nullable=0 metadata_preserved=1.
  • The arrow-dataset-scanner-test test translation unit compiles successfully with C++20.
  • The Arrow C++ shared library built and installed successfully.
  • clang-format==18.1.8 --dry-run --Werror and git diff --check passed.

The full local test executable was not linked because this minimal build lacked the optional Boost test dependency; the focused upstream regression is included for CI.

Are there any user-facing changes?

Yes. Dataset projections that cast a field now retain that field's declared nullability and metadata. The output data type, values, and cast behavior are unchanged.

Was AI used for this PR?

In accordance with the AI generation guidelines, AI was used for implementation, tests, and description. The behavior was reproduced against the unmodified base and validated against a local Arrow build before submission.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

Copilot AI balanced review requested due to automatic review settings November 22, 2026 10:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Python] Nullability not maintained when casting a column

2 participants