Skip to content

fix(#426): insert_arrow and insert_dataframe check a uint64 column before the first batch is written - #427

Open
shaunpatterson wants to merge 2 commits into
ArcadeData:mainfrom
shaunpatterson:fix/insert-arrow-uint64-range
Open

shaunpatterson wants to merge 2 commits into
ArcadeData:mainfrom
shaunpatterson:fix/insert-arrow-uint64-range

Conversation

@shaunpatterson

@shaunpatterson shaunpatterson commented Nov 21, 2026 •

Copy link
Copy Markdown

What does this PR do?

insert_arrow and insert_dataframe checked a uint64 column for a value above 2^63 - 1 as each batch was converted. A bad value in a later batch left the earlier batches committed, and on the JVM wheel a batch that held a null went down the Python-ints path and raised OverflowError. how-to/dataframes-and-columns.adoc ("raises ValueError", "refused before anything is written") and the insert_columns docstring promise a ValueError with nothing written; the insert_dataframe docstring did not list the case, and now does.

insert_table now calls _check_uint64_range once on the whole table before the first batch, for a plain and a dictionary-encoded uint64 column, with the message insert_columns already uses. The per-batch check in _column_values (native only) is removed, since it can no longer fire; insert_columns keeps its own check for direct callers.

Related issues

Closes #426

Performance changes

One pyarrow.compute.max per uint64 column over the whole table, a single vectorized pass that replaces the per-batch one on the native wheel. Nothing else changes for other column types.

Docs

The ValueError entry of the insert_dataframe docstring (which insert_arrow refers to) now lists the uint64 range and says it is raised before anything is written. how-to/dataframes-and-columns.adoc already described this behavior; the code now matches.

Checklist

  • A regression test fails on the old code and passes on the new (test_a_uint64_past_the_signed_range_in_a_late_batch_is_refused_before_any_row_is_written, with and without a null, plus a dictionary-encoded column and the largest allowed value; on the published native wheel 26.11.1.post4 with the change applied by hand, the two late-batch cases fail without it and the 163 tests of the three files pass with it)
  • The test suite that the change can affect passes (test_dataframe_columns_behavior.py, test_dataframe_ingest.py, test_insert_columns.py, test_arrow_temporal_column_jvm.py, test_arrow_vector_column_jvm.py: 200 tests on the JVM engine)
  • uvx pre-commit run --files <changed files> passes

… column before the first batch is written

The range check ran as each batch was converted, so a bad value in a later batch left the earlier ones
committed, and on the JVM wheel a batch with nulls raised OverflowError instead of ValueError. The whole
table is checked up front, dictionary-encoded columns included.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

insert_arrow/insert_dataframe: a uint64 value beyond the signed range is refused after earlier batches were committed

2 participants