Repository navigation
fix(#426): insert_arrow and insert_dataframe check a uint64 column before the first batch is written - #427
Open
shaunpatterson wants to merge 2 commits into
Conversation
… column before the first batch is written The range check ran as each batch was converted, so a bad value in a later batch left the earlier ones committed, and on the JVM wheel a batch with nulls raised OverflowError instead of ValueError. The whole table is checked up front, dictionary-encoded columns included.
…he ValueErrors raised before a write
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
insert_arrowandinsert_dataframechecked a uint64 column for a value above 2^63 - 1 as each batch was converted. A bad value in a later batch left the earlier batches committed, and on the JVM wheel a batch that held a null went down the Python-ints path and raisedOverflowError.how-to/dataframes-and-columns.adoc("raisesValueError", "refused before anything is written") and theinsert_columnsdocstring promise aValueErrorwith nothing written; theinsert_dataframedocstring did not list the case, and now does.insert_tablenow calls_check_uint64_rangeonce on the whole table before the first batch, for a plain and a dictionary-encoded uint64 column, with the messageinsert_columnsalready uses. The per-batch check in_column_values(native only) is removed, since it can no longer fire;insert_columnskeeps its own check for direct callers.Related issues
Closes #426
Performance changes
One
pyarrow.compute.maxper uint64 column over the whole table, a single vectorized pass that replaces the per-batch one on the native wheel. Nothing else changes for other column types.Docs
The
ValueErrorentry of theinsert_dataframedocstring (whichinsert_arrowrefers to) now lists the uint64 range and says it is raised before anything is written.how-to/dataframes-and-columns.adocalready described this behavior; the code now matches.Checklist
test_a_uint64_past_the_signed_range_in_a_late_batch_is_refused_before_any_row_is_written, with and without a null, plus a dictionary-encoded column and the largest allowed value; on the published native wheel 26.11.1.post4 with the change applied by hand, the two late-batch cases fail without it and the 163 tests of the three files pass with it)test_dataframe_columns_behavior.py,test_dataframe_ingest.py,test_insert_columns.py,test_arrow_temporal_column_jvm.py,test_arrow_vector_column_jvm.py: 200 tests on the JVM engine)uvx pre-commit run --files <changed files>passes