Repository navigation
Release 3.4.0: background file imports and SaaS connections (db.imports, db.connections) - #32
Merged
Merged
Conversation
…ts, db.connections) db.imports creates an import, uploads its files (S3 multipart parts to presigned URLs on a thread pool, parallelism 4, each part retried, or the bytes through the API), previews the mapping with personal-data findings, starts the run and follows it (get, list, wait, cancel, retry, delete); import_files does it in one call with mappings, links between files and progress. Files can be paths, bytes or binary file objects; paths and seekable files are read per part, so large files never sit in memory. db.connections lists the connector catalog and push templates and manages connections: create, update, delete(delete_data=), test, preview, fields, review_fields, sync, runs, pause, resume and rotate_webhook_token. Agent connections return their token once. Both are RushDB Cloud only: in direct mode every method raises CloudOnlyError before sending anything. Times are in seconds.
1pxone
force-pushed
the
feat/imports-and-connections
branch
from
November 12, 2026 16:38
86d02af to
0bf1dc1
Compare
1pxone
marked this pull request as ready for review
November 12, 2026 16:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Python parity with rush-db/javascript-sdk#19: background file imports (
db.imports) and SaaS connections (db.connections) for the platform's new data sources (rush-db/platform#36). Both are RushDB Cloud only.Before publishing
/api/v1/imports,/connectionsand/connectors/specs.versionlines inpyproject.tomlanduv.lockwill conflict; keep 3.4.0.import_fileswith a CSV path and a gzipped NDJSON file object.connections.create→test→sync→imports.waitwith a Stripe test key.v3.4.0to publish.What
db.imports:create,add_files,remove_file;upload, which follows the server's upload mode:s3-multipart: parts go to presigned URLs on a thread pool (parallelism=4), each part is retried (retries=3, backoff 0.25 s × 2ⁿ up to 8 s), and the upload completes with the ETags;direct: the bytes go through the API;preview,start,get,list(kind=, connection_id=, limit=, skip=),wait(interval=, timeout=),cancel,retry,delete;import_files: create → upload → start → optional wait in one call, withmappings(keyed by file name or position),linksbetween files andon_progress.db.connections:catalog,push_templates;list,get,create(params),update(id, params),delete(id, delete_data=False);test,preview,sync(streams=, full=),runs,pause,resume,rotate_webhook_token;fieldsandreview_fields;connectorId: "agent") returnagentTokenonce.rushdb/models/ingest.py, plusFINISHED_IMPORT_RUN_STATUSES.RushDB._send_rawsends a raw body to an absolute URL with only the headers the caller passes. It uses the standard library only; the package still has no runtime dependencies.Differences from the TypeScript SDK
str/Path),bytes/bytearray/memoryview, or a binary file object. Paths and seekable files are read per part inside the worker threads, so a large file never sits in memory. A non-seekable stream is read in order, needs itssize, and is not retried when uploaded directly.interval,timeout,wait_interval,wait_timeout). TypeScript uses milliseconds.import_files: a plainwait: bool, pluswait_intervalandwait_timeout.waittimeout raisesTimeoutError;ValueErrorbefore anything is sent;RushDBErrorwith.status;Content-Type: application/octet-stream, becauseurllibwould otherwise add a form content type. The platform's presigned part URLs don't sign that header.CLOUD_ONLY_METHODS.Testing
tests/test_imports.pyandtests/test_connections.py(new, mocked HTTP) cover:waitand its timeout, andimport_fileswith mappings and links;tests/test_direct_mode.pychecks that every new method raisesCloudOnlyErrorwithout sending anything.Content-Length.