Skip to content

Release 3.4.0: background file imports and SaaS connections (db.imports, db.connections) - #32

Merged
1pxone merged 1 commit into
mainfrom
feat/imports-and-connections
Nov 12, 2026
Merged

1pxone merged 1 commit into
mainfrom
feat/imports-and-connections

Conversation

@1pxone

@1pxone 1pxone commented Nov 11, 2026

Copy link
Copy Markdown
Member

Python parity with rush-db/javascript-sdk#19: background file imports (db.imports) and SaaS connections (db.connections) for the platform's new data sources (rush-db/platform#36). Both are RushDB Cloud only.

Before publishing

  • Platform first: rush-db/platform#36 must be merged and deployed to RushDB Cloud. These methods call /api/v1/imports, /connections and /connectors/specs.
  • Merge order: merge Release 3.3.0: catalog lookups (search_values, get_schema_summary, resolve_names, get_join_candidates) #31 (3.3.0) first, then rebase this PR. Only the version lines in pyproject.toml and uv.lock will conflict; keep 3.4.0.
  • CI should pass: black, isort, ruff, mypy, pytest on 3.10–3.13.
  • Try against Cloud once deployed:
    • import_files with a CSV path and a gzipped NDJSON file object.
    • An S3 multipart upload of a file larger than one part.
    • connections.create → test → sync → imports.wait with a Stripe test key.
  • Tag: after merging, tag v3.4.0 to publish.

What

  • db.imports:
    • create, add_files, remove_file;
    • upload, which follows the server's upload mode:
      • s3-multipart: parts go to presigned URLs on a thread pool (parallelism=4), each part is retried (retries=3, backoff 0.25 s × 2ⁿ up to 8 s), and the upload completes with the ETags;
      • direct: the bytes go through the API;
    • preview, start, get, list(kind=, connection_id=, limit=, skip=), wait(interval=, timeout=), cancel, retry, delete;
    • import_files: create → upload → start → optional wait in one call, with mappings (keyed by file name or position), links between files and on_progress.
  • db.connections:
    • catalog, push_templates;
    • list, get, create(params), update(id, params), delete(id, delete_data=False);
    • test, preview, sync(streams=, full=), runs, pause, resume, rotate_webhook_token;
    • field review: fields and review_fields;
    • agent connections (connectorId: "agent") return agentToken once.
  • Types: DTOs as TypedDicts in rushdb/models/ingest.py, plus FINISHED_IMPORT_RUN_STATUSES.
  • HTTP: RushDB._send_raw sends a raw body to an absolute URL with only the headers the caller passes. It uses the standard library only; the package still has no runtime dependencies.
  • README: a "File imports and connections (RushDB Cloud)" section, and a row in the Cloud vs self-hosted table.

Differences from the TypeScript SDK

  • Upload data: a file path (str/Path), bytes/bytearray/memoryview, or a binary file object. Paths and seekable files are read per part inside the worker threads, so a large file never sits in memory. A non-seekable stream is read in order, needs its size, and is not retried when uploaded directly.
  • Units: all times are in seconds (interval, timeout, wait_interval, wait_timeout). TypeScript uses milliseconds.
  • Waiting in import_files: a plain wait: bool, plus wait_interval and wait_timeout.
  • Errors:
    • a wait timeout raises TimeoutError;
    • bad input raises ValueError before anything is sent;
    • HTTP failures raise RushDBError with .status;
    • 429, 5xx and network errors are retried; other 4xx responses are not.
  • Content type on S3 parts: part uploads send Content-Type: application/octet-stream, because urllib would otherwise add a form content type. The platform's presigned part URLs don't sign that header.
  • Direct-mode guard: each method checks direct mode itself, before any upload touches storage, and all 29 methods are listed in CLOUD_ONLY_METHODS.

Testing

  • tests/test_imports.py and tests/test_connections.py (new, mocked HTTP) cover:
    • a multipart upload with a retried part, and a direct upload;
    • upload sources: path, bytes, seekable and non-seekable file objects;
    • wait and its timeout, and import_files with mappings and links;
    • connections CRUD, sync and field review.
  • tests/test_direct_mode.py checks that every new method raises CloudOnlyError without sending anything.
  • black, isort, ruff and mypy pass; pytest has 168 passing, with 58 live-server tests skipped as in CI.
  • Also checked by hand against a local HTTP server:
    • a multipart upload from a path sent three parts without an auth header, and completed with the ETags;
    • a direct upload sent the bytes with bearer auth and the right Content-Length.

@1pxone 1pxone closed this Nov 12, 2026
@1pxone 1pxone reopened this Nov 12, 2026
…ts, db.connections)

db.imports creates an import, uploads its files (S3 multipart parts to
presigned URLs on a thread pool, parallelism 4, each part retried, or the
bytes through the API), previews the mapping with personal-data findings,
starts the run and follows it (get, list, wait, cancel, retry, delete);
import_files does it in one call with mappings, links between files and
progress. Files can be paths, bytes or binary file objects; paths and
seekable files are read per part, so large files never sit in memory.

db.connections lists the connector catalog and push templates and manages
connections: create, update, delete(delete_data=), test, preview, fields,
review_fields, sync, runs, pause, resume and rotate_webhook_token. Agent
connections return their token once.

Both are RushDB Cloud only: in direct mode every method raises
CloudOnlyError before sending anything. Times are in seconds.
@1pxone
1pxone force-pushed the feat/imports-and-connections branch from 86d02af to 0bf1dc1 Compare November 12, 2026 16:38
@1pxone
1pxone marked this pull request as ready for review November 12, 2026 16:38
@1pxone
1pxone merged commit c9f2a56 into main Nov 12, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant