Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 105 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ What each offers:
| Managed embeddings: managed indexes, `records.vector_search` by text `query` | ✓ | — |
| Natural-language search on SQL (`ai.search`, `ai.run_sql`) | ✓ | — |
| Relationship-pattern suggestions (`relationships.patterns.*`) and join suggestions (`ai.get_join_candidates`) | ✓ | — |
| Background file imports (`imports.*`) and SaaS connections (`connections.*`) | ✓ | — |
| Project settings (`settings.get`), dashboard, team access | ✓ | — |

In direct mode a Cloud-only method raises `CloudOnlyError` (with `method` and `reason`) before
Expand Down Expand Up @@ -241,6 +242,110 @@ db.records.import_csv(

---

## File imports and connections (RushDB Cloud)

Large files and SaaS data load in the background: the client hands the data over and the run
continues on the server. Every file import and every connection sync is an **import run**; follow it
with `db.imports.get(id)` or `db.imports.wait(id)` until its status is `completed`, `partial`,
`failed` or `cancelled`.

### Import files

```python
# One call: create → upload every file → start (→ wait)
run = db.imports.import_files(
[
{'data': 'customers.csv', 'label': 'Customer'}, # a path: name and size come from it
open('orders.csv.gz', 'rb'), # a binary file object works too
],
mappings={0: {'keyColumn': 'id'}}, # re-importing upserts by `id`
links=[{'source': 1, 'sourceColumn': 'customer_id', 'target': 0, 'targetColumn': 'id', 'type': 'PLACED_BY'}],
parallelism=4,
wait=True,
wait_interval=2,
on_progress=lambda p: print(f"{p['loaded']}/{p['total']}" if p['phase'] == 'upload' else p['run']['status']),
)
print(run.data['recordsWritten'], run.data['summary'])
```

Or step by step, to show a preview before starting:

```python
import os

created = db.imports.create([{'name': 'orders.csv', 'size': os.path.getsize('orders.csv')}]).data
db.imports.upload(created['id'], created['files'][0], 'orders.csv', parallelism=4, retries=3)
preview = db.imports.preview(created['id']).data # column types, suggested key and links, personal data
columns = preview['files'][0]['columns'] # each has `suggestion` ('include' | 'skip' | 'hash') and `pii` when it looks personal
mapping = {
'skip': [c['name'] for c in columns if c['suggestion'] == 'skip'],
'hash': [c['name'] for c in columns if c['suggestion'] == 'hash'], # keyed hash: joins, not readable
}
if preview['files'][0].get('suggestedKey'):
mapping['keyColumn'] = preview['files'][0]['suggestedKey']
db.imports.start(created['id'], files=[{'id': created['files'][0]['id'], 'mapping': mapping}])
finished = db.imports.wait(created['id'], timeout=30 * 60).data # TimeoutError after 30 minutes
```

`upload` follows the upload mode the server returns: `s3-multipart` sends parts to their presigned
URLs (`parallelism` at a time on a thread pool, default 4; each failed part retried, `retries`
default 3) and completes with their ETags; `direct` sends the bytes through the API. Upload data can
be a file path (`str` or `Path`), `bytes`/`bytearray`/`memoryview`, or a binary file object (read
from its current position). A non-seekable stream (a pipe, a socket) needs its `size` up front in
`import_files`, and a direct upload of one is not retried.

Also: `imports.list(kind=, connection_id=, limit=, skip=)`, `cancel`, `retry`, `delete`,
`add_files`, `remove_file`.

### Connections

```python
import os

catalog = db.connections.catalog().data # connectors with their streams and config fields

created = db.connections.create({
'connectorId': 'hubspot',
'auth': {'type': 'bearer', 'token': os.environ['HUBSPOT_TOKEN']},
'mode': 'scheduled',
'scheduleMinutes': 60,
'streams': [{'stream': 'contacts', 'policy': {'mode': 'auto', 'exclude': ['mobilephone']}}],
}).data
connection_id = created['connection']['id']
db.connections.test(connection_id)
db.connections.preview(connection_id, 'contacts') # first records, nothing written

# Field review: profiles with personal-data findings (masked examples), the policy, unreviewed fields
review = db.connections.fields(connection_id, 'contacts', limit=100).data
db.connections.review_fields(
connection_id,
'contacts',
[{'name': f['name'], 'action': f['suggestion']} for f in review['fields']], # or your own choices
)

sync_run = db.connections.sync(connection_id).data # or sync(connection_id, streams=['contacts'], full=True)
db.imports.wait(sync_run['id'])
```

A stream's `policy` decides which fields reach the graph: `auto` keeps everything except `exclude`
and `hash`; `review_fields` switches it to an `allowlist` of the fields marked `include` or `hash`,
and fields that appear later wait in the stream's `unreviewed` list until the next review.

On-prem databases sync through `rushdb-agent`: create an agent connection and hand the token
(shown once) to the agent.

```python
agent = db.connections.create({'connectorId': 'agent', 'name': 'Warehouse', 'agent': {'intervalMinutes': 15}}).data
print(agent['agentToken']) # connection['agent'] shows what the agent last reported
```

Also: `list`, `get`, `update`, `delete(id, delete_data=True)`, `runs(id, limit=, skip=)`, `pause`,
`resume`, `rotate_webhook_token` (a new webhook URL, or a new agent token), `push_templates()`
(Zapier, Make and n8n recipes for push connections). Credentials are sealed on the server and never
returned.

---

## SearchResult

`db.records.find()` returns a `SearchResult` — a list-like container with pagination metadata.
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "rushdb"
version = "3.3.0"
version = "3.4.0"
description = "RushDB SDK for Python: the client for RushDB, the database for AI (RushDB Cloud or self-hosted)"
authors = [
{name = "RushDB Team", email = "hi@rushdb.com"}
Expand Down
6 changes: 6 additions & 0 deletions src/rushdb/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
"""

from .api.ai import AIAPI
from .api.connections import ConnectionsAPI
from .api.imports import ImportsAPI
from .api.query import QueryAPI
from .api.records import pick_record_id
from .api.relationship_patterns import RelationshipPatternsAPI
Expand All @@ -25,6 +27,7 @@
ValueMatch,
ValueMatchesData,
)
from .models.ingest import FINISHED_IMPORT_RUN_STATUSES
from .models.property import (
Property,
PropertyValuesData,
Expand Down Expand Up @@ -91,4 +94,7 @@
"ApiResponse",
"AIAPI",
"TextIndexesAPI",
"ImportsAPI",
"ConnectionsAPI",
"FINISHED_IMPORT_RUN_STATUSES",
]
4 changes: 4 additions & 0 deletions src/rushdb/api/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
from .ai import AIAPI
from .connections import ConnectionsAPI
from .imports import ImportsAPI
from .labels import LabelsAPI
from .properties import PropertiesAPI
from .query import QueryAPI
Expand All @@ -18,4 +20,6 @@
"RelationsAPI",
"RelationshipPatternsAPI",
"TextIndexesAPI",
"ImportsAPI",
"ConnectionsAPI",
]
Loading
Loading