Feeds
Stream each feed's real-time events and download its hourly and daily Parquet exports.
Each feed is an attribute of the client. On Client it is a Feed, and on AsyncClient an AsyncFeed, generic in the feed's event type, such as Feed[ProxyEvent]. Both classes are exported for type annotations, but don't create feeds yourself.
Every feed has a real-time stream and hourly Parquet exports that roll up into daily exports. Streaming and exports need separate scopes on the API key. The event types are listed under Event types.
| Attribute | Event type | Stream scope | Export scope |
|---|---|---|---|
proxies | ProxyEvent | PROXY_FIREHOSE | PROXY_FEEDS |
anonymizers | AnonymizerEvent | ANONYMIZERS_STREAM | ANONYMIZERS_FEED |
torrents | TorrentEvent | TORRENTS_STREAM | TORRENTS_FEED |
helios_http | HeliosHTTPEvent | HONEYPOT_HTTP_STREAM | HONEYPOT_HTTP_FEED |
helios_https | HeliosHTTPSEvent | HONEYPOT_HTTPS_STREAM | HONEYPOT_HTTPS_FEED |
helios_dns | HeliosDNSEvent | HONEYPOT_DNS_STREAM | HONEYPOT_DNS_FEED |
helios_adb | HeliosADBEvent | HONEYPOT_ADB_STREAM | HONEYPOT_ADB_FEED |
proxiescarries residential, mobile, and datacenter proxy sightings.anonymizerscarries VPN and anonymizer IP ranges.torrentscarries torrents seen on the BitTorrent DHT and trackers.- The four Helios feeds carry honeypot captures: HTTP requests, TLS ClientHellos, tunnel destinations, and Android Debug Bridge commands. Despite its name,
helios_dnsdoes not carry DNS queries.
See Firehose, Feeds, and Helios for the data behind each feed.
Streams
stream
Opens the feed's real-time stream and yields an iterator over its events, where ModelT is the feed's event type.
@contextmanager
def stream(self) -> Generator[Iterator[ModelT], None, None]
@asynccontextmanager
async def stream(self) -> AsyncGenerator[AsyncIterator[ModelT], None]
- Connection: The connection opens when the
withblock is entered and closes when it exits, even after abreak. The server keeps a stream open for up to 30 minutes and then closes it, which ends the iteration without an error. Opening a stream is retried like any other feed request. - Errors: If the stream cannot be opened, entering the block raises an
APIErrorfor a non-success status, such as 403 without the feed's scope or 429 when the key already has too many open streams, andAPIConnectionErrororAPITimeoutErrorwhen the connection fails. A connection that fails during iteration raisesAPIConnectionError, and an event that cannot be decoded raisesSynthientError. - Reconnecting: The API has no resume cursor, so events sent between connections are missed.
Open the stream again to reconnect:
while True:
with client.proxies.stream() as events:
for event in events:
print(event.ip, event.provider, event.type)
The loop reconnects immediately after a clean close. Opening a connection already uses the SDK's bounded retries, including Retry-After handling. Any error that remains after those retries, or occurs while reading the stream, stops this consumer. A 429 without Retry-After can indicate an exhausted monthly request budget or too many open streams; resolve that condition before reconnecting.
To keep consuming after a failure, catch the error and wait before reconnecting:
import time
while True:
try:
with client.proxies.stream() as events:
for event in events:
print(event.ip, event.provider, event.type)
except synthient.SynthientError as error:
if isinstance(error, synthient.APIError) and error.status_code in {401, 403}:
raise
print("proxy stream:", error)
# Avoid reconnecting in a tight loop.
time.sleep(2)
- Errors: A 401 or 403 stops the loop, because reconnecting can't fix a rejected key or a missing scope.
- Reconnecting: Any other failure raises a
SynthientError, and the loop waits 2 seconds before reconnecting.
With AsyncClient, enter the stream with async with and iterate with async for:
async with client.proxies.stream() as events:
async for event in events:
print(event.timestamp, event.ip, event.provider, event.type)
Streams
with client.proxies.stream() as events:
for event in events:
print(event.ip, event.provider, event.type, event.country_code, event.asn)
HeliosHTTPEvent.details and HeliosHTTPSEvent.details are None when there is no parsed request or handshake, so check them before use.
Exports
- Hourly exports: An hourly export is available until its day's daily export is rolled up, shortly after 00:30 UTC the next day. After that, requests for the hour raise an
APIErrorwith status 400. - Daily exports: While a daily export is being rolled up, requests for it raise an
APIErrorwith status 404 andretry_afterset.
Export IDs
Methods that take an export accept:
"latest", the feed's freshest export: the newest hourly export during the day, or the daily export once that day's hours are rolled up.- A UTC day as
"YYYY-MM-DD"or adate, for a daily export. - A UTC hour as
"YYYY-MM-DD/H", where a zero-padded hour is also accepted, or a timezone-awaredatetime, for an hourly export. Thedatetimeis converted to UTC.
Notes on these IDs:
- Input:
Export.idvalues can be passed as they are. There are no named helpers for these IDs. - Datetimes: Because
datetimeis a subclass ofdate, adatetimealways selects an hourly export, so call.astimezone(UTC).date()on it to select its UTC day. - Errors: A naive
datetimeor any other string raisesValueError, and an export ID of another type raisesTypeError, before a request is sent.
from datetime import UTC, datetime, timedelta
now = datetime.now(UTC)
yesterday = client.proxies.export_meta((now - timedelta(days=1)).date())
last_hour = client.proxies.export_meta(now - timedelta(hours=1))
print(yesterday.id, last_hour.id)
list_exports
Returns one page of the feed's exports, newest first.
def list_exports(self, *, limit: int | None = None, cursor: str | None = None) -> ExportPage
async def list_exports(
self, *, limit: int | None = None, cursor: str | None = None
) -> ExportPage
- Name
limit- Type
- int | None
- Description
The page size, from 1 to 500.
Noneuses the API default of 100. A value below 1, or one that is not an integer, raisesValueError.
- Name
cursor- Type
- str | None
- Description
Continues a listing from a previous page's
next_cursor.
page = client.proxies.list_exports(limit=24)
for export in page.exports:
print(export.id, export.kind, export.row_count)
print("next cursor:", page.next_cursor)
- Paging: Use
iter_exportsto iterate over every page.
iter_exports
Iterates over the feed's exports, newest first.
def iter_exports(
self, *, limit: int | None = None, cursor: str | None = None
) -> Iterator[Export]
async def iter_exports(
self, *, limit: int | None = None, cursor: str | None = None
) -> AsyncIterator[Export]
for export in client.proxies.iter_exports():
print(export.id, export.kind, export.size_bytes)
On AsyncFeed, call it without await and iterate with async for:
async for export in client.proxies.iter_exports():
print(export.id, export.kind, export.size_bytes)
- Paging: It fetches pages of
limitlazily as iteration proceeds, starting fromcursor. It stops fetching when the loop breaks. - Results: On the first page, the newest hourly export has the ID
"latest", a moving alias for the freshest export. To keep a stable ID, build it from the export'sdateandhour, such as"2026-10-14/21". - Errors: If the API repeats a cursor, it raises
SynthientErrorafter that page's exports.
export_meta
Returns the checksum, size, row count, and Parquet schema of an export.
def export_meta(self, export: str | date) -> ExportMeta
async def export_meta(self, export: str | date) -> ExportMeta
meta = client.proxies.export_meta("latest")
print(meta.id, meta.rows, meta.size, meta.checksum)
for column in meta.parquet_schema.fields:
print(column.name, column.type)
download_url
Returns a presigned URL for an export's Parquet file.
def download_url(self, export: str | date) -> str
async def download_url(self, export: str | date) -> str
from urllib.parse import urlsplit
url = client.proxies.download_url("latest")
print(urlsplit(url).hostname)
- Lifetime: The URL is valid for 24 hours.
- Security: The URL grants access without an API key, so treat it as a secret.
- Use: It suits tools that read Parquet over HTTPS directly.
- Errors: A redirect to anything other than an
httporhttpsURL, or fromhttpstohttp, raisesSynthientError.
download
Downloads an export's Parquet file and yields the streaming httpx.Response.
@contextmanager
def download(self, export: str | date) -> Generator[httpx.Response, None, None]
@asynccontextmanager
async def download(self, export: str | date) -> AsyncGenerator[httpx.Response, None]
- Reading: Read the body with
response.iter_bytes(), orresponse.aiter_bytes()onAsyncFeed. - Cleanup: The response closes when the block exits.
- Security: The API key is sent only to the Synthient API, never to the storage host that serves the file, and errors never include the presigned URL.
- Errors: A non-success response from either raises an
APIError, and a missing or invalid redirect raisesSynthientError. Reading the body is plain httpx, so a failure partway through raises an httpx exception, such ashttpx.ReadTimeoutorhttpx.ReadError, rather than aSynthientError, and leaves a partial file. Downloads from the storage host are not retried.
The SDK does not verify the file. Compare its SHA-256 with Export.checksum or ExportMeta.checksum, and ask for the checksum and the file by the same stable ID, because "latest" can move to a newer export between the two calls:
import hashlib
from datetime import UTC, date, datetime, timedelta
from pathlib import Path
import synthient
def save_export(feed: synthient.Feed[synthient.ProxyEvent], export: date, path: Path) -> None:
meta = feed.export_meta(export)
digest = hashlib.sha256()
with feed.download(export) as response, path.open("wb") as file:
for chunk in response.iter_bytes():
file.write(chunk)
digest.update(chunk)
if digest.hexdigest() != meta.checksum:
raise ValueError("checksum mismatch")
yesterday = datetime.now(UTC).date() - timedelta(days=1)
save_export(client.proxies, yesterday, Path("proxies.parquet"))