From the client perspective, Spark Connect mostly behaves as any other GRPC client and can be configured as such. However, to make it easy to use from different programming languages and to have a homogeneous connection surface this document proposes what the user surface is for connecting to a Spark Connect endpoint.
Similar to JDBC or other database connections, Spark Connect leverages a connection string that contains the relevant parameters that are interpreted to connect to the Spark Connect endpoint
Generally, the connection string follows the standard URI definitions. The URI
scheme is fixed and set to sc://. The full URI has to be a
valid URI and must
be parsed properly by most systems. For example, hostnames have to be valid and
cannot contain arbitrary characters. Configuration parameters are passed in the
style of the HTTP URL Path Parameter Syntax. This is similar to the JDBC connection
strings. The path component must be empty. All parameters are interpreted case sensitive.
sc://host:port/;param1=value;param2=value
| Parameter | Type | Description | Examples |
| host | String | The hostname of the endpoint for Spark Connect. Since the endpoint has to be a fully GRPC compatible endpoint a particular path cannot be specified. The hostname must be fully qualified or can be an IP address as well. |
myexample.com 127.0.0.1 |
| port | Numeric | The port to be used when connecting to the GRPC endpoint. The default value is: 15002. Any valid port number can be used. | 15002 443 |
| token | String | When this param is set in the URL, it will enable standard bearer token authentication using GRPC. By default this value is not set. Setting this value enables SSL. | token=ABCDEFGH |
| use_ssl | Boolean | When this flag is set, will by default connect to the endpoint using TLS. The assumption is that the necessary certificates to verify the server certificates are available in the system. The default value is false | use_ssl=true use_ssl=false |
| user_id | String | User ID to automatically set in the Spark Connect UserContext message. This is necessary for the appropriate Spark Session management. This is an *optional* parameter and depending on the deployment this parameter might be automatically injected using other means. |
user_id=Martin |
| user_agent | String | The user agent acting on behalf of the user, typically applications
that use Spark Connect to implement its functionality and execute Spark
requests on behalf of the user. Default: _SPARK_CONNECT_PYTHONin the Python client |
user_agent=my_data_query_app |
| session_id | String | In addition to the user ID, the cache of Spark Sessions in the Spark Connect
server uses a session ID as the cache key. This option in the connection string
allows to provide this session ID to allow sharing Spark Sessions for the same users
for example across multiple languages. The value must be provided in a valid UUID
string format. Default: A UUID generated randomly |
session_id=550e8400-e29b-41d4-a716-446655440000 |
| grpc_max_message_size | Numeric | Maximum message size allowed for gRPC messages in bytes. Default: 128 * 1024 * 1024 |
grpc_max_message_size=134217728 |
| grpc_keepalive_enabled | Boolean | Whether the client sends gRPC/HTTP2 keepalive PINGs to detect a silently-dead
connection (e.g. a NAT gateway or load balancer dropping an idle connection mapping
without closing the socket), so a blocked call fails with an error instead of hanging
forever. Can be turned off as an escape hatch, e.g. in environments prone to long stalls
(GC pauses, etc.) that could otherwise trip a false-positive disconnect. Default: true |
grpc_keepalive_enabled=false |
| grpc_keepalive_time_ms | Numeric | Idle time (in milliseconds) before the client sends a keepalive PING. A Spark Connect
server tolerates client PINGs no more often than every 10s; setting this below that floor
will get the connection torn down as "too_many_pings". Default: 60000 |
grpc_keepalive_time_ms=30000 |
| grpc_keepalive_timeout_ms | Numeric | Time (in milliseconds) the client waits for a keepalive PING ack before considering
the connection dead. Default: 20000 |
grpc_keepalive_timeout_ms=10000 |
| grpc_keepalive_without_calls | Boolean | Whether to keep sending keepalive PINGs when there are no in-flight RPCs on the
connection. Default: true |
grpc_keepalive_without_calls=false |
Below we capture valid configuration examples, explaining how the connection string will be used when configuring the Spark Connect client.
The below example connects to port 15002 on myhost.com.
server_url = "sc://myhost.com/"The next example configures the connection to use a different port with SSL.
server_url = "sc://myhost.com:443/;use_ssl=true"server_url = "sc://myhost.com:443/;use_ssl=true;token=ABCDEFG"The next example tunes the gRPC keepalive settings, e.g. to detect a dead connection faster than the 60s/20s default, or to disable it entirely.
server_url = "sc://myhost.com:443/;grpc_keepalive_time_ms=30000;grpc_keepalive_timeout_ms=10000"server_url = "sc://myhost.com:443/;grpc_keepalive_enabled=false"As mentioned above, Spark Connect uses a regular GRPC client and the server path cannot be configured to remain compatible with the GRPC standard and HTTP. For example the following examples are invalid.
server_url = "sc://myhost.com:443/mypathprefix/;token=AAAAAAA"