langchain.utilities.spark_sql.SparkSQL¶
class langchain.utilities.spark_sql.SparkSQL(spark_session: Optional[SparkSession] = None, catalog: Optional[str] = None, schema: Optional[str] = None, ignore_tables: Optional[List[str]] = None, include_tables: Optional[List[str]] = None, sample_rows_in_table_info: int = 3)[source]¶
Methods
__init__([spark_session, catalog, schema, ...])
from_uri(database_uri[, engine_args])
Creating a remote Spark Session via Spark connect.
get_table_info([table_names])
get_table_info_no_throw([table_names])
Get information about specified tables.
get_usable_table_names()
Get names of tables available.
run(command[, fetch])
run_no_throw(command[, fetch])
Execute a SQL command and return a string representing the results.
__init__(spark_session: Optional[SparkSession] = None, catalog: Optional[str] = None, schema: Optional[str] = None, ignore_tables: Optional[List[str]] = None, include_tables: Optional[List[str]] = None, sample_rows_in_table_info: int = 3)[source]¶
classmethod from_uri(database_uri: str, engine_args: Optional[dict] = None, **kwargs: Any) → SparkSQL[source]¶
Creating a remote Spark Session via Spark connect.
For example: SparkSQL.from_uri(“sc://localhost:15002”)
get_table_info(table_names: Optional[List[str]] = None) → str[source]¶
get_table_info_no_throw(table_names: Optional[List[str]] = None) → str[source]¶
Get information about specified tables.
Follows best practices as specified in: Rajkumar et al, 2022
(https://arxiv.org/abs/2204.00498)