Working with YDB databases
YDB can act as an external data source for another YDB database. This section discusses the organization of collaboration between two independent YDB databases in federated query processing mode.
To connect to an external YDB database from another YDB database acting as the federated query engine, the following steps need to be performed on the latter:
-
Prepare authentication data to access the remote YDB database. Currently, in federated queries to YDB, the only available authentication method is login and password (other methods are not supported). The password to the external database is stored as a secret:
CREATE OBJECT ydb_datasource_user_password (TYPE SECRET) WITH (value = "<password>");
-
Create an external data source describing the external YDB database. The
LOCATION
parameter contains the network address of the YDB instance to which the network connection is made. TheDATABASE_NAME
specifies the name of the database (e.g.,local
). For authentication to the external database, theLOGIN
andPASSWORD_SECRET_NAME
parameters are used. Encryption of connections to the external database can be enabled using theUSE_TLS="TRUE"
parameter. If encryption is enabled, the<port>
field in theLOCATION
parameter should specify the gRPCs port of the external YDB; otherwise, the gRPC port should be specified.CREATE EXTERNAL DATA SOURCE ydb_datasource WITH ( SOURCE_TYPE="Ydb", LOCATION="<host>:<port>", DATABASE_NAME="<database>", AUTH_METHOD="BASIC", LOGIN="user", PASSWORD_SECRET_NAME="ydb_datasource_user_password", USE_TLS="TRUE" );
-
Deploy the connector and configure the YDB dynamic nodes to interact with it. Additionally, ensure network access from the YDB dynamic nodes to the external data source (at the address specified in the
LOCATION
parameter of theCREATE EXTERNAL DATA SOURCE
request). If network connection encryption to the external source was enabled in the previous step, the connector will use the system's root certificates (more details on TLS configuration can be found in the guide on deploying the connector). -
Execute a query to the external data source.
Query syntax
To retrieve data from tables of the external YDB database, the following form of SQL query is used:
SELECT * FROM ydb_datasource.`<table_name>`
Where:
ydb_datasource
- identifier of the external data source;<table_name>
- full name of the table within the hierarchy of directories in the YDB database, e.g.,table
,dir1/table1
, ordir1/dir2/table3
.
If the table is at the top level of the hierarchy (not belonging to any directories), it is permissible not to enclose the table name in backticks "`":
SELECT * FROM ydb_datasource.<table_name>
Limitations
There are several limitations when working with external YDB data sources:
-
External sources are available only for reading data through
SELECT
queries. The federated query processing engine currently does not support queries that modify tables in external sources. -
The YDB federated query processing system is capable of delegating the execution of certain parts of a query to the system acting as the data source. Query fragments are passed through YDB directly to the external system and processed within it. This optimization, known as "predicate pushdown", significantly reduces the volume of data transferred from the source to the federated query processing engine. This reduces network load and saves computational resources for YDB.
A specific case of predicate pushdown, where filtering expressions specified after the
WHERE
keyword are passed down, is called "filter pushdown". Filter pushdown is possible when using:Description Example Filters like IS NULL
/IS NOT NULL
WHERE column1 IS NULL
orWHERE column1 IS NOT NULL
Logical conditions OR
,NOT
,AND
WHERE column IS NULL OR column2 is NOT NULL
Comparison conditions =
,<>
,<
,<=
,>
,>=
with other columns or constantsWHERE column3 > column4 OR column5 <= 10
Supported data types for filter pushdown:
YDB Data Type Bool
Int8
Uint8
Int16
Uint16
Int32
Uint32
Int64
Uint64
Float
Double
String
Utf8
Supported data types
When working with tables located in the external YDB database, users have access to a limited set of data types. All other types, except for those listed below, are not supported. Type conversion is not performed, meaning that the columns of the table from the external YDB database retain their type after being read by the YDB database processing the federated query.
YDB Data Type |
---|
Bool |
Int8 |
Int16 |
Int32 |
Int64 |
Uint8 |
Uint16 |
Uint32 |
Uint64 |
Float |
Double |
String |
Utf8 |
Date |
Datetime |
Timestamp |
Json |