The Flat File integration is used to upload a single flat file (like csv, parquet, and feather) into the data lineage graph.
|source||The name of the source, see sources||my-source|
|Namespace||Namespace for the connection, see namespaces||default|
The dbt integration can be run as a standalone python library to extract data lineage from flat files like csv and parquet.
The library is available via pip
pip install grai_source_flat_file
More information about the API is available here.
The library is split into a few distinct functions but if you only wish to extract nodes/edges you can do so as follows:
from grai_source_flat_file import FlatFileIntegration from grai_schemas.v1.source import SourceV1 source = SourceV1(name="my-source", type="my-type") integration = FlatFileIntegration(source=source, namespace="dbt", file_name="/path/to/file.csv") nodes, edges = integration.get_nodes_and_edges()