Support ABFSS and Hadoop Logical Relation in Column-level lineage #1008@wjohnson Introduces an extractDatasetIdentifier that uses similar logic to InsertIntoHadoopFsRelationVisitor to pull out the path on the HDFS compliant file system; tested on ABFSS and DBFS (Databricks FileSystem) to prove that lineage could be extracted using non-SQL commands.
Add Kusto relation visitor #939@hmoazam Implements a KustoRelationVisitor to support lineage for Azure Kusto's Spark connector.
Add ColumnLevelLineage facet doc #1020@julienledem Adds documentation for the Column-level lineage facet.
Include symlinks dataset facet #935@pawel-big-lebowski Includes the recently introduced SymlinkDatasetFacet in generated OpenLineage events.
Add support for dbt 1.3 beta's metadata changes #1051@mobuchowski Makes projects that are composed of only SQL models work on 1.3 beta (dbt 1.3 renamed the compiled_sql field to compiled_code to support Python models). Does not provide support for dbt's Python models.
Support Flink 1.15 #1009@mzareba382 Adds support for Flink 1.15.
Add Redshift dialect to the SQL integration #1066@mobuchowski Adds support for Redshift's SQL dialect in OpenLineage's SQL parser, including quirks such as the use of square brackets in JSON paths. (Note, this does not add support for all of Redshift's custom syntax.)
Make the timeout configurable in the Spark integration #1050@tnazarew Makes timeout configurable by the user. (In some cases, the time needed to send events was longer than 5 seconds, which exceeded the timeout value.)
Add a dialect parameter to Great Expectations SQL parser calls #1049@collado-mike Specifies the dialect name from the SQL engine.
Fix Delta 2.1.0 with Spark 3.3.0 #1065@pawel-big-lebowski Allows delta support for Spark 3.3 and fixes potential issues. (The Openlineage integration for Spark 3.3 was turned on without delta support, as delta did not support Spark 3.3 at that time.)