Snowflake open-sources PG Lake to bridge Postgres and Iceberg
Snowflake has released PG Lake, an extension that embeds DuckDB and an Iceberg catalog into Postgres. The tool allows users to query Parquet files in S3 using standard SQL, eliminating the need for complex ETL pipelines.
Snowflake open-sourced PG Lake in November, an extension that integrates an Iceberg catalog and an embedded DuckDB engine directly into Postgres. The tool enables users to create tables using Iceberg, storing data as Parquet files in S3, and query them using standard SQL. PG Lake comprises approximately 12 sub-extensions designed to handle SQL syntax conversion, function mapping, and version management. The development process took two years, with the project originally created at Crunchy Data before the company joined Snowflake.
The extension bridges the gap between OLTP and OLAP systems, allowing transactional databases to handle analytical workloads natively. This capability eliminates the need for complex ETL pipelines, reducing operational overhead and cost. Craig Kirsteins positioned PG Lake as a 'Swiss army knife' for data engineering, noting that it simplifies workflows by embedding analytical capabilities within the transactional layer. Snowflake also announced a database mirroring feature that continuously replicates Postgres data to Snowflake in real time, further integrating the two platforms.