My Digital Garden

Using optimize write on Apache Spark to produce more efficient tables - Azure Synapse Analytics

Using optimize write on Apache Spark to produce more efficient tables - Azure Synapse Analytics (DaniBunny, )

rw-book-cover

Metadata

  • Author: DaniBunny
  • Full Title: Using optimize write on Apache Spark to produce more efficient tables - Azure Synapse Analytics
  • Category: #articles
  • Document Tags: data-engineering
  • Summary: The Optimize Write feature in Apache Spark helps increase efficiency by reducing the number of small files written and optimizing file sizes. It can be enabled on Delta Lake tables for better performance in analytical workloads. Configurations can be set to control this feature and improve data processing on Azure Synapse Analytics.
  • URL: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/optimize-write-for-apache-spark

Highlights