XComs are a mechanism for between tasks.
When we talk about "exclusive" XCom usage, we refer to the practice of restricting data access to specific tasks or ensuring that only certain keys are utilized to avoid "polluting" the metadata database. 1. Avoiding Database Bloat
Apache Airflow XComs should be reserved exclusively for small metadata pointers, such as S3 keys or row IDs, to prevent metadata database bottlenecks. For large data transfers, utilizing custom XCom backends for object storage like S3 or GCS is recommended to optimize DAG performance. Read more on best practices at Astronomer Documentation Apache Airflow XComs — Airflow 3.2.0 Documentation
@task def consumer_1(data): # data is automatically pulled only to this task print(data)
Airflow Xcom Exclusive -
XComs are a mechanism for between tasks.
When we talk about "exclusive" XCom usage, we refer to the practice of restricting data access to specific tasks or ensuring that only certain keys are utilized to avoid "polluting" the metadata database. 1. Avoiding Database Bloat airflow xcom exclusive
Apache Airflow XComs should be reserved exclusively for small metadata pointers, such as S3 keys or row IDs, to prevent metadata database bottlenecks. For large data transfers, utilizing custom XCom backends for object storage like S3 or GCS is recommended to optimize DAG performance. Read more on best practices at Astronomer Documentation Apache Airflow XComs — Airflow 3.2.0 Documentation XComs are a mechanism for between tasks
@task def consumer_1(data): # data is automatically pulled only to this task print(data) airflow xcom exclusive
Copyright © 2020 ToolRocket Ltd. All Rights Reserved.
Privacy | Terms & Conditions