Dynamically generated tasks in Airflow 2.2.5 are moved to "REMOVED" state and breaks down the GANTT chart
Airflow Version : 2.2.5
Composer Version : 2.0.19
We have a task group which creates the tasks dynamically using for
loop. Within the taskgroup we are making use of BigQueryTableDeleteOperator
to delete the tables.
Issue: We noticed that once the tables are deleted, all the tasks moved to REMOVED
state, hence breaking the GANTT chart
with error message of Task not found
.
Before the task run : Image 1
After the task runs: Image 2
As shown above, before the taskgroup run it shows all the tables to deleted represented by each task . In this example 2 tasks.
Once the task runs into success and the table is deleted, those tasks are removed.
Sharing the piece of code below :
for table in tables_list:
table_name = projectid + '.' + dataset + '.' + table
if table not in safe_tables:
delete_table_task = bigquery_table_delete_operator.BigQueryTableDeleteOperator( task_id=f"delete_tables_{table_name}",
deletion_dataset_table=f"{table_name}",
ignore_if_missing=True)
list_operator += [delete_table_task]
list_operator
print(list_operator)
dummy_task >> list_operator
tables_list
: List of tables to be deleted
safe_tables
: List of tables not
to be deleted
Please let me know what we are missing here which is causing the issue.
Comments
Post a Comment