2022-08-30

Dynamically generated tasks in Airflow 2.2.5 are moved to "REMOVED" state and breaks down the GANTT chart

Airflow Version : 2.2.5
Composer Version : 2.0.19

We have a task group which creates the tasks dynamically using for loop. Within the taskgroup we are making use of BigQueryTableDeleteOperator to delete the tables.

Issue: We noticed that once the tables are deleted, all the tasks moved to REMOVED state, hence breaking the GANTT chart with error message of Task not found.

Before the task run : Image 1

enter image description here

After the task runs: Image 2

enter image description here

As shown above, before the taskgroup run it shows all the tables to deleted represented by each task . In this example 2 tasks.

Once the task runs into success and the table is deleted, those tasks are removed.

Sharing the piece of code below :

for table in tables_list:
            table_name = projectid + '.' + dataset + '.' + table
            if table not in safe_tables:
                delete_table_task = bigquery_table_delete_operator.BigQueryTableDeleteOperator( task_id=f"delete_tables_{table_name}",
                                                                                            deletion_dataset_table=f"{table_name}",  
                                                                                            ignore_if_missing=True)
                list_operator += [delete_table_task]

list_operator
print(list_operator)
    
dummy_task >> list_operator

tables_list : List of tables to be deleted

safe_tables : List of tables not to be deleted

Please let me know what we are missing here which is causing the issue.



No comments:

Post a Comment