Airflow网络服务器
DAG暂停:
Airflow Scheduler继续运行DAG
我全新安装了Airflow。所有暂停的示例都正常工作。即使在网络上发布PAAGED,新的样本DAG也会继续运行。 PAUSING / unPAUSING在默认示例中工作正常。 其他问题是,它每6小时安排一次。为什么要继续尝试运行DAG?
SWAT_Tutorial_01.py
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta
default_args = {'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime(2018, 3, 16),
'email': ['sample@scholastic.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
# 'catchup': False,
# 'queue': 'bash_queue',
# 'pool': 'backfill',
# 'priority_weight': 10,
# 'end_date': datetime(2016, 1, 1),
}
dag = DAG( 'SWAT.Tutorial_01', default_args=default_args, schedule_interval=timedelta(hours=6))
# t1, t2 and t3 are examples of tasks created by instantiating operators
t1 = BashOperator( task_id='print_date', bash_command='date', dag=dag)
t2 = BashOperator( task_id='sleep', bash_command='sleep 5', retries=3, dag=dag)
t3 = BashOperator( task_id='hello', bash_command='echo "Hello World"', retries=3, dag=dag)
t2.set_upstream(t1)
t3.set_upstream(t1)
计划程序日志(不断重复)
File Path PID Runtime Last Runtime Last Run
---------------------------------------------- ----- --------- -------------- ----------
/home/airflow/airflow/dags/SWAT_Tutorial_01.py 16930 0.00s
================================================================================
[2018-03-17 23:41:45,352] {jobs.py:1404} INFO - Heartbeating the process manager
[2018-03-17 23:41:45,352] {jobs.py:1440} INFO - Heartbeating the executor
[2018-03-17 23:41:46,354] {jobs.py:1404} INFO - Heartbeating the process manager
[2018-03-17 23:41:46,354] {dag_processing.py:559} INFO - Processor for /home/airflow/airflow/dags/SWAT_Tutorial_01.py finished
[2018-03-17 23:41:46,356] {dag_processing.py:627} INFO - Started a process (PID: 16932) to generate tasks for /home/airflow/airflow/dags/SWAT_Tutorial_01.py - logging into /var/log/airflow/scheduler/2018-03-17/SWAT_Tutorial_01.py.log
[2018-03-17 23:41:46,357] {jobs.py:1440} INFO - Heartbeating the executor
[2018-03-17 23:41:47,358] {jobs.py:1404} INFO - Heartbeating the process manager
[2018-03-17 23:41:47,359] {jobs.py:1440} INFO - Heartbeating the executor
[2018-03-17 23:41:48,360] {jobs.py:1404} INFO - Heartbeating the process manager
[2018-03-17 23:41:48,360] {dag_processing.py:559} INFO - Processor for /home/airflow/airflow/dags/SWAT_Tutorial_01.py finished
[2018-03-17 23:41:48,362] {dag_processing.py:627} INFO - Started a process (PID: 16934) to generate tasks for /home/airflow/airflow/dags/SWAT_Tutorial_01.py - logging into /var/log/airflow/scheduler/2018-03-17/SWAT_Tutorial_01.py.log
[2018-03-17 23:41:48,363] {jobs.py:1440} INFO - Heartbeating the executor
答案 0 :(得分:0)
这里只是一些猜测。
我认为气流不会停止回填数据,即使你暂停dag,这些工作可能是回填工作。
您的代码似乎没有禁用回填(catch_up = False
)