Airflow UI无法手动触发dag

时间:2018-12-03 19:41:24

标签: python airflow

版本:

  • 气流v1.10.1
  • SQL Server后端
  • Web服务器和调度程序在同一主机上运行
  • LocalExecutor

使用UI(按钮)手动触发时,简单的hello world DAG将不会运行。通过命令行运行时,同一示例运行良好。希望允许用户使用UI来触发作业。这是一个错误。

示例Hello World DAG已测试:

from datetime import datetime
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.python_operator import PythonOperator

def print_welcome():
    return 'Welcome!'

dag = DAG('say_welcome', description='Simple tutorial DAG',
          schedule_interval='0 12 * * *',
          start_date=datetime(2017, 3, 20), catchup=False)

dummy_operator = DummyOperator(task_id='say_welcome_dummy_task', retries=3, dag=dag)

hello_operator = PythonOperator(task_id='say_welcome_task', python_callable=print_welcome, dag=dag)

dummy_operator >> hello_operator

从命令行测试输出。

(airfow_v1_venv) sshuser@ed41-kp06sp:~/airflowv1/dags$ airflow trigger_dag say_welcome
[2018-12-03 19:38:34,679] {__init__.py:51} INFO - Using executor LocalExecutor
[2018-12-03 19:38:34,956] {models.py:271} INFO - Filling up the DagBag from /home/sshuser/airflowv1/dags
[2018-12-03 19:38:35,071] {cli.py:241} INFO - Created <DagRun say_welcome @ 2018-12-03 19:38:34+00:00: manual__2018-12-03T19:38:34+00:00, externally triggered: True>

使用用户界面触发时记录

    context)
  File "/home/sshuser/airfow_v1_venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 467, in do_executemany
    cursor.executemany(statement, parameters)
IntegrityError: (pyodbc.IntegrityError) ('23000', u"[23000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Violation of PRIMARY KEY constraint 'PK__task_ins__9BEABD04E2A8D429'. Cannot insert duplicate key in object 'dbo.task_instance'. The duplicate key value is (say_welcome_task, say_welcome, Dec  3 2018  7:40PM). (2627) (SQLExecDirectW)") [SQL: u'INSERT INTO task_instance (task_id, dag_id, execution_date, start_date, end_date, duration, state, try_number, max_tries, hostname, unixname, job_id, pool, queue, priority_weight, operator, queued_dttm, pid, executor_config) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)'] [parameters: (('say_welcome_task', 'say_welcome', datetime.datetime(2018, 12, 3, 19, 40, 9, 787000, tzinfo=<Timezone [UTC]>), None, None, None, None, 0, 0, u'', 'sshuser', None, None, 'default', 1, None, None, None, bytearray(b'\x80\x02}q\x00.')), ('say_welcome_dummy_task', 'say_welcome', datetime.datetime(2018, 12, 3, 19, 40, 9, 787000, tzinfo=<Timezone [UTC]>), None, None, None, None, 0, 3, u'', 'sshuser', None, None, 'default', 2, None, None, None, bytearray(b'\x80\x02}q\x00.')))]

1 个答案:

答案 0 :(得分:1)

在我看来,就像您尝试覆盖旧的say_welcome DAG。

创建一个名为say_welcome_v1的镜头并试一试。

创建新的DAG时,必须更改其名称,以便可以在元数据库中对其进行区分。因此,每当DAG中发生更改时,在DAG名称末尾使用_v1_v2等的约定。

由于您遇到的错误是完整性错误,因此当您尝试使用与已经存在的其他事物相同的主键将某些事物插入数据库时​​会发生此错误。新的DAG与旧的同名,很可能是错误。

如果您没有任何值得保留历史记录/日志的旧DAG运行,则只需使用airflow resetdb后跟airflow initdb即可重置数据库并从头开始。

从Airflow 1.10版及更高版本开始,您还可以使用airflow delete_dag my_dag_id从元数据库中删除旧的DAG ID。