如何在气流中正确使用S3ToSpreadSheetOperator?

时间:2019-11-13 12:43:46

标签: airflow airflow-operator

我正在尝试使用AirfloweBook_plugin从S3存储桶中的JSON文件传输到Excel文件。但是,当插件尝试读取文件时出现错误

from airflow.plugins_manager import AirflowPlugin
from spreadsheet_plugin.operators.s3_to_spreadsheet_operator import S3ToSpreadsheetOperator
from datetime import datetime, timedelta
from airflow import DAG
import boto3

start_date = datetime.combine(datetime.today() - timedelta(1), datetime.min.time()) 
default_args = {
  'owner': 'me',
  'start_date': start_date,
  'retries': 1,
  'max_active_runs': 1,
  # 'on_failure_callback': task_fail_slack_alert
}

dag = DAG(dag_id='s3ToExcel',
  default_args=default_args,
  schedule_interval='@once',
  catchup=False
)


downloadExcel = S3ToSpreadsheetOperator(
  task_id='downloadExcel',
  input_s3_conn_id='aws_default',
  input_s3_bucket='bucket_name',
  input_s3_key='key_name/file_name',
  input_file_type='json',
  output_destination='s3',
  output_conn_id='aws_default',
  output_s3_bucket='bucket_name',
  output_s3_key='key_name',
  output_box_folder_id='1',
  output_box_file_name='file_name',
  output_format='excel',
  dag=dag
)

S3ToSpreadsheetOperator.py内出现错误:

    df = self.read_file(input_key)
  File "/Users/sauravfitogram/airflow/plugins/spreadsheet_plugin/operators/s3_to_spreadsheet_operator.py", line 173, in read_file
    df = pd.read_json(input_key.key
AttributeError: 'str' object has no attribute 'get_contents_as_string'df = pd.read_json(input_key
    AttributeError: 'str' object has no attribute 'get_contents_as_string'

无法确定要为插件设置的正确参数。

0 个答案:

没有答案