mercredi 12 juin 2019

Check S3 File Size Using DAG In Airflow

I'm currently trying to write a dag to copy files from one S3 location to another. Before I can copy the files, I need to check the corresponding reject file to see if it is empty. The logic I need to follow is:

if rej_file_size = 0kb:
    Copy files from S3_loc_1 to S3_loc_2
else:
    Fail dag

I'm pretty new to dag and python, so what would be the best way for me to take a file from a location and see if it is empty (0kb) or not (>0kb)

Thank you for your time

Aucun commentaire:

Enregistrer un commentaire