mercredi 21 août 2019

Hive: Using if-else to check for file presence in a directory

I've this requirement, to create a hive table on top of a file present in a Azure blob directory.

Below piece of code is working fine:

DROP TABLE IF EXISTS DB.TSTTBL;

CREATE EXTERNAL TABLE DB.TSTTBL(
    ID INT,
    CLSS_CODE STRING,
    RANK INT,
    SUBJ_TKN INT,
    CHS_NAME STRING)
COMMENT 'Creating a table on top of a file.'
ROW FORMAT DELIMITED
    FIELDS TERMINATED BY '|'
    LINES TERMINATED BY '\n'
STORED AS TEXTFILE
LOCATION 'wasbs://alpha@mydrive.blob.core.windows.net/hive/sampledata1/'

Path structure:

wasbs://alpha@mydrive.blob.core.windows.net/hive/sampledata1/:

samplefile1.txt
samplefile2.txt
samplefile3.txt

The code works fine if the file is present at the specified location but fails if it doesn't. I want to make it gracefully exit/fail.

Is there a way to add if-else condition in the query something like:

if ('wasbs://alpha@mydrive.blob.core.windows.net/hive/sampledata1/' | wc -l >= 1)
then 
  CREATE EXTERNAL TABLE ...
else
  exit 1

Not sure is there's a way to achieve this in a hive query. Not looking for a UDF usage. Thanks for your time.

Aucun commentaire:

Enregistrer un commentaire