jeudi 3 juin 2021

Create enumerated folder path on Azure Data Lake

I am working on creating an enumerated folder path system on Azure Data Lake, where if the folder already exists, it creates a new folder to insert the file into. The first folder starts with a number 1, followed by 2 etc. (thus += 1). One way would be to loop through the folderNumber until it is available to create and another way I would imaging to read the highest folderNumber and add +=1 to it.

I have tried my hand at it but have failed so far. Any recommendations how to achieve this.

intro code
    #Connect to DataLake
    datalake_service_client = DataLakeServiceClient.from_connection_string("{connection string}")
    
    # Instantiate a FileSystemClient
    file_system_client = datalake_service_client.get_file_system_client("Test")

    folderNumber = 1
    if not file_system_client.get_file_client(f"/{folderNumber}/file.csv").exists():
        file_client = file_system_client.create_file(f"/{folderNumber}/file.csv")

    else:
        folderNumber +=1

Thank you.

Regards,

Aucun commentaire:

Enregistrer un commentaire