Python script logging (else statement not doing anything)

I'm hoping someone can help me out because for the life of I can't figure this out. I'm still very much an amateur in Python and had a go at writing my own script to clear out my downloads folder and log some messages to a log file as plan the run the script as an scheduled task (my downloads folder gets quite messy so why not automate cleaning it up). My code is as follows:

import os
import logging
import shutil

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s:%(name)s:%(levelname)s:%(message)s')
file_handler = logging.FileHandler('cleardownloads.log')
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)

folder = 'D:\\Downloads'
list_folder = os.listdir(folder)
number_files = len(list_folder)
#print(number_files)
for file in list_folder:
    file_path = str(os.path.join(folder, file))
    if number_files > 0:
        if os.path.isfile(file_path or os.path.islink(file_path)):
            logger.info(f'deleting file: {file_path}')
            os.unlink(file_path)
        elif os.path.isdir(file_path):
            logger.info(f'deleting folder: {file_path}')
            shutil.rmtree(file_path)
    else:
        logger.info(f'{file_path} is empty, nothing to delete')

The issue i'm having is the final 'else' statement, its not doing anything, the message is not being written to log file and even if I change that to a print statement nothing gets printed to the console. My Downlads folder is empty and printing out the len even shows 0 (commented out line in my code). If I create some files and then run the script the messages get logged correctly e.g.:

2021-02-11 23:27:54,174:__main__:INFO:deleting folder: D:\Downloads\dsfds
2021-02-11 23:33:45,460:__main__:INFO:deleting folder: D:\Downloads\fdggf
2021-02-11 23:33:45,460:__main__:INFO:deleting folder: D:\Downloads\fgfg
2021-02-11 23:42:22,086:__main__:INFO:deleting file: D:\Downloads\fgd.txt
2021-02-11 23:43:34,894:__main__:INFO:deleting file: D:\Downloads\fdsf.txt

Anyone know what i'm missing here?

Thanks in advance