How can I hide comment count in the wordpress loop only when there are no comments

I am showing the comment count in the wordpress loop in the category.php.

I would like to only show the count if it is not 0.

I have tried a number of if statements but cannot find something that works to check comment count while in the loop and only show if it is not 0, or more than 0, or not empty...etc.

To print the comments (in the loop) I am using - <?php comments_number( '0', '1', '%' ); ?>

I would like to wrap it in an if statement but

<?php if (comments_number() != 0 ) {...

(or != 'No Comments' or empty or any number of other things I've tried!) doesn't work - in fact it seemed to print the comment number from within the if statement which I didn't think was possible but clearly I am missing some knowledge here that's having me get it wrong. I also tried:

<?php if (have_comments() != 0 ) {...

Can anyone help show me what if statement would be required around my comment number print in order to hide it if there are 0/No Comments. Thank you

Edit: I also tried changing the '0' to '' in comments_number but as expected, that still prints the comment wrapper which I don't want as the wrapper is styled. I want it to not print anything at all if there are no comments.