Is it possible to return compared value + a string from a ternary operation without repeating said value?

I'm trying to find an easier solution to a problem.

Problem:

I want to attempt and simplify this but I have no idea where to start.

let days = Math.floor(distance / (1000 * 60 * 60 * 24));
if(days > 0) {
    days = days + "d";
}

Attempt:

I was thinking I could use ternary operators to return the calculation + "d" like so:

let days = Math.floor(distance / (1000 * 60 * 60 * 24)) === 0 ? Math.floor(distance / (1000 * 60 * 60 * 24)) + "d" : "";

this is however very messy in my opinion and I can't figure out another way.

Current structure

I am currently calculating days, hours, minutes and seconds for a timer like this:

let distance = expiry - now;
let days = Math.floor(distance / (1000 * 60 * 60 * 24));
let hours = Math.floor((distance % (1000 * 60 * 60 * 24)) / (1000 * 60 * 60));
let minutes = Math.floor((distance % (1000 * 60 * 60)) / (1000 * 60));
let seconds = Math.floor((distance % (1000 * 60)) / 1000);

After that I want to only show days if it's greater than 0 or minutes if it's greater than 0 and so on. I'm currently doing it with a bunch of if statements and a boolean to check if a value greater than 0 has been found already. Like so:

let isSet = false;

if (days > 0 && !isSet) {
    current = days + "d";
    isSet = true;
}

if (hours > 0 && !isSet) {
    current = hours + "h";
    isSet = true;
}

if (minutes > 0 && !isSet) {
    current = minutes + "m";
    isSet = true;
}

if (seconds > 0 && !isSet) {
    current = seconds + "s";
    isSet = true;
}

if (seconds < 0 && !isSet) {
    current = "expired";
    isSet = true;
}

This does however feel very repetitive and wrong (even if it works).