How to add multiple if statements in lambda?

I have a dataframe containing dictionaries. It looks something like this:

df
{key: NaN, key:values...}
{key: values, key:NaN, key:values}
Nan
{key: values, key:values...}
Nan

I aim to split the values from each dictionary into a separate column. In the same time I have to account for the values that are Nan. This is what I have so far:

df[col_name].apply(lambda d: set(d.values()) if d==d else d)

It works great, however there is one problem when some of the values in my dictionary are NaN, it outputs the, like this {NaN}. So the output can look like this:

key
NaN
{NaN}
{NaN}
key
key
NaN
{NaN}

How can I change my lambda so that if a the only value is NaN, it outputs NaN and not {Nan}. This would be the correct result:

key
NaN
NaN
NaN
key
key
NaN
NaN