Outlook VBA ignores original If argument when Else argument is added

Here is the original code that is working as intended:

Dim ContactsFolder As Folder
Set ContactsFolder = Session.GetDefaultFolder(olFolderContacts)
Dim SearchContactName As String
Set myItem = Outlook.CreateItem(olMailItem)
Dim NewContactEmail As String
Dim NewMail As Outlook.MailItem
Set NewMail = Application.ActiveInspector.currentItem

' Ask user for inputs
SearchContactName = InputBox("What is the name of your contact you want to comment to on Trello?")

Dim Contact As ContactItem

'search contacts'
For Each Contact In ContactsFolder.Items
    If Contact.FullName = SearchContactName Then
        If Contact.IMAddress = "" Then 'add trello card link'
            Contact.IMAddress = InputBox("What is the Trello card e-mail of this contact?")
            Contact.Save
            myItem.To = Contact.IMAddress
        Else
            myItem.To = Contact.IMAddress
        End If
    End If
Next

myItem.Body = NewMail.Body
myItem.Subject = NewMail.Subject

myItem.Display

My intention for this is that when it finds a contact, it will see if it has anything in the IMaddress. If it does have a link, then that will go into the "To:" of the e-mail. If it doesn't, outlook will prompt the user for a link, save the contact information and add the link to the "To:" of the e-mail. Like I said before, this code works as intended for any contacts that are in outlook, with or without a IMaddress.

The trouble I run into is when there is no contact information saved. Here is the code as stripped down as I could replicate the error:

Sub LeaveAComment()
Dim ContactsFolder As Folder
Set ContactsFolder = Session.GetDefaultFolder(olFolderContacts)
Dim SearchContactName As String
Set myItem = Outlook.CreateItem(olMailItem)
Dim NewContactEmail As String
Dim NewMail As Outlook.MailItem
Set NewMail = Application.ActiveInspector.currentItem

' Ask user for inputs
SearchContactName = InputBox("What is the name of your contact you want to comment to on Trello?")

Dim Contact As ContactItem

'search contacts'
For Each Contact In ContactsFolder.Items
    If Contact.FullName = SearchContactName Then
        If Contact.IMAddress = "" Then 'add trello card link'
            Contact.IMAddress = InputBox("What is the Trello card e-mail of this contact?")
            Contact.Save
            myItem.To = Contact.IMAddress
        Else
            myItem.To = Contact.IMAddress
        End If
    Else
        If MsgBox("Contact does not exist. Would you like to add one?", vbYesNo, "Confirm") = vbYes Then
            End
        Else
            End
        End If
    End If
Next

myItem.Body = NewMail.Body
myItem.Subject = NewMail.Subject

myItem.Display

It is when I add the

Else
        If MsgBox("Contact does not exist. Would you like to add one?", vbYesNo, "Confirm") = vbYes Then
            End
        Else
            End
        End If
End If

that I run into problems. Now when I run the macro, the msgbox will trigger no matter if the Contact already exists or not. For some reason the original If Contact.FullName = SearchContactName is being overridden by the Else, even though the Else shouldn't be triggered (at least in my thought process). I've tried reversing the logic and doing:

For Each Contact In ContactsFolder.Items
    If Contact.FullName <> SearchContactName Then

But that brought out the same outcome.

Any thoughts?