Problem looping through files (using Powershell) and moving them one at a time, based on error result

I am using Powershell to validate multiple XML files against multiple XSDs; this portion of the code is working as expected, however I also need to move any XML which fails to validate to an "Invalid" folder. I am attempting to loop through these files using ForEach, and then - using an If statement - to move any file which errored. My problem is that all files are being moved, whether or not they errored.

I've written this loop in as many different ways as I could conceive, but I'm not getting the result I expect. (I have also scoured the web for days to find the answer.) I need ForEach to apply the code to each file, one at a time. Is this a problem with my syntax? Perhaps I'm missing something very obvious, but I'm now at a loss.

I am using this function (found on Stack Overflow, and as shown here, slightly tweaked) to validate the XMLs.

function Test-XmlFile
{
    <#
    .Synopsis
        Validates an xml file against an xml schema file.
    .Example
        PS> dir *.xml | Test-XmlFile schema.xsd
    #>
    [CmdletBinding()]
    param (     
        [Parameter(ValueFromPipeline=$true, Mandatory=$true)]
        [SupportsWildcards()]
        $SchemaFile,

        [Parameter(ValueFromPipeline=$true, Mandatory=$true)]
        [SupportsWildcards()]
        [alias('Fullname')]
        $XmlFile,

        [scriptblock] $ValidationEventHandler = { Write-Error $args[1].Exception }
    )

    begin {
        $schemaReader = New-Object System.Xml.XmlTextReader $SchemaFile
        $schema = [System.Xml.Schema.XmlSchema]::Read($schemaReader, $ValidationEventHandler)
    }

    process {
        $ret = $true
        try {
            $xml = New-Object System.Xml.XmlDocument
            $xml.Schemas.Add($schema) | Out-Null
            $xml.Load($XmlFile)
            $xml.Validate({
                    throw ([PsCustomObject] @{
                        SchemaFile = $SchemaFile
                        XmlFile = $XmlFile
                        Exception = $args[1].Exception
                    })
                })
        } catch {
            Write-Error $_
            $ret = $false
        }
        $ret
    }

    end {
        $schemaReader.Close()
    }
}

And here is how I am selecting XMLs for validation against their given schemas.

$allfiles = "..\Schema Validation\XMLs\*.xml"

$xml1 = Get-ChildItem $allfiles -Recurse | Select-String "<UniqueElement>" -List | Resolve-Path
$xml2 = Get-ChildItem $allfiles -Recurse | Select-String "<UniqueElement>" -List | Resolve-Path

$xsd1 = "..\Schema Validation\Schemas\Schema1.xsd"
$xsd2 = "..\Schema Validation\Schemas\Schema2.xsd"

And here is the ForEach loop that's not working for me. (In its current configuration, though I've written it a dozen different ways.)

ForEach ($xml in $xml1) {
$xml | Test-XmlFile $xsd1
If ($Error) {
$Error[0].Exception, "`r" | Out-File "..\Schema Validation\Results\log.txt"
Move-Item $xml -Destination "..\Schema Validation\Invalid"
}}

The ForEach loop above is also repeated for the $xml2 and $xsd2 variables. (And as you can see from the Out-File, I'm also capturing the exception message in a text file for a log of sorts.)

I expected only those XMLs which error and hit an exception to be moved, due to the "If ($Error)" statement and the fact that I'm attempting to loop through the files one at a time; however, what happens is that any XML which contains the unique string that identifies it as part of the $xml1 or $xml2 group is moved to the Invalid folder, error or no error. So what painfully obvious thing am I missing?? (Incidentally, the exception text populates the error log as expected, so at least that part is working as I hoped.)