Perl directory is getting past if(! -d) statement?

Okay so I have a program that basically looks into a passed in directory, if any file names match a pattern I will make a directory and move that specific file and any that matches it (regardless of extension) into that directory. Now if they don't match I should move them into the PassedInDir/misc/ directory.

I have a condition in both cases to avoid passing in any directory (as my program isn't ready to deal with those yet) something like if( ! -d $fp).

Everything works fine when I run it the first time in the directory. However when I run it again on the same directory (which should now only contain directories) I get the Error Could not move file assignmentZ to destination DataB/misc at projectSorter.pl line 16.. AssignmentZ is a directory however its somehow getting past the (!-d) in the second case.

#!/usr/bin/perl -w
use File::Copy;
if(@ARGV < 1){
    print "\nUsage: proj6.pl <directory>\n\n";
    exit;
}
die("\nDirectory $ARGV[0] does not exist\n\n") if( ! -e $ARGV[0]);
opendir( DIR, $ARGV[0]) or die("\nCould not open directory $ARGV[0]\n\n");
while(($fp = readdir(DIR))){
    if($fp =~ m/proj(.*)\./){
        (! -d "$ARGV[0]/assignment$1") && (mkdir "$ARGV[0]/assignment$1");
        move("$ARGV[0]/$fp" , "$ARGV[0]/assignment$1") or die("Could not move file $fp to destination $ARGV[0]/assignment$1");
    }
    elsif(! -d $fp){  #gets past here!!!
        (! -d "$ARGV[0]/misc") && (mkdir "$ARGV[0]/misc");
        move("$ARGV[0]/$fp" , "$ARGV[0]/misc") or die("Could not move file $fp to destination $ARGV[0]/misc");
    }
}

It the only directory to do it out of the ones previously made by running my program once. I am curious about why this is happening.