as the binary is already loaded into memory

That’s not the reason why it continues. It’s because there’s still a file descriptor open to rm.