That's what due process means. If you deprive someone of life, liberty, or property, without having convicted him of a crime, then you are denying him due process, and violating his rights under the Fifth Amendment.
We don't get to sentence someone to prison, or a fine, or any other punishment, until that person has been properly convicted of a crime, in accordance with the processes that we have established to do this, and in accordance with the protections that we give to those accused of a crime.
Asset forfeiture, as currently practiced, is an attempt to get around due process, to deprive someone of property without convicting that person of a crime, or even giving him a fair chance to defend himself against the accusation. As such, it rather blatantly violates the Fifth Amendment, both in letter and in intent.