Basically, my scheme relies on the client supporting JavaScript (in my implementation I simply resort back to plain-text if Javascript is disabled or not supported) and having some sort of hashing (I use, and will use in this page, MD5) implementation available.
What happens, is when the client registers their password is stored in the database as the HMAC-MD5 hash of the password keyed with a concatenation of the username (lowercase) and a server-wide, constant salt (prevent rainbow table lookups if the database is compromised - not that they couldn't just authenticate with the salted hash out of the database, but at least it stops recovery of their plain-text password, which may be used elsewhere). From here, a secure authentication would go like this:
This design provides a few key advantages
Keep in mind, that the challenge must be stored or kept track of server side to provide protection against replay attacks. As well, if the initial password transfer at registration is compromised at all, this whole solution is pretty useless. I suggest using some sort of symmetric encryption or the like to at least attempt to provide some sort of security at this step.
If you're interested in trying to implement this design, I might suggest looking at Paul Johnston's site where he has BSD-licensed Javascript code to perform MD4, MD5 and SHA1 hashes.
If you're reading this and seriously analyzing this page and worrying about things like the randomness/uniqueness of the challenges and how secure the random number generator is, the possible implications of mixing multiple layers of hashing, etc, you may want to look into simply getting an SSL certificate or something similar rather than using something like this. Or use both. Just understand that securing your authentication doesn't do much if you have other easier attack vectors open or if you're trying to hide information (it will still be transmitted plain text). You're only as strong as your weakest link. If you're still allowing remote root logins with a password of 'flower' securing your site is pretty useless.
Once you're authenticated, if you had any sort of secure symmetric algorithm, you could use any piece of information that both ends know but that wasn't transmitted as a key. For example, if you transmitted the hash of the username instead of the plain text during authentication, and had the plain and hashed version stored server-side, the client and server could then use any sort of symmetric algorithm with the username as the key and provide encrypted communications for all sorts of page contents, etc, without SSL.
Another cool idea: Create a random challenge with each request and send it via SMS to the user (make this optional?) and require its entry along with the username (you could require the password (something you have (phone), something you know (password)) but sans password provides a low-risk way to log in on untrusted machines, as the SMS code is effectively a one-time pad).
I am not a cryptographer. I am not a security specialist. I have no formal training in any of this. I cannot guarantee the accuracy of statements made here or the security of this design. I will not be responsible if it fails.