Workgroup mode will not work with Windows security for the wire. Windows needs to own both sides of the communication (client and server) on the same Active Directory domain (or between trusted domains).
A quick overview of options is in the online help - search on Network Protocol Bindings
Some background on the WCF feature leveraged: