PuTTY

PuTTY is a perfectly portable application. The champion of Telnet / SSH / sFTP clients needs no installation.

update 10/5
I received an email from Lolitka mentioning that PuTTY will save session date to the Windows registry and asking if there was a way to bypass this and save the information on the flash drive. The easiest way is to not save sessions, but to invoke Putty with command-line options to automatically connect to the host. There is no way to actually save a session to the USB device that I'm aware of.

Filezilla

I also recommend Filezilla in my standard desktop software category. Simple and clean, it runs perfectly off a USB drive.

Pidgin

You're on the road and need to have a quick chat with your family. You can't install software on the machine you're working on. What do you do?

Chat on all the networks (AIM, ICQ, MSN, Jabber, and Yahoo! Messenger) with one client that you carry in your pocket. Pidgin has a plugin architecture that extends to every new chat server or protocol that comes up including Skype.