You are somewhat ambiguous with your terminology - do you want a client app that runs on the user s machine, or do you want an app that runs on the server and the user connects via remote desktop (RDP)?
If you are talking about a client app that communicates to the server via WCF etc., then yes it will be faster than a standard web app, although it will still be slower than a native desktop app. It will be faster than the web app not just because of the lack of postbacks, but also because you will be sending pure data through the wire, not a massive amount of HTML/Javascript combined with your data. With a client app, you have several options so consider them carefully - do you want Silverlight, WPF, or a native WinForms app? Each have their positives and negatives.
If you were talking about having a client app running on the server which the user then access via RDP, then you have other considerations to think of. For any more than two concurrent users you will need to consider buying CALs so the users can connect to the server. At this point you should also be considering whether you should be running a terminal server or Citrix type setup instead of using remote desktop.
Edit
When using WCF over a WAN (internet) you will certainly have to consider how you will secure it. WCF makes it trivial to secure the channel, but you need to consider how you will do authentication - there are a couple of different ways, but you can easily google that stuff yourself. The method you choose will be important due to the limited resources or skill-sets of the users.
As for what you write it in, you can t argue with Winforms if that is where your experience is. Personally, i would never again use ASP.NET/Ajax/etc for a web type application, it would be WPF or Silverlight all the way (i would only use ASP.NET for simple web sites). You can use the express (free) versions of Visual Studio to write it in, you don t need Expression (it s just a nice to have, and is more aimed at the design side than the actual coding side). Deploying the app need not be difficult - Silverlight or WPF xbap are delivered via the web, the user has to do nothing (except for the simple install of the Silverlight plugin or installing the right .Net framework for WPF - check this link). Winforms or stand-alone WPF require slightly more work, but you can avoid most issues by writing a good installer.
Whichever you choose, make sure you don t under estimate the time for development (because you will have a bit of a learning curve), and also make sure you budget enough time for testing it - especially the security side of it :)