Basically, you should follow the "best practices" provided by Microsoft. That is, all files should go in their corresponding directories. You always have the right to write files in the application data directory, so you put the files your application will need to modify in there.
One could argue the .exe file is actually data for your processor to chew and be tempted to install the application in the application data directory... I don't know why that's not a good idea, but I'm pretty sure it isn't... Maybe Windows 7 will have rules that say "any file you don't modify must not have writing permissions or I'll throw exceptions all over your code"
Seriously, if you run your application on an Windows XP computer that has limited access to the file system (say, a manager decides that salesman should not have write access to their program files directory...), you'll have just the same problems.
The problem now is the default rights that changed. Now, even when you are administrator of your computer, the processes you start do not have those rights by default. They must be given explicitly. Then whenever you tried to run your application and forgot to say "run as admin" everything would crash, so the UAC pops a window in your face to ask you "do you want to allow this?". If you are administrator, you can say yes and go on with your life, otherwise you need to type in the account info of an admin. That is all good and is actually just a fancy name for what Linux does by default (Actually, on Linux you can login as root and launch processes as root without a problem, you just always login as a normal user)...
The main advantage of this is that any virus that comes on your computer can delete you application files, your desktop icons, etc. But they cannot touch anything that would crash the system. They cannot install themselves in key parts of the system or replace key dlls with viruses and stuff. Much more secure. Obviously, if my application data disappears, I don't care much that the computer won't boot anymore, I lost all my work and I'll have to reinstall the machine anyway! But it's still more secure...
Then comes the real problem. You must put a manifest to your application to tell the OS what rights you need, otherwise you'll get an exception. Microsoft quickly understood that people would rather have a security risk than not to have a software at all and all those "legacy" (as they call them) applications that don't even have a manifest have cost billions to write! Who would buy a brand new OS that nothing works on? So they made lots and lots of exceptions to make sure the application you had will still work.
That's where the file system virtualization concept appeared. Whenever an application with no manifest will try to write to a directory it has no right on (that works for the registry as well), the files will actually be in an hidden folder (virtual store) somewhere in your application data directory. You application thinks everything works fine, but it's actually not doing what you told it at all!
Now that makes for a "legacy" installation program that doesn't crash, but doesn't do what you expected... But remember those "lots and lots of exceptions"? Executables that were recognized as installers get full rights on the file system. I have no idea how they made that secure, but if I worked for Viruses inc. I'd be digging that hole to make my virus look like a legacy installer...
