Deploying only the compiled App.exe would in the simplest cases work, but it is better practice to always create setup package. It is a clean and familiar method for users where they don't have to create the application folder/copy the file/make shortcuts themselves, and keeping track of all what was done, and also automates uninstalls/upgrades. What if your application produces data files and user just places the exe on desktop? What a mess that would make.
VB2005 Express doesn't make a setup when you 'Build', it just builds the current application to the output path.
To make setup click 'Publish' or go Application Properties to Publish tab, which creates the ClickOnce deployment setup. This result in several files needed for install. To make a single file (for download and local install) there is the possibility to zip them as self-extractable.
Another option is to find and employ a MSI installer as your deployment tool, these often produce a single setupApp.exe. There exist one in VB2005 Professional and some standalones if you search web. MSI was also the only option in earlier Visual Studio versions before ClickOnce technology was invented.