Delivering a Windows XP system to a customer, and don't want them to fiddle with settings?
First, you probably want to have your system log on automatically.
The Microsoft article on enabling autologin goes through the steps in detail. Essentially, start regedit and set HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon AutoAdminLogon (SZ) to 1.
Then add whatever you want on startup to the start menu 'Startup' folder. (Documents and Settings\USERNAME\Start Menu\Programs\Startup)
This is also a good time to clear the startup menu of any other pesky programs. To eliminate system wide startup items easily you can try MSConfig (from run), but I prefer to edit the registry directly.
Check HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run and RunOnce and RunOnceEx. If you just want to eliminate the taskbar icons all together you can try editing HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer and add a DWORD NoTrayItemsDisplay and set it to 1.
You can disable the windows keys etc using HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout\Scancode Map
For a detailed discussion see Tim's Northcode blog post, who suggests "Scancode Map"=hex:00,00,00,00,00,00,00,00,09,00,00,00,00,00,5b,e0,00,00,5c,e0,00,00,5d,e0,00,00, 44,00,00,00,1d,00,00,00,38,00,00,00,1d,e0,00,00,38,e0,00,00,00,00.
Note that this stops you from being able to use any of the windows special keys. So be careful!
Finally you want to lock down the user access rights. There are many ways to do this, the easiest is with the Group Policy editor. This Microsoft article on the Group Policy Editor describes how to get there, in essence: Start,run,MMC,Add a snap-in,'Group Policy Object Editor'.
Now you have options to edit under "User Configuration" and "Administrative Templates". Under "Start Menu and Taskbar" you can find options to disable Search, Help, etc. from the start menu. Under "Desktop" you can remove the various icons, and disable the desktop. Under "System" you can set which programs people can run. You might want to be careful not to disable regedit until you are 100% certain it is correctly setup.