Accuracy depends on a wide variety of factors coming together at a particular location and time. They include distortion of GPS signals as they travel through the ionosphere and errors in the position (ephemeris data) transmitted by GPS satellites.
Here are some factors recreational users will probably find most significant in everyday settings:
Number and Position of GPS Satellites
For most end users, accuracy is heavily influenced by having usable signals from several GPS satellites instead of just a few. Positions reported by your GPS receiver will probably be more accurate if it's using at least 5 or 6 satellites and not just three or four.
The position of satellites being used is also significant. Your calculated location is likely to be more accurate if the satellites you're using are farther apart (their signals coming to you from widely different directions).
Moving Around / Ephemeris Data
If you're moving around, the set of satellites your GPS receiver is able to use will probably change as you pass by buildings, hills and other obstructions. You'll get better accuracy if your receiver has already acquired ephemeris data from most of the satellites that are currently in your area. That way, it can quickly switch to using other satellites as signals shift in and out of view.
GPS receivers can gain additional accuracy by applying WAAS corrections to data they're receiving from GPS satellites. Where available, WAAS can provide corrections for errors in a particular satellite's timing and orbital data as well as localized corrections for distortions caused by the ionosphere.
GPS Receiver Quality
The quality of algorithms and circuitry used for processing satellite signals has an impact on accuracy. Most consumer grade receivers can do a pretty good job at getting your position within about 25 feet much of the time. For applications where accuracy is more critical, more sophisticated equipment can provide better results.
Finally, it should be mentioned that the U.S. government has the ability to degrade the accuracy of GPS signals using Selective Availability. This was an issue prior to May 1, 2000. However, it's not expected that this will ever be used again on a global basis.