Could be the battery is weak, or the connections were preventing it from charging properly. I'd have the battery checked/tested before dropping the money on a new starter (AutoZone or PepBoys will usually test the battery for free). If the car starts fine after the battery has been "recharged" (using a battery charger), I'd check to make sure all the cable connections between the battery, alternator, and starter are clean and tight. Finally, if the battery still doesn't stay charged from using the car normally, I'd have the alternator checked out, or look into upgrading it.
Did you clean the connecting surfaces of the battery terminals and the connectors on the clamps while they were off?