You should have mentioned what model/engine size. I'll take a guess and say you have the 2.2L 4 cylinder. If that's the case, you are looking at a throttle body, not a carburetor, since the 90 Cavaliers had TBI units. You should see injector "pods" in the throttle body.
A hard start can be caused by a number of problems. First off - when's the last time this car has had a tune up? Crudded up plugs, bad wires, cracked/worn cap/rotor in the distributor, clogged air filter - all of these things can cause a cold start problem, even if the engine runs decently warmed up. I'd go through that first, as well as checking the condition of the battery terminals/ground - resistance on the electrical side can make it more difficult to start as well.
Next could be the possibility that the injector(s) in your TBI unit are leaking down - if the fuel pressure is insufficient due to a leak, you can get a hard start condition until the injectors are fully pressurized and firing properly. Try this - the next morning, turn the key to "run" wait 3-5 seconds, then try turning the engine over. If it starts a little bit faster, you have a fuel pressure issue, most likely because of a leaking regulator or injector.
To answer your question - no, you shouldn't be pumping your gas pedal - a TBI unit should fire up just like any other fuel injection system.