I sympathize with you. (I'm in a rural area, but it appears that the problems are similar.) I've managed to find sources for most things I need but I really have to plan ahead and make heavy use of mail order (via the net), eBay, and overseas vendors.
For most fasteners, I've found a couple of industrial supply places where I can get almost any bolt/nut/screw/washer - very high quality, most sizes & threads, and priced by the pound. However, each trip costs me about 3 hours so I have to stockpile the stuff myself. Our local NAPA has things like Plastigauge, or will order it overnight. Their list prices are high but it's not hard to get at least 30 off from them if you have part numbers ahead of time etc.
Some things I get from the net - small, specialized things mostly. I've found a few large independent hardware stores with huge fastener stocks and a couple even offer bar stock - brass, aluminum, steel, and some plastics so I can sometimes make what I need. Again, I usually have to spend a half day to get to one. We have Home Depot & Lowes but I seldom get anything I can use from them. Same with the large car parts chains except for NAPA.
Some of the aggie places (e.g. Tractor & implement dealers) are very helpful, but probably hard to find in NYC. I have several friends with commercial shops who help, but again - it's a trek.
I have a good British car place about 3 hours south in New Hampshire. They're a Moss dealer and will mail. I also have several Internet guys I use for things I can't find elsewhere like Hylomar and similar.
My biggest problem is that I can only get things by mail. UPS, Fedex, et al won't deliver because of our road. A neighbor a few miles away will receive things for us but I hate to impose too often.
It's ironic. When I was a kid, NYC was THE place to go for "foreign car" parts. We frequently went down there. I don't remember all the places (was Inskip there?) but one was Hoffman Motors, mostly for BMW parts, and we once went to Luigi Chinettis for a single valve for someone's 4 cylinder Ferrari. ($50! Robbery!)
I don't know what to suggest, other than research and then some carefully planned trips to suppliers outside the "City".