Lots of pedestrians (and jay walking). Lots of construction. Potholes. Cyclists. Narrow streets which functionally have about 1.75 lanes a lot of the time. Buses and bus stops. Double parking delivery vehicles. Detours. Etc.
Consider these recurring issues experienced by many of these companies’ vehicles:
Disengage for a recklessly behaving road user
Disengage for hardware discrepancy
Disengage for unwanted maneuver of the vehicle
Disengage for a perception discrepancy
Disengage for incorrect behavior prediction of other traffic participants
Heavy pedestrian traffic
Cyclist
Traffic light detection
Construction
Localization divergence
Poor lane markings
Vehicle cut in
Cyclist riding from sidewalk into crosswalk
Vehicle in cross-traffic ran red light
Software crash
Unexpected steering due to path change
Strong unexpected braking
Also rain, snow...
As I keep saying, I think "safety" is a bit of a red herring. You can program them not to bump into things. You can't necessarily program them to not be nightmare drivers for everyone around them.