Suspension knocks?
 

I have a very subtle knocking sound that is not engine-related and no regular frequency. It goes away during stops. I can hear it as the car starts moving from a dead stop and it continues until the road noise drowns it out. It's almost as if one of the springs broke and the two ends are rubbing against each other, but the springs look good.

Diagnosing noise from description is a bliss, but sway bar end links, upper strut stops and engine motor mount (near timing chain) are the prime suspects

I removed both front wheels. I rocked the steering left and right, and I think the knocking is coming from the driver's side sway bar. What's the risk if I leave it alone?

You may be referring to the driver side tie rod end. Sway bar end links making most noise when going over speed bumps , especially with one wheel only.

If it is tie rod ends (which is not likely based on your description), unlikely you would hear a noise all the time while moving. It would present as unusual wear on the tires and drifting left or right to where you have to correct with the steering wheel to keep the car going straight.
My guess is either control arm bushings, or sway bar end links or sway bar bushings (which make the most noise on choppy pavement or going slow over bumps).
Sway bar end links are cheap and easy to replace. Sway bar bushings are challenging as you really need to drop the subframe for access to the bolts on the driver's side (iirc).
With any of these, the result is premature wear of tires, and likely other bushings in the front suspension due to more movement. Like doctor_J said, though, diagnosis over the internet is not easy.

I tested my car by opening the front door to get more leverage and rocking the body up and down. The bad sway bar end link began to clank, leading to replacement (if was "visually good", no torn boot) another signof bad link is an "old bed spring" noise produced when driver sits on the seat first time in a day