PSA: To make your USB drive audio files load faster, remove embedded album art
 

I noticed that some albums loaded faster than others, and I think I finally figured out why. I used Mp3tag to remove the album art from the slower-loading ones, some as large as 10MB, and now they load instantly.

Shortening the file name did not help. Removing the album art did.

Also if you're having issues with songs being loaded in their correct order, make sure that they are tagged correctly with their tracklist numbers and then use DriveSort (available on Windows) once they're copied onto the drive to "sort" them correctly.