In Prisoner of Azkaban, Lupin tells Harry the dementors affect him worse than the others because he has horrors in his past that others haven’t got. But why would his worst experience be the night his parents died, something he doesn’t actually witness, remember, or understand? Surely that’s nothing compared to the encounter with the giant-ass snake that nearly killed him in the Chamber of Secrets.
But what if, as a horcrux, he’s experiencing the effects from Voldemort’s perspective? Riddle’s worst fear is death. The night he tried to kill Harry, he very nearly died, was ripped from his body, and lost all his power. Not only is it Riddle’s worst memory, it’s also much worse than anything Harry’s ever experienced, so it’s what the dementors latch on to. He collapses under the combined pressure of Riddle’s horror and his own shock at having the them tap into the other soul co-existing in his body.