Why anything done on comments_array hook gets reset?

I am writing a plugin that will allow simple comment upvoting.

I am trying to sort comments by the number of upvotes. But anything I do on the comments_array Hook gets somehow reset. My question is where and how?

Here is my code:

The var_dump($arraya) outputs a modified array in the proper order, but the comments show on a page as if the filter was not run.