Get authors by term id or slug

This question sounds wired but I am in a situation where I need to list all author who posted in a particular term.

What I did so far, I am able to get all posts under a particular term and stored the author id into an array. But, the problem is that to create pagination over it. Does anyone know any better way to do that or any query to achieve that or any functions to solve my issue(I tried to find it out but did not get any good solution)?