Get terms that are associated with products from current category

I need to list all custom terms (brands) that are associated with products from the category that is currently being viewed. E.g. I have these brands created:

On ‘Shirts’ category page, only Shirts Brands A, B and C are displayed.

This is how I did it:

Of course it works, however with 300+ brands created it sometimes takes 0,5 – 1,5 seconds for this loop to get processed, which is unnecessary lot. AFAIK there is no direct database relation between terms and product categories (only number of posts for each term), but is there a better way to do this with better performance?