You can't use the meta_query() in the get_term() function. But you can achieve the desired result by slightly modifying your codes.
First you need to fetch the post ids from desired custom post type.
$portfolio_ids = get_posts(
array(
'post_type' => 'portfolio',
'posts_per_page' => -1,
'fields' => 'ids', // return an array of ids
)
);
Now use the collected ids with the get_term() function.
$portfolio_categories = get_terms([
'taxonomy' => "category",
'object_ids' => $portfolio_ids,
'hide_empty' => true,
]);