Combine the olive oil and the orange zest in a small bowl and set aside.
Make the fregola salad
In a large saucepan over high heat bring the chicken broth to a boil. Add the pasta and cook, stirring occasionally, until tender but still firm to the bite, 10 to 12 minutes.
Drain the pasta and turn it onto a large baking sheet. Spread it out into a single layer and let cool for 10 minutes.
Meanwhile, using a small knife, cut all the peel and pith off the orange and grapefruit. Holding the fruit over a large bowl to catch any juice, cut between the membranes to release the segments into the bowl. Add the onion, mint, basil, fennel seeds, salt, pepper, and cooled fregola pasta to the segments and juice in the bowl.
Add the reserved orange oil to the pasta. Toss all the ingredients and serve.