Rinse the olives briefly under cool water in a colander to remove any packing brine or sediment. Drain well and pat dry with paper towels. If any olives are whole with large pits and you prefer, gently crush a few with the side of a knife to allow more marinade penetration (optional). Transfer the olives to a medium mixing bowl.

Prepare aromatics

Using a microplane or fine zester, remove the bright orange part of the orange peel to yield about 1 tablespoon of zest (avoid the bitter white pith). If using lemon, zest about 1/2 teaspoon. Lightly crush the garlic cloves with the flat of a knife to release aroma (you can mince them if you prefer more garlic flavor). Strip the rosemary leaves from two sprigs and leave one sprig whole for garnish if desired.

Combine marinade

In a small bowl or measuring cup, whisk together 1/2 cup extra-virgin olive oil, 1 teaspoon vinegar (optional), 1/2 teaspoon coarse sea salt, 1/2 teaspoon freshly cracked black pepper, 1/2 teaspoon red pepper flakes, the orange zest (1 tablespoon), and the lemon zest (if using). Add the lightly crushed garlic and torn rosemary leaves and stir to combine.

Marinate olives

Pour the oil and herb mixture over the drained olives in the mixing bowl. Add the whole rosemary sprig on top for extra fragrance. Toss gently with a spoon until all olives are coated. Transfer everything (olives, oil, aromatics) to a clean jar or airtight container, scraping any marinade from the bowl into the jar.

Rest and develop flavor

Seal the jar and let the olives sit at room temperature for at least 2 hours to absorb flavors. For best flavor, refrigerate and marinate for 24 to 48 hours; bring to room temperature before serving. Shake or stir the olives once or twice while marinating to redistribute oil and aromatics.

Serve

Remove the rosemary sprig if you prefer, or leave it in for presentation. Serve olives in a small bowl with some of the marinade spooned over them and an extra drizzle of EVOO if desired. Provide toothpicks or small forks for guests. Store leftovers refrigerated for up to 2 weeks; allow to come to room temperature before serving for best flavor.