An ill-disciplined second half saw Ulster snatch an 18-12 victory at the Liberty Stadium.
The home side had led 12-0 early in the second half thanks to four penalties from the prolific boot of Dan Biggar.
But the hosts lost their composure after the break to hand Paddy Jackson six attempts at goal, kicks he took with aplomb.
With both sides reduced to 14 men during a frantic final few minutes, the result remained in the balance until the last whistle. The Ospreys gave it everything in the final minute, Alun Wyn Jones leading the charge on his 150th appearance for the region, but it was not enough.
The losing bonus point took Ospreys top of the table though this was a disheartening result on the eve of the Heineken Cup.
It had looked a different story shortly after half-time against the team that ended last season top of the league.
Biggar kicked the opening points after 20 minutes but it was not until the end of the first half that he added a second kick.
However, the Wales fly-half doubled that led within five minutes of the second half to hand the Ospreys what looked to be a commanding lead in a game of precious few chances.
That prompted a response from Ulster and Ospreys lost their heads.
A series of penalties, including two at the scrum, allowed Jackson to firstly level the scores on 64 minutes and take the lead with two kicks in the final seven minutes.
The second of those kicks saw replacement prop Aaron Jarvis sin binned.
The Ospreys mounted a spirited late flurry that saw Stuart Olding binned but despite Jones, Richard Hibbard and Justin Tipuric all weighing in, they fell short.
