A hat-trick of tries as well as two penalties and a conversion saw teenager Tovey end the game with a match-winning 23 point-haul that equalled Percy Montgomery’s club record.
It capped a stunning fight-back by the Dragons who trailed 16-8 before scoring 18 unanswered points in as many minutes.
The victory lifted the Dragons ahead of Ulster into eighth place as the third ranked Welsh region.
And there was more good news at Rodney Parade as Wales star Kevin Morgan ended his seven-month lay-off through injury as a second-half replacement.
But it was another full-back, 19-year-old Tovey who stole the show after getting the Dragons on their way in the sixth minute.
Ulster were reduced to 14 men when Tom Court was shown a yellow card for obstructing Wayne Smith and the Dragons made the visitors pay.
Wing Richard Fussell provided the spark, beating two men before Ashley Smith’s pass sent Tovey over in the corner.
It was a perfect start but the Dragons struggled to build as Ulster, once back up to full-strength, took a hold of the game.
Hoani Macdonald’s error in midfield allowed Simon Danielli to score before Ian Humphreys and Clinton Schifcofske kicked Ulster into a 16-8 lead after an hour, with Wales prop Rhys Thomas also in the sin bin for obstruction.
Tovey reduced the damage with a penalty and once Thomas returned to the fray, the Dragons launched a blistering comeback.
Ulster captain Kieran Dawson was the third player to see a yellow card on 69 minutes as the Dragons pressed and once again the home side made the advantage count.
Smith and Joe Bearman went close before Tovey, now playing outside-half, scrambled over.
His conversion drifted across the posts but after landing a 77th-minute penalty, the youngster pounced on Danielli’s fumble to complete his hat-trick in the final moments of an exciting game.
