Skipper Chris Willmott stepped up to the penalty spot to send his Northampton Town side through to the LDV Vans Trophy Southern Area quarter-finals after a thrilling tie at Edgar Street.
After Lawrie Dudfield had put Northampton ahead 15 minutes from time and David Brown equalised 12 minutes later, the sides were unable to settle the contest in extra time and it was down to penalties.
And with Hereford leading 3-1 after three penalties each, the Bulls looked odds-on to progress.
But top scorer Steve Guinan blasted his kick over and Jamie Pitman's effort was saved by Lee Harper so, with Dudfield and Martin Smith finding the net, it was sudden death.
Trialist Nick Harrhy's effort was saved by Harper, but Willmot made no mistake, firing his effort into the corner of the net for a memorable win.
In reality, Northampton should have settled the game within the 120 minutes as they succeeded in hitting the woodwork four times in the course of an exciting tie.
Eddie Asamoah, Dudfield and Smith, twice, all saw efforts come back off the post or bar and be cleared to safety.
Smith's two efforts both came during the added 30 minutes.
But on a night when, accurate finishing was definitely not the order of the day, Hereford also managed to rattle the goal frame, when a superb move ended with Guinan driving a Ryan Green cross against the angle of post and bar.
Hereford had looked the likelier scorers as the game progressed, but Dudfield held off Tony James to fire in a low left-footed shot.
With time running out, Rob Purdie raced out of defence and passed to Pitman, who slipped the ball inside to Brown, who hammered in a shot from the edge of the area.