I can't seem to beat the guardian in this game. I know what to do but I'm having trouble with the associated Quick Time event associated with it, The button prompt flashes but no matter how fast I mash on the buttons I can't seem to push him over the ledge. Any tips?

That's kind of weird. Usually I just hang out near an edge, wait for him to pick me up and throw me behind him (now HE'S right in front of the ledge), then charge and attack him like crazy. This starts up the "oh shit he's going to step on me" QTE, which once you win, will send him flying (since he's near the edge).
 
I think....It's been a while since I played that one.