i dont know how adept you are in video stuff. so i'm just gonna start from the basics.
you cant use the mini display port, because thats an "out".
ok. so in theory..
using the HDMI output from the PS3, you can get an HDMI to firewire. which is a video input to your mac.
then you would need some sort of media server to stream the video from the ps3.
http://mediatomb.cc/
disclaimer. dont download that unless you looked into it. but thats just what i found.