The data needs to pass through more buffers to be read out on a larger sensor.
Imagine the sensor has 24 million pixels. There aren't 24 million wires connecting the sensor to the CPU. Instead, the data is read out sequentially through buffers, usually line by line. Most DSLRs even skip lines to speed up the process, negating any large sensor advantage you might have had.
On a smaller sensor the electrons have to travel less far so the read-out can be made much faster, as long as the CPU can handle it.
This is why larger sensors have more rolling shutter effects, lower frame rates and worse video.
This is why a GH2/GH3 works so well for video. The sensor is designed for video. The sensor is smaller and is hooked up to faster CPU's. Rather than skipping lines, the sensor does some clever binning of the pixels.
On point & shoot cameras, you just have the combination of a microscopic sensor and a processing unit that destroys most of the quality. Those videos usually look like they were shot with a phone.
