I got here while looking for a solution. I did not find anything until I found a solution by myself. Here it is, just for info. I hope it helps someone.

$( .open-fancybox ).fancybox({
     afterShow : function() {
        $( #mediaelement ).mediaelementplayer();
    }
});

<a class="open-fancybox" href="#fancybox">
<div id="#fancybox">
    <video id="#mediaelement" src="...mp4">
</div>

I recently encountered the same problem as you did on a project. In my case, fancybox properly loaded the mediaelement player in the modal window. However, on IE8, the flash player was presented and you could then see the controls get stripped away and never replaced.

After looking at the plugin a bit more, it seems mediaelement listens for an event fired by the flash player in order to initialize the controls for the player. Depending on a number of circumstances, I was finding that the player was emitting that event BEFORE the plugin was called, which meant that the plugin never determined the player to be ready.

The fix for me was to explicitly set the flashName option to the path of the swf. Then, I simply removed object markup from my source, allowing the plugin to create it.

Explicitly setting the path to the swf:

    <script>
    $(document).ready(function(){
      $( video ).mediaelementplayer( {
      flashName:  /path_to_mediaelement_swf/flashmediaelement.swf ,
      });
   });
  </script>

Video source:

<video width="320" height="240" poster="poster.jpg" controls="controls" preload="none">
    <!-- MP4 for Safari, IE9, iPhone, iPad, Android, and Windows Phone 7 -->
    <source type="video/mp4" src="myvideo.mp4" />
    <!-- WebM/VP8 for Firefox4, Opera, and Chrome -->
    <source type="video/webm" src="myvideo.webm" />
    <!-- Ogg/Vorbis for older Firefox and Opera versions -->
    <source type="video/ogg" src="myvideo.ogv" />
    <!-- Optional: Add subtitles for each language -->
    <track kind="subtitles" src="subtitles.srt" srclang="en" />
    <!-- Optional: Add chapters -->
    <track kind="chapters" src="chapters.srt" srclang="en" /> 
    <!-- Allow the plugin to generate the object markup, preventing the swf source from loading too early  -->

</video>

Hope this helps!

I have just had the same issue, it turns out than in IE 7/8 if you have na (the flash player) in a div that is display:none the flash player wont load the video correctly.

The fix for me was to take the display none off the div, and use jquery to hide all divs with a class of .hidden_video.

After doing this, the video loaded correctly in the fancybox.