Interactive Elements Motion Tracked Onto HTML5 Video – It’s Possible

Interactive Motion Tracking
I read an interesting article the other day about IKEA sticking with Flash to deliver its ideas and there’s a quote in it from their marketing manager Nicolas Johnsson that says ‘Mobile-friendly tech [HTML5] does not support streaming video, animation of hotspot positions on top of video or sophisticated animation’ and  it got me thinking about the features and functionality of the components of HTML5 when compared to Flash especially when it comes to interactive video. We can do all sorts of clever stuff with Flash video, from overlaying interactive elements and embedding hidden links to animating the entire video itself in 3D space.

It’s all very cool and something we have grown to expect – the immersive and beautifully crafted experiences that companies like IKEA are creating show off Flash to its full potential (and often take it further) – however having spent the past few weeks delving into HTML5 video I’m pretty disappointed not only with what’s on offer but the reliability and stability of it; having to encode multiple versions for different browsers is a pain and then finding that certain functionality you wrote that works in one browser just ever-so-slightly doesn’t work as expected in another.

So, with the words of Nicolas Johnsson ringing in my ears (specifically the animation of hotspot positions) I’ve worked hard to show that it can actually be done albeit without the stability and browser penetration that the Flash player offers.

The process to get here has been pretty convoluted, involving several different bits of software and libraries (Flash, After Effects, Adobe Edge, Greensock JS, my own bespoke JS video timeline library – I had to get my hands dirty on this one!) and yes, it had to involve Flash in order to gather and generate the tracking data. So whilst this demo may well be a world first (I think it might be, I haven’t found it done anywhere else online yet but do ping me if I’m wrong) it still feels like it couldn’t have been possible without a technology that it seeks to replace.

Some things to note: I have tested it on Safari 5.1.7, Chrome 19, IE 9 and Firefox 13 and it seems to work well. It doesn’t work on an iPhone because the video is loaded into and played in the iOS native video player (which operates outside of the realms of HTML control). I’m pleased to report it does work on the iPad – it has been a little unpredictable although I have now managed to reduce that unpredictability down to almost zero now. There is a lot more work needed to simplify and streamline the workflow – if you happen to root through the code you’ll find it’s not very neat right now and lots of the code methods could do with optimising but it’s my first stab at it! Also the video is from the Adobe blog Creating Interactive Video with After Effects and Flash .

You can click/tap several times on the motion tracked label and see some data about the rider – the last bit of data is a link to a YouTube video to demonstrate that links can also be placed in these overlaid elements. What’s also cool is that the tracked data is synchronised with the video timeline using Greensock’s TimelineMax so when a user scrubs back and forth using the video controls (which aren’t enabled here but you can right click the video to enable them) the overlaid elements will also scrub with it.

I hope this experiment demonstrates the possibilities that HTML5 video can offer – I’m pretty excited by these results and the possibilities it presents.

And please do leave your thoughts and experiences of it in the comments – I’m really interested to hear how this is working in your browser!

Click here to launch the Interactive Motion Track demo

Click here for a tutorial on how to get Greensock’s Tween library working in Adobe Edge.

Update July 2014: If you liked this post check out my more recent 3D motion tracking stuff.
Chris Gannon Edge Animate Templates

24 thoughts on “Interactive Elements Motion Tracked Onto HTML5 Video – It’s Possible

  1. Hey Renata,

    To be fair this demo wouldn’t have been possible without Greensock, After Effects and Flash (AE for generating tracking data, Flash for retrieving that data and generating the code for use in Adobe Edge and Greensock for its excellent TweenMax and TimelineMax libraries).

    So whilst HTML5 is getting interesting (although there is always the underlying worry and general sense of uncertainty that ‘standards’ and functionality can just be removed by W3C at any time) I certainly had to go round the houses to get this working.

    Thanks for stopping by!

  2. @designms00

    Good question! The method I have used to do this requires knowledge of After Effects, Flash and FLV events, Greensock GSAP TweenLite and TimelineMax, JSON, HTML5 video (and their cross-browser issues), jQuery and Javascript.

    I started writing a tutorial for this and I had to pause, partly due to work commitments and partly because I realised it’s pretty complicated (ideally it would be a tutorial in 3 stages) and I couldn’t quite work out how to format it in terms of the level of knowledge I should expect the reader to have.

    Currently I’m refining and streamlining this process (by creating a bespoke app to do some of the work, generate JSON etc) which should make it easier to generate the code generated from After Effects so I’m kind of still in the middle of its development.

    You should definitely check out that link to the Adobe blog about creating interactive video – it provides a lot of the concepts that I translated to HTML5.

  3. Ok thank you so much for your reply!
    I am seeking information about that because I saw that you could do something for html5 video and wanted to test it in an email with video

  4. @Robert Bue

    It took a few weeks to create all the component parts and then get them all integrated and working together. Obviously I had to work out HOW to do it as well which took some time to organise in my head.

    So it was a case of generating the tracking data in After Effects and converting it to a usable format (in this case I wrote an app in Flash/AIR that converted the data from the FLV cue points and generated JavaScript/Greensock TimelineMax code that I could cut and paste into Edge).

    Then I made allowances for any position offsets (with the overlay graphics), created all the different vendor video formats, wrote a video timeline JS ‘class’ that worked in tandem with the GSAP timeline and finally created the interactive animating overlay elements (the racing statistic data).

    I’ve done it a few times since and it’s taken less time – one of the main time consumers is the AE tracking as the reliability of the track depends on the quality of your footage – it’s all good fun though!

  5. Great stuff. I tested it on the iPad retina (first release, the iPad 3), and if you replay the video a few times, the overlay does not move anymore, and the timecode always prints 0. Too bad browser vendors are allowed to have these buggy implementations of HTML5, it really puts a burden on all developers.

    1. Interesting. Since I created this I’ve developed a new, more streamlined way of doing this (outside of Edge) with tighter, simpler code and so I’ll post a new version that also uses the latest Greensock and jQuery libraries to see if that improves things.

      Thanks for testing it and posting your feedback – its very useful!

  6. Hi Chris,

    I’m doing similar experiments, and I found that Video.CurrentTime is not very accurate in most browsers. IE9 and Chrome do fine, but Firefox and Safari on iOS are not accurate at all… Furthermore iOS does not support Canvas.DrawImage(Video)… Brilliant 😦

    Have you been dealing with this too?


  7. Hi Peter,

    I think one reason currentTime is sometimes not all that accurate may be due to the key frames in the encoded video being too far apart. I tend to encode my videos with a key frame every frame – it makes larger videos (file size) but when progressively downloading this is somewhat irrelevant.

    I’m not using canvas for these interactive video projects as I’ve found DIVs and Greensock’s TimelineMax sync’d up to the video timeline ( all controlled by Greensock’s ticker works pretty well. I’ve also found WebM and MP4 to be fairly inconsistent with Chrome now defaulting to WebM when possible (I’m not getting very nice encoding results with WebM right now – pretty grotty format IMO).

    Of course you get the best results when you encode the video yourself with plenty of key frames to ensure granular accuracy which of course is not always possible, but for something like a motion tracking project you would usually be in control of that end of things anyway.

    What kind(s) of thing are you working on and how are you encoding stuff?

  8. Hey Chris,
    I have been working on that same idea for quite some time now. The main issue here really is the fact that ios on iphone does not allow inline videos … I hope that this will change in the near future !
    You can have a look at my solution here: I hope you like it 🙂

    1. That’s very cool – nicely done! How are you generating the shape outlines? We should chat.

      Edit: I just realised your demo is not actual video playing – seems you are rendering video frames to PNG (with canvas overlays)?

      How do you propose to have sound in your interactive demo? Again, great work!

      1. The objects are pre-identified using curves with motion tracking tools and are manually adjusted to follow exactly the shapes.
        They are then displayed in the application on a canvas using stroke() on top of the video and are synchronized on every frame change.
        You have my email on my website if you want to chat 🙂

      2. Sorry, just noticed your edit 🙂 (and have no idea how to edit my first answer …)

        The video is an actual video element, it is just not added to the DOM and I am using drawImage() of the canvas to redraw each image at every frame.
        This give me a better control over what is displayed and I am able to apply some effects (that are not demoed on this demo yet :))

        IOS on Ipad do not allow to use that method though so it is an actual Video element playing the video …

        Sound is an interesting matter. If your video is pausable (like in this demo), it does not make sense to use the sound of the video, so you should manage audio elements separately.

      1. Thanks Marco 🙂
        Well, I don’t know where to find tutorials to do that. It is mainly a mix of Video and Canvas element playing together, checking out that kind of tutorial may be a good start ..

  9. Chris,
    I tried the video in Firefox 26, Chrome 31, Safari 5.1, and IE 11. I haven’t tried it on my android note II yet or my galaxy to see if it works there, but all of the aforementioned browsers had no problem running this video except for IE (go figure). IE just made it really choppy kind of like video game lag. I have to be honest in saying that I’m frustrated with the interactive video concept related to html 5. I was impressed with video ( ), but haven’t really looked closely at the coding on it yet. I can tell you that it appears to be making use of modernizer, and flickrApi. This is another place that has a video very similar to what I am looking do myself as soon as I can figure it all out. Unfortunately I’m not a developer and it’s difficult for my designer mind to rap around some of the developer concepts which doesn’t make this easy. Anyway, good work on this and when those tutorials are done, I would be very interested in seeing them.

  10. Hey Chris, I want to overlay interactive 3D on to a video. The video is a medical operation which I want to extend with a 3D model of the anatomy which may be moved 360. The 3D should only become interactive once a tag is clicked on and return to the video when clicked again?
    Some help would be gratefully appreciated.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s