Thursday, Sep 17. 2009

Xvfb and Firefox headless screenshot generator

I’ve been working on a project that its main purpose is to get a screenshot of a bunch of different sites. The requirements for this project were simple.

  • It has to be fast
  • It needs to run on linux

I did some research, trying to found the best approach, I already knew Xvfb but haven’t worked with it in a real project, there wasn’t much info about what I was trying to do, however I’ve found some other automated solutions made in .NET that worked on windows using IE as rendering engine, but I’ve found nothing that worked out of the box in linux :(

I did found some interesting setups but nothing similar to what I was trying to achieve, and all the directions I’ve got, pointed me to the same place Xvfb, so because I’m a hands-on guy, I quickly cloned a minimal debian box to start playing with this.

First I’ve installed Xvfb, firefox and some fonts

ivano ~ # apt-get install xvfb xfonts-base xfonts-75dpi xfonts-100dpi firefox

After some package downloads I’ve got everything installed, I’ve checked the man page and found:

-screen screennum WxHxD

This option creates screen screennum and sets its width, height,

and depth to W, H, and D respectively. By default, only screen 0

exists and has the dimensions 1280x1024x12.

So I did:

ivano ~ # Xvfb :1 -screen 0 1024x768x24 &
[1] 2665
ivano ~ # Could not init font path element /usr/X11R6/lib/X11/fonts/misc, removing from list!
Could not init font path element /usr/share/fonts/X11/cyrillic, removing from list!
Could not init font path element /usr/share/fonts/X11/Type1, removing from list!
Could not init font path element /usr/X11R6/lib/X11/fonts/Type1, removing from list!
Could not init font path element /var/lib/defoma/x-ttcidfont-conf.d/dirs/TrueType, removing from list!

I’ve got some warnings from X, but nothing lethal, so now I have a Virtual X server ready to throw anything at it.

ivano ~ # DISPLAY=:1 firefox http://google.com &

I have to set the DISPLAY variable to be the same as the virtual screen I’ve created, after this command firefox must be there so I need to get a screen shot of the virtual screen so I’ve used the image magick import tool

ivano ~ # DISPLAY=:1 import -window root google.com.png

When the import program ended I had a the following pg file:

Firefox capture within Xvfb

Firefox capture within Xvfb

This apparently worked well, however I did some more testing to found any possible problem and the only thing I’ve found was that firefox tried to restore the session because of the improper shutdown so I had to tweak the firefox profile to avoid this, apart from this there were no other significant issues.

I will talk more about how I implemented the backend and frontend for the entire solution.




6 Comments to “Xvfb and Firefox headless screenshot generator”

  • October 7, 2009 at 9:39 am
    Eric Clemmons says:

    What settings in the profile did you modify to prevent the “Firefox did not shutdown properly” errors?

  • October 7, 2009 at 9:54 am
    Ivan Villareal says:

    browser.sessionstore.resume_from_crash was set to false, what worked for me was to change some settings and add the extensions on another firefox instance and then copying all the .mozilla folder to the home directory of the user that will be running the Xvfb, I’ve also used an extension to make firefox tabless, because I didn’t found a way to close specific tabs.

  • January 24, 2010 at 11:55 pm
    methode says:

    Excellent tutorial!

    Ivan, a question: for some reason I get some weird colors instead of the real output when generating the image with import. I’m not sure why but I smell it’s because I killed Xvfb and that didn’t start up properly the next time. The interesting is, that for the first time I took the screenshot it all worked properly; so I wrote a bash script to start and stop (with kill) the X and export the image. I assume stopping the X was a very stupid idea.
    So now I’m trying to fix these weird colors: http://picasaweb.google.com/lh/photo/2qWs1qIeM1W7lMKIwoJnqQ?authkey=Gv1sRgCNeRqrGLyufi9AE&feat=directlink

    Any idea on this?

  • January 26, 2010 at 4:49 pm
    Ivan Villareal says:

    @methode
    No I don’t think killing Xvfb had anything to do with this weird output you are getting, this is because I did a larger app and I’m killing Xvfb everytime I need to restart the process and I didn’t had this problem.

    You can try to debug your issue by running manually every step, first run Xvfb and on a second terminal take a screenshot, if it looks ok, open firefox inside of that Xvfb session and take a screenshot again, an easier way to debug this is changing Xvfb to X, and do all your testing trough an ssh session, that way you can look how your script performs and get an idea of what is going wrong.

  • April 26, 2012 at 2:37 am
    Aloy says:

    Hi,
    I tried this idea. it’s working fine for me. But I want to get the sceenshot of entire page without scrolling it (like screencrab on firebug). any idea?

  • April 26, 2012 at 11:02 am
    Ivan Villareal says:

    you can’t use this approach for that, if you want the entire page you will need to render the page then capture that, you can use python bindings for webkit or mozilla to do this, there are some command line tools that are already working like https://github.com/AdamN/python-webkit2png/ and http://www.paulhammond.org/webkit2png/ you may still need to setup the xvfb though.

Post comment