Now Playing Tracks

How to import WordPress XML files

… or, how I brought an old python code back from the dead!

Inspired by Kenneth Reitz's recent post and spurred by recent events, I decided to turn an old python code I wrote a while back into something that can be (hopefully) easier to get to than by sheer luck.

I’m talking about ChoppedPress, my script that let’s you split WordPress exported XML files into smaller files that can be easily imported into new WordPress installations. I’m sure that some of you have experienced the frustration of not being able to import this xml file due to size upload constraints on your host providers… One of my close friends who provides mostly support for WordPress gave me the idea a while back and that is how the script came about. Little did I know that other people have find it useful too, specially for migrating away from WordPress! As a matter of fact, I too used it when I moved to Tumblr, but that another story.

So this afternoon I took some time during my lunch break to create a repository and put together some very basic structure to give ChoppedPress a proper home (yay GitHub Pages!!!). For the first time I also uploaded something I created to PyPi… Sure, this may not be a big deal to some of you out there, but I can hardly contain my excitement. :)

Overall, I’m still enjoying a nice buzz from the experience. Obviously, I look forward to comments, suggestions and/or improvements to the code, but more than anything, I hope this will be useful to you too!

(Source: )

Desculpe Pela Bagunça…

Sorry  Recentemente eu comecei o processo de migrar o meu blog para o Tumblr, e ao importar os posts antigos do WordPress eu sem querer iniciei uma enxurrada onde certos agregadores estão mostrando posts de 2007… Ontem eu também causei um problema similar no Twitter e Facebook

Por favor aceite os meus mais sinceros pedidos de desculpas pelo incoveniente… depois escrevo mais sobre a migração para o Tumblr!

Splitting WordPress Export File with ChoppedPress

Four Views of a Book Press by 802

You’re all probably tired of hearing me talking about my script to split the XML-like file that WordPress generates when you use the Export feature. Tonight I have finally added the final touches to it and can now share it here with you.

If you don’t remember, I wrote it because my good friend Evandro from QuartoEstudio.com is always getting asked by some of his clients to migrate an existing web blog to a new host/domain and hands him a tarball containing the exported content of a WordPress blog in XML format. The problem resides on the issue that some web hosting providers will limit how big a file can be uploaded via a POST method, and depending on how big this XML file is, you may have to manually break it into smaller files first.

Along the way I learned a bit more about OptParse, LXML and Beautiful Soup, so the entire experience was very productive and fun. In the end, I solved my issue by simple using the strip_cdata=False argument to the XMLParser class. Got to love when the solution is that simple!

So, without further ado, I present you ChoppedPress (download)! In the next few days (or weeks, depending on what my schedule looks like) I will make it more robust and add more error handling, etc. In the meantime, I hope you will find it useful.

# -*- coding: utf-8 -*-

"""
Copyright (c) 2010, Og Maciel 

All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted
provided that the following conditions are met:

    * Redistributions of source code must retain the above copyright notice, this list of
      conditions and the following disclaimer.
    * Redistributions in binary form must reproduce the above copyright notice, this list
      of conditions and the following disclaimer in the documentation and/or other materials
      provided with the distribution.
    * Neither the name of the Og Maciel nor the names of its contributors may be used to
      endorse or promote products derived from this software without specific prior written
      permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
“AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""

import os
import sys

try:
    from lxml import etree
except ImportError:
    print "Please install lxml and try again: http://codespeak.net/lxml/"
    sys.exit(-1)

from optparse import OptionParser

# This is the default xml tag for wordpress
WP_TAG = "/rss/channel/item"
OUTFILE = "outfile"
CHUNKSIZE = 2
TMPFILE = "/tmp/template.xml"

# Function found on ActiveState Code
# Licensed under the PSF license
# http://code.activestate.com/recipes/425397-split-a-list-into-roughly-equal-sized-pieces/
def split_seq(seq, size):
    newseq = []
    splitsize = 1.0/size*len(seq)
    for i in range(size):
            newseq.append(seq[int(round(i*splitsize)):int(round((i+1)*splitsize))])

    return newseq

def chopit(xmlfile, outfile=OUTFILE, xmltag=WP_TAG, chunksize=CHUNKSIZE):
    parser = etree.XMLParser(resolve_entities=False, encoding="utf-8", strip_cdata=False)
    doc = etree.parse(xmlfile, parser)

    matches = doc.xpath(xmltag)
    print "Found %s blog posts!" % len(matches)
    matcheslist = split_seq(matches, chunksize)

    channel = doc.getroot().find('channel')

    # Create an empty wordpress xml file
    for e in matches:
        channel.remove(e)
    doc.write(TMPFILE, encoding="utf-8", method="xml", pretty_print=True)

    # Now, create smaller wordpress xml files
    ctr = len(matcheslist)
    print "Breaking WordPress XML into %s smaller files." % ctr
    for entities in matcheslist:
        doc = etree.parse(TMPFILE)
        channel = doc.getroot().find('channel')
        for entity in entities:
            channel.append(entity)

        output = '%s%03d.xml' % (outfile, ctr)
        doc.write(output, encoding='utf-8', method="xml", pretty_print=True)
        print " - File %s has %s posts." % (output, len(entities))
        ctr -= 1
    print "Done!"

def main():

    description = "ChoppedPress lets you split the WordPress XML export file " 
    "into smaller files that can be used to import your posts, comments, tags" 
    " and categories into a new WordPress installation."

    usage = "Usage:  %prog  [[] [] []]"
    epilog = "Constructive comments and feedback can be sent to ogmaciel at gnome dot org."
    version = "%prog version 0.1"

    parser = OptionParser(usage=usage, description=description, epilog=epilog, version=version)
    parser.add_option('-i', '--infile', dest='infile', metavar='', help='The XML file generated by WordPress')
    parser.add_option('-o', '--outfile', dest='outfile', default='out', metavar='', help='The name for the smaller XML files. [default: %default]')
    parser.add_option('-t', '--tag', dest='tag', default='/rss/channel/item', help='The XML tag that represents your data. [default: %default]')
    parser.add_option('-n', '--number', dest='number', default=2, type=int, help='How many new files should be generated. [default: %default]')

    # Verify arguments
    (opts, args) = parser.parse_args()

    if not opts.infile:
        parser.print_help()
        sys.exit(-1)

    chopit(opts.infile, opts.outfile, opts.tag, opts.number)

if __name__ == "__main__":
    main()

The usage should be pretty straight forward too:

$ python choppedpress
Usage:  choppedpress  <INFILE> [[<OUTFILE>] [<TAG>] [<NUMBER>]]

ChoppedPress lets you split the WordPress XML export file into smaller files
that can be used to import your posts, comments, tags and categories into a
new WordPress installation.

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -i , --infile=
                        The XML file generated by WordPress
  -o , --outfile=
                        The name for the smaller XML files. [default: out]
  -t TAG, --tag=TAG     The XML tag that represents your data. [default:
                        /rss/channel/item]
  -n NUMBER, --number=NUMBER
                        How many new files should be generated. [default: 2]

Constructive comments and feedback can be sent to ogmaciel at gnome dot org.

If you have any constructive comment or suggestion, please drop me a line or comment here.

Command Line Tools with Python OptParse

It is not very uncommon for me to receive some code requests from friends looking for a little help to “make something easier”. I’m talking about those pesky day to day things that after a while makes you go: “If I could only push a button and have it done!”

This week I was chit-chatting with my good friend Evandro Pastor when he mentioned one of these situations that, according to him, makes his existence miserable. You see, Evandro is the brain, muscle and talent behind QuartoStudio, “a small studio located in the state of São Paulo, Brazil” specialized in creating websites and blogs for small to medium sized companies. We have worked together many times on several different projects, and through the years have become great friends.

Anyhow, Evandro was telling me how painful it is when a new client asks him to migrate an existing web blog to a new host/domain and hands him a tarball containing the exported content of a WordPress blog in XML format. The problem resides on the issue that some web hosting providers will limit how big a file can be uploaded via a POST method, and depending on how big this XML file is, you may have to manually break it into smaller files first.

So during my lunch break yesterday I took upon myself to create a script that would do this for him so he wouldn’t have to manually split files and worry about making sure all the right tags were present and closed appropriately. Of course I searched the web for a free/open source alternative first but after several minutes later I still had not found something that would do what he needed.

"I’ll just write one then!", I said to my self. My first stop was to read up on lxml since I figured that using something that could parse XML would be a better start than reading a file and searching for tags within the file. Armed with this knowledge and still enjoying the buzz from my freshly brewed cup of coffee, I set out to write a couple of methods that take in a XML file, the number of “chunks” you want to split it into, and I was done!

There was still a problem, however: Evandro would have to use the python command line to do what he needed. Writing a command line tool then became my next priority.

Since I have written command line tools in python before, my first impulse was to copy some of the code I had already written somewhere else and adapt it for this specific project… and so I did. Two minutes later I had my first draft of a command line tool that could be invoked from the shell the same way that cp or ls can. It was then that it dawned on me that I had never really taken the time to really learn about optparse, the “convenient, flexible, and powerful library for parsing command-line options.”

Of course, my first stop was to read through the “official” documentation. I then did a bit of Googling around and was pleased to find an amazing introductory article by Alexander Sandler. If you have ever wanted to learn how to write a command line tool and need to learn in detail about optparse, I highly recommend his article to supplement the official documentation.

So Tuesday night, as I sat down to watch Lost, I added the finishing touches to my script and was able to surprise Evandro with it the next morning! Unfortunately this story does not have a 100% happy ending… The script works as expected and it lets you configure how many “chunks” to break the XML into, what tag to look for (defaults to “/rss/channel/item”) and what to name the generated smaller files. However, since a post or comment can have embedded HTML tags, lxml is converting them to HTML entities… aaaand WordPress doesn’t like that! Aaaand it turns out that the XML file generated by WordPress is not really XML per se but its own WRX format. :/

Not everything is lost though. I did learn more about optparse and using lxml to parse XML files! By the way, if anyone has some more information on how I can make lxml ignore html tags, please drop me a line! :)

We make Tumblr themes