How to convert Python scripts to FME workflows

One of the FME trends we’ve noticed over the past few years is the requirement to convert old and clunky python scripts to FME workflows.  These python scripts do any number of things, from filtering data, changing field names, modifying geometries, moving files around, and all kinds of geoprocessing. 

Why should I convert my Python scripts to FME?

There are lots of reasons why one might want to move from Python to FME, but here are some of the most common that we’ve heard:

• Difficult to identify and debug when something goes wrong
• Hard to understand what it’s doing (possibly written by multiple people in the past)
• Inefficient and time consuming (performance concerns)
• Concerns about software upgrades and migrating to new versions of python scripts
• They don’t always do exactly what the users need, and are not always easily adaptable

How does FME solve these issues?  Considering the flexibility and graphical user interface of FME, many of those concerns about Python are solved by moving towards FME workflows.  The graphical user interface and clean log files allow for easier error trapping and debugging, the object-based workflow as well as bookmarks and annotations make it clean and easy to understand, and quite often, you can develop with FME in a much more streamlined way than you can with Python.

Sounds great, where do I start?

Here are some suggested steps for moving from Python to FME:

1. Run the Python script, if possible. Inspect the data before and after.
2. Read and understand what the Python script is doing. (If you are not a Python expert, that is OK, you can gather a lot of information just from looking at the source and destination data!)  Make a point-form list of the steps that the Python script is executing.
3. Analyze the source and destination data, as well as any transformations that are happening, paying close attention to the following:
   1. Schema and structure – How is the data organized? What are the fields called and what are the data types? 
   2. Geometry – How are the geometries being modified, if at all? What kind of geoprocessing is happening and what tools are being used?
   3. Projection – What coordinate system is the data in? Is it being reprojected?
   4. Attribute content – Are attributes being modified? Are any calculations being made?
   5. Location of the data – Where is the data coming from and going to?
   6. Frequency of the script – When and how often does the script need to be run?
4. Plan out your readers, writers and transformers in FME that can get the job done and start building your workspace!

Remember that oftentimes, reproducing all the steps from a Python script is NOT the most efficient way to do it in FME.  If we look at the big picture and use the right tools, we can often make a much simpler workflow in FME.

Finally, keep in mind that while it can be beneficial to replace many Python scripts with FME workflows, there is still a place for Python scripts, even within FME.  Keep an eye out for a future blog post, where we will talk about various ways we can integrate Python scripts into our FME workflows.

You may also like: