Home/Blog /How to convert Python scripts to FME workflows for efficient geoprocessing

How to convert Python scripts to FME workflows for efficient geoprocessing

One of the FME trends I've noticed over the past few years is the requirement to convert old and clunky python scripts to FME workflows.  These python scripts do any number of things, from filtering data, changing field names, modifying geometries, moving files around, and all kinds of geoprocessing. 

Why should I convert my Python scripts to FME?

There are lots of reasons why one might want to move from Python to FME, but here are some of the most common that I’ve heard:

  • Difficult to identify and debug when something goes wrong
  • Hard to understand what it’s doing (possibly written by multiple people in the past)
  • Inefficient and time consuming (performance concerns)
  • Concerns about software upgrades and migrating to new versions of python scripts
  • They don't always do exactly what the users need, and are not always easily adaptable

How does FME solve these issues?  Considering the flexibility and graphical user interface of FME, many of those concerns about Python are solved by moving towards FME workflows.  The graphical user interface and clean log files allow for easier error trapping and debugging, the object-based workflow as well as bookmarks and annotations make it clean and easy to understand, and quite often, you can develop with FME in a much more streamlined way than you can with Python.

Sounds great, where do I start?

Here are some suggested steps for moving from Python to FME:

  1. Run the Python script, if possible.  Inspect the data before and after.
  2. Read and understand what the Python script is doing.  (If you are not a Python expert, that is OK, you can gather a lot of information just from looking at the source and destination data!)  Make a point-form list of the steps that the Python script is executing.
  3. Analyze the source and destination data, as well as any transformations that are happening, paying close attention to the following:
    1. Schema and structure - How is the data organized?  What are the fields called and what are the data types? 
    2. Geometry - How are the geometries being modified, if at all? What kind of geoprocessing is happening and what tools are being used?
    3. Projection - What coordinate system is the data in?  Is it being reprojected?
    4. Attribute content - Are attributes being modified?  Are any calculations being made?
    5. Location of the data - Where is the data coming from and going to?
    6. Frequency of the script - When and how often does the script need to be run?
  4. Plan out your readers, writers and transformers in FME that can get the job done, and start building your workspace!

 

Remember that more often than not, reproducing all the steps from a Python script is NOT the most efficient way to do it in FME. If we look at the big picture and use the right tools, we can often make a much simpler workflow in FME.

Finally, keep in mind that while it can be beneficial to replace many Python scripts with FME workflows, there is still a place for Python scripts, even within FME. Keep an eye out for a future blog post, where we will talk about various ways we can integrate Python scripts into our FME workflows.

Need help converting your data in the most efficient format? Contact us!

By Allison Hughes

GIS Consultant

icon-info-gros
Want to know more ?
Send by EmailPrint
AutodeskSafe SoftwareCodebook

Partners

Autodesk
Consortech
6300 Auteuil. Suite 505
Brossard (QuEbec)
J4Z 3P2  Canada
Toll Free
1 888 276-0543
Follow us on LinkedIn
ConsortechETL Interest Group