In the past couple of years, there has been an increasing number of types of professionals using the FME data integration platform. Not only GIS analysts are using it, but IT professionals, database administrators, data science specialists, application experts, and developers are becoming daily FME users.
With this blossoming of professionals using FME, many have asked the question: Is FME a GIS tool or is it a Development tool?
- GIS analysts: FME offers great flexibility to GIS people to perform all kinds of analysis and quickly prototype complex data processing workflows.
- IT professionals: FME is a complete data integration platform, hence it must be controlled by the IT department and its processes should be vetted.
The truth is FME can be looked at in both ways. When an organization has been using it for a while, they often realize at some point that there is a need to add governance around the use and management of the platform, especially with the FME Server component. By this time, organizations have developed perhaps a few hundred processes, some of them provisioning data that is mission critical to daily operations and they now must ensure that the system is reliable and solid.
Different types of FME workspaces for different needs
FME is used in different kinds of ways and having a one-sized fits all strategy to its management may not be efficient. For instance, if a process is developed for one time use by one professional, it may not be necessary to undergo a complex testing strategy. On the other hand, when you need to upgrade a workspace that provides important data on a scheduled basis to a new version of FME, you would be glad to have that suite of test cases to ensure everything works properly. While it can take hours to document a workspace thoroughly, if you’re in the middle of a volatile prototype project, this would be cumbersome as your FME process is constantly changing. But a production workspace being run on FME server by a team of people should be easy to interpret by the whole team and therefore best practices (like annotations, design and naming schemes) becomes important.
When looking at how FME gets used by different groups and professionals, we like to classify the workspaces into three categories. While there is some overlap and blurred lines, this process really helps determine how to manage them.
Analytical Workspaces
Analytical FME workspaces are often data processes that may only be used one time, or very sporadically. If a workspace is only used once per year, and the incoming data structure is unreliable, then that process usually gets manually run by an FME expert who can adapt the process as needed. Some FME workspaces may be developed to do some sort of analysis, drawing data together from different sources, or a specialized data extraction that may only occur once. The data analyst that develops the process is responsible for validating the results before delivering to the end users and usually validates the results manually before sending them to end users/clients/end system. Since the workspace will generally not be reused, they are often not as well documented, applying all best practices and doing automated testing and building a library of test cases are not usually necessary and would be cumbersome while adding little value.
We would also categorize prototype workspaces in this as well. When we are trying out a concept, there is often a lot of restructuring, tearing down and building up. When a new process passes from prototype to something more operational, we need to ensure that we have applied our best practices, but while the process is still in development and remains volatile, it is often futile to ensure all parameters, paths, naming and annotations schemes have been followed.
Operational Workspaces – Monitored
When FME workspaces are run more frequently, on a schedule or regular basis, and provide important data, we are now talking about a workspace that is Operational. In this category, Operational Monitored workspaces may be running on FME Server, but they are still monitored by a professional. They may be run on a schedule, on a trigger or still run manually when new data arrives. These workspaces may represent one step in a process.
The data provisioned by these processes is important to the organization but is monitored by an expert either through notifications, or running the task manually, or as batch files. They are not the most mission critical. If a process fails one day, we can take the time to adjust and fixt the problem. A day or 2 or down time is not as mission critical.
These processes benefit from good documentation, annotations, and organization. They may be shared among different team members who may need to manage them.
Operational Workspaces – Enterprise
This category of data process is the most mission critical. These data processes are usually automated on FME Server and provide time critical or important data. The processes need to be reliable enough to provide data straight to the end users, either directly or fed into another application or portal without intervention from an FME expert. Down-time needs to be minimized. These processes make activities like maintenance and migration more challenging. While migration forward with FME usually goes well, it is always important to test processes. In this case, we should have good testing methods or test cases to help streamline the future migration process.
FME Governance… A matter of balance
FME offers a wide range of possibilities and one of its features is how easy and fast it is to adapt, tear parts down and modify it. It is part of why GIS people love it so much and why IT managers need to add governance rules around it. It is important to balance the ease of use and flexibility with the need for reliability and assurance, or in other words, managing the risk around the data process.
Need help to categorize and manage your FME workspaces?