Home/Blog /Making the Most of Data Fetching

Making the Most of Data Fetching

Data fetching plays a pivotal role in connecting the world around us. It’s the under-the-hood process that allows raw data to be used and consumed by a number of web services and mobile applications. It can take many forms, from a one-time manual download of a file to a scheduled and fully automated fetching task. With the growing number of open datasets available to the public, optimising online data fetching methods has become increasingly popular.

Of course, online data fetching extends beyond the realm of open data. A number of organisations host private repositories and grant access to their data to whomever is willing to pay. Whether online datasets are open or private, the fetching operation remains the same. Downloaded files can always be saved locally, but many web services and applications use ETL tools, such as FME, to perform single-step automated operations that fetch and transform datasets.

Whether you’re seasoned in the art of data fetching or looking to develop your first automated data fetching process, there are a number of questions you need to ask yourself. Among them are:

  • What data format am I fetching?
  • Where am I fetching it from?
  • Am I using the right tool/application?
  • Is my process flexible enough to accommodate changes to source paths or data structure?
  • Am I fetching the most recent dataset?
  • Do I need to validate the source data during the fetching process?

Because more and more services rely on the latest and greatest data available, addressing some of these questions may become a necessity. The following blog posts will explore the various elements of online data fetching and will give various tips and tricks to make the most out of this process.

In the meantime, should you need any help ''getting'' the data you need, feel free to contact us!

By Patrick Connolly-Boutin

GIS Consultant

icon-info-gros
Want to know more ?
Send by EmailPrint
AutodeskSafe SoftwareCodebook

Partners

Autodesk
Consortech
6300 Auteuil. Suite 505
Brossard (QuEbec)
J4Z 3P2  Canada
Toll Free
1 888 276-0543
Follow us on LinkedIn
ConsortechETL Interest Group