Tackling Memory Challenges in FME When Working with Large OpenStreetMap Datasets

OpenStreetMap (OSM) is an open geographic database widely used in all kinds of tools and applications. Given the open-source nature of OSM, it contains an extensive range of information and geometries. However, it’s not feasible to use this data as-is unless you transform it to the data schema the client has in mind. This way we can create a customized dataset that can be downloaded by end users. In this project, we use FME for the conversion of the raw OSM data to a dataset that will be used as the foundational layer for specialized maps, such as walking or cycling maps. In a first step, we need to download OSM data. In this case we use Geofabrik, a trusted source for OSM extracts. This raw data, covering five European countries, needs mapping and filtering to meet the specific project requirements. Due to the large size of the dataset, memory limitations in FME pose significant challenges, leading to performance issues (e.g. crashes). A key part of the presentation focuses on these challenges and practical strategies for optimizing FME workflows to prevent memory overload, including methods for chunking data, reducing memory usage with specific transformers, and so on. Finally, I will introduce a FME flow app that enables users to easily download specific bounding boxes of cleaned and processed OSM data. By the end of this session, attendees will gain valuable insights into working with OSM data, handling memory limitations and creating easy interfaces for users.