Today marks a milestone in the development of OpenStreetMap NextGen. After months of rigorous development, I conducted the 1st OpenStreetMap NextGen performance benchmark, a crucial step towards realizing the vision of a more robust, efficient, and user-friendly OpenStreetMap.
The focus of today’s benchmark was on evaluating static and unauthenticated requests. Since this core functionality is unlikely to change significantly during future development, it’s the perfect time to test it.
Future benchmarks will focus on timing authenticated requests as well as API 0.6.
What was measured
The benchmark analyzed request processing speed, excluding network and client latency. Both osm-ruby and osm-ng support the X-Runtime response header, which tracks how long it takes to process a request and generate a response.
Here’s a general breakdown of a typical static request processing:
- Grabbing configuration settings
- Checking for authorization (cookies, oauth, etc.)
- Configuring translations
- Rendering the HTML template
The setup
The benchmarking setup consisted of local machine testing, as well as official production and development websites.
I initially planned to run the benchmark solely on my local machine, following the official Docker instructions. However, I quickly discovered that the production deployment instructions were outdated and required some Ruby knowledge to fix, which I lacked. In particular, the instructions for replacing the Rails server with Phusion Passenger had been redirected to a generic support page.
osm-ng was launched in production mode with all Python and Cython optimizations enabled. Since we were only dealing with static requests, both local Postgres databases remained empty.
The benchmarking script
I created a basic HTTP benchmarking script that first warmed things up with a few requests before launching into the actual test. It then measured runtime times for a series of HTTP requests, and I repeated the benchmark multiple times for consistency.
A note before the results
It’s important to remember that OpenStreetMap NextGen processes static requests in a similar way to osm-ruby, and it does not (currently) introduce any new caching logic for templates, especially since that would significantly impact the benchmark results (and some people would consider it cheating). osm-ng remains completely backwards compatible with the existing OpenStreetMap platform. Additionally, it’s important to emphasize that the X-Runtime header used for benchmarking is agnostic to network latency, meaning it only measures the processing time on the server itself.
And the winner is…
Here’s a detailed breakdown of the results:
| Environment | Minimum Runtime (s) | Median Runtime (s) |
|---|---|---|
| Ruby (local) | 0.04264 | 0.04521 |
| Ruby (official) | 0.01892 | 0.02921 |
| Ruby (test) | 0.00913 | 0.01725 |
| Python | 0.00314 | 0.00325 |
As you can see, osm-ng consistently outperformed osm-ruby in all test scenarios. The fastest Ruby deployment had the minimum runtime of 0.00913 seconds, while osm-ng achieved the blazing-fast time of 0.00314 seconds, a remarkable 290% performance improvement.
Support the NextGen revolution
I’m truly convinced that OpenStreetMap NextGen will be a game-changer for OpenStreetMap, not just in terms of performance, but also in privacy, security, usability, and overall openness.
If you believe in this project as much as I do, please consider donating so I can keep working on it full-time! 🙏 Every contribution helps push us closer to that first stable release.
And a huge thank you to those who have already supported me!
Useful links
Today, we benchmarked not just a system, but the future. And the future is bright.