From 5571788386816c55ec4d8523ac5f48958f4b2713 Mon Sep 17 00:00:00 2001 From: mbthiery Date: Thu, 12 Oct 2023 17:33:38 -0400 Subject: [PATCH] Add protomaps devblog --- devblog/2023-10-16-protmaps-migration.mdx | 171 ++++++++++++++++++++++ 1 file changed, 171 insertions(+) create mode 100644 devblog/2023-10-16-protmaps-migration.mdx diff --git a/devblog/2023-10-16-protmaps-migration.mdx b/devblog/2023-10-16-protmaps-migration.mdx new file mode 100644 index 000000000..f6d3db400 --- /dev/null +++ b/devblog/2023-10-16-protmaps-migration.mdx @@ -0,0 +1,171 @@ +--- +layout: post +title: Saving $2,000 by migrating from Mapbox to Protomaps +date: 2023-10-16 +hide_table_of_contents: true +authors: [mbthiery] +--- + +## Intro + +At the end of September, the Helium Foundation changed our tile server from Mapbox to Protomaps. We +are currently on track to saving ~$2,000 dollars a month (from our most expensive month). Given how +important maps are to us and the community, we are thrilled to be able to save this much money and +want to share some insights we learned while executing this migration. Hopefully these insights can +help other developers who are thinking of making the leap. I’d recommend a developer use this as an +addendum to the existing protomaps setup documentation. + + + +## What is protomaps? + +[Protomaps](https://protomaps.com/) is an open source self-hosted way to serve vector tiles. The +tiles are built with the use of [OpenStreetMaps](https://openstreetmap.org/) and stored in the +[pmtile format](https://protomaps.com/docs/pmtiles). This is “a single-file archive format for +pyramids of tiled data”. PMTiles readers can then use HTTP range-requests to get the subset of data +it needs. + +All you need to do in order to use protomaps: + +- is to upload a single planet-scale (or regional) map to your CDN. +- use one of the multiple open-source front-end libraries to render the map itself. + +## Uploading tips + +You can upload your PMTiles to any S3 compatible cloud-storage platform that supports HTTP +range-requests. You can find a long +[list of possible cloud platforms](https://protomaps.com/docs/pmtiles/cloud-storage) from protomaps. +While those are all possible options, **Cloudflare is the recommended storage platform as it has no +bandwidth fees**. Ultimately, that is what we did. + +While Protomaps strongly recommends Cloudflare as a storage platform, it is much looser in its +recommendations on how to upload the file. It lists out the web interface (for small files), the +pmtiles CLI, rclone and finally the AWS CLI. **I’d recommend skipping straight to the AWS CLI**. +While the PMTiles CLI and rclone work great for small files, it seems that multi-part uploads +currently do not work. Despite trying multiple different upload configurations (e.g. chunk size, +maximum concurrency) these uploads would invariably fail with a `SignatureDoesNotMatch` error. +Ultimately, for large files (the planet-size file is 110GB), the AWS CLI is the only option that +works. + +To make the AWS CLI work all you need to do is: + +1. configure an AWS profile with the Access Key Id and Secret Key that the + [Cloudflare R2 API](https://developers.cloudflare.com/r2/api/s3/tokens/) provides you. +1. Upload your desired file to your R2 bucket + + ``` + aws s3 cp FILE.pmtiles s3://R2_BUCKET_NAME/ --endpoint-url https://CLOUDFLARE_ACCOUNT_ID.r2.cloudflarestorage.com --profile cloudflare + ``` + +## R2 Worker and CORS setup + +After uploading your pmtiles files to Cloudlflare R2, you’ll need to create a Worker to expose said +file to the public. Protomaps provides instructions on how to do so with both the web console and +through Wrangler. At the time of writing this, the instructions for the web console no longer match +the UI on Cloudflare. For this reason and for further CORS setup, **I’d recommend setting up your +Worker with Wrangler +[per the instructions](https://protomaps.com/docs/cdn/cloudflare#alternative:-use-wrangler)**. + +Your `wrangler.toml` already comes with an `ALLOWED_ORIGINS` var which you can set to enable CORS +for your front-end. As currently set up however, `ALLOWED_ORIGINS` only works on exact string +matches. If you are like the Helium Foundation and would like to also enable CORS for your preview +deployments, then you can edit the `PMTiles/serverless/cloudflare/src/index.ts` file with the +following code snippet. + +``` + if (typeof env.ALLOWED_ORIGINS !== "undefined") { + const previewRegex = /YOUR_REGEX/g; + const requestOrigin = request.headers.get("Origin") || ""; + + for (const origin of env.ALLOWED_ORIGINS.split(",")) { + if (origin === requestOrigin || origin === "*") { + allowed_origin = origin; + } + } + + if (!!requestOrigin.match(previewRegex)) { + allowed_origin = requestOrigin; + } + } +``` + +## Maplibre-libre-js + +Since we were already using Mapbox to render our front-end, we opted to use maplibre-gl-js which is +an open source fork of [mapbox-gl-js](https://github.com/maplibre/maplibre-gl-js). For the most +part, migrating from one to the other is as straightforward as installing the dependencies and +[adding the pmtiles protocol](https://protomaps.com/docs/frontends/maplibre#installation). There +were however two minor gotcha’s. + +### Watch your urls + +This one is more an indictment of myself versus the documentation itself, but be extra careful of +the URLs you use when adding the sources. If you’re like me and looking at the documentation which +includes the two following code samples: + +``` +{ + "sources": { + "protomaps": { + "type": "vector", + "tiles": ["https://mycdn.com/tileset/{z}/{x}/{y}.mvt"], + } + } +} +—-- +{ + "sources": { + "protomaps": { + "type": "vector", + "url": "pmtiles://https://example.com/example.pmtiles", + } + } +} +``` + +You might think that when serving the tileset locally, that the following is correct. + +``` +"tiles": [http://127.0.0.1:8080/world.pmtiles/{z}/{x}/{y}.mvt] +``` + +It however, most definitely is not. Instead, if you are serving files locally the following two +formats work. + +``` + sources: { + protomaps: { + type: "vector", + tiles: [ + "pmtiles://http://127.0.0.1:8080/world.pmtiles/{z}/{x}/{y}.mvt", + ], + }, + }, +—-- + sources: { + protomaps: { + type: "vector", + url: "pmtiles://http://127.0.0.1:8080/world.pmtiles", + }, + }, +``` + +### Update your styling + +Mapbox and Protomaps use two different sets of vector basemap layers. As such, if you are trying to +migrate from Mapbox to Protomaps your existing Mapbox styles will not render anything. I’d recommend +you use or adapt one of the existing `protomaps-theme-base` layers as shown +[in the docs](https://protomaps.com/docs/frontends/maplibre#vector-basemaps). Alternatively, feel +free to use +[our modified layers](https://github.com/helium/network-explorer/blob/main/src/components/HotspotsMap/mapLayersDark.tsx) +(note that we have removed a number of layers that were not of interest to us). + +## Conclusion + +While we ran into a hiccup or two while completing the migration, we couldn’t be happier with the +end result of our migration from Mapbox to Protomaps. While Mapbox has the benefit of being a nice +out of the box solution, when you have the time (a few days at most following the above advice), I’d +recommend you migrate over to Protomaps. Since our migration we have experienced equal or better +performance for a fraction of the cost. In recognition of this our CEO Abhay (who happened to work +with Brandon, the man behind protomaps) has decided to become a sponsor of the project. It's that +great!