Add protomaps devblog

devblog/2023-10-16-protmaps-migration.mdx

@@ -0,0 +1,171 @@
+---
+layout: post
+title: Saving $2,000 by migrating from Mapbox to Protomaps
+date: 2023-10-16
+hide_table_of_contents: true
+authors: [mbthiery]
+---
+
+## Intro
+
+At the end of September, the Helium Foundation changed our tile server from Mapbox to Protomaps. We
+are currently on track to saving ~$2,000 dollars a month (from our most expensive month). Given how
+important maps are to us and the community, we are thrilled to be able to save this much money and
+want to share some insights we learned while executing this migration. Hopefully these insights can
+help other developers who are thinking of making the leap. I’d recommend a developer use this as an
+addendum to the existing protomaps setup documentation.
+
+<!--truncate-->
+
+## What is protomaps?
+
+[Protomaps](https://protomaps.com/) is an open source self-hosted way to serve vector tiles. The
+tiles are built with the use of [OpenStreetMaps](https://openstreetmap.org/) and stored in the
+[pmtile format](https://protomaps.com/docs/pmtiles). This is “a single-file archive format for
+pyramids of tiled data”. PMTiles readers can then use HTTP range-requests to get the subset of data
+it needs.
+
+All you need to do in order to use protomaps:
+
+- is to upload a single planet-scale (or regional) map to your CDN.
+- use one of the multiple open-source front-end libraries to render the map itself.
+
+## Uploading tips
+
+You can upload your PMTiles to any S3 compatible cloud-storage platform that supports HTTP
+range-requests. You can find a long
+[list of possible cloud platforms](https://protomaps.com/docs/pmtiles/cloud-storage) from protomaps.
+While those are all possible options, **Cloudflare is the recommended storage platform as it has no
+bandwidth fees**. Ultimately, that is what we did.
+
+While Protomaps strongly recommends Cloudflare as a storage platform, it is much looser in its
+recommendations on how to upload the file. It lists out the web interface (for small files), the
+pmtiles CLI, rclone and finally the AWS CLI. **I’d recommend skipping straight to the AWS CLI**.
+While the PMTiles CLI and rclone work great for small files, it seems that multi-part uploads
+currently do not work. Despite trying multiple different upload configurations (e.g. chunk size,
+maximum concurrency) these uploads would invariably fail with a `SignatureDoesNotMatch` error.
+Ultimately, for large files (the planet-size file is 110GB), the AWS CLI is the only option that
+works.
+
+To make the AWS CLI work all you need to do is:
+
+1. configure an AWS profile with the Access Key Id and Secret Key that the
+   [Cloudflare R2 API](https://developers.cloudflare.com/r2/api/s3/tokens/) provides you.
+1. Upload your desired file to your R2 bucket
+
+   ```
+   aws s3 cp FILE.pmtiles s3://R2_BUCKET_NAME/ --endpoint-url https://CLOUDFLARE_ACCOUNT_ID.r2.cloudflarestorage.com --profile cloudflare
+   ```
+
+## R2 Worker and CORS setup
+
+After uploading your pmtiles files to Cloudlflare R2, you’ll need to create a Worker to expose said
+file to the public. Protomaps provides instructions on how to do so with both the web console and
+through Wrangler. At the time of writing this, the instructions for the web console no longer match
+the UI on Cloudflare. For this reason and for further CORS setup, **I’d recommend setting up your
+Worker with Wrangler
+[per the instructions](https://protomaps.com/docs/cdn/cloudflare#alternative:-use-wrangler)**.
+
+Your `wrangler.toml` already comes with an `ALLOWED_ORIGINS` var which you can set to enable CORS
+for your front-end. As currently set up however, `ALLOWED_ORIGINS` only works on exact string
+matches. If you are like the Helium Foundation and would like to also enable CORS for your preview
+deployments, then you can edit the `PMTiles/serverless/cloudflare/src/index.ts` file with the
+following code snippet.
+
+```
+      if (typeof env.ALLOWED_ORIGINS !== "undefined") {
+        const previewRegex = /YOUR_REGEX/g;
+        const requestOrigin = request.headers.get("Origin") || "";
+
+        for (const origin of env.ALLOWED_ORIGINS.split(",")) {
+          if (origin === requestOrigin || origin === "*") {
+            allowed_origin = origin;
+          }
+        }
+
+        if (!!requestOrigin.match(previewRegex)) {
+          allowed_origin = requestOrigin;
+        }
+      }
+```
+
+## Maplibre-libre-js
+
+Since we were already using Mapbox to render our front-end, we opted to use maplibre-gl-js which is
+an open source fork of [mapbox-gl-js](https://github.com/maplibre/maplibre-gl-js). For the most
+part, migrating from one to the other is as straightforward as installing the dependencies and
+[adding the pmtiles protocol](https://protomaps.com/docs/frontends/maplibre#installation). There
+were however two minor gotcha’s.
+
+### Watch your urls
+
+This one is more an indictment of myself versus the documentation itself, but be extra careful of
+the URLs you use when adding the sources. If you’re like me and looking at the documentation which
+includes the two following code samples:
+
+```
+{
+  "sources": {
+    "protomaps": {
+        "type": "vector",
+        "tiles": ["https://mycdn.com/tileset/{z}/{x}/{y}.mvt"],
+    }
+  }
+}
+—--
+{
+  "sources": {
+    "protomaps": {
+        "type": "vector",
+        "url": "pmtiles://https://example.com/example.pmtiles",
+    }
+  }
+}
+```
+
+You might think that when serving the tileset locally, that the following is correct.
+
+```
+"tiles": [http://127.0.0.1:8080/world.pmtiles/{z}/{x}/{y}.mvt]
+```
+
+It however, most definitely is not. Instead, if you are serving files locally the following two
+formats work.
+
+```
+    sources: {
+      protomaps: {
+        type: "vector",
+        tiles: [
+          "pmtiles://http://127.0.0.1:8080/world.pmtiles/{z}/{x}/{y}.mvt",
+        ],
+      },
+    },
+—--
+    sources: {
+      protomaps: {
+        type: "vector",
+        url: "pmtiles://http://127.0.0.1:8080/world.pmtiles",
+      },
+    },
+```
+
+### Update your styling
+
+Mapbox and Protomaps use two different sets of vector basemap layers. As such, if you are trying to
+migrate from Mapbox to Protomaps your existing Mapbox styles will not render anything. I’d recommend
+you use or adapt one of the existing `protomaps-theme-base` layers as shown
+[in the docs](https://protomaps.com/docs/frontends/maplibre#vector-basemaps). Alternatively, feel
+free to use
+[our modified layers](https://github.com/helium/network-explorer/blob/main/src/components/HotspotsMap/mapLayersDark.tsx)
+(note that we have removed a number of layers that were not of interest to us).
+
+## Conclusion
+
+While we ran into a hiccup or two while completing the migration, we couldn’t be happier with the
+end result of our migration from Mapbox to Protomaps. While Mapbox has the benefit of being a nice
+out of the box solution, when you have the time (a few days at most following the above advice), I’d
+recommend you migrate over to Protomaps. Since our migration we have experienced equal or better
+performance for a fraction of the cost. In recognition of this our CEO Abhay (who happened to work
+with Brandon, the man behind protomaps) has decided to become a sponsor of the project. It's that
+great!