Support CSV and other formats with data source preprocess
option
#635
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To enable support for formats that may contain geographic data, but not be formatted in the commonly supported formats (GeoJSON, MVT, etc.), this PR adds a data source
preprocess
option. This parameter can define a JS function to mutate the raw network response fetched by a data source, before it is processed further. This is similar to the existingtransform
function, with the difference thattransform
is called after initial response parsing (e.g. decoding TopoJSON and MVT to GeoJSON-style data).The
preprocess
function should return data in the format expected by the data source type, e.g. iftype: GeoJSON
, thenpreprocess
should return a GeoJSON object.For example, given a Who's On First CSV export (which contains geographic data in the form of census interior points) formatted like this:
We can use the
preprocess
function to remap those columns into a GeoJSON collection of points:Additional notes:
preprocess
has access to the same externally loadedscripts
as thetransform
function.preprocess
can return an immediate response, or aPromise
which resolves with a response, allowing for the potential use of async operations (while the initial tests are with CSV files, theoretically the functionality could be applied to shapefiles and other format parsing as well).preprocess
andtransform
functions in a single data source (if both are defined, both will execute), in practice there are probably few cases where it would be useful.