pixel-convolve

Note

This is one of 199 standalone projects, maintained as part of the @thi.ng/umbrella monorepo and anti-framework.

🚀 Please help me to work full-time on these projects by sponsoring me on GitHub. Thank you! ❤️

• About
  • Strided convolution & pooling
  • Normal map generation
• Status
• Installation
• Dependencies
• Usage examples
• API
• Authors
• License

About

Extensible bitmap image convolution, kernel presets, normal map & image pyramid generation. This is a support package for @thi.ng/pixel.

This package contains functionality which was previously part of and has been extracted from the @thi.ng/pixel package.

• Convolution w/ arbitrary shaped/sized kernels, pooling, striding
• Convolution kernel & pooling kernels presets
  • Higher order kernel generators (Gaussian, Lanczos)
• Image pooling filters (min/max, mean, adaptive threshold, custom)
• Image pyramid generation (w/ customizable kernels)
• Customizable normal map generation (i.e. X/Y gradients plus static Z component)

Strided convolution & pooling

Floating point buffers can be processed using arbitrary convolution kernels. The following convolution kernel presets are provided for convenience:

Kernel	Size
BOX_BLUR3	3x3
BOX_BLUR5	5x5
GAUSSIAN_BLUR3	3x3
GAUSSIAN_BLUR5	5x5
GAUSSIAN(n)	2n+1 x 2n+1
HIGHPASS3	3x3
LANCZOS(a,s)	as+1 x as+1
SHARPEN3	3x3
SOBEL_X	3x3
SOBEL_Y	3x3
UNSHARP_MASK5	5x5

Custom kernels can be defined (and code generated) using an array of coefficients and a given kernel size. See above presets and defKernel() for reference.

Furthermore, convolution supports striding (i.e. only processing & keeping every nth pixel column/row, aka downscaling) and pixel pooling (e.g. for ML applications). Available pooling kernel presets (kernel sizes must be configured independently):

Kernel	Description
POOL_MEAN	Moving average
POOL_MAX	Local maximum
POOL_MIN	Local minimum
POOL_NEAREST	Nearest neighbor
POOL_THRESHOLD(bias)	Adaptive threshold

Convolution can be applied to single, multiple or all channels of a FloatBuffer. See convolveChannel() and convolveImage().

See ConvolveOpts for config options.

import { floatBufferFromImage, FLOAT_RGB, imageFromURL } from "@thi.ng/pixel";
import { convolveImage, SOBEL_X } from "@thi.ng/pixel-convolve";

// convolutions are only available for float buffers (for now)
const src = floatBufferFromImage(await imageFromURL("test.jpg"), FLOAT_RGB);

// apply horizontal Sobel kernel preset to all channels
// downscale image by factor 2 (must be integer)
// scale kernel result values by factor 4
const dest = convolveImage(src, { kernel: SOBEL_X, stride: 2, scale: 4 });

Normal map generation

Normal maps can be created via normalMap(). This function uses an adjustable convolution kernel size to control gradient smoothness & details. Result X/Y gradients can also be scaled (uniform or anisotropic) and the Z component can be customized to (default: 1.0). The resulting image is in FLOAT_NORMAL format, using signed channel values. This channel format is auto-translating these into unsigned values when the image is converted into an integer format.

Step	Scale = 1	Scale = 2	Scale = 4	Scale = 8
0
1
2
3

import { ARGB8888, FLOAT_GRAY, floatBufferFromImage, imageFromURL } from "@thi.ng/pixel";
import { normalMap } from "@thi.ng/pixel-convolve";

// read source image into a single channel floating point buffer
const src = floatBufferFromImage(await imageFromURL("noise.png"), FLOAT_GRAY);

// create normal map (w/ default options)
// this results in a new float pixel buffer with FLOAT_RGB format
const nmap = normalMap(src, { step: 0, scale: 1 });

// pixel lookup (vectors are stored _un_normalized)
nmap.getAt(10, 10);
// Float32Array(3) [ -0.019607841968536377, -0.04313725233078003, 1 ]

// convert to 32bit packed int format
const nmapARGB = nmap.as(ARGB8888);

Status

STABLE - used in production

Search or submit any issues for this package

Installation

yarn add @thi.ng/pixel-convolve

ESM import:

import * as pc from "@thi.ng/pixel-convolve";

Browser ESM import:

<script type="module" src="https://esm.run/@thi.ng/pixel-convolve"></script>

JSDelivr documentation

For Node.js REPL:

const pc = await import("@thi.ng/pixel-convolve");

Package sizes (brotli'd, pre-treeshake): ESM: 2.27 KB

Dependencies

• @thi.ng/api
• @thi.ng/checks
• @thi.ng/errors
• @thi.ng/math
• @thi.ng/pixel

Note: @thi.ng/api is in most cases a type-only import (not used at runtime)

Usage examples

Three projects in this repo's /examples directory are using this package:

Screenshot	Description	Live demo	Source
	Interactive image processing (adaptive threshold)	Demo	Source
	2.5D hidden line visualization of digital elevation files (DEM)	Demo	Source
	Normal map creation/conversion basics	Demo	Source

API

Generated API docs

Authors

• Karsten Schmidt

If this project contributes to an academic publication, please cite it as:

@misc{thing-pixel-convolve,
  title = "@thi.ng/pixel-convolve",
  author = "Karsten Schmidt",
  note = "https://thi.ng/pixel-convolve",
  year = 2021
}

License

© 2021 - 2024 Karsten Schmidt // Apache License 2.0