A neural layer

Generated by tools/gen_example_pages.py from mnd_scripts/examples/example_neural_layer.mnd — the output below is a real run, not a transcription. Regenerating the page reruns it.

a complete quantized neural layer — matmul, bias, and the activation set (relu, sigmoid, tanh, softmax, layernorm) — all in integer arithmetic against a fixed-point scale.

The program#

# ============================================================================
# example_neural_layer.mnd
# ----------------------------------------------------------------------------
# FUNCTIONALITY: a complete quantized neural layer -- matmul, bias, and the
# activation set (relu, sigmoid, tanh, softmax, layernorm) -- all in integer
# arithmetic against a fixed-point scale.
#
# THE SCALE IS THE PROGRAM'S CONTRACT. `mnd::quantres=N` says every register
# integer is a real number times 10^N. With quantres=3, the integer 500 means
# 0.500 and sigmoid(0) prints as 500, not 0.5. The activation LUTs are shared
# byte-for-byte between this interpreter and the worker firmware, so a local
# run and a remote FPGA run produce the SAME integers -- which is what makes
# `maned-run --device ... --verify` a meaningful equality check rather than
# a tolerance comparison.
#
#   relu       max(x, 0), exact at any scale
#   sigmoid    fixed-point LUT, decoded at 10^quantres
#   tanh       fixed-point LUT, decoded at 10^quantres
#   softmax    takes {axis: n} object params
#   layernorm  zero-mean unit-variance per row, at 10^quantres
#
# RUN: ../maned-run examples/example_neural_layer.mnd
# Compare against a worker:
#   ../maned-run examples/example_neural_layer.mnd \
#       --device http://HOST:PORT --device-user U --device-pass env:PASS --verify
# ============================================================================
mnd::quantmax=1000;
mnd::quantmin=-1000;
mnd::quantres=3;          # values are scaled by 10^3 -- 1000 means 1.000

# Inputs are written at that scale: 0.100, 0.200, -0.300, 0.400.
in::x    = [[100,200],[-300,400]];
in::w    = [[1000,0],[0,1000]];      # identity at scale 1000
in::bias = [[50,-50],[50,-50]];      # +0.050 / -0.050

in::scale = 1000;          # 10^quantres, written out so the rescale is visible

calc::lambda_flow layer(x, w, bias, scale) {
    # --- affine part -------------------------------------------------------
    # Multiplying two operands that are each scaled by 10^quantres gives a
    # result at 10^(2*quantres). Rescaling is the PROGRAM'S job, not a hidden
    # coercion -- that explicitness is the whole point of a declared
    # quantization contract, and it is why the same integers come back from an
    # FPGA worker that has no divider on the critical path.
    x w matmul raw_product =
    raw_product scale div product =            # back down to 10^quantres
    product bias elemwise_add pre_activation =

    # --- activations -------------------------------------------------------
    pre_activation relu   activated_relu =
    pre_activation sigmoid activated_sigmoid =
    pre_activation tanh    activated_tanh =

    # --- softmax: object params come BEFORE the operator -------------------
    # Postfix means EVERY argument precedes its operator, and the `{...}`
    # configuration block is an argument. `{axis: 1} softmax`, never
    # `softmax {axis: 1}` -- the latter parses but the params arrive too late
    # and the operator silently uses its defaults.
    pre_activation {axis: 1} softmax attention_weights =

    # --- normalization -----------------------------------------------------
    pre_activation layernorm normalized =

    # --- a reduction to a single scalar figure of merit --------------------
    activated_relu tensor_sum total_activation =
} return raw_product, product, pre_activation, activated_relu, activated_sigmoid,
         activated_tanh, attention_weights, normalized, total_activation;

The run#

$ maned-run mnd_scripts/examples/example_neural_layer.mnd
flow 'layer' outputs:
raw_product : [2x2] = [100000, 200000, -300000, 400000]
product : [2x2] = [100, 200, -300, 400]
pre_activation : [2x2] = [150, 150, -250, 350]
activated_relu : [2x2] = [150, 150, 0, 350]
activated_sigmoid : [2x2] = [537, 537, 438, 587]
activated_tanh : [2x2] = [149, 149, -245, 336]
attention_weights : [2x2] = [500, 500, 354, 646]
normalized : [2x2] = [0, 0, -1000, 1000]
total_activation : [1] = [650]

Next#

• Language guide — every operation the examples use, in the full table.
• RPN syntax specification — the normative rules.