Beginner guide to Vectorization, Broadcasting & Indexing in Numpy

SIMD (single instruction, multiple data)

In previous decades, the rise of ML (DL) draw everyone from different background, many academia (logicians, mathematicians, neurologist etc), scientists/engineers (computing on Matlab, Octave) and especially programmers. Google the term “AI hype cycle” and you’ll see how Artificial General Intelligence(AGI) flip the coin & now we had narrow/weak ML or should i say Deep Artificial Neural Networks everywhere. In order to emerge complex systems, we need more than that, pattern formation, collective behavior, nonlinear dynamics, evolution & adaptation just to name a few. I am not here to show you those but humble building blocks that are part of ML system in production.

SIMD on Wiki

Many weak AIs depend upon vector/matrix friendly programming languages like Python & R. Both languages support arrays/vectors operation out of the box & we need more, Numpy to the rescue. This short article will cover 3 useful topics Vectorization, Broadcasting and Indexing.

Q: Let’s address the elephant in the room, why vector or multiple-dimension array processing ?

A: Vector processor, way faster than general purpose registers especially on modern CPU

Followings are practical usage

  1. Multimedia applications need fast arithmetic operations on the large scale of integers or floating-point processing
  2. Intensive GPU processing, like games, highest resolution or mining :P
  3. Machine learning, gradients and automatic differentiation, or simple Y=a + bX
Image source: NativeScript blog

Please dive detail of computer architectures at Flynn’s taxonomy. And feel free to skip two session if you’re familiar with basic concepts.

1. Vectors, Matrices, and Arrays

List, build-in data type of python can created either using list function or primitive

list([1,2,3])
[1,2,3]

Array

import numpy as nparr = np.array([1,2,3])type(arr)
# numpy.ndarray

Vector, one-dimensional array horizontally (row) or vertically (column).

import numpy as np

vector_row = np.array([1, 2, 3])
vector_column = np.array([[1],
[2],
[3]])

Matrix, just simple rectangular array with X rows & X columns.

import numpy as npmatrix = np.mat([[1, 2],
[1, 2],
[1, 2]])
type(matrix)
# numpy.matrix
array_matrix = np.array([[1, 2],
[1, 2],
[1, 2]])

Note: please don’t use this .mat() function but stick with .array() method since they are defecto standard in Numpy and many function/operation return array(numpy.ndarray) not matrix(numpy.matrix) data type.

A vector is comparable to a fixed-length array containing integer or floating-point values. — apple doc

array_matrix.size   
# will get the size(number of elements) of matrix
array_matrix.shape
# will get the shape (x,x,x) of matrix

2. Intro to NumPy Arrays

I assume you have basic python skills here, otherwise this freecodecamp course will help you. Let’s create evenly spaced values using arange() function and reshape into interesting shape of matrices.

arr  = np.arange(18).reshape(2, 3, 3)
arr2 = np.arange(18).reshape(3,2,3)
Arrange and Reshape from Numpy Lib

These two arrays are in 18 elements starting with zero, for first array you should be reading from left to right like “2 reshaped arrays, which are in 3 rows X 3 columns”. You got it for arr2 then “3 reshaped arrays in 2 rows X 3 columns format”.

Numpy arrays are extremely fast, store in continuous memory block of memory in short.

3. Vectorization

Vectorization is a method for achieving parallelism inside a single processor core. — Intel blog

To put into layman’s terms a single instruction perform over each individual element of an array on single core of CPU in parallel manner. Vectorized operation can replace traditional for loops, run faster, work on different matrix dimension size. Take away — it’s just array expression replacing explicit for-loop.

This practice of replacing explicit loops with array expressions is commonly referred to as vectorization. In general, vectorized array operations will often be one or two (or more) orders of magnitude faster than their pure Python equivalents, with the biggest impact [seen] in any kind of numerical computations. ~ Wes McKinney

import numpy as np

matrix = np.array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])

sqr = lambda i: i * i

vectorized_sqr = np.vectorize(sqr)

vectorized_sqr(matrix)

To explain about snippet, we are using lambda(nameless/anonymous function) to double the input value i and we pass that into Numpy’s vectorize method as instruction[lambda named sqr ] then pass the data [matrix].

Vectorize function is just for convenience it won’t run any faster and also np.array is already aware of vectorization.

Many examples in this article refer this variable called [matrix].

4. Broadcasting

From previous example np.vectorize( ) function work on different matrix dimensions, this is known as broadcasting. Meaning rules like same shape don’t constraint here to perform any operation.

matrix2 = np.arange(8).reshape(2,2,2)
# Reshaping into 2 matrices of 2rows X 2columns
vectorized_sqr(matrix2)
# Can still use this function with new shaped data

Simple example would be following snippet and you can find the theory behind it.

x = np.array([1,2,3])
y = np.array([10])
x + y
# array([11, 12, 13]) element wise sum operation performed

5. Indexing

Access the array elements arbitrarily permitting items to be access out of order and even repeatedly. — unknown source

We’ll start with very basic 1D and work up to multi-dimension arrays.

one_d = np.array([1,2,3,4,5])one_d[0]   # access from starting index = 1
one_d[-5] # reverse access from ending point = 1
one_d[4] # 5
one_d[-1] # 5

2 Dimension array

two_d = np.array([[1,2,3,4,5], [6,7,8,9,10]])two_d[1][2]   # 1 is second array, 2 for 3 index of that array= 8
two_d[-1][-3] # It's the same but access from ending point= 8

3 Dimension Array

3D array
three_d = np.arange(18).reshape(2,3,3)three_d[0][0][1]    # we will have= 1 three_d[-2][-3][-2] # same value= 1

Let’s jump into randomly accessing to an array. We will select 2 elements from the matrix, then we will access using first wrapper size(3), it will be both column/row .

n = np.arange(9).reshape(3,3)
# array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])
n[[0,1],[1,1]]
# Will get middle elements of first two array 1 & 4
n[[0,1,2],[1,1,1]]
# First we access row number then column of all elements
# According to our matrix [n]
# 1 index is [0,1]
# 4 index is [0,1]
# 7 index is [0,1]

To be fancy we will access group of them using normal list and np_array.

normal_list = list(range(1,10))np_array = np.arange(1,10)accessor = np.arange([2,3,4])normal_list[accessor]
TypeError: only integer scalar arrays can be converted to a scalar index
np_array[accessor]
# We will get index 2,3,4 of an np_array access with np_array
accessor_list = [2,3,4]np_array[accessor_list]
# It's valid, we access via normal python list to an np_array array

Let’s switch our gear to Boolean indexing, what we’d seen above is just Integer indexing. Boolean indexing is useful to get rid of unwanted data, replacing null/None or incomplete data, other tasks like masking, filtering etc.

m = np.array([[1,-2,3],[-2,4,3]])m[m < 2]
# array([ 1, -2, -2])
# We got elements those are last than 2
m[m > 2] * 2
# array([6, 8, 6])
# We can do on the fly calculation like this too

Slicing, we will be using : column operator to access row or column in steps manner, or all elements.

array[[row, column]] and array [start:end:step] can use interchangeable.

row = np.arange(25).reshape(5,5)
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19],
[20, 21, 22, 23, 24]])
# 3rd 4th column of that matrix
row[:,[2,3]] or row[:, 2:4:1]
# : in first parameter meaning for every row
array([[ 2, 3],
[ 7, 8],
[12, 13],
[17, 18],
[22, 23]])
# 4th 3rd column of that matrix
row[:,-2:-4:-1]
# Remember how we reverse by using negative value ?
# Here not only we access from last element, it's also return the result in that order
array([[ 3, 2],
[ 8, 7],
[13, 12],
[18, 17],
[23, 22]])

We can combine Boolean indexing with slicing too, it’s getting more interesting.

greater_than_4 = row > 4
array([[False, False, False, False, False],
[ True, True, True, True, True],
[ True, True, True, True, True],
[ True, True, True, True, True],
[ True, True, True, True, True]])
row[greater_than_4]
array([ 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24])
# Please notice that we'd flatten into one-dimension array
# Replacing any elements under4
# Condition, Data, Replacement for not meeting condition
np.where(greater_than_4, row, 0)
array([[ 0, 0, 0, 0, 0],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19],
[20, 21, 22, 23, 24]])

Put famous logical and/or/not, not only for filtering but also replacing & slicing.

zero_under_4_over_20 = np.logical_and(row > 4, row < 20)
array([[False, False, False, False, False],
[ True, True, True, True, True],
[ True, True, True, True, True],
[ True, True, True, True, True],
[False, False, False, False, False]])
row[zero_under_4_over_20]
# as usual we will get one-dimension array back
np.where(zero_under_4_over_20, row, 0)
array([[ 0, 0, 0, 0, 0],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19],
[ 0, 0, 0, 0, 0]])

I had covered some of advance indexing topic here but as always please reference official doc for more information.

6. Average, Variance & Standard Deviation

Let’s continue with useful Numpy function, using same old data [matrix]

np.max(matrix)
# Max value of an element
np.min(matrix)
# Min value of an element
np.mean(matrix)
# Return mean value
np.var(matrix)
# Return mean variance
np.std(matrix)
# Return standard deviation value

Another useful trick would be flatten the matrix into vector either row or column. Flatten and reshape are exactly the same except the one with -1 and column orientation.

matrix.flatten() 
# flatten into 1 dimension matrix or row vector
matrix.reshape(9)
# Since we refer to previous variable name called matrix, which had 9 elements, reshape into flat row vector
matrix.reshape(1,-1)
# -1 mean "as many as needed", but this would create nested array inside so i suggest we avoid this
matrix.reshape(9, -1)
# Reshape into flat column vector

References and useful resources

I hope you get the essence of vectorization and other tricks provided by Numpy. Happy learning, please provide your feedback on comment.

Full stack developer, base in Myanmar. Turn coffee into code, a husband, a father, a reader.