Operator Overloading in TensorFlow¶

Version 1.02

This notebook is part of the book Applied Deep Learning: a case based approach, 2nd edition from APRESS by U. Michelucci and M. Sperti.

Notebook Learning Goals¶

At the end of this notebook you will be able to undertsand the difference between operations like tf.multiply(x,y) and x*y when used with tensors or numpy arrays.

Libraries Import¶

import tensorflow as tf
import numpy as np

Operator Overloading¶

What is the difference between tf.add(x,y) and x+y? Or between tf.multiply(x,y) and x*y?

If at least one of the arguments x and y is a tensor (tf.Tensor) then tf.add(x,y) and x+y are equivalent. The reason you may want to use tf.add() is to specify a name keyword argument for the created op, which is not possible with the overloaded operator version. The keyword will be used in tensorboard when visualising the computational graph. If you don’t need it you can use the overloaded operator, meaning +.

Note that if neither x nor y is a tf.Tensor (for example if they are NumPy arrays) then x + y will not create a TensorFlow node. tf.add() always creates a TensorFlow node and converts its arguments to tf.Tensor objects.

The following operators are overloaded in the TensorFlow Python API:

neg (unary -)

abs (abs())

invert (unary ~)

add (binary +)

sub (binary -)

mul (binary elementwise *)

div (binary / in Python 2)

floordiv (binary // in Python 3)

truediv (binary / in Python 3)

mod (binary %)

pow (binary **)

and (binary &)

or (binary |)

xor (binary ^)

lt (binary <)

le (binary <=)

gt (binary >)

ge (binary >=)

See: https://www.tensorflow.org/api_docs/cc/group/math-ops.

eq ( binary == )

is not overloaded. x == y will simply return a Python boolean array. You need to use tf.equal() to do element-wise equality checks.

Programming Style¶

On the official documentation you can find the following style guideline:

Prefer using a Tensor's overloaded operators rather than TensorFlow functions. For example, prefer **, +, /, *, - and/or over tf.pow, tf.add, tf.div, tf.mul, tf.subtract, and tf.logical * unless a specific name for the operation is desired. 

Again name is referred to the use of TensorBoard.

REMEMBER: if your inputs are not tensors, you may end up with other datatypes other than tensors. With the tf.XXXX functions you always have a tensor at the end.

Example¶

x = np.array([1, 2, 3])
y = np.array([3, 4, 5])

x + y

array([4, 6, 8])

The sum is a numpy array.

type(x + y)

numpy.ndarray

z = tf.add(x, y)

type(z)

tensorflow.python.framework.ops.EagerTensor

In this case, also if the inputs were numpy arrays, the output is a tensor.

Applied Deep Learning - 2nd Edition