Contents

Computational Graphs - Exercises

Version 1.02

(C) 2020 - Umberto Michelucci, Michela Sperti

This notebook is part of the book Applied Deep Learning: a case based approach, 2nd edition from APRESS by U. Michelucci and M. Sperti.

# tensorflow setup

%tensorflow_version 1.x
import tensorflow as tf

TensorFlow 1.x selected.

Notebook Learning Goals

At the end of this notebook you will know how to create an easy computational graph with the basic types in TensorFlow.

To show you how the different datatypes worked in TensorFlow 1.X we will need this version and therefore at the beginning we use the magic command

%tensorflow_version 1.x

Since if we use TensorFlow 2.X we would have Eager execution enabled and the examples would not work.

Computational Graph - Sum of Two Variables

Let’s see some examples that will show you how you can solve the exercises that you will find in this notebook.

Let’s start with something easy.

\[ z = x_1+x_2 \]

This is simply the sum of two variables. Now let’s create the computational graph with different datatypes in TensorFlow to see the differences in how they work.

Graph with tf.constant

x1 = tf.constant(1)
x2 = tf.constant(2)
z = tf.add(x1, x2)

print(z)

Tensor("Add:0", shape=(), dtype=int32)

print(x1)

Tensor("Const:0", shape=(), dtype=int32)

Now let’s create a session and let’s evaluate the graph

sess= tf.Session()
print(sess.run(z))

3

print(sess.run(x1))

1

sess.close()

Graph with tf.Variable

x1 = tf.Variable(1)
x2 = tf.Variable(2)
z = tf.add(x1,x2)

print(z)

Tensor("Add_1:0", shape=(), dtype=int32)

#
# WARNING: you will get an error from this cell. This is normal and is designed
# to return an error. Continue to read below for the explanation.
#
sess = tf.Session()
print(sess.run(z))

---------------------------------------------------------------------------
FailedPreconditionError                   Traceback (most recent call last)
/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py in _do_call(self, fn, *args)
   1364     try:
-> 1365       return fn(*args)
   1366     except errors.OpError as e:

/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py in _run_fn(feed_dict, fetch_list, target_list, options, run_metadata)
   1349       return self._call_tf_sessionrun(options, feed_dict, fetch_list,
-> 1350                                       target_list, run_metadata)
   1351 

/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py in _call_tf_sessionrun(self, options, feed_dict, fetch_list, target_list, run_metadata)
   1442                                             fetch_list, target_list,
-> 1443                                             run_metadata)
   1444 

FailedPreconditionError: Attempting to use uninitialized value Variable_1
	 [[{{node Variable_1/read}}]]

During handling of the above exception, another exception occurred:

FailedPreconditionError                   Traceback (most recent call last)
<ipython-input-11-b1e96672aef6> in <module>()
      4 #
      5 sess = tf.Session()
----> 6 print(sess.run(z))

/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    954     try:
    955       result = self._run(None, fetches, feed_dict, options_ptr,
--> 956                          run_metadata_ptr)
    957       if run_metadata:
    958         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1178     if final_fetches or final_targets or (handle and feed_dict_tensor):
   1179       results = self._do_run(handle, final_targets, final_fetches,
-> 1180                              feed_dict_tensor, options, run_metadata)
   1181     else:
   1182       results = []

/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1357     if handle is None:
   1358       return self._do_call(_run_fn, feeds, fetches, targets, options,
-> 1359                            run_metadata)
   1360     else:
   1361       return self._do_call(_prun_fn, handle, feeds, fetches)

/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py in _do_call(self, fn, *args)
   1382                     '\nsession_config.graph_options.rewrite_options.'
   1383                     'disable_meta_optimizer = True')
-> 1384       raise type(e)(node_def, op, message)
   1385 
   1386   def _extend_graph(self):

FailedPreconditionError: Attempting to use uninitialized value Variable_1
	 [[node Variable_1/read (defined at /tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'Variable_1/read':
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/usr/local/lib/python3.6/dist-packages/traitlets/config/application.py", line 664, in launch_instance
    app.start()
  File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelapp.py", line 499, in start
    self.io_loop.start()
  File "/usr/local/lib/python3.6/dist-packages/tornado/platform/asyncio.py", line 132, in start
    self.asyncio_loop.run_forever()
  File "/usr/lib/python3.6/asyncio/base_events.py", line 438, in run_forever
    self._run_once()
  File "/usr/lib/python3.6/asyncio/base_events.py", line 1451, in _run_once
    handle._run()
  File "/usr/lib/python3.6/asyncio/events.py", line 145, in _run
    self._callback(*self._args)
  File "/usr/local/lib/python3.6/dist-packages/tornado/platform/asyncio.py", line 122, in _handle_events
    handler_func(fileobj, events)
  File "/usr/local/lib/python3.6/dist-packages/tornado/stack_context.py", line 300, in null_wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/zmq/eventloop/zmqstream.py", line 462, in _handle_events
    self._handle_recv()
  File "/usr/local/lib/python3.6/dist-packages/zmq/eventloop/zmqstream.py", line 492, in _handle_recv
    self._run_callback(callback, msg)
  File "/usr/local/lib/python3.6/dist-packages/zmq/eventloop/zmqstream.py", line 444, in _run_callback
    callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tornado/stack_context.py", line 300, in null_wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelbase.py", line 283, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
    handler(stream, idents, msg)
  File "/usr/local/lib/python3.6/dist-packages/ipykernel/kernelbase.py", line 399, in execute_request
    user_expressions, allow_stdin)
  File "/usr/local/lib/python3.6/dist-packages/ipykernel/ipkernel.py", line 208, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/usr/local/lib/python3.6/dist-packages/ipykernel/zmqshell.py", line 537, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 2718, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 2822, in run_ast_nodes
    if self.run_code(code, result):
  File "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py", line 2882, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-9-93cd6787ae35>", line 2, in <module>
    x2 = tf.Variable(2)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/variables.py", line 258, in __call__
    return cls._variable_v1_call(*args, **kwargs)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/variables.py", line 219, in _variable_v1_call
    shape=shape)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/variables.py", line 197, in <lambda>
    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/variable_scope.py", line 2519, in default_variable_creator
    shape=shape)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/variables.py", line 262, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/variables.py", line 1688, in __init__
    shape=shape)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/variables.py", line 1872, in _init_from_args
    self._snapshot = array_ops.identity(self._variable, name="read")
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/util/dispatch.py", line 180, in wrapper
    return target(*args, **kwargs)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/array_ops.py", line 203, in identity
    ret = gen_array_ops.identity(input, name=name)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/gen_array_ops.py", line 4239, in identity
    "Identity", input=input, name=name)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

The error message is normal… First you have to initialize the variables in TensorFlow! TensorFlow does not automatically initialize the variables.

sess.run(x1.initializer)
sess.run(x2.initializer)
print(sess.run(z))

3

init = tf.global_variables_initializer()

sess.run(init)
print(sess.run(z))

3

sess.close()

Graph with tf.placeholder

x1 = tf.placeholder(tf.float32, 1)
x2 = tf.placeholder(tf.float32, 1)

Note that in this declaration no values were passed. The “1” is the dimension of the placeholder, in this case a scalar!

z = tf.add(x1, x2)

print(z)

Tensor("Add_2:0", shape=(1,), dtype=float32)

sess = tf.Session()

feed_dict = {x1: [1], x2: [2]}
print(sess.run(z, feed_dict))

[3.]

sess.close()

Now we can also use vectors as input, to show that we don’t need to deal only with scalars.

x1 = tf.placeholder(tf.float32, [2])
x2 = tf.placeholder(tf.float32, [2])

z = tf.add(x1,x2)
feed_dict = {x1: [1,5], x2: [1,1]}

sess = tf.Session()
sess.run(z, feed_dict)

array([2., 6.], dtype=float32)

Exercises

Exercise 1 (Difficulty: easy)

Draw and develop in TensorFlow with tf.constant the computational graphs for the following operations

A) w1*x1 + w2*x2 + x1*x1

B) A*x1 + 3 + x2/2

Use as input values x1 = 5 and x2 = 6.

Exercise 2 (Difficulty: medium)

Draw and develop in TensorFlow with tf.Variable the computational graph for the following operation A*(w1*x1 + w2*x2).

Build the computational graph and then evaluate it two times (without re-building it) with the initial values in the same session

A) x1 = 3, x2 = 4

B) x1 = 5, x2 = 7

Exercise 3 (Difficulty: FUN)

Consider two vectors

x1 = [1,2,3,4,5],  x2 = [6,7,8,9,10]

Draw and build in TensorFlow the computational graph for the dot-product operation between the two vectors. If you don’t know what a dot-product is you can check it here (we covered that in our introductory week) .

Build it in two different ways:

A) Do it with loops. Build a computational graph that takes as input scalars and in the session/evaluation phase build a loop to go over all the inputs and then sums the results.

B) Do it in one shot with tensorflow. Build a computational graph that takes as input vectors and do the entire operation directly in TensorFlow.

Hint: you can use in TensorFlow two methods: tf.reduce_sum(tf.multiply(x1, x2)) or tf.matmul(x1, tf.reshape(x2, [-1, 1])). Try to understand why they work checking the official documentation.

Difference between run and eval

Consider the following code:

x1 = tf.constant(1)
x2 = tf.constant(2)
z = tf.add(x1, x2)
init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)
sess.run(z)

3

You can evaluate more nodes at the same time.

sess.run([x1,x2,z])

[1, 2, 3]

You can also use the eval() method to evaluate nodes, but you have to provide the session name.

z.eval(session = sess)

3

Dependencies between Nodes

Sometime TensorFlow may evaluate some nodes multiple times. Pay attention. Consider the following code:

c = tf.constant(5)
x = c + 1
y = x + 1
z = x + 2
sess = tf.Session()
print('y is',sess.run(y))
print('z is',sess.run(z))
sess.close()

y is 7
z is 8

Here x is evaluated twice! You can do it like this:

c = tf.constant(5)
x = c + 1
y = x + 1
z = x + 2
sess = tf.Session()
print('y,z are',sess.run([y,z]))
sess.close()

y,z are [7, 8]

Now x is evaluated only once.

How to Close a Session

You can close a session in this way:

sess = tf.Session()
# Do something here
sess.close()

You can also do it in this way:

with tf.Session() as sess:
    # Do something here

  File "<ipython-input-29-ede77bfb1f3b>", line 2
    # Do something here
                       ^
SyntaxError: unexpected EOF while parsing

You need some code to avoid the error. Consider the following code:

c = tf.constant(5)
x = c + 1
y = x + 1
z = x + 2
sess = tf.Session()
print('y is',sess.run(y))
print('z is',sess.run(z))
sess.close()

y is 7
z is 8

This can also be written as the following:

c = tf.constant(5)
x = c + 1
y = x + 1
z = x + 2
with tf.Session() as sess:
    print('y is',sess.run(y))
    print('z is',sess.run(z))

y is 7
z is 8

Now you don’t need to close the session explicitly. Is done for you.

Exercises

Exercise 4 (Difficulty: medium)

Write a function that build a computational graph for the operation x1+x2 where the input x1 and x2 are input with given dimensions. Your x1 and x2 should be declared as tf.placeholder. Your functions should accept as input:

  • dimensions of x1 as list, for example [3]

  • dimensions of x2 as list, for example [3]

The function should return a tensor z = x1 + x2. Then open a session and evaluate z with the following inputs:

  • x1 = [4,6,7], x2 = [1,2,9]

  • x1 = [1,2,....., 1000], x2 = [10001, 10002, ...., 11000]

and print the result.

Exercise 5 (Difficult: FUN)

Linear Regression with TensorFlow

https://onlinecourses.science.psu.edu/stat501/node/382/

Consider the following dataset:

x = [4.0, 4.5, 5.0, 5.5, 6.0, 6.5, 7.0]
y = [33, 42, 45, 51, 53, 61, 62]

We want to find the best parameters \(p_0\) and \(p_1\) that minimise the MSE (Mean Squared Error) for the data given, in other words we want to do a linear regression on the data \((x,y)\). Given that a matrix solution to find the best parameter is

\[ {\bf p} =(X^TX)^{-1} X^T Y \]

where \(X^T\) is the transpose of the matrix \(X\). The matrix \(X\) is defined as

\[\begin{split} X = \begin{bmatrix} 1 & x_1 \\ ... & ... \\ 1 & x_n \end{bmatrix} \end{split}\]

The matrix \(Y\) is simply a matrix \(n\times 1\) containing the values \(y_i\).

Dimensions are:

  • \(X\) has dimensions \(n\times 2\)

  • \(Y\) has dimensions \(n\times 1\)

  • \({\bf p}\) has dimensions \(2\times 1\)

Build a computational graph that evaluates \(\bf p\) as given above, given the matrices \(X\) and \(Y\). Note you will have to build the matrices from the data given at the beginning. If you need more information a beatifully long explanation can be found here https://onlinecourses.science.psu.edu/stat501/node/382/.