Network Automation The Secret 1667693521
Network Automation The Secret 1667693521
Network Automation The Secret 1667693521
Disclaimer
1. Context
1.1 Network Automation with Python course
1.2 Other references
2. Objectives
3. Example scenario
3.1 Example Real Production Scenario
7. What do I use?
References
Books
Blogs
This work was made by IeXa Academy, the authors are Aldo Menichelli, Giuseppe Citerna, Elvio
Sebastianelli, Claudio Scala.
If you want to cooperate on the document to add or modify content, please contact IeXa
Academy (click on this link)
Iexa Academy S.r.l. is in no way affiliated with any of the companies mentioned, including Cisco System Inc.
This document is not officially sponsored, affiliated, or endorsed by Cisco Systems Inc. Iexa Academy S.r.l.
respects the trademarks of all other companies and institutions mentioned.
Some logos, third-party trademarks, product names, trade names, corporate names and companies in this
product may be trademarks owned by their respective owners or registered trademarks of other
companies and are used for purely explanatory purposes, without any purpose of violation of the
Copyright rights in force by IEXA Academy S.r.l.
If this describes you and you want to learn the secret to understand automation concepts, this
document is for you.
Network Automation revolves around the practice of having software configure your network
devices - as opposed to you having to manually log on to each device and write lines of text or
edit files on that device.
Of course there’s a lot more to Network Automation than this, but this is the central idea.
The benefits of having software configure your network are obvious: repetitive tasks can be
performed without human fatigue or human error, configuration changes can be pushed out
much faster and with more predictable results, and it gives you the opportunity to integrate with
other IT workflow tools.
This is a preview of IeXa Academy’s Network Automation course, follow the link to have more
information.
You might find all the codes of this document in the following GitHub repository:
https://github.com/IexaCommunity/NetworkAutomation.TheSecret
Finally, we host the course in an innovative learning experience platform, that is LearnAmp,
which we can define as the Netflix for learners. It allows us to manage students, groups and
contents, to speed up your path!
This course guides the student from the setup of his own lab environment till the launch of a
500 lines long Python script!
This book contains the first attempt of structuring all those tools that we
encountered when we approached these new networking concepts. So it
represents our starting point for this document!
How exactly you indicate your “intention” varies depending upon the exact software technology
and application you choose, and this is going to be the focus of the rest of this article.
The goal of this article is to provide an overview of the Network Automation landscape. That
means that we are not going to give you a comprehensive-yet-incomprehensible story filled with
terminology, nor are we going to just point you at a bunch of GitHub code repositories and send
you on your way.
What we do, instead, is build a model, just like the ISO-OSI pile, which guides engineers into the
development of network protocols. This model’s aim is to give you a rundown of network
automations instruments, to help you define a toolbox from which you can design suitable
automation solutions to implement in real networks. We are going to detect the core layers and
their functionalities, because it wouldn’t be possible to analyze every existing tool, since there
are hundreds of these every day.
Here’s one perspective on how you might classify tools within the definition of their core
functionalities.
As we progress through this document, we will analyze each layer of this model and focus on
the dependencies between these tools.
3. Example scenario
To get started off on exploring networking automation and building proofs-of-concept, all you
need is a Linux computer. A Linux VM on your laptop or workstation will work just fine. You will
also need at least one network device to experiment with.
We will use this setup to make many examples with different automation tools: specifically, we
are going to run some scripts on our Control Machine, that is an Ubuntu 20.04 LTS virtual
machine, and a virtualized csr1000v, which run Cisco IOS XE 1671.
However, before diving in the theoretical aspects of Network Automation, we are going to see
what could be a real production environment.
The figure above depicts a Distribution switch connected to several Access switches - this is a
small segment of a typical 3-tier enterprise network, with layer-2 in the Access layer, and layer-3
in the Distribution layer.
This could be a possible scenario: a new VLAN, 220, has been created on the Access switch,
and some ports on the Access switch have been assigned to VLAN 220. The trunk port has also
been configured on the Access switch.
Your job could be to configure the Distribution switch for this new VLAN 220. This would involve
executing the following steps (Cisco devices assumed):
This scenario is a good example for demonstrating Network Automation because it is a typical
operation that must be carried out repeatedly as and when new VLANs are added. The steps
are simple enough, but susceptible to human error especially when executed by operations
personnel with limited skill and experience.
Every single network device has to perform at least three distinct activities:
● Process the transit traffic (that’s why we buy them) in the data plane;
● Figure out what’s going on around it with the control plane protocols;
● Interact with its owner (or NMS) through the management plane.
Routers are used as a typical example in every text describing the three planes of operation:
● Interfaces, IP subnets and routing protocols are configured through management plane
protocols, ranging from CLI to NETCONF and RESTful API;
● Router runs control plane routing protocols (OSPF, EIGRP, BGP …) to discover adjacent
devices and the overall network topology to understand which path the packets should
follow;
● Router inserts the results of the control-plane protocols into Routing Information Base
(RIB) and Forwarding Information Base (FIB). Data plane software or ASICs uses FIB
structures to forward the transit traffic;
● Management plane protocols like SNMP can be used to monitor the device operation, its
performance, interface counters, …
There are some advantages and disadvantages of having a distributed vs a central control
plane. One of the advantages of having a central controller is that we can configure the entire
network from a single device. This controller has full access and insight of everything that is
happening in our network.
SDN gives you a centralized control point for your network, but you still have to provide most of
the intelligence to deploy or change the underlay and overlay. That means you still have to tell
the control plane how to do what you want it to do.
Let’s start by defining Network Management, which means interacting with the network to
make it achieve all the expectations you (or your company) have. This happens when dealing
with different aspects of the network:
Another term that needs to be defined is Automation: it means using software to perform a task
you would otherwise do manually. Automation software might be built into a network node,
might be a purchased software platform, or might be a program or script you create yourself.
Speaking about the last part, our goal is to give you all the fundamentals to be able to
understand the underlying protocols used by products and to use them in your scripts and
programs to create your personal network automation solution.
Going on with definitions, let’s talk about Orchestration: in IT sense, it is the coordination of
many automated tasks, in a specific sequence, across many systems to accomplish a single
objective, also known as workflow.
Be careful not to confuse automation with orchestration: automation performs individual tasks,
orchestration automates workflows.
The real performer that makes actual automation and orchestration happen in the network is
programmability. Programmability makes it possible to send commands and parameters to make
everything happen and gets the configuration that is needed directly into the device, by taking
the device-agnostic service parameters and turning them into a network-specific configuration
that is tied to specific devices.
Network Programmability and Automation are two processes that work together to make human
intervention even less required in implementations and operations. In other words, we can say
that automation consists of taking the tools developed for network programmability and adding
intelligence to them so that they carry out tasks and make decisions on their own, without the
intervention of an individual.
In one way or another, the programmability and automation of networks has existed. However,
the difference with the before and now is the level of integration between platforms that you
want to reach. You want to reach a full level of orchestration in service provisioning or network
management. A manufacturer-independent orchestration using standard programmable
interfaces and protocols.
All that is left to do is to understand what are the approaches used by companies and tools
developers towards the network automation and programmability world we’ve just started to
see. There are many methods, open-source or vendor-specific solutions, but their explanation
and definition are beyond the scope of this document.
Right now we can identify two main paradigms and spend some words on them.
These are technologies that are concerned with connecting to the network device, gathering its
state, and changing its state. There are two sub-categories.
These are software interfaces that essentially mimic the human user: establish an SSH or telnet
or serial line connection to the network device, then send text commands (e.g. “show
inventory”), and then read (parse) the output printed by the device. Some examples in this
category are:
● Python (with the Paramiko or Netmiko libraries) - we’ll be talking about this
● Expect
API stands for application programming interface, which is a set of definitions and protocols for
building and integrating application software. APIs are sometimes thought of as contracts, with
documentation that represents an agreement between parties: if system 1 sends a remote
request structured a particular way, this is how system 2’s software will respond.
These are non text-based interfaces found on some device models that allow external software
to connect to the device and set/get its state using methods other than human CLI commands.
The phrase Model Driven Programmability is sometimes used to describe these technologies.
Some examples in this category are:
Note on SNMP: SNMP would fall under the subcategory of a non text-based interface to access
a device’s state. Over the many decades of its existence, though, the industry has used SNMP
heavily for device monitoring but not so much to change the device’s state. For this reason this
article will not talk about SNMP.
The Device Interface technologies we spoke about in the previous sections are low-level
interfaces, in the sense that they take care of moving configurations (desired states) to the
device, and in the reverse direction deliver the device status (operational state) back to the
controlling station. You could regard them as merely “transport” mechanisms.
● expressing the desired state of a device, usually using a language that is specific to the
system (we’ll see examples of this below)
● maintaining and managing device configurations (as “recipes” or “playbooks”)
● maintaining an inventory of devices in a formal and standardized manner
● Chef
● Puppet
● Ansible
One point to keep in mind about Configuration Management systems is that historically they
were conceived and developed for automating IT systems, e.g. servers. None of them were
specifically developed for automating network devices. This is a good thing, though - at a
conceptual level, there is little difference between configuring the Apache web server on a
bunch of servers, and configuring a QoS policy on a bunch of access switches. Having a single
tool that does both therefore makes a lot of sense.
In this framework we defined eight layers, related to the main functionalities we need when
talking about a Network Automation solution. Let’s define the role of each layer (we’ll mention
some of the tools here, don’t worry if you don’t know them yet, we’ll see them together in the
next chapters):
● Programming → At the core of this framework you need, this layer is the one that
contains all the languages that allow you to write code that is readable and reusable.
Not everyone in the industry has to put hands on scripts, you might just use something
that someone has already developed for others. So, for example, you might just work
with Ansible or other higher layer tools that exploit Python modules written by some
coworker. But you should know that actual programming is needed.
● Libraries → Libraries are a set of useful functions that eliminate the need for writing
codes from scratch. Since we are talking about network automation, we are going to use
many libraries that allow us to connect to network devices and to operate with higher
layer tools. For example, we are going to see Python libraries to connect through the
SSH protocol and to send and receive data from the device using Napalm, a framework
that relies on these lower modules.
● Protocols → Traditionally, administrators have used protocols like SNMP for monitoring
and a CLI to apply configurations. While those are still viable options on many network
devices, Cisco included, they aren't optimized for automation. The purpose of these new
● Encoding → A big part of letting computers perform tasks for us is providing data in a
format that computers can read. This is where YANG, YAML, JSON, and XML come into
the network programmability picture. To query or change data, a client sends, for
example, an XML-based call over one of the supported secure transfer methods, and the
server replies with XML-encoded data, if you are using NETCONF or RESTCONF as
protocols.
● Modeling → A Data Model is a set of rules that define how data should be structured,
formatted and represented, in order to be consumed. For all those data formats, valid
content is defined by YANG. This means that a YANG module defines, for example, the
fields needed to represent the configuration of an interface. Through that YANG module,
we can write an XML message with the correct tags to retrieve or modify the
configuration of the interface of the router and send it through the NETCONF message
structure.
● Automation Products → These are the actual tools and frameworks that allow users to
interact with various network operating systems through a unified API. Some of them use
Domain Specific Languages, like Ansible or Puppet, others are pure Python frameworks,
like Nornir and Napalm. They rely on the use of the other layers in many ways. Indeed,
we’ll see that they can actually skip layers in this model. For example, Ansible might use
just one Python library like Paramiko, or multiple automation frameworks through its
plugins.
● Transport Protocols → Going back to our more familiar traditional networking concepts,
we know that whatever is the content of the packets, they need to travel over networks.
And we need to create a channel between the control machine and the network device.
This channel is created through some application protocols of the ISO-OSI model, like
Telnet, SSH and HTTPS.
In the next paragraphs we are going to describe with more details the layers and some
examples of tools.
As we said, the network industry is fundamentally changing. Scripting or writing code to gather
information on the network, or to make change, isn’t new at all. It’s been done for years. There
are engineers who used to code in their language of choice, who learned to work with raw text
using complex parsing, regular expressions, and querying SNMP MIBs in a script. But working
with regular expressions and parsing text is time-consuming and tedious, it demands that the
network administrator knows the output of the commands, too dependent on the vendor, the
OS, or its release.
Many engineers could confirm that the automation concept is not unfamiliar to them, since
they’ve been automating operations through Batch, Bash, Except, VBScript in SecureCRT and
many other languages.
The innovation lies in the fact that today there are many freely available open source tools,
network devices APIs, vendor and community supported Python libraries that give you and
every other network engineer access to a growing ecosystem to jump start your network
automation journey. This ultimately means that you have to write less code than you would have
in the past, and less code means faster development and fewer bugs.
Programmability and automation, although it has become popular in recent years, are not new
concepts. Engineers have used the command line to program. Over the years there have been
resources like TCL (Tool Command Language) Scripting that interpret and execute CLI
commands to automate administrative tasks. Cisco has long offered its Cisco EEM (Embedded
Event Manager) tool embedded in the Cisco IOS operating system. With this tool it is possible to
automate actions based on events that occur in the devices. Also trigger scheduled actions.
The EEM (Embedded Event Manager) is a software component of Cisco IOS, XR, and NX-OS
makes life easier for administrators by tracking and classifying events that take place on a router
and providing notification options for those events. EEM allows you to automate tasks, perform
minor enhancements and create workarounds.
EEM uses event detectors and actions to provide notifications of those events. EEM detectors
can be:
Example: trigger down of a interface and take an action (no shut & send email)
Configuration:
event manager applet interface_Shutdown
event syslog pattern "Interface FastEthernet1/0, changed state to
administratively down"
action 1.0 cli command "enable"
action 1.5 cli command "config t"
action 2.0 cli command "interface fa1/0"
action 2.5 cli command "no shutdown"
action 3.0 cli command "end"
action 3.5 cli command "who"
action 4.0 mail server "192.168.1.1" to ".engineer@cisco.com." from
".EEM@cisco.com." subject ".ISP1_Interface_fa1/0_SHUT." body "Current
users $_cli_result"
TCL (Tool Control Language) is a scripting language used extensively by Cisco to facilitate the
testing and automating of various functions in the IOS.
In this example, a small TCL script is configured to check reachability of Branch devices from the
Core router.
R1#tclsh
R1(tcl)#
R1(tcl)#foreach VAR {
+>(tcl)#192.168.1.1
+>(tcl)#192.168.2.1
+>(tcl)#192.168.3.1
+>(tcl)#192.168.4.1
+>(tcl)#} { puts [exec "ping $VAR"] }
Python is the modern de facto choice. Let’s analyze its most used libraries for network
automation.
Paramiko is a Python library that takes care of low-level SSH protocol operation (e.g. negotiating
ciphers). Python programmers use Paramiko to implement SSH client and server functionality in
their programs, for connecting to remote computers.
Direct use of Paramiko itself is only intended for users who need advanced/low-level primitives.
We’ll see an example of the use of Paramiko in the next chapter.
You can install it on your control machine through pip, the package installer for Python:
Paramiko is particularly used from many of the upper layers in our framework: all the ones that
relies on SSH as transport protocol have as dependency the use of Paramiko, to set that
channel to send data.
Netmiko is a Python library that builds on Paramiko, and is specifically designed to provide SSH
connectivity to network equipment from different manufacturers.
Paramiko is more of a generic SSH module that you can use to automate specific SSH tasks. In
contrast, Netmiko is broader and well optimized for managing network devices such as switches
and routers. It supports a wide range of devices, and the Netmiko script is much shorter than the
Paramiko script.
Ncclient is a Python library for NETCONF clients. It maps the XML of NETCONF
to a Python API, and makes writing network-management scripts easier.
We’re going to focus on the NETCONF protocol in a few sections, so just keep
in mind this library operates to facilitate client-side scripting and application development
around it.
As you can see, the library relies on the usage of the Paramiko library to connect to devices
through SSH.
Requests is an elegant and simple HTTP library for Python, built for
human beings.
The protocols layer dictates a programmatic interface for accessing or building the abstraction
of a network. These protocols provide a complete framework for automating the provisioning,
operation and maintenance of networks.
They are built on the functions provided by the transport protocols, encoding and modeling
layers.
We have seen how, in the field of networking, the use of models (YANG) can help to describe
the elements of the networks, devices and relationships or services in which they are involved,
but how do we operationally then use these models on the equipment?
One of the methods is to use NETCONF, which is a network management protocol that allows
you to configure network elements.
NETCONF is a protocol standardized by the IETF as RFC 6241. It specifies a protocol that runs
between the control machine and the network device, and specifies methods for sending
configurations to the device, reading status from the device, and handling notifications from the
device. The desired configuration is NOT sent as configuration CLI lines; instead it is structured
in some specific format which we will see shortly. Similarly, status read back from the network
device is also structured in a particular way, not as the output of a CLI “show” command.
The NETCONF protocol only specifies and defines the operations that the control machine can
carry out (like get, get-config, edit-config, lock, unlock etc.), and the mechanism by
which these operation requests are delivered to the network device. The NETCONF RFC does
not define the contents of these requests themselves. That comes under the purview of YANG.
Let’s see an example of NETCONF communication between a network device (server) and a
client machine. In the following graph you can check the phases that elapse between the server
(router) and the client (control machine) when using NETCONF.
On top of the transport layer is the Message layer, which encompassed the RPC paradigm,
through the rpc and rpc-reply messages.
You might often read articles that define this protocol “polite”: this is because the server
requires an exchange of Hello messages before starting using rpc messages, in order to
provide its capabilities to the client. For capabilities we mean that the router informs the client
machine about the data models it knows and supports.
The operation layer describes all the actions that the client can execute remotely on the device
via an rpc message. For example, it may retrieve or edit the running configuration.
The important thing to understand is that a NETCONF message is a well-formed XML document,
stored inside the SSH payload.
Here’s an example of XML file containing an rpc message to retrieve informations about a
GigabitEthernet interface:
<?xml version="1.0"?>
<nc:rpc message-id="101" xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">
<nc:get>
<nc:filter type="subtree">
<native xmlns="http://cisco.com/ns/yang/ned/ios">
IeXa Academy wants to spread the networker method based on the analysis of the packets
captured through a sniffer. Since NETCONF uses SSH, the actual payload is encrypted, so we
are not able to see the XML content. But we can check some phases to help you get a better
understanding of the concept.
In the example we made, we are going to use NETCONF through the Python library ncclient. As
we said before, it allows to have a higher level of abstraction providing the facility of emulating a
NETCONF client.
Here’s the code we use to retrieve and print the capabilities of the network device:
from ncclient import manager
m = manager.connect(host='192.168.183.111', port=830,
username='enginesan',
password='cisco', device_params={'name': 'csr'})
print(m.server_capabilities)
When we run this script, here’s what we get on our control machine terminal:
The concept of network management that we have seen previously with NETCONF also applies
to RESTCONF, with some differences that we will see, especially in terms of the protocols used
and the exposure of services by the server (device).
Since it uses the HTTP protocol and its operations, RESTCONF handles API calls through URIs.
Here's how they are built:
Below you can find an example of packets exchanged between a network device (in this case a
Cisco router CSR1000V) and a client machine.
We know that RESTCONF uses HTTP, which uses TCP Layer 4 protocol. This means that there is
a handshake between the two sides to establish the connection.
But what we are focusing on is the HTTP GET Request sent by the machine:
● Message → it is a request from the client machine to the router, which acts as a server in
the Model-Driven Automation
● Operation → HTTP GET
● Content → URI related to the YANG model “Cisco-IOS-XE-interfaces.yang”
Going on, we are able to check the reply from the router, in a HTTP Response:
Again, n reference to our previous RESTCONF packet structure, we can define what is inside the
HTTP payload:
There are various data formats. For example, if you want to query the configuration of a Cisco
router, you will need to know in which format to send the request to the "interfaces" made
available by the router, in order to be able to package a "request" that the router can interpret
and consequently invoke the right commands to be able to complete actions and / or collect
data to send the applicant a correct "answer".
Among the various formats that can allow serialization/deserialization of data, the most used
formats are:
XML was born before, it is long (each item is surrounded by tags), complex to read and manage
compared to the others, but it is great for having highly structured data exchanges between
applications (machine-to-machine communications).
JSON is widely used for data transmission between applications/systems (especially web), due
to the fact that it is lighter and more readable. Its syntax is very similar to javascript, so it is very
simple JSON data in JavaScript objects.
JSON syntax includes the use of braces, square brackets, and quotation marks for its
representation.
YAML is very simple and often used by those who start programming. It is not a markup
language such as xml and json. It is minimalist and is in general the most human readable of
formats and is becoming increasingly successful among programmability developers (e.g.
network programmability).
The need to structure (model) data is not only present when it comes to databases, but in all
cases where data is used to perform some functions / operations and this obviously also
happens in the field of networking. Devices historicize data in the form of configurations,
parameters, events and network controllers manage data relating to the interactions between
the devices that make up the network and the services that use it.
5.5.1 YANG
YANG is the modelling language for the NETCONF protocol (NETwork CONFiguration) and
RESTCONF (Representational State Transfer Configuration Protocol), which we will see in a few
paragraphs, and uses a hierarchical scheme to describe the data, describing its syntax and
semantics.
A YANG model describes a particular component of configuration data, like a VLAN or an OSPF
router instance. Facts like “a permissible VLAN Identifier is an integer from 1 to 4094” and “an
OSPF instance contains a list of networks” are encoded in formal language inside a file with a
.yang extension. This file is then referred to as a “YANG model for VLANs” or “YANG model for
OSPF”.
YANG model files are created and published by the IETF, and also by individual equipment
manufacturers. As an example, look at the Cisco IOS XE YANG model for OSPF. Running the
YANG file through the pyang utility provides a tree view of the model. The figure below is a
partial, truncated view of this tree:
You can see a small part of the Cisco IOS XE YANG model for OSPF from that tree - an OSPF
router instance has an ID, may have an associated VRF explicitly configured, has an address
family, may have one or more neighbors explicitly configured, and may have one or more
networks configured. Each network inside an OSPF router instance contains an IP address,
mask and may have an area explicitly configured. Of course, this is only a tiny portion of the
OSPF model.
That’s about the model. How this model is used to actually send configuration to a device is the
subject of the following sections.
The network CLI isn’t going away anytime soon, nor are people going to start manipulating XML
directly for their network configuration data.
With the proper automation tool, we will be able to do thousands of tasks per day, something
that probably took months before. Automation will give us more time to think about network
architecture, design, and plan for its capacity.
Using the right automation tool can save companies lots of money in the process.
All the layers we’ve just seen are used by many frameworks and management systems. It would
be impossible to analyze them all, so let’s see some examples together.
NAPALM (Network Automation and Programmability Abstraction Layer with Multivendor support)
is a Python library that implements a set of functions to interact with different network device
Operating Systems using a unified API.
In other words, NAPALM abstracts the lower level semantics of device interaction, such as which
backend device library should be used (pyeapi, pynxos, netmiko etc), and which calls to make.
Napalm then presents a common set of methods that are agnostic of the platform type.
You can install it on your control machine through pip, the package installer for Python:
What NAPALM adds on top is additional features like configuration merge, replace, commit,
discard, and rollback which are not available in the other libraries we previously saw. Also
NAPALM uses declarative automation style. So users have to specify only the config not caring
about the exact commands sent to the device. NAPALM takes care of the “how” part or exact
commands to be sent to the device.
Being a Python library, NAPALM can be used directly in Python scripts or integrated with Ansible
(napalm-ansible module), Nornir (nornir_napalm plugin). This also means that it does not have to
use all the layers in our framework, but it could just use Python libraries we previously analize to
operate with network devices.
driver = get_network_driver('ios')
device = driver('192.168.183.111', 'enginesan', 'cisco')
device.open()
config = device.get_config()
print(config['running'])
This allows you to collect all the relevant informations you need to give to NAPALM to make it
work with various devices in a whole structured file.
The point of Nornir is to write code once that gets applied to all of your devices simultaneously.
But it is just a framework (like Ansible core) and someone still needs to do the heavy lifting like
connecting to network devices and executing commands on them.
First you define your inventory, host variables, group variables, and a set of default variables.
This is important because your code never has to worry about inventory. To begin with the
inventory consists of hosts, groups and defaults.
#nornir_config.yml
---
core:
num_workers: 20
inventory:
plugin: nornir.plugins.inventory.simple.SimpleInventory
options:
host_file: "inventory/hosts_netauto101.yml"
group_file: "inventory/groups.yml"
#groups.yml
1 ---
2 netauto101:
3 data:
4 domain: enginesan.local
5 asn: 1
#default.yml
1 ---
2 data:
3 domain: enginesan.local
Second, you choose the transport mechanism. And we mean that literally. You’ll choose which
library you want to carry out your task against all of your inventory. So maybe you want to use
Netmiko, or NAPALM, etc. For the record, these are called Connection Plugins.
NORNIR_CONFIG_FILE = "nornir_config.yml"
nr = InitNornir(config_file=NORNIR_CONFIG_FILE)
routers = nr.filter(role="router")
result = routers.run(task=networking.napalm.get,
getters=['config'])
print_result(result)
There are many other IT automation tools available, including more mature ones like Puppet
and Chef, so why would you choose Ansible? The main reason is simplicity.
Michael DeHaan, the creator of Ansible, already had a lot of experience with other configuration
management tools when he decided to develop a new one. He said that he wanted “a tool that
you could not use for six months, come back to, and still remember.”
DeHaan accomplished this by using YAML, a simple configuration language. Puppet and Chef,
on the other hand, use Ruby, which is more difficult to learn. This makes Ansible especially
appealing to system administrators. DeHaan also simplified Ansible deployment by making it
agentless. That is, instead of having to install an agent on every system you want to manage (as
you have to do with Puppet and Chef), Ansible just requires that systems have Python (on Linux
servers) or PowerShell (on Windows servers) and SSH.
Playbooks are written in YAML. A simple playbook could contain a single play, and a
complicated one to achieve a complex workflow could contain several plays. In essence, a
playbook is a list of instructions that you execute on remote systems (managed nodes).
Playbooks use modules, which are programs written in any language (but typically Python) that
implement the logic needed to achieve a particular task.
[ios]
csr1000v-1.enginesan.local
By default, Ansible could operate with just the usage of YAML playbooks and relies on
Paramiko’s SSH connection to send and retrieve data.
If you are curious about these concepts and want to learn more, check IeXa Academy’s Next
Generation Network Engineer course!
As a networker, you’re certainly familiar with the concept of transporting data across a network.
Secure Shell (SSH) and HyperText Transfer Protocol (HTTP) are the most common transports
for getting data to and from network devices, even within the network automation and
programmability world.
NB: Do not confuse the transport layer of the ISO-OSI networking model
with the transport layer of the network automation and programmability
stack.The two models do not contradict one another, the difference is in
the context in which they operate.
These protocols are the one that create the channel to connect to devices, where all the
network automation tools we saw can send their data.
SSH is the most common way of connecting devices securely and it provides command-line
access to devices by spawning a remote shell. SSH meets the three main requirements of
remote access:
Hypertext Transfer Protocol (HTTP) is a method for encoding and transporting information
between a client (such as a web browser) and a web server. HTTP is the primary protocol for
transmission of information across the Internet.
HTTP follows a request‑response paradigm in which the client makes a request and the server
issues a response that includes not only the requested content, but also relevant status
information about the request.
HTTP resources such as web servers are identified across the Internet using unique identifiers
known as Uniform Resource Locators (URLs).
HTTP defines methods to indicate the desired action to be performed on the identified
resource:
● The GET method indicates that you’re trying to get or retrieve data
● The POST method indicates that you’re trying to send data to the server
● The PUT method indicates that the target resource is updating some data
● The DELETE method indicates that you are deleting some data on the server
The most used version of HTTP is HTTPS (HTTP Secure), which is more secure because it
provides the encryption of the HTTP message. Technically speaking, HTTPS is not a separate
protocol from HTTP. It is simply using TLS encryption over the HTTP protocol.
This means that with HTTPS we can create an encrypted channel between our devices.
The network configuration we use is NAT, where the VM network adapter can access external
devices, but will receive an IP of the same subnet as the VMnet8 interface (192.168.183.0/24).
https://rednectar.net/2011/07/20/vmware-interfaces-tutorial/
To give you a better understanding of the tools we explained before, we are going to automate
a task in our topology using different methods:
Since we discussed the evolution of network automation, we are starting from Bash scripting,
then we will repeat the same task with newer automation products.
Furthermore, we are going to explain the encapsulation of the packets exchanged, in order to
understand the network automation model we described at the beginning.
Don’t worry if you’re not able to fully understand this whole document, you
could join us in the Next Generation Network Engineer path! All the scripts
present in this document will be discussed by the trainer during the lectures,
with more examples and exercises. Our students will face more complex
scenarios and projects.
You will also find many examples and articles about using regular expressions into scripts.
Those are special sequences of characters, commonly referred to as patterns. Patterns are used
to search through text to find matches.
HOST='192.168.183.111'
USER='enginesan'
PASSWD='cisco'
CMD='show interface g1'
(
echo open "$HOST"
sleep 2
echo "$USER"
sleep 2
echo "$PASSWD"
sleep 2
echo "$CMD"
sleep 2
echo "exit"
) | telnet > check_interface_status_bash.txt
sleep 10
echo exit
In this first example, we used Telnet to connect to the router, so we are able to check the
payload of the packets:
csr1000v-1#show interface g1
GigabitEthernet1 is up, line protocol is up
Hardware is CSR vNIC, address is 000c.2922.b24a (bia 000c.2922.b24a)
Description: LINK TO VMNET8
Internet address is 192.168.183.111/24
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
--- snip ---
To run this script, we need to make it executable and then we can launch it. We use the
following commands:
enginesan@ubuntu204:~/nap2/netauto101$ chmod + interface_config_expect.exp
enginesan@ubuntu204:~/nap2/netauto101$ ./interface_config_expect.exp
Again, we used Telnet to connect to the router, so we can see what is actually traveling in the
packet:
Username: enginesan
Password:
SecureCRT client for Windows, Mac, and Linux provides rock-solid terminal emulation for
computing professionals, raising productivity with advanced session management and a host of
ways to save time and streamline repetitive tasks. SecureCRT provides secure remote access,
file transfer, and data tunneling for everyone in your organization.
crt.Screen.Synchronous = True
' VBS sample script: get interface config from running configuration
Sub Main
' Connect using a session (by session name) - need to pre configure
session data (ex. ssh login password)
crt.session.Connect("/s csr1000v1")
crt.Screen.WaitForString "#"
' Choose the directory to save the logged file in and then
' Set the name of the log file name to "SessionName-MMDDYY-HHMM.txt"
' In this script the directory specified is "C:\vandyke"
'
Dim logfile
logfile = "securecrt_scripts\%S-%M%D%Y-%h%m%s.txt"
crt.Session.LogFileName = logfile
End Sub
To launch this script, we ran SecureCRT from our Windows host. We created a session to
connect to our csr1000v through SSH, with the following information:
● hostname
● IP address
● username
But before running it, we launch Wireshark on our Winfìdows host, being careful to select the
correct interface: since the csr1000v is virtualized in VMware Workstation, it is connected to the
VMnet8 network. The host machine has a virtualized NIC inside the same network, so that it can
communicate with the router:
Once we run the script, the prompt on SecureCRT will ask us the password to remotely access
to the router with our user:
The highlighted lines represent the Paramiko commands used in this script.
The management of the output that we receive from the router is not so obvious when using
Paramiko. We had to set three variables to capture it, in order to parse it later (lines 9-15):
● stdin which represents the stream from which a program reads its input data
● stdout which represents the stream to which a program writes its output data
● stderr, which is another output stream typically used by programs to output error
messages
Focus now on how we parse the output of the command. Since we were looking for a specific
string, we used a regular expression through the Python library re, defining first the pattern
(^GigabitEthernet1 is up.*) and then applying it to the output (lines 19-28).
Again, Paramiko allows us to connect to the device through the SSH protocol, so in our usual
Wireshark capture, we are not able to see the actual command and the relative output, but we
just find encrypted data. However, you might recognize the SSH phases:
The highlighted lines represent the Netmiko commands used in the script, the others belong to
built-in Python functions and the re library.
With Netmiko, the management of the device informations and the output of the command is
much easier for the network administrator, and this means that Netmiko scripts are shorter than
Paramiko ones when dealing with more complex instructions.
In the above picture, you can see the main differences between Paramiko and Netmiko:
● you have to import the relative library (remember that Netmiko uses automatically
Paramiko to set the SSH connection, even if you do not explicit it)
● with Netmiko you create a Python dictionary to save the device informarmations, while in
Paramiko you insert them in the function “connect()” parameters
● with Paramiko you have to manage different variables to save the output, while with
Netmiko you might just use a variable with the only command you need to use
NAPALM returns it in Python dictionary format, so parsing it is easier. Here in the example, we
used the json library to save the output in an external file, but the parsing is based on the
dictionary’s [key][value] structure.
At the beginning we import the libraries we need (lines 1-2). Then we use Napalm functions to
get the IOS driver and apply them to our router, passing its credentials for remote access as
parameters (lines 4-7). Next, we define the name of the interface we want to check as a variable
and pass it to another Napalm function to get its configuration (lines 8-9). We save the output
related to a local file (lines 10-12).
A big change here is how data is received from the device: it is not the mere CLI output we are
used to see and to parse with the previous libraries, since now it is saved in a structured Python
dictionary. Here’s the file containing the output:
enginesan@ubuntu204:~/nap2/netauto101$ cat
check_interface_status_napalm.txt
{"GigabitEthernet1": {"is_enabled": true, "is_up": true, "description":
"LINK TO VMNET8", "mac_address": "00:0C:29:22:B2:4A", "last_flapped":
-1.0, "mtu": 1500, "speed": 1000}}
enginesan@ubuntu204:~/nap2/netauto101$
This means that we do not need to use regular expression to parse it, but we just need to select
the righrìt the key of the dictionary to get its value (lines 14-17).
This time Napalm uses Netmiko, which uses Paramiko to connect in SSH to the router. You might
check below the exchanged packets.
In fact, the examples we’ve just seen are all automation scripts based on Screen Scraping,
which try to emulate humans while they are interacting with a device.
We used the online YANG catalog search engine to find the model “ietf-interfaces”
(https://yangcatalog.org/yang-search/module_details/ietf-interfaces@2018-02-20).
We are not going to use the whole tree of the YANG module, so we need to select just a portion
of it. Filtering is done by matching a specified attribute value of the tree.
Since we need to obtain the filter to make the request, we downloaded the YANG model and
used the pyang tool:
pyang -f sample-xml-skeleton ietf-interfaces.yang
The highlighted lines represent the Ncclient functions used. As you can see, we had to create
the filter (saved as “netconf_data” variable) which was used later in the “netconf_reply”
instruction.
There are many online tools that allow us to view the xml text in a more structured way. For
example, pasting this text on the website https://codebeautify.org/xmlviewer, we get:
● it is a rpc-reply message, since in the script we used a filter to ask details about the
interface GigabitEthernet1 (line 4)
● it contains the requested informations in a xml structure that follows the YANG model
“ietf-interfaces” (8-9)
The tag that contains the interface status is at line 18 (<enabled>). To get this value, we used the
indices of the related Python list as coordinates (in the script, line 24). Each pair of square
brackets represents a hierarchy level of the content included between the <data> tags (sìif you
check the above image, it is the block of lines 8-30). And each integer number inside the
brackets represents the tag position in the same hierarchy level.
So the first pair [0] represents the 0th tag in the first hierarchy level, which corresponds to the
<interfaces> tag. The second [0] represents the 0th tag in the second hierarchy level, which
corresponds to the <interfaces> tag. The third index [3] represents the 3rd tag in the third
hierarchy level, which corresponds to the <enabled> tag.
Also, we said that RESTCONF might use different encoding protocols, so this time we’ll ask to
get the informations in JSON.
The highlighted lines represent the Requests instructions used. As you can see, it is a GET
message, of which we have to specify the headers → url, authentication, accepted content
format (lines 4-11). The following lines are used to select the correct pair of key:value of the
JSON response we get, in order to understand the status of the interface.
And finally we are able to see the content of the packets through Wireshark, since we can save
and upload the encryption keys.
With this last script, we can end the rundown of examples on the usage of the different tools.
Following the “evolution” path, we passed from scripting languages that allow a kind of
automation that is very dependent on the device you’re trying to connect to and its OS, since
you need to know which are the commands you should give as input and what is the output, but
then we used newer libraries and frameworks which abstract the underlying hardware and focus
on the modeled data.
If you want to see other aspects of network automation, join the IeXa Academy community and
check our services and courses!
In IeXa Academy we believe in the power of method and mindset. The best approach, in our
opinion, is to take a step back and a paper sheet with a pen, no digital distractions.
You have to prepare a strategy, or a plan to action, in order to have the whole overview and
control on every bit of the project.
There are many considerations you might follow while choosing the solution toolset:
● First, you have to gather the requirements of the customer. Are you a freelancer to
automate some operating processes? Or you could be the boss yourself, but you still
need to define which are the needs for your network.consultant? Or maybe you are a
network engineer inside a company, whose boss asked
● The solution should be balanced in terms of available features and the ease of their
usage and implementation. Some ready-to-use open-source free tools might require
skills that you or your company do not have at the moment.
● “How about open source software?”, people might ask. We agree, with tools like Python
and Ansible you can implement some things really fast. The main challenge however is
that, as Ansible's key focus historically was on server automation, you need serious
programming skills to make it work for network automation and time to build what you
want.
● Solution reusability. The tasks are potentially applicable to many production networks.
One should keep this in mind.
● Support for multiple network platforms. Once you get into multi vendor and hybrid
networks, you have a problem: you first want to unify and centralize your configuration
changes. Second, you need something to automate jobs across your vendors and
thirdly, at some point you may need a ‘source-of-truth’.
● Real-world networks consist of many platforms. This is a thing to note as well.
● The documentation must be available for the chosen toolset.
Once you have all the details, you need to unwrap all the main requests in subtasks and
activities. This is where you need the paper sheet: you can use it to create kanban, wbs, graphs
or whatever representation you like, to identify your next steps. Take the framework, check the
functionalities you have and start defining the things you need.
This document includes the secret you need to decrypt the complexity created by the huge
number of tools we can find in the network automation field. It could be also secret to kill a
network engineer interview, since not so many professionals have the whole vision of how
things are working.
IeXa Academy is launching a brand new path, expecially focused on this part, that contains
many other details, examples, labs. Come check our website!
Of course there will also be other projects and we’ll go in more depth in the analysis of the
tools.
Don’t worry if you’re not able to fully understand this whole document, you
could join us in the Next Generation Network Engineer path! All the scripts
present in this document will be discussed by the trainer during the lectures,
with more examples and exercises. Our students will face more complex
scenarios and projects.
Blogs
● https://vnetman.github.io/network-automation/netconf/yang/ansible/chef/puppet/netmiko
/python/2019/10/18/network-automation-a-1000ft-view.html
● https://habr.com/en/post/534716/
● https://rednectar.net/2011/07/20/vmware-interfaces-tutorial/
● https://journey2theccie.wordpress.com/2020/04/07/automating-my-ccie-lab-pt-4-netmik
o-napalm-nornir/
● https://blog.ipspace.net/2019/09/paramiko-netmiko-napalm-or-nornir.html