Rasa Certification Workshop

Mady Mantha, Juste Petraityte, Karen White

RASA CERTIFICATION WORKSHOP

Agenda
Part 1: Welcome and deep dive into Rasa

● Welcome! 👋
● Intro to Rasa
● Conversation Design
● Deep dive into NLU
● Deep dive into dialogue policies
● CDD and testing
● Demo

Part 2: Adding custom actions, forms and improving your assistant using Rasa X

● Reviewing the previous day 📚

● Events and Slots
● Custom actions and Forms
● Improving the assistant using Rasa X
● Connecting the assistant to the external messaging channels
● What’s next
 RASA CERTIFICATION WORKSHOP

The Rasa Team

Mady Mantha Juste Petraityte Karen White

Sr. Technical Evangelist Head of Developer Relations Developer Marketing
Manager
 RASA CERTIFICATION WORKSHOP

How to get help

● Please ask your questions in the #workshop-help Slack channel

● A note on time zones:
○ The Rasa team is based across the US and in Berlin. We’ll do our best to answer questions within
team members’ working hours, but please keep in mind, some discussions may need to take place
async rather than in real time.
Intro to Rasa
 RASA CERTIFICATION WORKSHOP

Part 1 Roadmap

● First half: talk

● Second half: we’ll code together

● Recap: talk, with some time for questions

*More info blablabla

 5 LEVELS OF CONVERSATIONAL AI MATURITY MODEL

Contextual assistants are an important step on the journey to autonomous

organizations

Level 1:
Last 10 years
Command-based

Most common Level 2:

today “Chatbots” - FAQs

A few
examples
Level 3:
today Contextual Assistants

Level 4:
In 2-5 years
Consultative Assistants

Level 5:
In 5-10 years
Adaptive Assistants

# Automation Use Cases

RASA OPEN SOURCE

Rasa Open Source

Natural
Language
Understanding
“I want to change
my address”
Connector
ML-based
Modules
Dialogue
E.g. Live Chat, Management
“What’s your new App, Messenger Backend,
address?” database,
API, etc.
Response
Selection
Conversational AI is not easy
Real conversations don’t follow the happy path
 THE APPROACH

Build a minimum viable assistant with Rasa Open Source + improve it

using Rasa X inimu
m
prove
by
ve us
ing
ve us
ing
m o o
Build ssistant Im he
g to t t Impr sations Impr sations
ea i n er r
conve l users
viabl ta l k
tan conv t users
assis tes rea
with with

Quality of assistant

Rasa Open Source is an open

source framework for natural
language understanding, dialogue Rasa Open Source (Local)
Rasa X is a toolset used
management, and integrations.
to improve a contextual
Rasa X (Server)
assistant built using
Rasa Open Source.
NLU & Dialogue Management
 NLU Deep Dive

Natural Language Understanding (NLU)

Natural
Language
Understanding
How much did I
spend at Starbucks
last week?
ML-based
Connector Dialogue
Modules Management
Backend,
~~~ database,
API, etc.
Response
Selection
 NLU Deep Dive

Natural Language Understanding (NLU)

Goal: Extract structured information from messages

{...
"intent": {
"name": "search_transactions ",
"confidence": 0.96
},
How much did I spend "entities": [
at Starbucks last {
week? "entity": "vendor_name ",
"value": "Starbucks",
...
}
]
}
 NLU Deep Dive

Intents and Entities

Two of the most common and necessary types of information to

extract from a message

Intent: What the user

wants to achieve

How much did I spend at Starbucks last

week?
Entity: Key piece of
information
NLU DEEP DIVE

NLU Pipeline: Input and Output

Many components take the output of earlier components as input

Raw Text

Text Text
Tokenizer Tokens
Featurizer Representation

Entity Intent
Extractor Classifier

Entities Intent
 NLU DEEP DIVE

NLU Pipeline: Example

“How much did I spend
at Starbucks last
week?”
[how, much,
[(0, 421) 1
did, i,
(1, 48) 2
spend, Text
Tokenizer : :
at, Featurizer
(2, 90) 1
starbucks,
(2, 150) 1]
last, week]

Entity Intent
Extractor Classifier

vendor_name: starbucks search_transa

ctions
NLU DEEP DIVE

“Text Representation”
How do words turn into numbers?

[(0, 421) 1
(1, 48) 2
Text
Tokens Text Representation : :
Featurizer
(2, 90) 1
(2, 150) 1]

Entity Extractor Intent Classifier

Entities Intent
 NLU DEEP DIVE

Intent Classification & Entity Extraction

did I
bucks
?”
[(0, 421) 1
[how,much,
(1, 48) 2
did,i,spend, Text
zer Text Representation : :
at,starbucks Featurizer
(2, 90) 1
,last,week]
(2, 150) 1]

Entity Intent
Extractor Classifier

search_transa
vendor_name: starbucks
ctions
 NLU DEEP DIVE

Text Representation: Word Vectors

“Judge a Word by the Company it Keeps” — John R. Firth

“... sleep on a bed...”

“...sleep in a bed?”

sleep?
bed mattress
“...sleep on a mattress...”

“...mattress to sleep on…”

These would be two of many dimensions!

 NLU DEEP DIVE

Text Representation: How models use word vectors

Non-sequence model:

○ One feature vector per input (whole message)

○ Order of words not captured

Sequence model:

○ One feature vector per token (word) & feature vector for whole message
○ Word order captured
Configuration Files
 Livecoding

Let’s start coding!

1. Let’s use a sandbox branch instead of the master branch you cloned yesterday. Make sure you are inside
the financial-demo folder:
cd financial-demo

2. Fetch the remote branches:

git fetch origin

3. Checkout the sandbox branch named “workshop_day_2_sandbox”:

git checkout -b workshop_day_2_sandbox origin/workshop_day_2_sandbox
 Core Deep Dive

Domain

Defines the “world” of your assistant - what it knows, can understand, and can do

intents: What your bot can understand

…
entities: What your bot can remember
…
slots:
…
actions: What your bot can do (and say)
…
responses: What your bot can say (and
… display)
session_config:
…
What your bot considers a new
conversation
 LIVE CODING: MINIMUM VIABLE ASSISTANT

Project setup: Files

__init__.py an empty file that helps python find your actions

actions.py code for your custom actions

config.yml configuration of your NLU and dialogue models

credentials.yml details for connecting to other services

data/nlu.md your NLU training data

data/stories.md your stories

domain.yml your assistant’s domain

endpoints.yml details for connecting to channels like fb

messenger

models/<timestamp>.tar.gz your initial model

 NLU Deep Dive

NLU Pipeline

Defines how a user message is processed & what information is extracted

language: "en"

pipeline:
- name: "Whitespacetokenizer"
- name: "ConveRTFeaturizer"
- name: "RegexFeaturizer" Components
- name: "LexicalSyntacticFeaturizer"
- name: "CountVectorsFeaturizer"
 NLU Deep Dive

NLU Pipeline: Components

Built-in component types:

- Tokenizers
- Word Vector Sources
- Text Featurizers
- Entity Extractors
- Intent Classifiers
- Response Selectors

You can also add custom components for e.g.

- Sentiment analysis
- Spell checking
NLU DEEP DIVE

DIETClassifier: Intent Classification & Entity Extraction

The Short Story

Intent search_transactions
DIETClassifier
Entities vendor name: starbucks

Dense
Features Sparse
(Pretrained Features
Embeddings)

“How much did I spend at Starbucks last

week?”
NLU DEEP DIVE

DIET: Intent Classification & Entity Extraction entities

intent
 NLU DEEP DIVE

Feature: Customize What Features Go In!

sparse features
in pipeline

BERT/GloVe/pretrained
and continuous
features
 NLU DEEP DIVE

Entity Extraction: Other options

Possible Approaches:

1. Rule-based entity extraction

2. Direct structured prediction of entity based on sentence context

DIETClassifier Trained on your

MitieEntityExtractor
Pretrained data
SpacyEntityExtractor
DucklingHTTPExtractor
Predefined, Modifies extracted
EntitySynonymMapper
rule-based entities
Livecoding

Let’s create some intents and entities!

 Core Deep Dive: Policies

Policies
● Decide which action to take at every step in the conversation
● Each policy predicts an action with some probability. This is called core confidence.

Interpreter
“I want to know
1
how much I spent
2
at Starbucks last Message In
week”
3
Tracker Policy
Message Out
6
“You spent $12 at 5
Starbucks”
Action 4
 Core Deep Dive: Policies

Rule Based Policies

● MemoizationPolicy: Memorizes your stories

○ Makes predictions based on your max_history value
● AugmentedMemoizationPolicy: Memorizes your stories with a twist
○ As above, but if no match is found, it will forget certain events until a match is found
● FormPolicy: Fills required slots
○ Asks for information until all required slots are filled
● MappingPolicy: Intents trigger actions
○ Will execute the specified action regardless of context unless superseded by another policy
● FallbackPolicy: Failing gracefully
○ If model confidence is below a certain value, it will send the user a “fallback message”, e.g. “Sorry, I
didn’t understand you”
● TwoStageFallbackPolicy: Failing (more) gracefully
○ Attempts to disambiguate the user’s intent before sending a fallback message
 Core Deep Dive: Policies

Machine Learning Based Policies

These policies should be used in conjunction with rule-based policies

● KerasPolicy: Uses a standard LSTM to predict the next action

○ Learns the patterns of your stories
○ Good for handling stories that don’t exactly match your training data

● TED Policy: Uses Attention to Handle Uncooperative Dialogue

○ Requires fewer story examples of uncooperative user dialogue
■ e.g. users who go off on tangents instead of providing the requested information
○ Effectively “ignores” irrelevant parts of the dialogue
 Core Deep Dive: Policies

Multiple Policies

● The policy with the highest confidence wins.

● If the confidence of two policies is equal, the policy with the highest priority wins.

Policy priorities
(higher numbers = higher priority)
Rule-based
5. FormPolicy
policies have
4. FallbackPolicy, TwoStageFallbackPolicy
higher priority 3. MemoizationPolicy, AugmentedMemoizationPolicy
than ML-based 2. MappingPolicy
policies 1. EmbeddingPolicy, KerasPolicy, TED
Conversation Design
Core Deep Dive

Initially: Conversations as simple question / answer pairs

What can I help you with

today?

I just moved

What is your new address?

It’s 120 King St in New York

City.
Core Deep Dive

Real conversations are more complex: Rely on context

What can I help you with

today?

I just moved

What is your new address?

It’s 120 King St

Which city do you live in?

in New York City

 Conversation Design

Scope Conversation

How to get started with conversation design

The assistant’s purpose

Leverage the knowledge of domain experts

Common search queries

FAQs and wikis

 Conversation Design

Scope Conversation

Source: Botsociety
 Livecoding

Let’s create some stories together!

● Check account balance

● Search transactions
 Core Deep Dive

The importance of context

Shall we mail
Context: Mailing to an address letters to the same Previous Action
address too?

Which one do you

Dialogue Response We currently send
have on file at NLU
Management Generator letters to...
the moment?

Intent Action
clarification_question show_mailing_address
90.5% probability 92.5% probability
vector = (0, 11, 26, … , 1) vector =(2 , 212, 90, … , 5)
 Core Deep Dive

The importance of context

Shall I also update

Context: Update Billing Info your credit card Previous Action
details?

Which one do you

Dialogue Response We currently charge
have on file at NLU
Management Generator this credit card...
the moment?

Intent Action
clarification_question show_payment_info
90.5% probability 89.3% probability
vector = (0, 11, 26, … , 1) vector =(4 , 170, 90, … , 5)
 previous
Core Deep Dive

Dialogue Handling Action

How much did I spend

next
at Starbucks last week? State Action
SVM
Intent Recurrent NN
...
Entities

You spent $12 at

Starbucks last
week!

updated after next

Thanks.
State Action
 NLU Deep Dive

NLU: Output to Dialogue Management

entity: vendor_name
Natural Language value: Starbucks
Understanding …

name: search_transactions
confidence: 0.96

ML-based Dialogue
tor Modules Management
 Core Deep Dive

Dialogue Management (Core)

Natural
Language
Understanding
My new address is
222 Broadway NYC.
ML-based
Connector Dialogue
Modules Management
Backend,
~~~ database,
API, etc.
Response
Selection
Training Data
 Training Data for NLU and Dialogue Management

Machine learning models require training data that the models can
generalize from
NLU needs data in the form of examples for Dialogue management model needs
intents data in the form of stories
## intent:bot_challenge ## new to rasa at start
- are you a bot? * how_to_get_started{" user_type":
- are you a human? "new"}
- am I talking to a bot? - action_set_onboarding
- am I talking to a human? - slot{" onboarding": true}
- utter_getstarted_new
## intent:i_like_food - utter_built_bot_before
- I like [ apples](food) * deny
- my friend likes [ oranges](food) - utter_explain_rasa_components
- I'm a fan of [ pears](food) - utter_rasa_components_details
- do you like [ coffee](food) - utter_ask_explain_nlucorex
 Rasa Deep Dive

Rasa Open Source Reviewed

Natural
Language
Understanding
“How much did I
spend at Starbucks
last week?”
Connector
ML-based
Modules
Dialogue
E.g. Live Chat, Management
“You spent $12 at App, Messenger Backend,
Starbucks!” database,
API, etc.
Response
Selection
Deep dive into DIET, TED, and some testing!
 Core Deep Dive

Domain
Defines the “world” of your assistant - what it knows, can understand, and can do

intents: What your bot can understand

…
entities: What your bot can remember
…
slots:
…
actions: What your bot can do (and say)
…
responses: What your bot can say (and
… display)
session_config:
…
What your bot considers a new
conversation
 LIVE CODING: MINIMUM VIABLE ASSISTANT

Project setup: Files

__init__.py an empty file that helps python find your actions

actions.py code for your custom actions

config.yml configuration of your NLU and dialogue models

credentials.yml details for connecting to other services

data/nlu.md your NLU training data

data/stories.md your stories

domain.yml your assistant’s domain

endpoints.yml details for connecting to channels like fb

messenger

models/<timestamp>.tar.gz your initial model

Rasa Certification Workshop

In other words
 Glue is out, SuperGlue is in

@madymantha
NLU Deep Dive

DIET is our new neural network architecture for NLU

What is DIET?
● New state of the art neural network architecture for NLU
● Predicts intents and entities together
● Plug and play pretrained language models
 NLU DEEP DIVE

Feature: Customize what features go in!

sparse features
in pipeline

BERT/GloVe/pretrained
and continuous
@madymantha features
 How to use DIET in your Rasa project

Here’s an example config.yml

Before the DIET model, you can specify any

featurizer.
In our experiments, we use:
● Sparse features (aka no pre-trained model)
● GloVe (word vectors)
● BERT (large language model)
● ConveRT (pre-trained encoder for
conversations)

@madymantha
 Experiments on the NLU-benchmark dataset

● Repo is on github
● Domain: human-robot interaction (smart home setting)
● 64 different intents
● 54 different entity types
● ~26k labelled examples

Previous state of the art:

● HERMIT NLU (Vanzo, Bastianelli, and Lemon @ SIGdial 2019)
● uses ELMo embeddings

@madymantha
 Result 1: DIET outperforms SotA even without any pretrained embeddings

Previous state of the art: intent: 87.55 entities: 84.74

@madymantha
 Result 2: GloVe embeddings perform better than BERT

@madymantha
 Result 3: ConveRT embeddings perform best on the NLU-benchmark
dataset

@madymantha
 Result 4: DIET outperforms fine-tuning BERT

@madymantha
 Which featurizer is best depends on your dataset, so try different
ones!
We don’t believe in “one size fits all” machine
learning

● We aim to provide sensible defaults and

suggestions
● BUT even more important that these models
are easy to customize

Share your results and compare notes with 8000+

developers at forum.rasa.com

@madymantha
Transformer
Embedding Dialogue
policy (TED)
 NLU Deep Dive

Conversational AI requires NLU and Dialogue management

@madymantha
 Happy paths are already solved

@madymantha
 You can’t predict user behavior

@madymantha
 Users will interject and loop back to
earlier topics

@madymantha
 Real conversations don’t follow the
happy path

@madymantha
 People typically use a recurrent neural net (RNN) to model dialogue

W h1 W
y1
W
W W
h2 y2
W
W W y3
h3

@madymantha
 But not all input should be treated equally

https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html
@madymantha
 Transformers (AKA self-attention) are now state of the art for many
tasks

https://distill.pub/2016/augmented-rnns/
@madymantha
 We found out that the Transformer Embedding Dialogue policy can
untangle sub-dialogues

paper

@madymantha
Development Stages

Conversation Driven Development (CDD)

The challenge

When developing assistants, it’s impossible to anticipate all of the things your users might say.

The approach

A user-centric process: listening to your users and using those insights to improve your AI assistant.

Blog post: Conversation-Driven Development

 Conversation Driven Development

te
ew
re

k
ta

st

ac

x
a

vi

no

Fi
Te
Sh

Tr
Re

An
1 2 3 4 5 6

Rasa X
RASA TOOLS AND PRODUCTS

Successful production-ready bots follow engineering best practices

 TIME TO VALUE

From zero to production-ready deployment in under 5 minutes

@madymantha
 THE APPROACH

Continually improve your assistant using Rasa X

Ensure your new assistant passes tests

Collect conversations between
using continuous integration (CI) and
users and your assistant
redeploy it to users using continuous
deployment (CD)

Review conversations and

improve your assistant based
on what you learn
Testing
 Testing

Use the Rasa CLI to test your assistant

End to End NLU Core

Evaluation Evaluation Evaluation
Run through test conversations Split data into a test set or Evaluate your trained model on
to make sure that both NLU and estimate how well your model a set of test stories and
Core make correct predictions. generalizes using generate a confusion matrix.
cross-validation.

$ rasa test $ rasa test nlu -u $ rasa test core

data/nlu.md --config --stories
config.yml test_stories.md --out
--cross-validation results
 RASA DEEP DIVE

Interactive Learning: Talk to your bot yourself

● Correct your bot’s predictions as you go

● Save conversations as training stories or

E2E stories
 Testing

Run `rasa test` locally when building a minimum viable assistant

● Test your model after training to make development more productive and reliable
Adding custom actions and implementing forms
 RASA CERTIFICATION WORKSHOP

Agenda
Day 1: Welcome and deep dive into Rasa

● Welcome! 👋
● Intro to Rasa
● Conversation Design
● Deep dive into NLU
● Deep dive into dialogue policies
● CDD and testing
● Demo

Day 2: Adding custom actions, forms and improving your assistant using Rasa X

● Events and Slots

● Custom actions and Forms
● Improving the assistant using Rasa X
● Connecting the assistant to the external messaging channels
● What’s next
 RASA CERTIFICATION WORKSHOP

Day 2: Roadmap for the first half of the day

Part 1:

● Rasa events and slots: talk and coding together

● Custom actions: talk and coding together
● Short break 🧁

Part 2:

● Forms: talk and coding together

● Recap and Q&A
 RASA OPEN SOURCE

Rasa Open Source

Natural
Language
Understanding
“I want to change
my address”
Connector
ML-based
Modules
Dialogue
E.g. Live Chat, Management
“What’s your new App, Messenger Backend,
address?” database,
API, etc.
Response
Selection
 LIVE CODING: MINIMUM VIABLE ASSISTANT

Project setup: Files

__init__.py an empty file that helps python find your actions

actions.py code for your custom actions

config.yml configuration of your NLU and Core models

credentials.yml details for connecting to other services

data/nlu.md your NLU training data

data/stories.md your stories

domain.yml your assistant’s domain

endpoints.yml details for connecting to channels like fb

messenger

models/<timestamp>.tar.gz your initial model

Rasa events and slots
 Core Deep Dive

Events

● Internally, conversations are represented as a sequence of events.

● Some events are automatically tracked

Events Story

UserUttered ## new to rasa at start

SlotSet * how_to_get_started{" user_type":
SlotSet "new"}
- action_set_onboarding
BotUttered - slot{" onboarding": true}
UserUttered - utter_getstarted_new
Restarted * restart
- action_restart
 Core Deep Dive

Tracker

● Trackers maintain the state of a dialogue between the assistant and the user.
● It keeps track of events, slots, session info etc.
{
"sender_id": "default", Session info
"slots": {
"user_type": "new",
"onboarding": true}, Slots
"latest_message": {...},
"events": [
...
{"event": "action", Events:
"name": "action_set_onboarding",
"policy": "MemoizationPolicy",
...
"confidence": 0.92}, SlotSet
... ...
{"event": "user", UserUttered
"text": "\/restart",...},
...
...
]
}
 Core Deep Dive

Slots
Your bot’s memory

Hi, my name is Tom Hi

Hello! How can I help? Hello! How can I help?

I would like to check my I would like to check my

account balance account balance

Sure. Can you tell me your What is your name?

surname?
 Core Deep Dive

Slots

Your bot’s memory

● Can store:
○ user-provided info
○ info from the outside world
● Can be set by:
○ NLU (from extracted entities, or buttons)
○ Custom Actions
● Can be configured to affect or not affect the dialogue progression
 Core Deep Dive

Slots types
Text and List slots influence conversation path based on whether it is set or not.

“I am based in London” “Are there available appointments on

Monday or Wednesday this week?”
 Core Deep Dive

Slots types
Categorical, Boolean and Float slots influence conversation path based on the value
of the slot.

“I am looking for a restaurant in a “Find me a restaurant within 2 mile

low price range.” radius.”
 Core Deep Dive

Slots types
Unfeaturized slots don’t have any influence on the dialogue.

“My name is Sarah.”

 Core Deep Dive

Slots
There are a few different ways how slots can be set

● Slots set from NLU

## intent:search_transactions
- how much did I spend at [Target](vendor_name) this week?
- what is my typical spending at [Amazon](vendor_name)?
 Core Deep Dive

Slots
There are a few different ways slots can be set

● Slots set by clicking buttons

utter_ask_transfer_form_confirm:
Would you like to transfer $100 to Tom?
- text: “Would you like to transfer $100 to Tom?”
buttons: No, cancel the
Yes
- title: Yes transaction
payload: /affirm
- title: No, cancel the transaction
payload: /deny
 Core Deep Dive

Slots
There are a few different ways how slots can be set

● Slots set by custom actions

## account balance story

* check_account_balance
- action_account_balance
- slot {"account_balance": 1000}
- slot {"amount_transferred": None}
 Livecoding

Let’s define the slots for our Financial Bot assistant!

1. Let’s use a sandbox branch for today’s exercise. First, make sure your are in a financial bot directory:
cd financial-demo

2. Pull the latest updates pushed to the repo:

git pull

3. Checkout the sandbox branch named “workshop_day_2”:

git checkout workshop_day_2
Custom actions
 Core Deep Dive

Custom Actions: Connecting to the outside world

Natural
Language
Understanding
My new address is
222 Broadway NYC.
ML-based
Connector Dialogue
Modules Management
Backend,
~~~ database,
Natural API, etc.
Response
Language
Selection
Generation
 Core Deep Dive

Actions
Functions your bot runs in response to user input.
Four different action types:
● Utterance actions: `utter_`
○ send a specific message to the user
○ specified in `responses` section of the domain
● Retrieval actions: `respond_`
○ send a message selected by a retrieval model
● Custom actions: `action_`
○ run arbitrary code and send any number of messages (or none).
○ return events
● Default actions:
○ built-in implementations available but can be overridden
○ E.g. action_listen, action_restart, action_default_fallback
 Core Deep Dive

Custom Actions: The Action Server

Custom action code is run by a webserver called the action server

● How custom actions get run:

○ When a custom action is predicted, Rasa will call the endpoint to your action server
○ The action server:
■ runs the code for your custom action
■ [optionally] returns information to modify the dialogue state
● How to create an action server:
○ You can create an action server in any language you want!
○ You can use the Rasa-SDK for easy development in Python
 Core Deep Dive

Custom Actions: Examples

Custom actions can do whatever you want them to! e.g.

● Send multiple messages

● Query a database
● Make an API call to another service
● Return events e.g.
○ Set a slot
■ e.g. based on a database query)
○ Force a follow up action
■ force a specific action to be executed next
○ Revert a user utterance
■ I.e. remove a user utterance from the tracker
 Core Deep Dive

Custom Action Example: Query a Database

using the Rasa SDK

class ActionCheckAddress(Action):
def name(self) -> Text:
“Should I also
return "action_check_address"
update your
mailing address?”
def run(self,
dispatcher: CollectingDispatcher,
tracker: Tracker,
domain: Dict[Text, Any]) -> List[Dict[Text, Any]]:

idnum = tracker.get_slot('person_id')
“What do you have q = "SELECT Address FROM Customers \
on file?” \WHERE CustomerID=1;'{0}'".format(idnum)
result = db.query(q)

return [SlotSet(
"address",
result if result is not None else "NotOnFile")]
 Let’s Code

Let’s implement a custom for our financial assistant

We will implement a custom action which will allow users to check their account
balance:

Hello

Hello! How can I help?

I would like to check my

account balance

Your account balance is

$10.
Forms
 Core Deep Dive

Forms
Rasa Forms allow you to describe all happy paths with a single story

## Happy path 1:
U: I would like to make a money transfer
B: Towards which credit card would you like to make a payment?
U: Towards my justice bank credit card
B: How much do you want to pay? ## pay credit card happy path
U: $100
* pay_cc
B: The transaction has been scheduled.
- cc_payment_form
- form{"name": "cc_payment_form"}

## Happy path 2: - form{"name": null}

U: I would like to make a 100$ money transfer
B: Towards which credit card would you like to make a payment?
U: Towards my justice bank credit card
B: The transaction has been scheduled.
 Core Deep Dive

Forms
The structure of the form

## pay credit card happy path

* greet
- utter_greet
* pay_cc User input

- cc_payment_form
Form action
Form activated - form{"name": "cc_payment_form"}

Form deactivated - form{"name": null}

 Core Deep Dive

Forms
FormAction is a custom action which allows you to set the required slots and
determine the behaviour of the form

The main components of the form action:

● name - defines the name of the form actions

def name(self) -> Text:

" " "Unique identifier of the form" " "

return "cc_payment_form"
 Core Deep Dive

Forms
FormAction is a custom action which allows you to set the required slots and
determine the behaviour of the form

The main components of the form action:

● required slots - sets the list of required slots

def required_slots(tracker: Tracker) -> List[Text]:

" " "A list of required slots to fill in" " "
return [“credit_card”, “amount_of_money”]
 Core Deep Dive

Forms
FormAction is a custom action which allows you to set the required slots and
determine the behaviour of the form

The main components of the form action:

● submit - defines the output of the form action once all slots are filled in

def submit(self, args) -> List[Dict]:

" " "Defines what the form has to do after all slots are filled in" " "
dispatcher.utter_message(“The payment is confirmed”)
return [AllSlotsReset()]
 Core Deep Dive

Forms: Custom slot mappings

The slot mappings define how to extract slot values from user inputs.

With slot mappings you can define how certain slots can be extracted from:

- Entities
- Intents
- Support free text input
- Support yes/no inputs

def slot_mappings(self) -> List[Text, Union[Dict, List[Dict]]]:

" " "Defines what the form has to do after all slots are filled in" " "
return {
“confirm”: [
self.from_intent(value:True, intent=”affirm”),
self.from_intent(value:False, intent=”deny”)
]
}
 Core Deep Dive

Forms: Validating user input

Validate method in form action allows you to check the value of the slot against a set of
values.

I would like to make a $100 payment. validate

“Minimum balance”: 85
slot: amout_of_money “Current balance”: 550
 Core Deep Dive

Forms
Custom Form Action

Custom Form Action allows you to add new methods to a regular FormAction.

class CustomFormAction(FormAction):

def name(self):
return “”

def custom_method(self, args):

...
return
 Core Deep Dive

Forms
Handling unhappy paths

Users are not always cooperative - they can change their mind or interrupt the form.

● Create stories to handle the interruptions:

## pay credit card unhappy path

* pay_cc
- cc_payment_form
- form{"name": "cc_payment_form"}
* chitchat
- utter_chitchat
- cc_payment_form
- form{"name": null}
 Core Deep Dive

Forms

Handling unhappy paths

Users are not always cooperative - they can change their mind or interrupt the form.

● Use action action_deactivate_form to handle the situation where users decide not to proceed with the form:

## chitchat
* pay_cc
- cc_payment_form
- form{"name": "cc_payment_form"}
* stop
- utter_ask_continue
* deny
- action_deactivate_form
- form{"name": null}
 Core Deep Dive

Forms
How this affect domain and policy configuration?

Domain Policy configuration

forms: policies:
cc_payment_form - name: FormPolicy
 Let’s Code

Let’s implement a for paying out a credit card

We will implement a form which will enable our assistant to collect necessary information to pay a credit
card:
Hello

Hello! How can I help?

I would like to pay my credit card

Towards which account would you like to make a payment?

Iron bank

How much would you like to transfer?

$1000

When would you like the transaction to be made?

Today

Do you want to schedule a $1000 transfer to iron bank for today?

Yes
 Part 4: Messaging Channels,
Conversation Driven Development,
and Rasa X
 Agenda

Theme for this section: Connecting with your users

I. Conversation Driven Development

A. Combining user input and software engineering best practices at every stage of the development
cycle
B. Conversations = training data
II. Rasa X
A. A tool for Conversation Driven Development
B. Launching Rasa X locally
C. Reviewing conversations
III. Messaging Channels
A. Live coding: connecting Telegram
IV. Review
Conversation Driven Development
WHY CDD

CDD captures the lessons we’ve learned as a community

If you’ve built conversational AI before, you know that:

● It is very hard.
● Building a prototype is not the hard part.
● The hard parts all show up when you want to go from a
prototype to something you’d want to ship.

Conversation-Driven Development should:

● Help all of us build better conversational AI.

● Save newcomers from having to learn this the hard way.
 What is the opposite of conversation driven development?

● Developing your assistant for months

in isolation before testing it with users
● Only looking at top-level metrics, not
conversations themselves
● Autogenerating training data that
doesn’t reflect ways users really
interact
● Pushing changes to production
without running tests
 Conversation-Driven Development

te
ew
re

k
ta

st

ac

x
a

vi

no

Fi
Te
Sh

Tr
Re

An
1 2 3 4 5 6
 Share

Users will always surprise you.

So get some test users to try your prototype as early

as possible.

Projects that ship without a bunch of

test users are bound to fall down in
production.
 Review

At every stage of a project, it is worth reading what users are

saying.

Avoid getting caught up in metrics right away. Conversations

are valuable data.
Annotate

Using a script to generate

synthetic training data

Turning real messages into

training examples
Rasa X Tour

Real Conversations > Synthetic Data

During development, training data is created by:

● Writing down hypothetical conversations and training examples

● Studying existing chat transcripts or knowledge bases
● Consulting subject matter experts

That’s a great start, but all of those scenarios are based on human:human conversations.

Human:bot conversations have different patterns - users talk to a bot differently than they do a human

The best training data is generated from the assistant’s

actual conversations
 Quality of data, by source

1 Human + bot conversations

Human + human 2
conversations
Quality

3 Manufactured data

Documentation, FAQs 4
 By the time you go to production, most of your training data should
come from human:bot interactions

1 Human + bot conversations

Proportion

Human + human 2
conversations

3 Manufactured data

Documentation, FAQs 4
 Test

Professional teams don’t ship applications without

tests.

Use whole conversations as end-to-end tests

Run them on a continuous integration (CI) server.

PrototypeTrack

Use proxy measures to track which conversations are

successful and which ones failed.

‘Negative’ signals are useful too, e.g. users not getting back in
touch with support.
Prototype
Fix

Study conversations that went smoothly and ones that failed.

Successful conversations can become new tests 🎉

Fix issues by annotating more data and/or fixing your code 🔧

CDD in practice
PROCESS

It’s not a linear process: you’ll find yourself jumping between these
actions

share review annotate test track fix

TEAM

Some actions require software skills, others a deep understanding of

the user

Image credit: Freepik

 CDD for teams

Track Review Share Test Annotate Fix

Product Design Development

● Product ● UX Designers ● Engineers
Managers ● Conversation ● Data Scientists
● Innovation Designers ● Annotators
and Business ● Content
Leaders Creators
Rasa X is a tool to help
teams do CDD
Rasa X Tour

Rasa X turns conversations into training data

�� ��
Messaging Share your
Conversations Conversations
channels
Rasa X bot

Review 👁‍🗨 Annotate ✏ Improve

💪
PRODUCT

Rasa X layers on top of Rasa Open Source

Rasa Open Source: Framework for Rasa X: Tool for

building AI assistants Conversation-Driven Development

≈≈
 Rasa X Local mode vs Server mode

Local Mode:

● Great for initial testing, familiarizing yourself with Rasa X

● Conversations get saved to a local database

Server Mode:

● Great for serving your bot and collecting conversations from many testers or users, 24/7
● Production-ready & scalable
● Deploys Rasa X (and your assistant) using Docker Compose or Kubernetes
● Conversations get saved to a production database
● Includes a Git integration (integrated version control)
Share your bot with testers using just a link
Review conversations coming in from every channel
Annotate the messages coming in
Push new training data to git and trigger your CI pipeline

#232a2e
 Track failures and successes

Use the API to automatically tag conversations, or add Turn successful conversations into new
tags manually as you read end-to-end tests
 Installing Rasa X Locally

Checkout the master branch

git checkout master

Install Rasa X in your virtual environment:

pip3 install rasa-x --extra-index-url https://pypi.rasa.com/simple

Launch Rasa X in the browser by running the following command in your project directory:

rasa x
 Testing Locally: ngrok

Creates a secure, publicly accessible tunnel URL connected to a process running on localhost.

Allows you to develop as though your application were running on a server instead of locally.
 Starting ngrok

1. Start Rasa X. When you start Rasa X locally, it runs on localhost:5002

2. Open a new terminal window and start ngrok. Specify that ngrok should listen for the process running on
port 5002
a. ngrok http 5002
3. Save the URL generated by ngrok
4. Generate a Share your Bot link in Rasa X.
5. In the Share Your Bot URL, replace localhost:5002 with your ngrok URL
http://92099d45e813.ngrok.io/guest/conversations/production/76a25a7880ac40f5bda7b42d650bc
a75
Messaging Channels
RASA OPEN SOURCE

Rasa Open Source

Natural
Language
Understanding
“I want to change
my address”
Message
ML-based
Channels
Dialogue
E.g. Live Chat, Management
“What’s your new App, Messenger Backend,
address?” database,
API, etc.
Response
Selection
MESSAGING CHANNELS

Built in Channels

Telegram
 Messaging Channels

● Channel connection details are kept in

the credentials.yml file
● When you move to production, you’ll
want to re-create these credentials
securely on your server
● After you update credentials.yml, start (or
restart) the server with rasa run
 Telegram

Register a new bot:

- Go to https://web.telegram.org/#/im?p=@BotFather
- Type /newbot
- Name shows up next to the bot’s profile icon
- Username is the bot’s handle, and ends in _bot

credentials.yml

Telegram:
access_token: "your API token"
verify: "bot’s username (ends in _bot)"
webhook_url: "https://<ngrok-url>/webhooks/telegram/webhook"
 Testing Telegram locally

1. Start the rasa server (this command starts the server without a chat session on the command line)
a. rasa run
2. Start ngrok on port 5005
a. ngrok http 5005
3. Add the ngrok URL to your Telegram webhook URL - save the file
4. Restart the Rasa server (so changes to credentials.yml get picked up)
a. /stop
b. rasa run
5. Make sure your action server is running!
6. Try talking to your bot on Telegram
What’s Next?
NEXT STEPS

Get people to chat with your assistant and keep improving it!

Where to go from here?

1. Look at conversations and ask:

how does your assistant struggle?
2. Learn how to solve problem from
Rasa Docs and Community Forum
3. Improve assistant
4. Test and deploy your update
5. Repeat 😎
 THE APPROACH

Continually improve your assistant using Rasa X

Ensure your new assistant passes tests

Collect conversations between
using continuous integration (CI) and
users and your assistant
redeploy it to users using continuous
deployment (CD)

Review conversations and

improve your assistant based
on what you learn
 THE APPROACH

The path to a contextual assistant

No iterations

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

A few iterations

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

Many iterations

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 THE APPROACH

The path to a contextual assistant

A lot of iterations

Quality of assistant

Rasa Open Source (Local)

Rasa X (Server)
 Build a minimum viable assistant with Rasa Open Source + improve it
using Rasa X
um y sing sing
minim nt ove b ove u ns ove u ns
u i l d Impr h e I m p r I m p r
B
e ass
ista ng to t t ersat
io rsatio
Talk to viabl ta l k i
tan conv t users conve l users
assis tes rea
your bot with with

Quality of assistant

Rasa Open Source is an open

source framework for natural
language understanding, dialogue Rasa Open Source (Local)
Rasa X is a toolset used
management, and integrations.
to improve a contextual
Rasa X (Server)
assistant built using
Rasa Open Source.
Build a minimum viable assistant with Rasa Open Source + improve it using Rasa X

um y sing sing
Share minim nt ove b ove u ns ove u ns
u i l d Impr h e I m p r I m p r
your bot B
e ass
ista ng to t t ersat
io rsatio
viabl ta l k i
tan conv t users conve l users
assis tes rea
with with

Quality of assistant

Rasa Open Source is an open

source framework for natural
language understanding, dialogue Rasa Open Source (Local)
Rasa X is a toolset used
management, and integrations.
to improve a contextual
Rasa X (Server)
assistant built using
Rasa Open Source.
 Build a minimum viable assistant with Rasa Open Source + improve it using
Rasa X
um y sing sing
minim nt ove b ove u ns ove u ns
u i l d Impr h e I m p r I m p r
B
e ass
ista ng to t t ersat
io rsatio
viabl ta l k i
tan conv t users conve l users
assis tes rea
with with
Real users
on Telegram

Quality of assistant

Rasa Open Source is an open

source framework for natural
language understanding, dialogue Rasa Open Source (Local)
Rasa X is a toolset used
management, and integrations.
to improve a contextual
Rasa X (Server)
assistant built using
Rasa Open Source.
DEVELOPMENT WORKFLOW

End-to-end workflow for managing assistant updates

Recap
 RECAP

What we’ve learned so far

● Set up
● Deep dive of NLU and dialogue management
● Build an MVP assistant
● Share your assistant with the outside world
● Make iterative improvements and take your assistant to the next level
 RASA OPEN SOURCE

Rasa Open Source

Natural
Language
Understanding
“How much did I
spend at Starbucks
last week?”
Connector
ML-based
Modules
Dialogue
E.g. Live Chat, Management
“You spent $12 at App, Messenger Backend,
Starbucks!” database,
API, etc.
Response
Selection
 NLU Deep Dive

Intents and Entities

Two of the most common and necessary types of information to

extract from a message

Intent: What the user

wants to achieve

How much did I spend at Starbucks last

week?
Entity: Key piece of
information
 Dialogue Management Deep Dive: Policies

Policies

● Decide which action to take at every step in the conversation

● Each policy predicts an action with some probability. This is called core confidence.

Interpreter
“I want to know
1
how much I spent
2
at Starbucks last Message In
week”
3
Tracker Policy
Message Out
6
“You spent $12 at 5
Starbucks”
Action 4
 Core Deep Dive

Domain
Defines the “world” of your assistant - what it knows, can understand, and can do

intents: What your bot can understand

…
entities: What your bot can remember
…
slots:
…
actions: What your bot can do (and say)
…
responses: What your bot can say (and
… display)
session_config:
…
What your bot considers a new
conversation
 Core Deep Dive: Policies

Multiple Policies

● The policy with the highest confidence wins.

● If the confidence of two policies is equal, the policy with the highest priority wins.

Policy priorities
(higher numbers = higher priority)
Rule-based
5. FormPolicy
policies have
4. FallbackPolicy, TwoStageFallbackPolicy
higher priority 3. MemoizationPolicy, AugmentedMemoizationPolicy
than ML-based 2. MappingPolicy
policies 1. EmbeddingPolicy, TEDPolicy
 NLU Deep Dive

DIET is our new neural network architecture for NLU

What is DIET?
● New state of the art neural network architecture for NLU
● Predicts intents and entities together
● Plug and play pretrained language models
 We found out that the Transformer Embedding Dialogue policy can
untangle sub-dialogues

paper

@madymantha
 Conversation Design

Scope Conversation
How to get started with conversation design

The assistant’s purpose

Leverage the knowledge of domain experts

Common search queries

FAQs and wikis

 Training Data for NLU and Dialogue Management

Machine learning models require training data that the models can
generalize from
NLU needs data in the form of examples for Dialogue management model needs
intents data in the form of stories
## intent:bot_challenge ## new to rasa at start
- are you a bot? * how_to_get_started{" user_type":
- are you a human? "new"}
- am I talking to a bot? - action_set_onboarding
- am I talking to a human? - slot{" onboarding": true}
- utter_getstarted_new
## intent:i_like_food - utter_built_bot_before
- I like [ apples](food) * deny
- my friend likes [ oranges](food) - utter_explain_rasa_components
- I'm a fan of [ pears](food) - utter_rasa_components_details
- do you like [ coffee](food) - utter_ask_explain_nlucorex
 LIVE CODING: MINIMUM VIABLE ASSISTANT

What is a Minimum Viable Assistant (MVA)?

● A basic assistant that can handle the most important happy path stories.
○ Happy path: If your assistant asks a user for some information and the user
provides it, we call that a happy path.
○ Unhappy path: All the possible edge cases of the bot
 Testing

Use the Rasa CLI to test your assistant

End to End NLU Core

Evaluation Evaluation Evaluation
Run through test conversations Split data into a test set or Evaluate your trained model on
to make sure that both NLU and estimate how well your model a set of test stories and
Core make correct predictions. generalizes using generate a confusion matrix.
cross-validation.

$ rasa test $ rasa test nlu -u $ rasa test core

data/nlu.md --config --stories
config.yml test_stories.md --out
--cross-validation results
 Core Deep Dive

Custom Actions: Connecting to the outside world

Natural
Language
Understanding
My new address is
222 Broadway NYC.
ML-based
Connector Dialogue
Modules Management
Backend,
~~~ database,
Natural API, etc.
Language
Generation
 Core Deep Dive

Events
● Internally, all conversations are represented as a sequence of events.
● Some events are automatically tracked

Events Story

UserUttered ## new to rasa at start

SlotSet * how_to_get_started{" user_type":
SlotSet "new"}
- action_set_onboarding
BotUttered - slot{" onboarding": true}
UserUttered - utter_getstarted_new
Restarted * restart
- action_restart
 Core Deep Dive

Tracker
● Trackers maintain the state of a dialogue between the assistant and the user.
● It keeps track of events, slots, session info etc.
{
"sender_id": "default", Session info
"slots": {
"user_type": "new",
"onboarding": true}, Slots
"latest_message": {...},
"events": [
...
{"event": "action", Events:
"name": "action_set_onboarding",
"policy": "MemoizationPolicy",
...
"confidence": 0.92}, SlotSet
... ...
{"event": "user", UserUttered
"text": "\/restart",...},
...
...
]
}
 Core Deep Dive

Slots
Your bot’s memory

● Can store:
○ user-provided info
○ info from the outside world
● Can be set by:
○ NLU (from extracted entities, or buttons)
○ Custom Actions
● Can be configured to affect or not affect the dialogue progression
 Core Deep Dive

Actions
Things your bot runs in response to user input.
Four different action types:
● Utterance actions: `utter_`
○ send a specific message to the user
○ specified in `responses` section of the domain
● Retrieval actions: `respond_`
○ send a message selected by a retrieval model
● Custom actions: `action_`
○ run arbitrary code and send any number of messages (or none).
○ return events
● Default actions:
○ built-in implementations available but can be overridden
○ E.g. action_listen, action_restart, action_default_fallback
 Core Deep Dive

Custom Actions: Examples

Custom actions can do whatever you want them to! e.g.

● Send multiple messages

● Query a database
● Make an API call to another service
● Return events e.g.
○ Set a slot
■ e.g. based on a database query)
○ Force a follow up action
■ force a specific action to be executed next
○ Revert a user utterance
■ I.e. remove a user utterance from the tracker
 Core Deep Dive

Forms
The structure of the form

## pay credit card happy path

* greet
- utter_greet
* pay_cc User input

- cc_payment_form
Form action
Form activated - form{"name": "cc_payment_form"}

Form deactivated - form{"name": null}

 Core Deep Dive

Forms
Handling unhappy paths

Users are not always cooperative - they can change their mind or interrupt the form.

● Use action action_deactivate_form to handle the situation where users decide not to proceed with the form:
## chitchat
* request_restaurant
- restaurant_form
- form{"name": "restaurant_form"}
* stop
- utter_ask_continue
* deny
- action_deactivate_form
- form{"name": null}
 Rasa X Tour

Rasa X turns conversations into training data

�� ��
Messaging Share your
Conversations Conversations
channels
Rasa X bot

Review 👁‍🗨 Annotate ✏ Improve

💪
 Rasa X Tour

Real Conversations > Synthetic Data

During development, training data is created by:

● Writing down hypothetical conversations and training examples

● Studying existing chat transcripts or knowledge bases
● Consulting subject matter experts

That’s a great start, but all of those scenarios are based on human:human conversations.

Human:bot conversations have different patterns - users say different things when they are talking to a bot.

The best training data is generated from the assistant’s

actual conversations
 Conversation-Driven Development

te
ew
re

k
ta

st

ac

x
a

vi

no

Fi
Te
Sh

Tr
Re

An
1 2 3 4 5 6
 Build a minimum viable assistant with Rasa Open Source + improve it
using Rasa X
um y sing sing
minim nt ove b ove u ns ove u ns
u i l d Impr h e I m p r I m p r
B
e ass
ista ng to t t ersat
io rsatio
viabl ta l k i
tan conv t users conve l users
Talk to assis tes rea
with with
your bot

Quality of assistant

Rasa Open Source is an open

source framework for natural
language understanding, dialogue Rasa Open Source (Local)
Rasa X is a toolset used
management, and integrations.
to improve a contextual
Rasa X (Server)
assistant built using
Rasa Open Source.
 Build a minimum viable assistant with Rasa Open Source + improve it
using Rasa X
um y sing sing
minim nt ove b ove u ns ove u ns
u i l d Impr h e I m p r I m p r
B
e ass
ista ng to t t ersat
io rsatio
viabl ta l k i
tan conv t users conve l users
assis tes rea
with with
Share
your bot

Quality of assistant

Rasa Open Source is an open

source framework for natural
language understanding, dialogue Rasa Open Source (Local)
Rasa X is a toolset used
management, and integrations.
to improve a contextual
Rasa X (Server)
assistant built using
Rasa Open Source.
 Build a minimum viable assistant with Rasa Open Source + improve it
using Rasa X
um y sing sing
minim nt ove b ove u ns ove u ns
u i l d Impr h e I m p r I m p r
B
e ass
ista ng to t t ersat
io rsatio
viabl ta l k i
tan conv t users conve l users
assis tes rea
with with
Real users

Quality of assistant

Rasa Open Source is an open

source framework for natural
language understanding, dialogue Rasa Open Source (Local)
Rasa X is a toolset used
management, and integrations.
to improve a contextual
Rasa X (Server)
assistant built using
Rasa Open Source.
DEVELOPMENT WORKFLOW

End-to-end workflow for managing assistant updates

DEVELOPMENT WORKFLOW

How Rasa fits into your stack

DEVELOPMENT WORKFLOW

How Rasa fits into your stack