Receive DTMF and Speech Input Using Python

Overview

You can use speech input or dual-tone multi-frequency (DTMF) tones (a.k.a. Touch-Tone) to route callers or otherwise change call flows for applications such as interactive voice response (IVR), virtual assistants, and mobile surveys.

Prerequisites

To get started, you need a Plivo account — sign up with your work email address if you don’t have one already. You must have a voice-enabled Plivo phone number to receive incoming calls; you can rent numbers from the Numbers page of the Plivo console, or by using the Numbers API. If this is your first time using Plivo APIs, follow our instructions to set up a Python development environment and a web server and safely expose that server to the internet.

Detect DTMF inputs

How it works

This example shows a multilevel IVR phone application that uses digit press input captured using the GetInput XML element. A virtual assistant answers incoming calls and offers the caller three choices: “Press 1 for your account balance. Press 2 for your account status. Press 3 to speak to a representative.” If the caller enters 1 or 2, the application will retrieve the requested information and play the caller a text-to-speech message. If the caller presses 3, the application will redirect the caller to the second branch, which offers two new choices: “Press 1 for sales. Press 2 for support.” The application then connects the caller with the requested department.

Code

Create a file called detect_dtmf.py and paste into it this code.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112

#! / usr / bin / python
# -* - coding: utf - 8 -* -
from flask import Flask, Response, request, url_for
from plivo import plivoxml
# Welcome message, first branch
WelcomeMessage = "Welcome to the demo. Press 1 for your account balance. Press 2 for your account status. Press 3 to speak to a representative"
# Message for second branch
RepresentativeBranch = "Press 1 for sales. Press 2 for support"
# Message that Plivo reads when the caller does nothing
NoInput = "Sorry, I didn't catch that. Please hang up and try again"
# Message that Plivo reads when the caller presses a wrong digit
WrongInput = "Sorry, that's not a valid input"
app = Flask(__name__)

@app.route("/multilevelivr/", methods=["GET", "POST"])
def ivr():
    element = plivoxml.ResponseElement()
    if request.method == "GET":
        response = (
            element.add(
                plivoxml.GetInputElement()
                .set_action("https://<ngrok_identifier>.ngrok.io/multilevelivr/firstbranch/")
                .set_method("POST")
                .set_input_type("dtmf")
                .set_digit_end_timeout(5)
                .set_redirect(True)
                .set_language("en-US")
                .add_speak(
                    content=WelcomeMessage, voice="Polly.Salli", language="en-US"
                )
            )
            .add_speak(content=NoInput)
            .to_string(False)
        )
    print(response)
    return Response(response, mimetype="text/xml")


@app.route("/multilevelivr/firstbranch/", methods=["GET", "POST"])
def firstbranch():
    response = plivoxml.ResponseElement()
    digit = request.form.get("Digits")
    print("digit pressed: {digit}")
    if digit == "1":
        text = "Your account balance is $20"
        params = {"language": "en-GB"}
        response.add(plivoxml.SpeakElement(text, ** params))
    elif digit == "2":
        text = "Your account status is active"
        params = {"language": "en-GB"}
        response.add(plivoxml.SpeakElement(text, ** params))
    elif digit == "3":
        element = plivoxml.ResponseElement()
        response = (
            element.add(
                plivoxml.GetInputElement()
                .set_action("https://<ngrok_identifier>.ngrok.io/multilevelivr/secondbranch/")
                .set_method("POST")
                .set_input_type("dtmf")
                .set_digit_end_timeout(5)
                .set_redirect(True)
                .set_language("en-US")
                .add_speak(
                    content=RepresentativeBranch, voice="Polly.Salli", language="en-US"
                )
            )
            .add_speak(content=NoInput)
            .to_string(False)
        )
        print(response)
        return Response(response, mimetype="text/xml")
    else:
        response.add(plivoxml.SpeakElement(WrongInput))
        print(response.to_string())
        return Response(response.to_string(), mimetype="text/xml")

@app.route("/multilevelivr/secondbranch/", methods=["GET", "POST"])
def secondbranch():
    response = plivoxml.ResponseElement()
    digit = request.form.get("Digits")
    from_number = request.form.get("From")
    print("digit pressed: {digit}")
    if digit == "1":
        response.add(
            plivoxml.DialElement(
                action="https://<ngrok_identifier>.ngrok.io/multilevelivr/action/",
                method="POST",
                redirect=False,
                caller_id=from_number,
            ).add(plivoxml.NumberElement("<number_1>"))
        )
        print(response.to_string())
        return Response(str(response), mimetype="text/xml")
    elif digit == "2":
        response.add(
            plivoxml.DialElement(
                action="https://<ngrok_identifier>.ngrok.io/multilevelivr/action/",
                method="POST",
                redirect=False,
                caller_id=from_number,
            ).add(plivoxml.NumberElement("<number_2>"))
        )
        print(response.to_string())
        return Response(str(response), mimetype="text/xml")
    else:
        response.add(plivoxml.SpeakElement(WrongInput))
        print(response.to_string())
        return Response(response.to_string(), mimetype="text/xml")


if __name__ == "__main__":
    app.run(host="0.0.0.0", debug=True)

Save the file and run it.

$ python detect_dtmf.py

You should see your application in action at http://localhost:5000/multilevelivr/.

Control the gathering of DTMF inputs

You can improve DTMF collection by using attributes available for the GetInput XML element, such as digitEndTimeout, numDigit, finishOnKey, and executionTimeout.

digitEndTimeout sets the maximum time interval between successive digit inputs. The default value is auto and other allowed values are 2 to 10 seconds. If the user provides no new digits within the digitEndTimeout period, the digits entered to that point will be processed.

numDigits sets the maximum number of digits the user can provide on the current call. The default value is 32 and the allowed values are 1 to 32.

If the user provides more digits than the value of numDigits, Plivo will send only the number of digits specified as numDigits to the action URL; additional digit inputs will be ignored. For example, if numDigits is specified as “4” and the user enters five digits, the last digit will be ignored.

finishOnKey defines a key that users can press to submit the digits they entered. The default value is # and additional allowed values are 0-9, *, <empty string>, and ”none.” When you set the value to <empty string> or “none,” DTMF input collection ends depending on the digitEndTimeout or the numDigits attribute.

Note: These three attributes apply to input types dtmf and dtmf speech and do not apply to input type speech. If all three of these attributes are specified, the priority is for finishOnKey.

executionTimeout sets the maximum time during which Plivo detects input. You can use this timeout to tell the application to process the next element in the XML response when a user doesn‘t provide input during the call. The default value is 15 seconds, and allowed values are 5 to 60 seconds.

Detect speech input

The GetInput XML element can also capture speech input.

How it works

Receive speech

This example shows how to implement a simple IVR phone tree. A virtual assistant answers the call and offers the caller two choices: “Say sales to talk to a sales representative. Say support to talk to a support representative.”

If the caller says “sales,” the caller will be connected to a sales representative; if the caller says “support,” they will be connected to a support representative.

Create a Flask application to detect speech input

Create a file called detect_speech.py and paste into it this code.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71

#! / usr / bin / python
# -* - coding: utf - 8 -* -
from flask import Flask, Response, request, url_for
from plivo import plivoxml
# Welcome message, first branch
WelcomeMessage = "Welcome to the demo. Say sales to talk to a sales representative. Say support to talk to a support representative"
# Message that Plivo reads when the caller does nothing
NoInput = "Sorry, I didn't catch that. Please hang up and try again"
# Message that Plivo reads when the caller speaks something unrecognized
WrongInput = "Sorry, that's not a valid input"
app = Flask(__name__)

@app.route("/ivrspeech/", methods=["GET", "POST"])
def ivr():
    element = plivoxml.ResponseElement()
    if request.method == "GET":
        response = (
            element.add(
                plivoxml.GetInputElement()
                .set_action("https://<ngrok_identifier>.ngrok.io/ivrspeech/firstbranch/")
                .set_method("POST")
                .set_input_type("dtmf")
                .set_digit_end_timeout(5)
                .set_redirect(True)
                .set_language("en-US")
                .add_speak(
                    content=WelcomeMessage, voice="Polly.Salli", language="en-US"
                )
            )
            .add_speak(content=NoInput)
            .to_string(False)
        )
    print(response)
    return Response(response, mimetype="text/xml")

@app.route("/ivrspeech/firstbranch/", methods=["GET", "POST"])
def secondbranch():
    response = plivoxml.ResponseElement()
    speech = request.form.get("Speech")
    from_number = request.form.get("From")
    print("Speech Input is: {speech}")
    if speech == "1":
        response.add(
            plivoxml.DialElement(
                action="https://<ngrok_identifier>.ngrok.io/ivrspeech/action/",
                method="POST",
                redirect=False,
                caller_id=from_number,
            ).add(plivoxml.NumberElement("<number_1>"))
        )
        print(response.to_string())
        return Response(str(response), mimetype="text/xml")
    elif speech == "2":
        response.add(
            plivoxml.DialElement(
                action="https://<ngrok_identifier>.ngrok.io/ivrspeech/action/",
                method="POST",
                redirect=False,
                caller_id=from_number,
            ).add(plivoxml.NumberElement("<number_2>"))
        )
        print(response.to_string())
        return Response(str(response), mimetype="text/xml")
    else:
        response.add(plivoxml.SpeakElement(WrongInput))
        print(response.to_string())
        return Response(response.to_string(), mimetype="text/xml")


if __name__ == "__main__":
    app.run(host="0.0.0.0", debug=True)

Save the file and run it.

$ python detect_speech.py

You should see your application in action at http://localhost:5000/ivrspeech/.

Speech recognition attributes

Speech models

Different applications may benefit from different automatic speech recognition (ASR) models, which you can specify using the the GetInput XML element‘s speechModel attribute. By default, it has a value of default, which is suitable for long-form audio, such as dictation, but you can also try command_and_search for shorter audio clips, such as when you expect callers to use voice commands or voice search, or phone_call, if you want to transcribe audio from a phone call. Explore the models and see which works best for your use case.

Example XML:

<Response>
<GetInput action="https://<yourdomain>.com/action/" method="POST" inputType="speech" speechModel="command_and_search" redirect="true">
<Speak>Welcome to the demo. Say sales to talk to a sales representative. Say support to talk to a support representative</Speak>
</GetInput>
<Speak>Sorry, I didn't catch that. Please hang up and try again later.</Speak>
</Response>

Hints

You can use the hints attribute to potentially improve speech transcription results by defining words and phrases that are common in your use case. For example, a call center where callers use voice commands to connect to various departments can use the names of the departments as hints.

  • Allowed values: a non-empty string of comma-separated phrases
  • Limitations are:
    • Phrases per request: 500
    • Characters per phrase: 100
    • Characters per request: 10,000

Example XML:

<Response>
<GetInput action="https://<yourdomain>.com/action/" method="POST" inputType="speech" hints="sales,support" redirect="true">
<Speak>Welcome to the demo. Say sales to talk to a sales representative. Say support to talk to a support representative</Speak>
</GetInput>
<Speak>Sorry, I didn't catch that. Please hang up and try again later.</Speak>
</Response>

Controlling the gathering of speech input

You can improve the functionality of speech input collection by using GetInput XML attributes such as speechEndTimeout, language, profanityFilter, and executionTimeout.

speechEndTimeout sets the time that Plivo waits for more speech input after silence is detected. The default value is auto; other allowed values are 2 to 10 seconds. If the user doesn‘t provide new speech input within the speechEndTimeout period, the speech collected to that point will be processed.

language specifies the language and national/regional dialect of the audio to be recognized on calls. The default language for speech detection is en-US. You can choose your preferred language from the list of supported languages.

profanityFilter: If a user speaks any profane words, Plivo can filter them out during transcription if you set this attribute to true. The profanity filter applies only to single words — it doesn‘t work for a combination of words. The default value is false.

Note: These three attributes apply to input types speech and dtmf speech and do not apply to input type dtmf.

executionTimeout sets the maximum time during which Plivo detects input. You can use this timeout to tell the application to process the next element in the XML response when a user doesn‘t provide input during the call. The default value is 15 seconds, and allowed values are 5 to 60 seconds.

Example XML:

<Response>
<GetInput action="https://<yourdomain>.com/action/" method="POST" inputType="speech" speechEndTimeout="5" language="en-IN" profanityFilter="true" executionTimeout="25" redirect="true">
<Speak>Welcome to the demo. Say sales to talk to a sales representative. Say support to talk to a support representative</Speak>
</GetInput>
<Speak>Sorry, I didn't catch that. Please hang up and try again later.</Speak>
</Response>

Real-time speech recognition

You can use the interimSpeechResultsCallback attribute to perform real-time speech recognition. If you specify a URL for your application server to this attribute, you can receive real-time callbacks of the user’s recognized speech while the user is still speaking on the call. Plivo sends the transcribed result to your server URL with attributes such as UnstableSpeech, Stability, StableSpeech, and SequenceNumber.

  • UnstableSpeech holds the interim transcribed result of the user’s speech, which may be refined when more speech is collected from the user.
  • Stability is an estimate of the likelihood that the recognizer will not change its guess about the interim UnstableSpeech result. Values range from 0.0 (completely unstable) to 1.0 (completely stable).
  • StableSpeech holds the stable transcribed result of the user’s speech.
  • SequenceNumber holds the sequence number of the interim speech callback, which helps you order incoming callback requests.

Example XML:

<Response>
<GetInput action="https://<yourdomain>.com/action/" method="POST" interimSpeechResultsCallback="https://<yourdomain>.com/interimcallback/" interimSpeechResultsCallbackMethod="POST" inputType="speech" redirect="true">
<Speak>Welcome to the demo. Say sales to talk to a sales representative. Say support to talk to a support representative</Speak>
</GetInput>
<Speak>Sorry, I didn't catch that. Please hang up and try again later.</Speak>
</Response>

Data logging preferences

You can use the GetInput XML element’s log attribute to manage input logging preferences. It defaults to true, but if you define it to false, logging will be disabled and Plivo will not log digit and speech input.