Having VC to ask for confirmation for almost understood commands.

[marcusvdt]

This is for having VC to execute commands when you can't get it to understand a command phrase. Kind of creates a second level of confidence, but with confirmation required.
Pre-reqs:
1-Python plugin enabled
2-Put this function inside an existing python script that you may be already using. If you still don't have any python script being loaded on the startup of VC, then you will need to create a new python script and set it to start with VC. This is beyond the scope of this thread.
3-Enable prefix mode. This is actually not 100% needed, but since I'm using prefix mode, my command has been designed to work on prefix mode. If you don't want to use prefix mode, please edit the IF logic on the command "Low confidence - 1st step".

More about your need for this (or not):
There are many times I say a command and it is not executed because the confidence was too low. This usually happens because of background noise, like people talking, kids playing or high volume in the TV.
So I built this command so I can make VC to execute commands even if the confidence is lower than the one that you set in Options.
If you say the same command phrase two times (you can adjust how many times) and the confidence in these two times are lower than the confidence you set in VC options, the script will tell you that it is guessing you are requesting such command and then ask for a confirmation.
Both the minimum confidence and the number of repeated times can be set in the script (variables confidence_threshold and repeated_times). These variables must be adjusted as to avoid false positives.
If you confirm the almost understood command shall be executed, then it is executed.

Not sure this is the best way to do it, but it's the way I found to be better.


python script (place in a python script that you will call at VC startup or create a new one that will be called at startup)

Code Select Expand


#below is part of main program (not a function), but has been put here to make it easier to understand what it refers to.
#it is run only at startup.
global phrase_previous_almost
phrase_previous_almost=""
global repeated_times
repeated_times=0

def almost_understood(phrase_almost_fnct="", confidence_pct_fnct=0):
    global repeated_times
    global repeated_threshold
    global phrase_almost
    global phrase_previous_almost
    phrase_almost=phrase_almost_fnct
    global confidence_pct
    confidence_threshold=30 #minimum confidence required so the spoken phrase is not ignored
    repeated_threshold=1    #minimum times the same phrase should be repeated and almost understood so VC asks if the command should be executed.
    confidence_pct=num(confidence_pct_fnct)
    print "repeated_times"
    print repeated_times
    print "phrase_almost"
    print phrase_almost
    print "phrase_previous_almost"
    print phrase_previous_almost
    if confidence_pct>=confidence_threshold:
        print "confidence_pct>=confidence_threshold"
        if phrase_previous_almost==phrase_almost:
            repeated_times=repeated_times+1
            print "phrase_previous_almost==phrase_almost"
            print "repeated_times"
            print repeated_times
            if repeated_times>=repeated_threshold:
                print "repeated_times>=repeated_threshold will return 1"
                #here it means we said have repeated the same phrase 2 times with a confidence lower than required confidence,
                #but higher than our minimum tolerance threshold.
                #Then we will return a good value.
                repeated_times=0
                phrase_previous_almost=""
                return 1
        else:
            print "phrase_previous_almost!=phrase_almost will reset counter and return 0"
            repeated_times=0
            phrase_previous_almost=phrase_almost
    else:
        print "confidence_pct<confidence_threshold will reset the counter and return 0"
        repeated_times=0
        phrase_previous_almost=""
    print "repeated_times"
    print repeated_times
    print "phrase_almost"
    print phrase_almost
    print "phrase_previous_almost"
    print phrase_previous_almost
    return 0
   



def num(s):
    try:
        return int(s)
    except ValueError:
        try:
            return float(s)
        except ValueError:
            return float(s.replace(",","."))



These are the required command groups in VC.

Code Select Expand

<?xml version="1.0" encoding="utf-16"?>
<!--VoxCommando 2.1.5.2-->
<commandGroup open="True" name="Low Confidence" enabled="True" prefix="" priority="0" requiredProcess="" description="">
  <command id="661" name="Prefix Active" enabled="true" alwaysOn="False" confirm="False" requiredConfidence="0" loop="False" loopDelay="0" loopMax="0" description="">
    <if ifBlockDisabled="False" ifNot="False">
      <ifType>(A)==(B)</ifType>
      <ifParams>{LastEvent}&amp;&amp;VC.Prefix.End</ifParams>
      <then>
        <action>
          <cmdType>Results.SetVar</cmdType>
          <params>
            <param>prefixactive</param>
            <param>0</param>
          </params>
          <cmdRepeat>1</cmdRepeat>
        </action>
      </then>
      <else>
        <action>
          <cmdType>Results.SetVar</cmdType>
          <params>
            <param>prefixactive</param>
            <param>1</param>
          </params>
          <cmdRepeat>1</cmdRepeat>
        </action>
      </else>
    </if>
    <event>VC.Prefix.End</event>
    <event>VC.Prefix.Start</event>
  </command>
  <command id="769" name="Low confidence - 1st step" enabled="true" alwaysOn="False" confirm="False" requiredConfidence="0" loop="False" loopDelay="0" loopMax="0" description="">
    <if ifBlockDisabled="False" ifNot="False">
      <ifType>(A)==(B)</ifType>
      <ifParams>{Var.prefixactive}&amp;&amp;1</ifParams>
      <then>
        <action>
          <cmdType>PY.ExecString</cmdType>
          <params>
            <param>result=almost_understood("{1}","{2}")</param>
          </params>
          <cmdRepeat>1</cmdRepeat>
        </action>
      </then>
      <else />
    </if>
    <if ifBlockDisabled="False" ifNot="False">
      <ifType>(A)==(B)</ifType>
      <ifParams>{LastResult}&amp;&amp;1</ifParams>
      <then>
        <action>
          <cmdType>VC.TriggerEvent</cmdType>
          <params>
            <param>2nd.step.VC.Reco.TooLow</param>
            <param>{1}</param>
            <param>{2}</param>
          </params>
          <cmdRepeat>1</cmdRepeat>
        </action>
      </then>
      <else />
    </if>
    <event>VC.Reco.TooLow</event>
  </command>
  <command id="752" name="Low confidence - 2nd step" enabled="true" alwaysOn="False" confirm="False" requiredConfidence="0" loop="False" loopDelay="0" loopMax="0" description="">
    <action>
      <cmdType>TTS.SpeakSync</cmdType>
      <params>
        <param>I think I heard you asking this: {1}. Can I go ahead? You can say: yes please</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>Results.SetVar</cmdType>
      <params>
        <param>lowconfidencecmd</param>
        <param>{1}</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>Group.Disable</cmdType>
      <params>
        <param>Low Confidence</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>Group.Enable</cmdType>
      <params>
        <param>Confirmar low confidence</param>
        <param>30</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>VC.TriggerEvent</cmdType>
      <params>
        <param>EnableCustomConfirmation</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>VC.TellVox</cmdType>
      <params>
        <param>Confirma o comando</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <event>2nd.step.VC.Reco.TooLow</event>
  </command>
</commandGroup>

Code Select Expand

<?xml version="1.0" encoding="utf-16"?>
<!--VoxCommando 2.1.5.2-->
<commandGroup open="True" name="Confirmar low confidence" enabled="False" prefix="" priority="0" requiredProcess="" description="">
  <command id="685" name="Confirma o comando?" enabled="true" alwaysOn="False" confirm="True" requiredConfidence="60" loop="False" loopDelay="0" loopMax="0" description="">
    <action>
      <cmdType>VC.TellVox</cmdType>
      <params>
        <param>{Var.lowconfidencecmd}</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>Group.Disable</cmdType>
      <params>
        <param>Confirmar low confidence</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>Group.Enable</cmdType>
      <params>
        <param>Low Confidence</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <phrase>Confirma o comando</phrase>
  </command>
  <command id="612" name="Reenable Low Confidence" enabled="true" alwaysOn="False" confirm="False" requiredConfidence="0" loop="False" loopDelay="0" loopMax="0" description="">
    <action>
      <cmdType>Group.Enable</cmdType>
      <params>
        <param>Low Confidence</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>Group.Disable</cmdType>
      <params>
        <param>Confirmar low confidence</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <event>*.Confirma o comando?</event>
  </command>
</commandGroup>

Code Select Expand

<?xml version="1.0" encoding="utf-16"?>
<!--VoxCommando 2.1.5.2-->
<commandGroup open="True" name="Custom confirmation" enabled="False" prefix="" priority="0" requiredProcess="" description="">
  <command id="712" name="comando de confirmação" enabled="true" alwaysOn="True" confirm="False" requiredConfidence="80" loop="False" loopDelay="0" loopMax="0" description="">
    <action>
      <cmdType>VC.ConfirmYes</cmdType>
      <params />
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>OSD.AddText</cmdType>
      <params>
        <param>&gt;Confirmado!</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <phrase>yes please</phrase>
  </command>
</commandGroup>


Last edit to attach the final version as per conversation below. It's working now as of today July/9/2015.

[Haddood]

@marcus
http://voxcommando.com/forum/index.php?topic=1690.0 

the reason I avoided requesting confirmation and opted to auto-execute repeated commands, is after testing with confirmation I found VC becoming annoying when I have friends over .. it kept on begging pardon for things people even did not say.

Apart from false positives, even with prefix mode VC will still hear everything, see this
http://voxcommando.com/forum/index.php?topic=2065.msg17902#msg17902

I hope one day James will apply the idea in that thread , as I think it will help open-air mic setup a lot ...

on the other hand I will study your solution, it might be more elegant, more efficient or have some aspects that can improve the previously shared one 

[marcusvdt]

@Haddood, thanks for the comments and for the links. I'll take a look at them as soon as I can.

Most times I can't find the exact thing I'm looking for, then I find it easier to write it in python. I think python adds huge power to what can be done with VC and sometimes will have more flexibility about what can be done.

I was using a simpler way, but then I was having a crazy behavior where VC was executing commands that nobody said. I solved it by adding three levels of checking before it asks for a confirmation:
1-The prefix must have been said previously than the VC.Reco.TooLow (as I'm in prefix mode). This is done by detecting VC.Prefix.Start and VC.Prefix.End events.
2-The command with low confidence must be equal to the last one.
3-The confidence of these low confidence commands must be lower than the default confidence level, but higher than a certain second threshold (which I have set to a very low value, but works for me because the previous checks will avoid false positives 99% of the times.)
Also, I have a different treatment (and much simpler) for VC.NotRecognized events in another command. If the prefix has been said detected by VC as part of the not recognized command, but the command itself triggers the event VC.NotRecognized, then I just tell the user a random message like "I did not understand your command, please repeat.". I only say this when a prefix start event has been triggered before the VC.NotRecognized, hence I can kind of tell if the user was really trying to ask something to VC. When a VC.NotRecognized event is triggered by random noise, there are less chance of this background noise to contain a sound similar to my prefix.

The only problem I'm seeing with these techniques is that I probably don't completely understand the logic behind the events for VC.Prefix.Start and VC.Prefix.End. Sometimes the VC.Prefix.Start event appears and no corresponding VC.Prefix.End event is triggered until I have tried other commands. Sometimes I wait many seconds before I try again, and no new VC.Prefix.Start is triggered, which I find weird. I expected it would exist a timeout for every VC.Prefix.Start and a correponding VC.Prefix.End, but somehow it does not happen sometimes.

The other problem I see is that I don't know if it is possible to allow for the confirmation command to be said without a prefix. My understanding it that the same way we can override a prefix for a group, we should be able to override the prefix for a group so this group has no prefix. I can't see why this is not available now.
I find it very annoying to say:
"Maria, confirmar" instead of just "confirmar".

[marcusvdt]

The attached screenshots show 2 examples where the prefix events behave differently of what I expected.
Maria and Querida are prefixes.

[nime5ter]

The other problem I see is that I don't know if it is possible to allow for the confirmation command to be said without a prefix. My understanding it that the same way we can override a prefix for a group, we should be able to override the prefix for a group so this group has no prefix. I can't see why this is not available now.
I find it very annoying to say:
"Maria, confirmar" instead of just "confirmar".

If that's really necessary for an entire group, set all the commands in that group to "always on".
http://voxcommando.com/mediawiki/index.php?title=Logical_Command_Builder#Command_Settings

[Haddood]

1-The prefix must have been said previously than the VC.Reco.TooLow (as I'm in prefix mode). This is done by detecting VC.Prefix.Start and VC.Prefix.End events.
2-The command with low confidence must be equal to the last one.
3-The confidence of these low confidence commands must be lower than the default confidence level, but higher than a certain second threshold (which I have set to a very low value, but works for me because the previous checks will avoid false positives 99% of the times.)

that is exactly what the command in the link do ...

The other problem I see is that I don't know if it is possible to allow for the confirmation command to be said without a prefix. My understanding it that the same way we can override a prefix for a group, we should be able to override the prefix for a group so this group has no prefix. I can't see why this is not available now.
I find it very annoying to say:
"Maria, confirmar" instead of just "confirmar".

check this
http://voxcommando.com/forum/index.php?topic=1602.msg14008#msg14008
it allows me to issue multiple commands without using the prefix every time , it might give what you want as an alternative for nime5ter suggestion

[marcusvdt]

@nime5ter, Thanks!
Based on what you said, I've changed the confirmation command to a different group. This confirmation command will be always on, but on a disabled group. The command that deals with the situation where a confirmation will be required will also enable that group so the confirmation can be said.
Inside the confirmation command itself, the group will be disabled back again.

I tried it to no success. I'm sorry, but I'm new to this and I could be misunderstanding something.
1-The confirmation message has been displayed
2-I said confirmar (which is the word to confirm)
3-The confirmation has been ignored.

new testing command (I'll update the 1st post once I understand what is happening).

Code Select Expand

<?xml version="1.0" encoding="utf-16"?>
<!--VoxCommando 2.1.5.2-->
<commandGroup open="True" name="Low Confidence" enabled="True" prefix="" priority="0" requiredProcess="" description="">
  <command id="682" name="Prefix Active" enabled="true" alwaysOn="False" confirm="False" requiredConfidence="0" loop="False" loopDelay="0" loopMax="0" description="">
    <if ifBlockDisabled="False" ifNot="False">
      <ifType>(A)==(B)</ifType>
      <ifParams>{LastEvent}&amp;&amp;VC.Prefix.End</ifParams>
      <then>
        <action>
          <cmdType>Results.SetVar</cmdType>
          <params>
            <param>prefixactive</param>
            <param>0</param>
          </params>
          <cmdRepeat>1</cmdRepeat>
        </action>
      </then>
      <else>
        <action>
          <cmdType>Results.SetVar</cmdType>
          <params>
            <param>prefixactive</param>
            <param>1</param>
          </params>
          <cmdRepeat>1</cmdRepeat>
        </action>
      </else>
    </if>
    <event>VC.Prefix.End</event>
    <event>VC.Prefix.Start</event>
  </command>
  <command id="771" name="Low confidence - 1st step" enabled="true" alwaysOn="False" confirm="False" requiredConfidence="0" loop="False" loopDelay="0" loopMax="0" description="">
    <if ifBlockDisabled="False" ifNot="False">
      <ifType>(A)==(B)</ifType>
      <ifParams>{Var.prefixactive}&amp;&amp;1</ifParams>
      <then>
        <action>
          <cmdType>PY.ExecString</cmdType>
          <params>
            <param>result=almost_understood("{1}","{2}")</param>
          </params>
          <cmdRepeat>1</cmdRepeat>
        </action>
      </then>
      <else />
    </if>
    <if ifBlockDisabled="False" ifNot="False">
      <ifType>(A)==(B)</ifType>
      <ifParams>{LastResult}&amp;&amp;1</ifParams>
      <then>
        <action>
          <cmdType>VC.TriggerEvent</cmdType>
          <params>
            <param>2nd.step.VC.Reco.TooLow</param>
            <param>{1}</param>
            <param>{2}</param>
          </params>
          <cmdRepeat>1</cmdRepeat>
        </action>
      </then>
      <else />
    </if>
    <event>VC.Reco.TooLow</event>
  </command>
  <command id="754" name="Low confidence - 2nd step" enabled="true" alwaysOn="False" confirm="False" requiredConfidence="0" loop="False" loopDelay="0" loopMax="0" description="">
    <action>
      <cmdType>TTS.Speak</cmdType>
      <params>
        <param>I think I heard you asking this:</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>Results.SetVar</cmdType>
      <params>
        <param>lowconfidencecmd</param>
        <param>{1}</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>Group.Disable</cmdType>
      <params>
        <param>Low Confidence</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>Group.Enable</cmdType>
      <params>
        <param>Confirmar low confidence</param>
        <param>30</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>VC.TellVox</cmdType>
      <params>
        <param>Confirma o comando</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <event>2nd.step.VC.Reco.TooLow</event>
  </command>
</commandGroup>


Code Select Expand

<?xml version="1.0" encoding="utf-16"?>
<!--VoxCommando 2.1.5.2-->
<commandGroup open="True" name="Confirmar low confidence" enabled="False" prefix="" priority="0" requiredProcess="" description="">
  <command id="685" name="Confirma o comando?" enabled="true" alwaysOn="True" confirm="True" requiredConfidence="60" loop="False" loopDelay="0" loopMax="0" description="">
    <action>
      <cmdType>VC.TellVox</cmdType>
      <params>
        <param>{Var.lowconfidencecmd}</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>Group.Disable</cmdType>
      <params>
        <param>Confirmar low confidence</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <action>
      <cmdType>Group.Enable</cmdType>
      <params>
        <param>Low Confidence</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <phrase>Confirma o comando</phrase>
  </command>
  <command id="612" name="Reenable Low Confidence" enabled="true" alwaysOn="False" confirm="False" requiredConfidence="0" loop="False" loopDelay="0" loopMax="0" description="">
    <action>
      <cmdType>Group.Enable</cmdType>
      <params>
        <param>Low Confidence</param>
      </params>
      <cmdRepeat>1</cmdRepeat>
    </action>
    <event>Confirm.Cancel.Confirma o comando?</event>
  </command>
</commandGroup>

[nime5ter]

Take a look at the xml command group here: http://voxcommando.com/forum/index.php?topic=1117.msg9410#msg9410

It takes advantage of the always on feature, confirmation events, and the VC.ConfirmYes action.

http://voxcommando.com/mediawiki/index.php?title=Actions#VoxCommando

[marcusvdt]

Thanks, I'll look into it. Could you please help me on how or why sometimes there are no events for prefix end? This is affecting also my treatment to not recognized commands.
Also do you have any idea as to why my phrase to confirm a command has been recognized but ignored?

[jitterjames]

I believe the commands with no prefix end are caused by a bug that was introduced in version 2.1.5.0 when I added support for commands with only payloads and no phrases. http://voxcommando.com/mediawiki/index.php?title=ChangeLog#Version_2.1.5.0
It occurs when you say only the prefix and then don't follow up with a recognized command.  I will try to fix this for the next release.  You will know it is happening because the history window shows "can't find command: The given key was not present in the dictionary."

Your confirmation is being ignored because you are in standby.  This is very clearly shown in your history image.  I'm not sure how you could have missed it.  You have even got a red arrow pointing at it marked '3'. 

In order to give a confirmation without saying a prefix, when you are in standby, you'll need to create a custom confirmation command* that is set to "always on".  You were just provided a link to an example of how to do this by Nime5ter. 

Note that if you use the same phrases in your custom confirmation command as in your confirmation settings in options, you may run into problems because it won't know which command to execute.  You have 3 options.

1 - disable confirmation in options
2 - use different phrases
3 - (might work) raise the priority of your group that contains the custom confirm command.

* To be clear: when I talk about a "confirmation command",I am not talking about a command that requires confirmation. I am talking about a command that allows you to provide confirmation for another command that requires it. 

[marcusvdt]

James, thanks for the answers.

As for the bug with the prefix end event, yes, I noticed it always happen together with that message. I'll keep my eyes on next releases.

As for the confirmation command, I misunderstood what nime5ter said and I have created a command with the checkboxes "always on" and "must confirm" checked and I was considering this was my custom confirmation command. So I was expecting that this command would respond to the normal confirmation phrase without a prefix because the "always on" checkbox was marked.
As per your explanation and the example itself, I've created a custom confirmation command that does not accept the same word that I have setup in my options for command confirmations. This custom command has "always on" checked. Once called, it will fire an action VC.ConfirmYes that will confirm my previous confirmation command.
To avoid this custom confirmation command from being heard when I'm not in a situation that requires the custom confirmation, I put it on a disabled group. The group will be enabled only temporarily to allow for the custom confirmation. Not 100% sure however if this is effective or not because the command has "always on".

I expect the result of the above setup will make the confirmation to work both with "Maria, confirmar" which is the prefix + default confirmation phrase, and also should work with the phrases that are part of the new custom confirmation command.



I'm testing the new commands and I'll update the 1st post with the final version. I'll see if I can get an optional setting on the python script to make it try to identify similar phrases that have been spoken and then suggest the most likely match. I expect that the check for the prefix will be the key to avoid false positives.


Thanks for the help.

[jitterjames]

To avoid this custom confirmation command from being heard when I'm not in a situation that requires the custom confirmation, I put it on a disabled group. The group will be enabled only temporarily to allow for the custom confirmation. Not 100% sure however if this is effective or not because the command has "always on".

"Always On" only means that no prefix is required for this command even when you are in standby.  It does not mean that the command will be recognized when VC is off or if the Group is off.  There is no reason to have to guess about this.  There is a rollover on this checkbox which says "Execute command, even when in standby mode".  And in the wiki it says

Always On: If this box is checked, this command will execute even when you are in standby mode, and without the need for a prefix. An obvious use for this feature is for the "Pay attention" command, but you can use it for other commands if you wish.

http://voxcommando.com/mediawiki/index.php?title=Logical_Command_Builder