ONNX Chat

TutorialGenerate Code

The ai chat command allows you to interact w/ ONNX models from the command line.

Prerequisites

Before you begin, you'll need to install the ai CLI and set up an ONNX model.

Install the ai CLI
Setup ONNX and Phi-3 Model

Prompts

One prompt

ai chat --model-path @mp --user "What is the capital of France?"

Interactive chat

ai chat --model-path @mp --interactive

System prompts

ai chat --model-path @mp --interactive --system @prompt.txt

User and system prompts

ai chat --model-path @mp --interactive --system @prompt.txt --user "Tell me a joke"

Answers and chat history

Output answer to a file

ai chat --model-path @mp --interactive --output-answer answer.txt

Output chat history to a file

ai chat --model-path @mp --interactive --output-chat-history history.jsonl

Input chat history

ai chat --model-path @mp --interactive --input-chat-history history.jsonl

The ai dev new allows you to generate sample code that demonstrates how to use ONNX models.

Prerequisites

Before you begin, you'll need to install the ai CLI and set up an ONNX model.

Install the ai CLI
Setup ONNX and Phi-3 Model

C#GoJavaJavaScriptPython

List samples

List available templates

ai dev new list

AI - Azure AI CLI, Version 1.0.0
Copyright (c) 2024 Microsoft Corporation. All Rights Reserved.

Name                                                      Short Name                                       Language                        
------------------------------------------------------    ---------------------------------------------    --------------------------------
Environment Variables                                     .env                                                                             
Azure AI Inference Chat Completions (Streaming)           az-inference-chat-streaming                      C#, Python                      
Helper Function Class Library                             helper-functions                                 C#                              
OpenAI Assistants                                         openai-asst                                      C#, JavaScript, Python          
OpenAI Assistants (Streaming)                             openai-asst-streaming                            C#, JavaScript, Python          
OpenAI Assistants (w/ Code Interpreter)                   openai-asst-streaming-with-code                  C#, JavaScript, Python          
OpenAI Assistants (w/ File Search)                        openai-asst-streaming-with-file-search           C#, JavaScript, Python          
OpenAI Assistants (w/ Functions)                          openai-asst-streaming-with-functions             C#, JavaScript, Python          
OpenAI Assistants Webpage                                 openai-asst-webpage                              JavaScript, TypeScript          
OpenAI Assistants Webpage (w/ Functions)                  openai-asst-webpage-with-functions               JavaScript                      
OpenAI Chat Completions                                   openai-chat                                      C#, Go, Java, JavaScript, Python
OpenAI Chat Completions (Streaming)                       openai-chat-streaming                            C#, Go, Java, JavaScript, Python
OpenAI Chat Completions (w/ Data + AI Search)             openai-chat-streaming-with-data                  C#, Go, Java, JavaScript, Python
OpenAI Chat Completions (w/ Functions)                    openai-chat-streaming-with-functions             C#, Go, JavaScript, Python      
OpenAI Chat Webpage                                       openai-chat-webpage                              JavaScript, TypeScript          
OpenAI Chat Webpage (w/ Functions)                        openai-chat-webpage-with-functions               JavaScript, TypeScript          
OpenAI Chat Webpage (w/ Speech input/output)              openai-chat-webpage-with-speech                  TypeScript                      
OpenAI Chat Webpage (w/ Functions + Speech)               openai-chat-webpage-with-speech-and-functions    TypeScript                      
Phi-3 Chat Completions (w/ ONNX)                          phi3-onnx-chat-streaming                         C#                              
Phi-3 Chat Completions (w/ ONNX + Functions)              phi3-onnx-chat-streaming-with-functions          C#                              
Semantic Kernel Chat Completions (Streaming)              sk-chat-streaming                                C#                              
Semantic Kernel Chat Completions (w/ Data + AI Search)    sk-chat-streaming-with-data                      C#                              
Semantic Kernel Chat Completions (w/ Functions)           sk-chat-streaming-with-functions                 C#                              
Semantic Kernel Chat Completions (w/ Agents)              sk-chat-with-agents                              C#                              
Speech-to-text                                            speech-to-text                                   C#, Python                      
Speech-to-text (w/ Continuous recognition)                speech-to-text-continuous-reco                   C#, Python                      
Speech-to-text (w/ File input)                            speech-to-text-with-file                         C#, Python                      
Speech-to-text (w/ Keyword detection)                     speech-to-text-with-keyword                      C#, Python                      
Speech-to-text (w/ Translation)                           speech-to-text-with-translation                  C#, Python                      
Text-to-speech                                            text-to-speech                                   C#, Python                      
Text-to-speech (w/ File output)                           text-to-speech-with-file                         C#, Python

List only C# samples

ai dev new list --csharp

Filter the list by name

ai dev new list onnx --csharp

Generate, build, and run a sample

Generate sample code

ai dev new phi3-onnx-chat-streaming --csharp
cd phi3-onnx-chat-streaming-cs

See the code; learn how it works...

Program.cs
OnnxGenAIChatCompletionsStreamingClass.cs
get-phi3-mini-onnx.cmd
Phi3ChatStreaming.csproj

Deep dive on how it works

Install dependencies

dotnet restore

Get models

./get-phi3-mini-onnx.cmd

Run the sample

ai dev shell
dotnet run

List samples

List available templates

ai dev new list

AI - Azure AI CLI, Version 1.0.0
Copyright (c) 2024 Microsoft Corporation. All Rights Reserved.

Name                                                      Short Name                                       Language                        
------------------------------------------------------    ---------------------------------------------    --------------------------------
Environment Variables                                     .env                                                                             
Azure AI Inference Chat Completions (Streaming)           az-inference-chat-streaming                      C#, Python                      
Helper Function Class Library                             helper-functions                                 C#                              
OpenAI Assistants                                         openai-asst                                      C#, JavaScript, Python          
OpenAI Assistants (Streaming)                             openai-asst-streaming                            C#, JavaScript, Python          
OpenAI Assistants (w/ Code Interpreter)                   openai-asst-streaming-with-code                  C#, JavaScript, Python          
OpenAI Assistants (w/ File Search)                        openai-asst-streaming-with-file-search           C#, JavaScript, Python          
OpenAI Assistants (w/ Functions)                          openai-asst-streaming-with-functions             C#, JavaScript, Python          
OpenAI Assistants Webpage                                 openai-asst-webpage                              JavaScript, TypeScript          
OpenAI Assistants Webpage (w/ Functions)                  openai-asst-webpage-with-functions               JavaScript                      
OpenAI Chat Completions                                   openai-chat                                      C#, Go, Java, JavaScript, Python
OpenAI Chat Completions (Streaming)                       openai-chat-streaming                            C#, Go, Java, JavaScript, Python
OpenAI Chat Completions (w/ Data + AI Search)             openai-chat-streaming-with-data                  C#, Go, Java, JavaScript, Python
OpenAI Chat Completions (w/ Functions)                    openai-chat-streaming-with-functions             C#, Go, JavaScript, Python      
OpenAI Chat Webpage                                       openai-chat-webpage                              JavaScript, TypeScript          
OpenAI Chat Webpage (w/ Functions)                        openai-chat-webpage-with-functions               JavaScript, TypeScript          
OpenAI Chat Webpage (w/ Speech input/output)              openai-chat-webpage-with-speech                  TypeScript                      
OpenAI Chat Webpage (w/ Functions + Speech)               openai-chat-webpage-with-speech-and-functions    TypeScript                      
Phi-3 Chat Completions (w/ ONNX)                          phi3-onnx-chat-streaming                         C#                              
Phi-3 Chat Completions (w/ ONNX + Functions)              phi3-onnx-chat-streaming-with-functions          C#                              
Semantic Kernel Chat Completions (Streaming)              sk-chat-streaming                                C#                              
Semantic Kernel Chat Completions (w/ Data + AI Search)    sk-chat-streaming-with-data                      C#                              
Semantic Kernel Chat Completions (w/ Functions)           sk-chat-streaming-with-functions                 C#                              
Semantic Kernel Chat Completions (w/ Agents)              sk-chat-with-agents                              C#                              
Speech-to-text                                            speech-to-text                                   C#, Python                      
Speech-to-text (w/ Continuous recognition)                speech-to-text-continuous-reco                   C#, Python                      
Speech-to-text (w/ File input)                            speech-to-text-with-file                         C#, Python                      
Speech-to-text (w/ Keyword detection)                     speech-to-text-with-keyword                      C#, Python                      
Speech-to-text (w/ Translation)                           speech-to-text-with-translation                  C#, Python                      
Text-to-speech                                            text-to-speech                                   C#, Python                      
Text-to-speech (w/ File output)                           text-to-speech-with-file                         C#, Python

List only Go samples

ai dev new list --go

Filter the list by name

ai dev new list onnx --go

Generate, build, and run a sample

... 🚧 UNDER CONSTRUCTION ...

List samples

List available templates

ai dev new list

AI - Azure AI CLI, Version 1.0.0
Copyright (c) 2024 Microsoft Corporation. All Rights Reserved.

Name                                                      Short Name                                       Language                        
------------------------------------------------------    ---------------------------------------------    --------------------------------
Environment Variables                                     .env                                                                             
Azure AI Inference Chat Completions (Streaming)           az-inference-chat-streaming                      C#, Python                      
Helper Function Class Library                             helper-functions                                 C#                              
OpenAI Assistants                                         openai-asst                                      C#, JavaScript, Python          
OpenAI Assistants (Streaming)                             openai-asst-streaming                            C#, JavaScript, Python          
OpenAI Assistants (w/ Code Interpreter)                   openai-asst-streaming-with-code                  C#, JavaScript, Python          
OpenAI Assistants (w/ File Search)                        openai-asst-streaming-with-file-search           C#, JavaScript, Python          
OpenAI Assistants (w/ Functions)                          openai-asst-streaming-with-functions             C#, JavaScript, Python          
OpenAI Assistants Webpage                                 openai-asst-webpage                              JavaScript, TypeScript          
OpenAI Assistants Webpage (w/ Functions)                  openai-asst-webpage-with-functions               JavaScript                      
OpenAI Chat Completions                                   openai-chat                                      C#, Go, Java, JavaScript, Python
OpenAI Chat Completions (Streaming)                       openai-chat-streaming                            C#, Go, Java, JavaScript, Python
OpenAI Chat Completions (w/ Data + AI Search)             openai-chat-streaming-with-data                  C#, Go, Java, JavaScript, Python
OpenAI Chat Completions (w/ Functions)                    openai-chat-streaming-with-functions             C#, Go, JavaScript, Python      
OpenAI Chat Webpage                                       openai-chat-webpage                              JavaScript, TypeScript          
OpenAI Chat Webpage (w/ Functions)                        openai-chat-webpage-with-functions               JavaScript, TypeScript          
OpenAI Chat Webpage (w/ Speech input/output)              openai-chat-webpage-with-speech                  TypeScript                      
OpenAI Chat Webpage (w/ Functions + Speech)               openai-chat-webpage-with-speech-and-functions    TypeScript                      
Phi-3 Chat Completions (w/ ONNX)                          phi3-onnx-chat-streaming                         C#                              
Phi-3 Chat Completions (w/ ONNX + Functions)              phi3-onnx-chat-streaming-with-functions          C#                              
Semantic Kernel Chat Completions (Streaming)              sk-chat-streaming                                C#                              
Semantic Kernel Chat Completions (w/ Data + AI Search)    sk-chat-streaming-with-data                      C#                              
Semantic Kernel Chat Completions (w/ Functions)           sk-chat-streaming-with-functions                 C#                              
Semantic Kernel Chat Completions (w/ Agents)              sk-chat-with-agents                              C#                              
Speech-to-text                                            speech-to-text                                   C#, Python                      
Speech-to-text (w/ Continuous recognition)                speech-to-text-continuous-reco                   C#, Python                      
Speech-to-text (w/ File input)                            speech-to-text-with-file                         C#, Python                      
Speech-to-text (w/ Keyword detection)                     speech-to-text-with-keyword                      C#, Python                      
Speech-to-text (w/ Translation)                           speech-to-text-with-translation                  C#, Python                      
Text-to-speech                                            text-to-speech                                   C#, Python                      
Text-to-speech (w/ File output)                           text-to-speech-with-file                         C#, Python

List only Java samples

ai dev new list --java

Filter the list by name

ai dev new list onnx --java

Generate, build, and run a sample

... 🚧 UNDER CONSTRUCTION ...

List samples

List available templates

ai dev new list

AI - Azure AI CLI, Version 1.0.0
Copyright (c) 2024 Microsoft Corporation. All Rights Reserved.

Name                                                      Short Name                                       Language                        
------------------------------------------------------    ---------------------------------------------    --------------------------------
Environment Variables                                     .env                                                                             
Azure AI Inference Chat Completions (Streaming)           az-inference-chat-streaming                      C#, Python                      
Helper Function Class Library                             helper-functions                                 C#                              
OpenAI Assistants                                         openai-asst                                      C#, JavaScript, Python          
OpenAI Assistants (Streaming)                             openai-asst-streaming                            C#, JavaScript, Python          
OpenAI Assistants (w/ Code Interpreter)                   openai-asst-streaming-with-code                  C#, JavaScript, Python          
OpenAI Assistants (w/ File Search)                        openai-asst-streaming-with-file-search           C#, JavaScript, Python          
OpenAI Assistants (w/ Functions)                          openai-asst-streaming-with-functions             C#, JavaScript, Python          
OpenAI Assistants Webpage                                 openai-asst-webpage                              JavaScript, TypeScript          
OpenAI Assistants Webpage (w/ Functions)                  openai-asst-webpage-with-functions               JavaScript                      
OpenAI Chat Completions                                   openai-chat                                      C#, Go, Java, JavaScript, Python
OpenAI Chat Completions (Streaming)                       openai-chat-streaming                            C#, Go, Java, JavaScript, Python
OpenAI Chat Completions (w/ Data + AI Search)             openai-chat-streaming-with-data                  C#, Go, Java, JavaScript, Python
OpenAI Chat Completions (w/ Functions)                    openai-chat-streaming-with-functions             C#, Go, JavaScript, Python      
OpenAI Chat Webpage                                       openai-chat-webpage                              JavaScript, TypeScript          
OpenAI Chat Webpage (w/ Functions)                        openai-chat-webpage-with-functions               JavaScript, TypeScript          
OpenAI Chat Webpage (w/ Speech input/output)              openai-chat-webpage-with-speech                  TypeScript                      
OpenAI Chat Webpage (w/ Functions + Speech)               openai-chat-webpage-with-speech-and-functions    TypeScript                      
Phi-3 Chat Completions (w/ ONNX)                          phi3-onnx-chat-streaming                         C#                              
Phi-3 Chat Completions (w/ ONNX + Functions)              phi3-onnx-chat-streaming-with-functions          C#                              
Semantic Kernel Chat Completions (Streaming)              sk-chat-streaming                                C#                              
Semantic Kernel Chat Completions (w/ Data + AI Search)    sk-chat-streaming-with-data                      C#                              
Semantic Kernel Chat Completions (w/ Functions)           sk-chat-streaming-with-functions                 C#                              
Semantic Kernel Chat Completions (w/ Agents)              sk-chat-with-agents                              C#                              
Speech-to-text                                            speech-to-text                                   C#, Python                      
Speech-to-text (w/ Continuous recognition)                speech-to-text-continuous-reco                   C#, Python                      
Speech-to-text (w/ File input)                            speech-to-text-with-file                         C#, Python                      
Speech-to-text (w/ Keyword detection)                     speech-to-text-with-keyword                      C#, Python                      
Speech-to-text (w/ Translation)                           speech-to-text-with-translation                  C#, Python                      
Text-to-speech                                            text-to-speech                                   C#, Python                      
Text-to-speech (w/ File output)                           text-to-speech-with-file                         C#, Python

List only JavaScript samples

ai dev new list --javascript

Filter the list by name

ai dev new list onnx --javascript

Generate, build, and run a sample

... 🚧 UNDER CONSTRUCTION ...

List samples

List available templates

ai dev new list

AI - Azure AI CLI, Version 1.0.0
Copyright (c) 2024 Microsoft Corporation. All Rights Reserved.

Name                                                      Short Name                                       Language                        
------------------------------------------------------    ---------------------------------------------    --------------------------------
Environment Variables                                     .env                                                                             
Azure AI Inference Chat Completions (Streaming)           az-inference-chat-streaming                      C#, Python                      
Helper Function Class Library                             helper-functions                                 C#                              
OpenAI Assistants                                         openai-asst                                      C#, JavaScript, Python          
OpenAI Assistants (Streaming)                             openai-asst-streaming                            C#, JavaScript, Python          
OpenAI Assistants (w/ Code Interpreter)                   openai-asst-streaming-with-code                  C#, JavaScript, Python          
OpenAI Assistants (w/ File Search)                        openai-asst-streaming-with-file-search           C#, JavaScript, Python          
OpenAI Assistants (w/ Functions)                          openai-asst-streaming-with-functions             C#, JavaScript, Python          
OpenAI Assistants Webpage                                 openai-asst-webpage                              JavaScript, TypeScript          
OpenAI Assistants Webpage (w/ Functions)                  openai-asst-webpage-with-functions               JavaScript                      
OpenAI Chat Completions                                   openai-chat                                      C#, Go, Java, JavaScript, Python
OpenAI Chat Completions (Streaming)                       openai-chat-streaming                            C#, Go, Java, JavaScript, Python
OpenAI Chat Completions (w/ Data + AI Search)             openai-chat-streaming-with-data                  C#, Go, Java, JavaScript, Python
OpenAI Chat Completions (w/ Functions)                    openai-chat-streaming-with-functions             C#, Go, JavaScript, Python      
OpenAI Chat Webpage                                       openai-chat-webpage                              JavaScript, TypeScript          
OpenAI Chat Webpage (w/ Functions)                        openai-chat-webpage-with-functions               JavaScript, TypeScript          
OpenAI Chat Webpage (w/ Speech input/output)              openai-chat-webpage-with-speech                  TypeScript                      
OpenAI Chat Webpage (w/ Functions + Speech)               openai-chat-webpage-with-speech-and-functions    TypeScript                      
Phi-3 Chat Completions (w/ ONNX)                          phi3-onnx-chat-streaming                         C#                              
Phi-3 Chat Completions (w/ ONNX + Functions)              phi3-onnx-chat-streaming-with-functions          C#                              
Semantic Kernel Chat Completions (Streaming)              sk-chat-streaming                                C#                              
Semantic Kernel Chat Completions (w/ Data + AI Search)    sk-chat-streaming-with-data                      C#                              
Semantic Kernel Chat Completions (w/ Functions)           sk-chat-streaming-with-functions                 C#                              
Semantic Kernel Chat Completions (w/ Agents)              sk-chat-with-agents                              C#                              
Speech-to-text                                            speech-to-text                                   C#, Python                      
Speech-to-text (w/ Continuous recognition)                speech-to-text-continuous-reco                   C#, Python                      
Speech-to-text (w/ File input)                            speech-to-text-with-file                         C#, Python                      
Speech-to-text (w/ Keyword detection)                     speech-to-text-with-keyword                      C#, Python                      
Speech-to-text (w/ Translation)                           speech-to-text-with-translation                  C#, Python                      
Text-to-speech                                            text-to-speech                                   C#, Python                      
Text-to-speech (w/ File output)                           text-to-speech-with-file                         C#, Python

List only Python samples

ai dev new list --python

Filter the list by name

ai dev new list onnx --python

Generate, build, and run a sample

... 🚧 UNDER CONSTRUCTION ...