Azure Inference Chat Streaming in Python

This page was automatically generated by AI; not yet reviewed for accuracy...

The content and code samples on this page were generated by using the ai CLI with customized prompts in this repository.

It's cool, but, it's experimental. 😁

Please review the content and code before using it in your application.

This sample demonstrates how to use the Azure AI Inference Chat API with streaming in a Python console application.

main.py
requirements.txt

How to generate this sample

Command

ai dev new az-inference-chat-streaming --python

Output

AI - Azure AI CLI, Version 1.0.0
Copyright (c) 2024 Microsoft Corporation. All Rights Reserved.

This PUBLIC PREVIEW version may change at any time.
See: https://aka.ms/azure-ai-cli-public-preview

Generating 'az-inference-chat-streaming' in 'az-inference-chat-streaming-py' (3 files)...

main.py
requirements.txt

Generating 'az-inference-chat-streaming' in 'az-inference-chat-streaming-py' (2 files)... DONE!

main.py

STEP 1: Import required libraries and modules.

main.py

from azureml_chat_completions_streaming import AzureAIInferenceChatCompletionsStreaming
import os
import sys

STEP 2: Define the main function that reads the configuration settings from environment variables.

main.py

def main():
    chat_api_key = os.getenv("AZURE_AI_CHAT_API_KEY", '<insert your Azure AI Inference API key here>')
    chat_endpoint = os.getenv("AZURE_AI_CHAT_ENDPOINT", '<insert your Azure AI Inference endpoint here>')
    chat_model = os.getenv('AZURE_AI_CHAT_MODEL', '')
    chat_system_prompt = os.getenv('SYSTEM_PROMPT', 'You are a helpful AI assistant.')

STEP 3: Validate the configuration settings.

main.py

    ok = all([chat_api_key, chat_endpoint, chat_system_prompt]) and \
         all([not s.startswith('<insert') for s in [chat_api_key, chat_endpoint, chat_system_prompt]])
    if not ok:
        print(
            'To use Azure AI Chat Streaming, set the following environment variables:' +
            '\n- AZURE_AI_CHAT_API_KEY' +
            '\n- AZURE_AI_CHAT_ENDPOINT' +
            '\n- AZURE_AI_CHAT_MODEL (optional)' +
            '\n- SYSTEM_PROMPT (optional)')
        sys.exit(1)

STEP 4: Initialize the AzureAIInferenceChatCompletionsStreaming class with the configuration settings.

main.py

    chat = AzureAIInferenceChatCompletionsStreaming(chat_endpoint, chat_api_key, chat_model, chat_system_prompt)

STEP 5: Implement an input loop to obtain user input.

main.py

    while True:
        user_input = input('User: ')
        if user_input == 'exit' or user_input == '':
            break

STEP 6: Use the helper class to get the assistant's response and display responses as they are received.

main.py

        print('\nAssistant: ', end='')
        response = chat.get_chat_completions(user_input, lambda content: print(content, end=''))
        print('\n')