Recently, the Azure OpenAI service from Microsoft has become Generally Available. This is the service that gives you access to OpenAI large language models (LLMs), such as ChatGPT, to use with your own apps that you are building in Azure. One common growing concern with ChatGPT as a whole is data privacy because of users entering in sensitive information to the prompts. You can think of Azure OpenAI as a self-hosted instance of ChatGPT where you can train models based on your own business data and that data is not used to help train the public model. You can read more about the data privacy of Azure OpenAI here: Data, privacy, and security for Azure OpenAI Service – Azure Cognitive Services | Microsoft Learn
Microsoft also released a new project that I thought was very exciting in which they provide a sample of an application that can consume your own internal data, index it, and allow you to use a chat interface for users to query the data. You can find the project here.
The example they provide uploads company pdfs about the particular organizations HR benefit plans so that users can ask questions in a chat. The chat even makes citations about the content where it surfaced the information.
I spun up my own instance of the sample just to test it out. I ended up using their sample data and also injected some of my own to see how quickly it could respond and cite the new information. My findings were the following:
- The service is able to cite the benefit information of the sample data very accurately and can show you the evidence it searched through to provide its results
- After I added a new document to the dataset, it took about 1-2 minutes before it was able to cite the data. I did not have to restart/reboot the service or anything like that
- I uploaded the popular feature matrix: https://m365maps.com/ and I was able to get acuate answers when I was asking it questions about M365 licensing.
Can this tap into a SharePoint environment?
The short answer is yes. It can use SharePoint document libraries to surface answers across pdfs, csvs, word docs, etc. that you have stored. It currently cannot surface information about SharePoint Lists. Here is the full tutorial on how to set that up: SharePoint indexer (preview) – Azure Cognitive Search | Microsoft Learn
Additionally this service can support data from:
- Azure Storage
- Azure SQL
- Azure Cosmos DB
How much does it cost?
The resources that get spun up in Azure are as follows:
Using the Azure Calculator, I put in some fake estimates and saw the cost was roughly $132/month. There is an extreme disclaimer here that this all depends on usage. This include both how often users are leveraging the chat and how many documents you are scanning. As you can see below, the Form Recognizer has the highest cost where the pricing model is $50 per 1k pages scanned. You could switch to the free tier where the limit is 500 pages per month as a secondary option to help lower cost. Its likely, you would have more upfront cost with the initial ingestion.
As always, the best practice should be to spin up the service, add some documents, and test it with a pilot group of users to get a better sense of cost. I was able to spin up the service and spend less than $20 to do my testing.
ChatGPT As a Service?
This technology got my mind going on the possibilities for MSPs. Its very possible you could turn this into a service for customers if they have a lot of data in Azure or SharePoint Online. Being able to index and query their own internal documentation would be a gamechanger. From a pricing model, you could charge a flat fee if you believe the cost over time will be pretty consistent or you could just charge it completely based on usage with markup as you might do today for Azure Services. It would also be a great way to get a customer to start adopting services in Azure if they are not already. The benefits of the infrastructure would be seen at an individual user level, making it a very powerful introduction to Azure services.
The only hesitation I have for something like this is wondering if it will be baked into the new Microsoft 365 Co-pilot services we will see in the future. It seems like low hanging fruit to index SharePoint Online information for advanced search and chat. Other data sources like SQL DB’s and Azure Storage are still completely on the table though in my opinion.
Additional Resources
Microsoft Mechanics Video
My Video: