AI-Powered Knowledge Base for Product Managers

Tweet                                                                            

AI-Powered Knowledge Base for Product Managers

Author: Rick Hightower

This article originally appeared on August 7, 2023 on LinkedIn.

Building an AI-powered Knowledge Base for Product Managers

An IBM study shows product managers are early adopters of generative AI. They rank in the top-ten professions that use AI. The report states that 21% of product managers use AI daily. Product Managers are leading the AI charge.

As AI’s role in product management increases, product managers must learn how to use AI to stay competitive. Product managers using AI differ from their traditional counterparts by applying their technical expertise to harness AI’s potential in enhancing product management processes.

Let’s cover some practical ways to use generative AI in your product management role.

How AI is Revolutionizing Product Management

As technology advances, industries find new ways to incorporate AI into their processes, including product management. Companies succeed or fail based on effective product management. Product management is essential to any successful SaaS company or software product company.

The challenge is consolidating real-time product information across numerous tools and sources. Let’s explore the struggles and challenges of using various tools. Let’s also examine how AI revolutionizes product management to enhance collaboration, decision-making, and overall product quality.

Challenges in getting the correct contextual information to inform decision making

Product managers need to find relevant information to make key decisions. The problem is that the product data might be spread across multiple portals, documents, and systems. This results in lower productivity or poor choices. The required knowledge is not at your fingertips.

Consolidating real-time information across tools like Figma, ServiceNow, Slack, JIRA, GitHub (PRs, releases), CloudWatch (logs, metrics), DataDog, customer metrics, project plans, WIP tracking, business analyst spreadsheets, market data, product data, and customer feedback channels is a daunting task for any product manager. Challenges in finding information can affect product quality and timelines. Then if you do collect all of the data, the next challenge is finding the right product data to make informed decisions.

Having a wealth of knowledge and information is worth little if you can’t find it when needed. It has to be accessible, discoverable, and up-to-date. Having an abundance of information is only valuable with recency. Old data can be as helpful as drinking spoiled milk. Having the data and being unable to find and use it is also useless. Getting data from multiple sources to make informed product decisions is often manual and time-consuming. Correlating data across disparate systems is also tricky.

AI Knowledge Base Solutions

One solution to these challenges is generative AI. Large language models (LLMs) can be fine-tuned for domain-specific text, allowing for deeper understanding. Semantic search with embeddings offers agility and scalability. Semantic search lets product managers easily access essential information with a simple search. Now it is easy to find that needle in the haystack answer to inform product decisions quickly.

Continuous knowledge management is also crucial for quality insights. This practice allows AI to provide relevant insights continually. The data must be collected and groomed. Generative AI is only as good as the contextual information you can give. You must groom the knowledge base for AI and refine and build the knowledge base with AI.

Main AI Options

Generative AI tools like ChatGPT, Bard, Perplexity, Poe, and Anthropic, can be used for generation, classification, extraction, and summarization, all of which are essential components of product management. AI can streamline product development, enhance collaboration, and improve decision-making in SaaS and software product companies. All tech companies, really, and these days companies are not using technology for logistics, planning, operations, automation, data analysis, etc.

The main AI options for product management are fine-tuning LLMs, semantic search, AI-enhanced search, and smart agents:

• Fine-tuning LLMs can be effective but may have scalability concerns.
• Semantic search with embeddings is more flexible but requires well-maintained knowledge bases for specialized understanding and semantic search with text embeddings.
• AI-enhanced search, using text embeddings and HyDE, can help to comb through tons of data and answer direct questions from multiple documents in various systems.
• Smart agents that specialize in a specific task

• uses vector lookup mix-ins to offer workflow context and look up related documentation spread across many different systems

• prompt engineering automation, a chain of thought, tree of thought, and synthetic prompts can customize AI for task/use cases for best results.

• LLMs can only provide additional insights through additional context.

• Vector mix-ins

• Trained with synthetic textbooks or high-quality data mixed in based on the workflow context

With generative AI, the key to getting good answers is the right contextual knowledge. The correct contextual information means good quality data related to the task at hand–focused data. The main AI options for product management include fine-tuning LLMs for specialized understanding, but scalability concerns may arise, and the knowledge of how to do this remains very specialized. Semantic Search with Embeddings is flexible, but they rely on well-maintained knowledge bases. The quality and recency of the data are essential to get the best answers—garbage in equals garbage out. Maintaining a high-quality knowledge base requires rigorous data hygiene and frequent scraping, scrubbing, and ingestion.

Using AI-enhanced search (e.g., Embeddings, HyDE) allows you to comb through many data sources and answer direct questions spread across multiple documents in various systems. Think of it as English as the “new programming language” you ask, and it goes and finds it.

It can answer particular questions: “Who won the Super Bowl 1966? Who won the MVP? Give me the teams involved and which team won the coin toss?” As long as you provide the proper context to the LLMs, it can provide additional insights. Smart Agents with Vector lookup mix-ins offer the most relevant parts of workflow for context and get specially tasked via prompt engineering automation, chain of thought, tree of thought, synthetic prompts, and customizing AI for task/use cases for best results.

Smart agents powered by LLMs and Idea databases allow you to be the Tony Stark of product managers, but just J.A.R.V.I.S aspect, no hand-rocket propulsion.

Understanding AI Enhanced Search, Vector Databases, and Text Embeddings

AI-enhanced search is a form of semantic search that relies heavily on looking at a multidimensional vector called text embedding. Think of text embedding as an idea key. With this idea key, you can look at parts of a conversation or document, in the case of a chatbot, or some other workflow, in the case of an agent. Then the agent can see which parts of this workflow or conversation are the most relevant to the current step in the workflow and mix that content in with the instruction to the LLM. A vector database is an idea database that does not look up things by keys or queries but by taking the text embedding, which is an idea key, and looking up other ideas.

The first step in an AI-enhanced search would be to take the question and ask the generative AI (LLM) to create what an ideal answer would be like (HyDE), or perhaps several variations of an ideal answer. Then, it will take that same question and create many queries to downstream search engines or specialized databases. It can also use idea databases (vector databases). Or it can even take the English question and form a series of queries and searches. Then it takes these searches that it just generates and uses them to perform searches or queries in various downstream systems.

Then it compares the text embedding (idea key) you made for the ideal answer (or answers) and sorts the results from the searches based on how closely it matches the ideal answer (called HyDE). Then it gives you an answer by combining excerpts from several documents found in various systems.

It quickly automates the search generation, collection, sorting, and quickly sifting through vast amounts of data. What could take you hours or days to find through searching each system now takes mere minutes or seconds. This lets you use plain English and get complex answers from many downstream systems. This data can be up to the second if needed.

Idea databases (Vector databases) are staples of creating smart agents and any advanced prompt engineering. Remember, the key to using LLMs is context—having the correct contextual data for the task at hand.

An intelligent agent can act in specific roles. Perhaps the agent acts as a UX expert who knows all of your style guides or a sports fanatic who knows all the team stats for fantasy football. Imagine creating high-quality virtual textbooks on performing complex tasks with sample input and output and a wealth of role-based knowledge that you can seed that agent with. Then seed the agent with role-level instructions. When the agent hits a task, it can decide which parts of the conversation or workflow are the most important (using the same idea key and idea comparison above). It then loads the task-related contextual data from the virtual textbooks (or other sources using an AI-enhanced search) which have been stored in your idea database. It generates an idea key for the task at hand, pulls the parts of the workflow that match, and loads enriched specialized contextual information from the idea database. Thus for each step, it is always passing the right context to the LLM to perform the next task.

To make the agents even more brilliant, you can provide feedback loops to give them a score on their current task or even have them create multiple solutions, follow various paths, score all of the outcomes, and pick the highest score from the ones that did not fail. This is approaching Tree of Thought levels. With minimal feedback, proper prompt engineering, and mixing in the right ideas with a feedback loop, I have seen the accuracy of complex tasks move from about 70% to 95% plus. Using this technique, I have automated complex tasks that would take a person months (and more recently years) to do that can be done in mere hours. The accuracy can increase with more time and better training and feedback loops.

Conclusion

In conclusion, successful product management is the key to product development and continuous improvements and has tremendous ROI. The role of the product manager continues to become more complex. The abilities of the product manager are only enhanced through AI integration. It is already essential for real-time access to information. Careful consideration of approaches and rigorous best practices must be maintained to ensure that AI provides value and improves the overall quality of the product. For technology companies and companies that rely on technology, implementing AI for product management is a give. It is now a competitive advantage. Later product managers and companies that don’t adopt it will be left behind. Generative AI continues to shape the future of the industry. As a CIO, CTO, VP of Product Management, Director of Product Management, or Product Manager, staying ahead of the curve and incorporating AI into your product management process is essential.

Product managers are increasingly incorporating AI systems into their work. The new trend is that Product Managers using AI can also assist product teams in capitalizing on AI’s commercial potential. Some question if all product managers will one day be AI product managers. AI Product managers differ from their traditional counterparts by leveraging their technical expertise to harness AI’s potential in enhancing product management processes.

Follow-up and references

Product management teams often face challenges in consolidating real-time information across various tools such as Figma, ServiceNow, Slack, JIRA, GitHub, CloudWatch, DataDog, custom metrics, project plans, WIP tracking, and customer feedback channels. These challenges can affect product quality and timelines. AI solutions can help address these issues and streamline product development, enhance collaboration, and improve decision-making. Some AI solutions and insights include:

-Large language models and semantic search:These technologies can improve accessibility to information by understanding natural language queries and providing relevant results. -Fine-tuning on domain-specific text:This approach allows AI models to better understand the specific domain, leading to more accurate and relevant search results. -Semantic search with embeddings:This method enables agility and scalability by representing text in a high-dimensional space, allowing for efficient similarity calculations between queries and documents 1. -Continuous knowledge management:Maintaining up-to-date and well-organized knowledge bases is crucial for providing quality insights and ensuring the effectiveness of AI solutions.

There are two main AI options for product management teams:

-Fine-tuning LLMs:Large language models can be fine-tuned for specialized understanding, but they may have scalability concerns. -Semantic Search with Embeddings:This approach offers flexibility and requires well-maintained knowledge bases for optimal performance.

Product managers are increasingly incorporating AI systems into their work. The new trend is that Product Managers using AI can also assist product teams in capitalizing on AI’s commercial potential. Some question if all product managers will one day be AI product managers.

Resources

Products and companies that can assist product managers in enhancing their decision-making, collaboration, and overall efficiency include:

-Product Launch AI:(This AI-assisted tool) provides practice scenarios for product management, helping product managers refine their skills and anticipate real-world situations. -ClickUp:(ClickUp) integrates AI to enhance project management and productivity, aiding product managers in streamlining workflows and fostering collaboration. -Jam:(Jam) utilizes AI to enhance communication and collaboration within product management teams. -ChatGPT:(ChatGPT), with code interpreter capabilities, assists in generating human-like text and analyzing spreadsheets, aiding in tasks like report writing or data insights discovery. -Canva:(Canva) leverages AI to enable product managers to create visually appealing marketing materials with ease. -TLDV:TLDV offers AI-powered video summarization, helping product managers comprehend lengthy videos such as meetings efficiently. -Notion:Notion employs AI for note-taking and collaboration, enhancing organization and task management.

• **Otter.AI:**Otter.AI is an AI-driven transcription service, aiding in the documentation of meetings and interviews. -Collato:Collato incorporates AI features to facilitate project management and collaboration.

Follow-up articles from the author

• Java Open AI Client
• Using ChatGpt embeddings and Hyde to improve search results
• Anthropics Claude Chatbot Gets Upgrade
• Elon Musks XAi’s new frontier for artificial intelligence
• Using Mockito to test JAI Java Open AI Client
• Fine-tuning journey with Open AI API
• Using Open AI to create callback functions, the basis for plugins
• Using Java Open AI Client Async
• Fastest Java JSON Parser
• Java Open AI API Client on Github
• Medium: Introducing Java Open AI Client
• Medium: Using ChatGPT, Embeddings, and HyDE to Improve Search Results
• Chain of Thought
• Synthetic Prompts

About the Author

Rick Hightower is a seasoned technology leader and innovator with over two decades of experience in software development and AI integration. Rick has been at the forefront of developing cutting-edge AI solutions for enterprise clients.

Rick is also known for his contributions to open-source projects, including the development of JAI (Java AI Open API Client), which simplifies the integration of OpenAI’s API into Java applications. His expertise spans across various AI domains, including natural language processing, machine learning, and cognitive computing.

A frequent speaker at technology conferences and a prolific writer, Rick shares his insights on AI trends, product management, and software development through his articles on LinkedIn and Medium. His passion for AI and its practical applications in business continues to drive innovation in the field of artificial intelligence.

Recent Consulting ExperienceAI Documentation and Analysis Tool (2023-2024)• Developed an AI system using advanced APIs for generating documentation and analyzing sensitive documents.

• Created entity extraction and classification tools utilizing Chain of Thought and synthetic prompts for legal document use cases. • Implemented advanced techniques for AI-enhanced search using Text Embeddings and vector sorting. • Established vector databases for analyzing code bases and product documentation to extract business rules. • Engineered a tool with a feedback validation loop, improving output accuracy from 70% to 90% for complex diagram generation.AI-based Subject Matter Expert System (Dec 2023 - March 2024)• Implemented an AI-based SME system using various LLMs, Vector Databases, and frameworks. • Developed a full-stack solution with React frontend, Go middleware, and Python-based RAG Agent LLM layer. • Deployed the system on a major cloud platform, integrating various services for document indexing and code repository analysis. • Led the transition from a Flask-based RAG system to GRPC and cloud-based pub/sub for enhanced scalability.Legal AI Startup (June 2023 - August 2023)• Utilized AI and prompt engineering for efficient legal document evaluation and entity extraction. • Created a demo showcasing automated processes for investors and potential clients.

Tweet                                                                            

comments powered by Disqus