Finding a Good Partner to Build Your Next Generative AI Project

When you're looking to partner with a professional services firm to build a generative AI solution, selecting one with the right mix of skills and approaches is essential. With the rapid pace of AI advancement, it's easy to get lost in technical jargon and complex proposals. By focusing on a few key areas and asking targeted questions, you can identify a firm that understands both the technology and your business goals.

Here is a breakdown of the questions we consider essential to guide your decision-making process.

Balancing Skills

What to consider: Data scientists are skilled with statistics and forecasting, but generative AI projects often place more emphasis on traditional app development skills, particularly where solutions involve documents and process automation rather than numerical analysis. From time to time we've seen data scientists engaged unnecessarily to build apps containing generative AI, so it's worth keeping this in mind.

What to ask: "How does your team combine data science know-how with app development skills? Can you give examples of past projects where both were needed and used effectively? For each role on the project, help me understand what value they bring."

Development Stages

What to consider: It's important to know how your project will move through development, testing, and production, especially as test data used to build search indexes may not be the complete corpus of information you ultimately want to draw from.

What to ask: "Can you walk me through how you handle development, testing, and production environments? What steps do you take to ensure quality at each stage?"

Training and Testing Prompts

What to consider: You want the AI to respond well, so effective prompt instructions and testing are key.

What to ask: "How do you train and test prompts? Do you use AI to evaluate responses? Can you give examples of how this works for chatbots and other tasks like document processing?"

Model Cost and Performance

What to consider: Balancing cost and performance is important. A practical approach is to start with the most capable model (often the most expensive), and then once the solution is proven, test with less expensive models to see how they perform. This is worth the effort when run costs are significant enough to justify optimisation time.

What to ask: "How do you figure out which AI models give the best performance for the price? Once a model is proven, how do you optimise costs?"

Solution Architecture

What to consider: Every component in the architecture should be necessary and justified. There is a lot of hype in this space, and analytics vendors are frequently recommended as a gateway to the LLM, often unnecessarily. This tends to come up more frequently from firms experienced in "traditional AI" solutions such as machine learning, compared with those that have traditionally specialised in app development. For more on how Hypergen approaches architecture decisions, see our Why Us page.

What to ask: "Can you explain why each part of your proposed architecture is needed? If you're suggesting an analytics platform needs to be included in the design, why is it essential for us?"

Low Code vs. Full Code

What to consider: Different projects call for different approaches, and you want one that fits your situation, balancing capability, supportability, and time to go-live. This is something we explore in depth during our AI Accelerate Workshop, where we help teams identify the right approach for their specific use cases.

What to ask: "How do you decide if a low-code or full-code solution is right for us? What are the pros and cons of each in our case?"

Continual Improvement and Tuning

What to consider: Feedback is vital for making generative AI solutions better over time, especially with chatbots, and also other AI applications to a varying extent.

What to ask: "How do you collect and use user feedback to improve chatbot responses? What metrics do you track to see how well the chatbot is performing?"

Testing Chatbots

What to consider: Testing should match your success criteria. In the case of chatbots, it is common for users to ask questions post-launch that were not considered or optimised for during solution design.

What to ask: "How do you make sure your testing aligns with what success looks like for us? Can you show examples of the prompts and test cases you'll use?"

Updating Search Indexes

What to consider: Keeping the search index updated is important in any application that relies heavily on search (RAG architecture). Stale or incomplete indexes lead to poor AI responses, so it's worth understanding how your vendor handles this.

What to ask: "How do you handle updates to the search index when content changes? How do you decide what content gets indexed? Is there a method to automate this re-indexing, and remove content that is no longer relevant or out of date?"

Managing Permissions

What to consider: Protecting sensitive content is non-negotiable in most situations. Any AI solution that surfaces organisational data needs robust permission controls to ensure the right people see the right information.

What to ask: "How do you manage permissions to ensure only authorised users can access sensitive content? What security measures are in place to protect data?"

Bringing It Together

By asking these kinds of questions, you can dig into whether a firm is the right fit for your project and whether they are equipped to deliver what you are looking for. The goal is not to test the vendor, but to understand how they think, how they work, and whether their approach aligns with your priorities.

At Hypergen, we work through these considerations with our clients from the outset. If you'd like to see examples of how we've approached real AI projects, take a look at our project portfolio.