AWS
Generative AI powered Intelligent Document Processing
Steps to use Amazon Bedrock to get process Documents.
How to Design a Proper GenAI Proof of Concept (PoC)
Our VP of Engineering outlines the questions a properly designed proof of concept (PoC) needs to answer, particularly for complex technologies like Generative AI.
Nathan Wilkerson, VP of Engineering
Apr 24, 2024
A common issue we’ve seen working with large and small enterprises, particularly as they look to test Generative AI use cases, is the poorly designed PoC. Often this is a result of misaligned expectations around what a PoC should and should not do. Often a PoC is considered a beta version of an application, which means the product teams are expecting that they can take a PoC to production with minimal additional development.
As a result, the team doing the PoC build will spend months testing different foundational models for the highest level of incremental accuracy, or else optimizing data pipelines for cost and performance. The more time passes, the more momentum is lost and the needs of the business change before you have even had a chance to validate if doing a full application buildout makes sense.
What we propose is that a PoC should not be approached as a not-quite-ready version of a finished application. Rather it should be a quick validation test around two key areas - technical validation and business validation:
WHAT DOES A PROPER POC ANSWER?
|
Technical validation (can we do it?) |
Business validation (is it worth doing?) |
|
Do we have the right data? If not, can we get it? |
Is the output of the model better than whatever we use now? |
|
Do we know how to get the data from the source to where the inferencing/decisioning happens? |
Can the business actually use that output? |
|
Can we deliver the output to downstream applications and users that need it? |
Does this provide a rough idea of what the costs will be to run this application? |
|
Do we have the right internal resources to maintain this application (e.g., data engineers and data scientists)? |
Are these costs less than the expected incremental revenues/cost savings? |
When we build PoCs for our clients, we set up secured sandboxes in their AWS cloud environment to ensure no secure data leaks out during testing, but we are not spending a long time recreating the production environment. Rather than fine tuning models from scratch, we leverage pre-built AWS APIs for GenAI that often deliver near or even better accuracy at a much lower price.
Once we have technical and business validation, we then take those learnings to build production applications with highly performant and reliable data pipelines and carefully analyze what frameworks or model libraries make the most sense to finetune given technical, business, and regulatory constraints.
If your business is looking to test GenAI use cases, reach out to us. As AWS-certified experts in machine learning and GenAI, we can help your product teams move faster than ever from idea, to proof of concept, to production.