We've taken a week or so off.
Hope everybody had a good Thanksgiving.
We just got back.
We're kind of ready to hit the ground running
for the last like two weeks,
two, three weeks of like full weeks of December.
So today we have our lead developer, Luchen,
who's coming all the way from Montreal, Canada
and excited to have him on here.
He's gonna be talking about
how we do our automated test workflow using some AI bots.
And I'm gonna turn it over to him.
And then we may do some follow-up questions,
but he's got a little presentation to walk through.
And yeah, I'm excited to hear about it.
So over to you, Luchen.
Hey, what's up guys?
So I was developing this.
This is gonna be a very like technical,
like we're gonna go pretty deep in this.
So I was developing this AI chat bot for a client
and they have like fairly complex workflow.
We use like rag.
We have like an ingestion pipeline for their documents
and then they're able to ask questions
about their documents.
So developing like the prompts and the whole like AI agent,
the whole flow was like,
we would have to like manually test it.
So that was a bit tedious.
And I was thinking of,
we have like our like tried and true API test,
like framework and like our whole way of doing things.
So I was thinking, how can I reuse the lessons learned
from it to put in place like a testing approach
for AI chat bots and save myself some time and headache
and not have to do as much like manual flow.
So this is more like a backend oriented presentation.
So let's go through it.
Do you know?
So why this was hard?
Like it's really about like reducing friction
and removing as much like manual testing as we can.
So it's quite complex to like upload like files
and we'd have to think in advance of how to,
like which cases we want to test for.
Like some documents are hard to read
or maybe they contain irrelevant info.
So thinking in advance of like kind of this setup
for each test case and having to do over and over
and kind of keeping it in working memory
is a bit time consuming.
And then long feedback loop,
like all of the manual steps I would have to take
and then every time that we like test manually
we might miss edge cases
and not go through all of the cases.
So we might kind of break something
when we modify the prompt or something like that.
And the LLM responses vary.
So sometimes you're not really sure,
like, is this a good response?
So this is kind of the challenges going into it
and like how our feature works.
I kind of gave you a overview.
Users, they upload some medical documents
like a doctor visit, like doctor notes or things like that.
And then we have like another user that asks questions
about their documents.
Maybe they want to get like, I don't know,
a quick overview of like the most severe conditions
or something like that.
And then the LLM reads files content and answers queries.
So our backend has like tool calling and connections.
So we had like some chatbots
that they're just about like taking an input
and then doing an action on it.
But this, since it has like a more complex flow
that also goes into a consideration
like how we're gonna test this.
And we have like a capability here,
like file selection that a user can like choose
specific files to ask questions about.
That's kind of like a special feature
that we're gonna be testing in our backend.
And so in our normal API tests,
we try to get as much, as close as we can
to like the real behavior.
So we use like a real database,
but it's like a separate database.
It's like a test database that gets wiped
in between each run,
but it's a real database.
So if we write like bad SQL queries,
then we're gonna catch that in our test database,
which you wouldn't get if you would mock
like in some teams, in some approaches, they do that.
So the thing is about being pragmatic
about like what we're gonna mock
and what we're gonna let...
um let be like the real back-end behavior so
we i already put in place like um a test for the data ingestion pipeline
so i already know that it's like writing to the database in a specific table
like the file content takes um certain like pdf files and then parses
them and for each page it takes the page content and that
becomes like a row in the database and then another row
for the next page etc so the ingestion
piece is already tested so there's no need to like include it
to be like kind of end-to-end we we're gonna go with a starting point that like
the database is already pre-populated with
um file contents and we have control over
what files and what's their content so that way we can
start designing test cases um
then we make real api calls through the controller so we
actually let it um call open ai and then we capture the
llm responses and then we we put it in a structured file
so that we can review each test case like
what was the setup the scenario and then what was the actual
response so in the setup like the
files that we start off with in the knowledge base
we kind of hard code this and so we started with like hard coding
two um so simulate as if they already went
through the ingestion pipeline uh we have full control over the contents
and then if we want we can push it further but i'm just going to show you
like a basic one um in our case like we're adding two
files and then we choose to ask a question about only
one file and then we assert that um the second
file is not included in the response so that the
tool calling and like everything internally is hooked up properly
and that way we get the maximum confidence that our api is like is
working properly so to make this real api call we
we actually call the like chat controller kind of manually
and it goes through the real logic it calls the
the llm with the real file contents and so it's kind of like the complete
flow uh for that like the chat api
and then once we get that back we uh we just like create like a markdown
file that says like what it was uh the expected behavior
like the second file content shouldn't be there the first file content should
be there uh the actual response that we got from
the llm like the real uh open ai call and then what we can
check for manually and so this makes us like
way faster to test and we can add like other cases
uh with full control and confidence that like
our api is working properly and we can like repeat it as much as we
want like when we tweak prompts we can see the effect almost in real
time and it's the real behavior and it's like
real readable to me what's important in writing
test is that it's like readable kind of like
like plain english like a story you know we started off with
two medical files and we chose to read one and then we got
this output and then we can evaluate if it's if it makes sense
so file selection feature uh kind of went already through this
current state so we have like multiple test scenarios
um and another thing that i didn't put in the slides that
we actually skipped this from continuous integration because
since this test makes like a real api call it's going to be slower
and sometimes it could be also flaky and it might like
require manual review so it doesn't really make sense to put in continuous
integration this is a for for now our setup is
if you're working in the in the chatbot api
you're going to be running it manual you choose to like
turn it on and run it manually every time
um but it doesn't make sense to put in in ci right now there could be a
separate setup for that and this next step i i tweak with this
but like have another llm to like read
the output and like expect it in actual and and tell us like if it passed or
failed um i kind of play with this a little bit
it's a bit uh
flaky sometimes there's some prompt engineering to do but that can be like a bit of a time saver
um so when to use this it's good for like um when the api is like pretty simple it has just
like one case then it's okay to like uh test it manually but almost like you're almost never
going to be in that kind of situation so i would say it's almost always like worth it to write
tests because you're saving yourself so much effort and you're getting more confidence about
the stability of what you're building uh features where you need many test scenarios so faster
iteration is really the key here and confidence by using like a realistic uh case uh simple mpi
influence ui specific there's i'm not really touching like um front-end tests in this
presentation this is just back-end so but we have like another way of testing uh which i
won't go into right now so let me go in um in the test here i'll show you the code
um so here we have the the controller so here we have like the database like a helper
for like the the real database that gets like wiped this is where we do the wiping of the
database and then in here like we set up the user with their like case and like a
chat like an empty chat history and then we create like the the file so notice already it just kind
of reads as a like plain english like story we create a user we create a ptsd file for them
we create a file about chronic pain and then we're asking what are the current treatment approaches
and then we're like mocking this and then we select just the ptsd file and then we call
the controller the api endpoint and then we get the chat messages just to like unwrap the
the response or like get them from the database because we save the chat history
so we save like the messages that was uh responded with and then we we write like okay
what was the setup about what is this test about we have like these files and then we selected
this file we have this query and then the expected behavior it should only be about the ptsd file
and not about like mention like a chronic pain and then just the utility like a helper
to write to a file and then i created this just to play around with this but this is like the
the second llm call to like validate so we have let me just run it as a demo
so it goes ahead it makes the call here takes like five seconds and then here i'll kind of
not scroll we're getting a bit of a spoiler it fails so test result is written here let me go in
like preview mode so this is what like if you're gonna do manual review this kind of what it looks
like you can just open it after each run and then see okay you get like a refresher in case you
weren't reading the code um so we have this this is the setup and then we expect this and we should
not see like knee pain and then we can review what the what the chat is actually going to be
responding so the treatment we have some medicine cognitive therapy and pdsd nightmares blah blah
so i'm not seeing anything neat and we see the source that's great uh so it seems like it's
passing so uh my setup like saved me a bunch of time of like not having to do this through the ui
and getting a really controlled um experience and this i went with the test result file
um you know that's another one validate response content so this makes like another um
like llm call you validate test results analyze the markdown report
and then it should mention items should not mention check these topics are absent
and it's a bit flaky like it tells us some things um so it's it's not perfect some prompt
engineering has to be done here so i say it's good enough for now but um there
we'll keep like incrementing on it,
but right now it saves a lot of time
and the kind of foundation, the framework is set up
for us to add like more scenarios
because we could have like adding
like different data sources to the reg,
like not only from like documents,
uploaded medical documents,
you can also ask questions from like call transcripts,
maybe like the medical provider did a call with the client
and that was like transcribed
and we wanna ask questions about that as well.
So that would be very easy to add from here.
And from here, like tests are readable.
So the main points is that it's gonna increase
your like feedback loop,
keep as much as you can, not mocked.
So that way it gives you confidence
that everything internally is hooked up together properly
and that your prompts are doing what they should
and make it as readable as you can,
like kind of like plain English story,
more or less as much as possible.
So that's it for me.
If you guys have like questions from the team
or anything like that, I'll be welcome.
That's great.
Thanks for doing that.
This is, I know we did something
kind of similar to this as well.
Sorry, when you were running the test,
like we did something where we were essentially
scraping even the JPEGs of these documents
that weren't readable.
It's like another test.
You remember between you and I,
we created like a test suite
because we were basically having the same job
or the same tool call would go off
and like process lots of different types of files.
So what you could do is we wrote like multiple tests
and then gave it, basically use this prompt,
but run it against these five different files.
That way you could just tweak the prompts in one place
and then it would, you could kind of run it again
against all five different scenarios.
I think, is that similar in some ways
what you're talking about here?
Because I think we did something similar to that.
Yeah, what you're talking about
is like on the ingestion piece
and we had like a really nice setup there.
Even we had like a kind of reporting dashboard
like that showed a preview of the page.
Like that was, it's kind of like a picture of the page
and like the doctor had some handwriting and whatever.
And you could see like the preview and output
of the doc ingestion.
That was quite good.
And we even like leveraged AI
to like build that like simple reporting dashboard.
That was a really good setup on the ingestion.
I don't know if we would want to like,
kind of find it again,
like kind of parse through the code to try to find it.
I didn't prepare like a presentation about that piece,
but that was a really good setup as well.
Cause we had just in one,
like you could just scroll the page
and see like the multiple different cases,
like multiple different,
like a page is like blank or it has illegible handwriting
or another one that has like legible handwriting,
one that had like forms filled out.
So we had all of these just kind of lined up
and we were seeing like when we tweak the prompt
to parse these pages,
then how the doc ingestion pipeline would work.
And it worked really well.
Nice.
Yeah, I think this is helpful when you're,
you know, the typical scenario is like we have a client
who is using chatGBT on their own, right?
They're just going to chatGBT.com
and they're like throwing in a few files
and they're just like asking questions against it.
And so now when you try to bring that to like a company,
it makes it more complicated
cause you're trying to standardize a lot of things.
So we have to, that's where the rag piece comes in.
So this has been a helpful tool I know for you
cause you can quickly get lost
in sort of an infinite amount of like variables here
because you're working with like natural language
versus like a very digital, you know,
like either it's true or false,
like this kind of natural language responses.
So having this type of thing, I think it's helpful
because I think if I understood your demo there,
like you were running the test,
it was coming back response
and then you were even having it validate,
like use the response that was coming back
and then you're using AI to kind of interpret that.
Is that what was happening?
Yes.
Like I said, that piece is like a little bit flaky
and I wouldn't like trust it a hundred percent yet
until I iterate more and like-
Still AI, you wouldn't want to-
Yeah, so like what we had in the ingestion pipeline
is probably the best trade-off
that you see like all the cases in one line
and like all the outputs
and then you can just give it like thumbs up.
Um, that that's kind of like, uh, emerging, like pattern that, uh, I was
reading recently they called evals and datasets that there's a bunch of like
good documentation online about it.
But yeah, that's the main idea.
Like you, you set up your kind of scenarios you're expected and your
actual, and then you just kind of like eyeball them quickly and see like,
okay, yeah, each one of them is good.
Um, or some of them are not good.
And then you go back and tweak.
So yeah, like just not having like so many variables and being like kind of
spread thin and holding all the kind of working context in your mind.
It's really good for that.
Nice.
That's good.
Yeah.
If you guys, um, IRA and Keith just joined Leon, do not, if you guys have
any questions that we know in the chat, I can bring you on and we can just chat.
I think this is kind of casual.
Um, I know we're doing a couple of implementations like this, um,
on some other projects, but yeah.
If you guys have questions for, uh, let me know.
I just need to bring you on here.
Let me, uh, awesome.
Um, yeah, thanks for putting that together.
And let me just see it real quick.
Um, so, um, yeah, do you guys have any questions, thoughts?
I know some of you guys joined halfway through, so that
probably made no sense.
Uh, halfway through.
Any other thoughts, questions?
I know IRA, you've been doing a lot with the same client.
Um, I don't know if you've touched any of these, these, uh, AI automated tests yet.
No, not yet, but this was, uh, some really good context for me.
Sweet.
Yeah, that's true.
I really liked your approach.
Like, uh, I was even like, uh, you mentioned evolves.
I was, I was reading about it and like, uh, I think the whole way you are
thinking about testing, it's more or less like how they suggest that we test it.
So I think it's very, very, uh, interesting.
I'm curious to, to use it in the Hackstack project for, for the chat part that you
have, it's a kind of a bit forgotten because we haven't touched it for some
time, but I think once we, we have some time to get back at it, I will, I will
for sure want to, to try some of it because that's a very core problem of
working with LLMs, like it's, it can be like very inconsistent and like, for
instance, when you are, when you want to, to given in our case, for instance, we
have the ability to kind of, uh, do, do estimates.
So we do a budget for the customer.
Like we can make calculations wrong.
Like it can be wrong on the, on this part and we need for, for, for it to be
consistent.
So I really liked the overall approach in general.
I'm curious to test it.
Yeah.
It really shines when you have like tool calling.
So let's say like you kind of pre load the database with like your financial
reports and then you like prompted, like, Hey, like calculate this.
And there was like no other way that it could possibly know, except
for making the tool calls.
So you're, you're set up for testing that.
There's so much we could talk about as far as the rag stuff goes.
Um, so this is like just totally like an intro to it, but yeah, Greg.
Uh, Hey, I joined halfway through, sorry.
Um, there are people doing stuff like this.
Even when I caught Lucian, there's a local startup here where I'm at called
ordinal and their whole thing is bringing all the city documentation, the, all the
different, you know, volumes and volumes of ordinances and all that stuff.
And basically sort of digitizing that and then having an LLM go through it and send
it back to the LLM again.
And, and, you know, a lot and be like, is this right?
Are you sure it's right?
Where is this at?
If you don't know, say, you don't know.
So I think it's definitely one of the interesting use cases to get something
from the LLM and then send it back to the same one or a different one and go, okay,
what do you think?
This is what I wanted.
Did you give it to me or not?
And I'm like, Oh yeah, no, sorry.
It was good.
And you're absolutely right.
And that's a great point.
It kind of brings me like, cause you can use this approach, like on the ingestion
pipeline as well as on the, on the chat side.
And, you know, I was reading about like different patterns, like you're, you're
mentioning, it's kind of like chaining, you know, like you read it and then you
put it in the, in the, in the, in the, in the, in the, in the, in the, in the, in the
pass it to another step like the the output is like oh review this does it make sense and
what's what's really cool also is that you can use this approach for like testing just
intermediate steps you know still making the real response but you know if you're gonna write like
fewer tests I would say just like the full like the whole end-to-end thing is a good starting point
and that's good as well because I think when you're also developing with AI
we've it's it's very collaborative with the clients like it's it's kind of like you're
working with this like this other third party which is the AI so we're trying to like train
it and then it they'll go and use it a bunch of times so when they come back and say hey I'm
getting a but when I ask this question like I always get this so that would be a good point
to like okay let's add that to the test and then make sure that it tests against that I think that's
kind of the workflow is that what you're thinking which is like yeah it's almost like you don't even
invent and it's almost like you just take like the client's words and that becomes the test you know
you're just asking them like hey how are you using it today and they'll show you a few examples that's
when you paid like the most attention and then you just like kind of forward that and they become
like test cases and it like kind of next level is putting in place kind of client driven or like
user driven evals like in uh cloud or in like an open ai like in the normal chat there's like a
thumbs up thumbs down when you get a response and you can give like feedback so I'm sure you can use
these to like drive up like your your test further like oh we didn't catch this kind of edge case so
so that can be a valuable data data point could you set it up in a way and motivate them to use
your own client rather than just going straight to chat tpt and then you can actually capture
the input and see what they did wrong yeah I mean like since since we're building like their custom
so we have like full control and I'm thinking like some users in some scenarios like we know in
advance like some users like kind of like the power users so their feedback is different
disproportionately impactful so we can even kind of put like if you will like user feature flags like
the feedback mechanism is only shown to some users and normally like they would use the thumbs up
thumbs down only when something gets wrong so so you're sure to get like kind of good quality data
cool
very cool anything else we can still hang out and chat but just related to that that's good