has his AI joining.
That's funny.
But Janata is not here today.
And we do have one new person, Ira, who's here.
So he's kind of taking Janata's place for the live call here.
He just joined two weeks ago.
So we're doing a trial project right now.
And we've been fast-tracking him to get caught up on ShapeUp.
But so far, he's been crushing it.
He's been doing a great job.
So we just wanted to hear it direct from here
as we go through it.
Yeah, and no better way to get caught up
than to just dive into the middle of a session
together, right?
So that's awesome.
Cool.
All right.
And hey, Lucian and Chris.
Nice to see you guys again.
Good to see you, too.
So OK, so Bruce, do you want to start us off
on what are you guys working on?
And then lead into what do you have so far?
And then we'll do like before, and we'll jump straight in.
Yeah, this is great.
I think so we have a couple of things.
And we even had a call to try to frame up this call
to figure out what we do to get the most out of it.
I think we definitely have one project we could go deep on.
It's not related to Tribe.
It's another client's project.
But it's got some very unique rabbit holes and things
that some new tech that we've not worked with before.
So there's a lot of unknowns.
And I thought one way to get to that project
is just talk about the process of how we worked with the client
and how we got to the point of having this project.
Because I think there's some work,
there's some pre-work that could have been done potentially
weeks ago that we've inadvertently
left to inside the project.
So now the clock started, so to speak.
And we're now feel like we're, I mean,
I would probably guess we're like maybe 40%, 50% shaped.
60% is probably a lot less, like it's less optimistic.
So we're trying to figure out, because our process,
as you know, is a little bit unique in that we
have the client relationship.
I'll say one thing that has worked extremely well
is I have, on two different clients,
I have just pulled them right into the entire process.
I've gotten on a three-hour call with some of them,
four-hour call, and we breadboard, we shape,
we set the appetite.
They see the whole process.
And that has been incredibly eye-opening for them
and for us to get super clear.
And so we're working more and more that way
where we just bring them into the team.
Because there's not a way to sort of make it all packaged
and then just show them.
You kind of have to bring them to the desk.
Because they are kind of the key decision maker,
especially when it comes to scope and trade-offs
and out-of-bounds stuff.
So that's where we're finding a lot of value.
So already, I mean, this is only our second month,
third call, like crazy amount of value already.
Awesome.
Like worth all of our time and energy to do it.
And so what I'm seeing now,
now that we've kind of got a good process,
I'm trying to make sure that we sort of prep
the process that happens before this.
And how much do we frame at a time?
Because at some point we kind of say,
okay, this is four weeks.
And then the clients lock in and they commit to it.
And we have an agreement that says, this is four weeks.
Here's the dollars that have to be paid.
And then we lock in at four weeks.
And so then we now need to go like,
we want to try to keep to a four-week calendar
as well as a four-week scope as well.
Because if we run over in calendar or budget,
like in hours, it costs us.
And not just in the team,
because now we can't like,
other projects get pushed back.
We get over, you know, people start getting overwhelmed.
Totally.
But also then, you know, we just want to be as,
that's part of it.
There's a healthy,
one of the healthy boundaries is that timeline.
Like you have the four weeks, we're on week two.
Like we only have two weeks, like, you know,
like we have to make some trade-offs now
to get this thing shipped
instead of just letting it, you know,
get into scope creep.
And then, so I can, if it's helpful,
I made a little diagram of just kind of
how our process is working.
I can just, I just sketched it out on a little whiteboard,
but I can show it to you.
And then I can show you how we got to this project
or we can pause for a second
if you have any questions so far.
So, so let's see.
So what we're trying to get to here is,
you're trying to work out what the process looks like
of like, what do you do with the client
before this four-week clock starts?
And what do you do with them
after the four-week clock starts?
That kind of the question is like,
what kind of work is happening at which stage
in that life cycle?
Right.
And figuring out,
so we typically do an engagement.
That's kind of a one, one or two week, uh, process where we just connect with them.
We figure out the lay of the land.
What is the project?
We kind of identify two or three potential pitches.
There'll be like, we just frame two or three potential projects and say, Hey,
we could, we could work with you if you want, these are the three next things.
We kind of hear what their biggest pain points are, sketch out a few things.
And so we've put some time and energy into it doing a little bit of breadboarding,
a little bit of, you know, digging into their code.
Um, sometimes it's a brand new app.
Other times in this case, it was an existing app.
Um, they initially came in for us to say, Hey, come and just
rebuild our entire app, right?
Um, that's not always the best option.
So we initially, one of the big decisions was like, do we need to rebuild it?
Like, is this solid?
And we, once we got into it, we were actually like very impressed with the
previous developers and how they had built it and the same similar tech stack to us.
So we made a strong argument for them to keep it and just to keep it adding onto it.
Cause it's in production.
They've been using it for four years.
There's a lot of value to it.
Um, and so it's that kind of thing where just figuring out like, like kind of
getting some strategy on like, how do they keep building like in a new,
get them into this new process.
We're doing a lot of education, teaching them the process of how we
shape and frame and all those things.
Um, but, uh, that's so that we, we kind of did this, that call, and then we kind
of came out of that with, with like a handful of projects of which we identified
one four week project, um, that they said, okay, this is the next most important
thing.
So we, we focused on that now we're like a week and a bit into that project.
Um, which we have pretty much all of June to finish.
Um, so we're in it right now.
And, um, so that would be a great outcome of this call as well.
It's just like also getting clear on that project in particular.
So that when we leave this call, we're, we're even, you know, more clear about it.
Um, so that'd be a by-product.
That'd be great.
Yeah, totally.
Um, my first instinct in cases like this is always to go straight into the work.
And then when we've looked, when we've all seen the same work together, then we
can have the reflection on, could we have known something earlier if we had tried
to do a different type of session earlier, or maybe we couldn't have even
known that, and this is just how it is.
Uh, you know what I mean?
Like that?
Cause it's, it's honestly, it's pretty hard to, for me to say something that's
going to actually be helpful, like in, in, in general terms about process.
Do you know what I mean?
Um, but, um, but I'm happy to look at the diagram if you want to just kind of like.
If you want to, you know what I mean?
Like give it 10 minutes or whatever, and just see if there's anything to react to
there and then I pretty much said it all.
Um, this is just the exact thing.
We sort of our whole pipeline here is, uh, just going through, um, Oh, sorry.
Let's see.
Let me leave that.
Okay.
Sorry.
I have a sharing on Slack as well.
Um, this is, you know, we do a kickoff with them.
We do some internal sessions and then we kind of review and get two or three
projects is basically just what I said.
There's some details here.
We're still kind of breadboarding this out as far as like, there's, you know,
some, some content stuff we're handing them, like we sent them some of your
content on YouTube, but like, you need to watch this video, like up to speed.
Um, you know, we do this figure out lay of land problem solution, just try to like
find out what's going to be the hard stuff.
Um, and then going to sort of internal review.
And so what we're finding though, is like, this kind of starts like the
framing piece of which, you know, we still have to shape it.
Cause there's a lot of work as we're finding working with you.
Like this is a big, a big process, the shaping piece.
And then there's sort of the build and the ship, which I actually made it almost
like this, where it's like frame shape and then build ship, like that's how we teach.
That's how, that's how we teach it in shaping in real life is that it's
one hill and we do it the same way.
Cool.
Yeah.
Very similar.
Awesome.
And then, and we thought like in this side of the idea, like this is where we're
breadboarding doing design reviews.
Like this is where we're doing super quick, you know, little prototypes
to like get proof of concepts out.
Um, and, and then of course this is just, you know, the, the
build and test and ship it.
Yeah.
This is just, this is my thoughts of kind of how our process has changed
because before it was shaped the entire thing, what we thought was shaping,
which was not really shaping.
It was like really framing.
Um, then we would just get into it.
We would just go from that sort of framing directly to build.
Um, and then we would try to ship it and then we'd run into all these errors
of like, oh no, we missed this requirement.
We, this thing doesn't work.
This formula.
Exactly.
That's the, the lack of shaping manifests as all of those time bombs going off.
Right.
The hidden time bombs that you didn't know were there.
Yeah.
Yes.
So, um, so this was me sort of thinking of like, what's our new way of doing it?
And this is how we've done this recent project because, um, we took them on.
Like while but like between while we were been working with you. So we've tried to follow. This is as close as we can
That's it. This is just my own
Yeah, this this this in broad outlines is much closer to what to what we're teaching and shaping in real life now
It also corresponds. I mean even when you're not working with a client when you have a
If you're in a if you're in a in a product
Company and you have a little bit like this like kind of like the salespeople and the rev team are kind of figuring out what they
Want next and then you have like more the engineering side that has to go figure out how to make that happen
What you end up having is this conversation which is very similar to the conversation you're having with the client
Which is like what are kind of like the two or three things that are actually most important for the upcoming slice of time?
That's going to become available right and like what's what's that thing? That's most important next and that framing step is
It's
mainly
What is the what is actually most important out of the things we think we could do to do next and then the other piece?
of it is
There's a first it's your first line of defense of like that's insane
That's feels like it's not even possible in a in in four weeks
Do you know what I mean?
Like you don't you still don't know what to do
But it's your first line of defense about like no, we're not gonna do the entire app in four weeks
You know what? I mean?
And there's a little bit of a sense of like this thing that we're talking about involves
Like some science project that we've never done before
You know what? I mean? And then that's going to influence the way that you the way that you go into the project
So but I think you're very much on track there and then as you have more specific questions about what about this
What about that then we you know, then we can get into that
Right. Okay, that's cool
So would it be helpful to
Just show you what we have framed and shaped so far. Yeah, exactly. Let's just look at what you have
Okay
So, I mean I'll share you guys this is
Lucian's been on this this project, but I'll share real quick since I have it up
We
So
generally
We kind of just pitched like the the project here that the is that
This is for a very company called bear tells ratings. They work with VA. So for veterans and they're helping them
basically file a claim to to get
compensation for
You know and service related injuries. So the problem is that they have you know
Potentially like 20 plus files, you know interviews with this person
There's a lot of data around it
and so the case managers today are like
Pulling files down to their computer and they're like dropping them into chat GPT or and they're trying to like
Get some value out of using AI to help summarize or just read through because some of these can be like a hundred pages long
Uh-huh, and you can't miss like oh there was this one symptom this guy mentioned in you know
This one call it has to be very precise
And they're trying to basically compile all the information to then file this claim and then work with them to get it approved so
They're they're doing like a hundred or so hundred fifty or fifty claims
I think a month right now and they're wanting to scale that
Right now this is where like a big bottleneck is like the case managers themselves are getting blocked
So they identified this through a number of conversations. So once we presented this to them, they were like, yes exactly
This is what we have
And then
We had some different things you're getting into different scopes, but I'll show this
Here's like our and we do have your we got this. So we'll jump into this as well
and
Just kind of talk through like what's happening even today like they're uploading, you know
We did a bit of very light breadboarding and even Lucian and I were sort of redoing all this this morning
You know, this is this is what they do today without what you've without what you're building, uh-huh
This is the current current. Very good. Yeah. Yeah. Yeah, and so, you know, you upload files. They upload a transcript
they are using some open AI right now to
create a summary or analyze or create statements and then they save that to their database and there's like a
This is just the different tables that how everything links. I mean the in the Postgres database
How do they save it to the database they have like some app where they have to enter stuff
No, this is like so we started drawing it
When the
Transcript here's uploaded
Yeah, let me grab this. This was supposed to be up here
So that this is like on upload goes to s3
They have a triggered like an event queue which then goes to processes PDFs and it does all this different
The other things to it and then after this once it creates these sort of these outputs of these like three different
types
then it just saves the results basically back to the database.
So, it's all automated.
Uh-huh. So, what is the thing, if this starts with upload,
so, this breadboard that we're looking at here that's with the orange pieces,
where does this slot into the original current that you were showing me?
I know.
It's like this.
So, basically, the app is here.
They're in the interface here, and they can upload a file,
and that would kick off this guy here.
Right. So, the question I'm asking is, after the OpenAI step,
when they are asking for a summary, they somehow upload that as a file?
What's the link between the OpenAI step you showed me
and them uploading a file into their current system?
So, I think this is actually where OpenAI is happening,
kind of on this processing level, because it's doing the processing here.
There's no interaction.
After it goes into the database, nothing else happens to it.
That's kind of where it ends today.
So, they, okay, hold on, I misunderstood.
They are using OpenAI behind the scenes through the API
in their current implementation of this.
I see.
There's the automatic processing of transcripts,
and there's also a custom GPT just on the chat GPT website.
Okay.
So, they're uploading all this stuff.
Then this process PDF,
is this, are you summarizing some things that happen with user interactions,
or is this literally all happening as an automated flow behind the scenes?
So, for the transcripts, they are getting a transcript from their call.
They have like a call platform, but the result of that is basically
like a transcript comes out of that, and then that gets sent.
So, I think it can create, it just stores summaries of that potentially in the database.
So, yeah.
Uh-huh, okay.
I see that you shared a Figma with me.
I just put that in there for anyone else who wanted to be in this instead of.
Let me, so let's see.
Last time when we did this, you didn't bring any breadboards in, you know,
and so I was kind of working from a blank canvas.
Yeah.
What I'm inclined to do here is, how can we do this here?
Maybe I can sign into the Figma here, and then I can maybe just kind of copy and paste,
and we can do some things on the side.
Yeah, you just move it to the side for sure.
And this one, we've kind of done a few, this is like version three of this whole document.
So, it's very flexible right now.
Like, it's definitely not nailed down, and we sort of like pulled all the pieces apart this morning
to take it, you know, to like reassemble this, and that's where we are at this exact moment.
So, we jumped from that call to this call.
Okay, cool.
Let me just, I'm just wrestling with Figma a little bit to log in.
It looks like I need to request access from the...
Yeah, if you do that, here we go, approve.
There we go.
That's to view, let me give you...
That's a surprisingly good workflow that actually just worked.
That's the kind of thing that usually is.
Can you edit though?
Let's see.
You can only view and comment on this board.
Start editing.
I'm sending a request for that.
Oh, here we go.
I can just set you to editor.
Okay, it should be good now.
It's reloading.
Bing, bang, boom.
Cool, okay, I see all the tools.
I'm going to...
I'll stop sharing.
Okay, I'm going to take over for a moment, but then I'm going to hand it back to you
because you're still giving me background and we're still not very far.
But what I wanted to do was already kind of start just updating this picture
with what's in my head.
So this is...
I'm just going to do a...
So just give me a moment as I...
I'm not completely used to to figma. This is very good. Okay, good. I'm just going to
Can I do a is there such thing as a line? Oh, that's a connector
Yeah, there is a line connectors, but it's hidden somewhere. Okay, i'm gonna cheat and just do this. It's perfect. Okay
um
and uh
and then uh
This is the equivalent of a frame. Yeah, this is a section
Um, i'm just going to call this current prime. You can also do it outside of the section just to
if you want
Okay, and then i'm just going to grab
This which what we was what we were looking at
And drop it here. And i'm just trying to figure out like, um
uh
so
They hit upload
And then upload files
This is like the dialogue where they are actually choosing the file and then they're hitting upload and then s3 upload
Is this this is um, not actually like a user interface place. This is something that's happening behind the scenes
behind the scenes
Uh-huh. Good. Okay, so i'm just gonna um
I'm, just gonna do a dumb simple thing
I'm, just gonna make this gray and we're gonna say if it's gray
It's not user facing and it's something that's happening behind the scenes and this is going to help us. Um,
Uh understand a little bit better what's going on. Um,
So, uh when you hit upload it's going to upload to
to s3
Um, i'm just going to write this a little bit more like as a command
um
and
this
triggered
event queue, so, um,
This means that now like, um, there's a there's a there's an event in the queue
It's kind of like a job of like this this uploaded stuff hasn't been processed yet it needs to be processed
Yeah, I see I see lucian nodding, okay good, um, so then this processing is still happening behind the scenes, yeah
Okay, so i'm also going to change this
um
and um
Uh, and this is something that we can
unpack then so so
This process step
How much of this is happening
In there's parts of this that are happening in openai and there's and then there's some output from that and then there is this
kind of cover sheet
so I just want to like unpack this a little bit because it kind of feels like there's a
This to me feels murky and like there's a lot hiding in here
And and my guess is this is where some of the complexity is. Um,
uh, so, uh
Creates cover sheet i'm guessing this is after
The ai processing is done
Yeah lucian
The cover sheet on the uploading, uh full transparency I did like, um
evaluate through like the transcript flow, but the
Upload file flow that I didn't like look at that in detail. So yeah, so so should we be looking should we should we be looking?
Yeah, sorry, should we be looking at a different flow then?
um
And uh, and and actually let me let me step back a second
So the thing that I started to do here, but I want to step back
So I don't go deep down the wrong thing. That isn't the thing that we're trying to solve
is um, what I what I saw here was like
Just a lack of clarity around what's front end and what's back end?
um
Ah, but I you did this here
Yeah, I we started we heard you the first time you mentioned the gray
I think we missed the
And then I just when we started to jump around then then I just lost it, but you already did this very good
Okay, well, no, we did we missed the s3 and the processing on the right there
So we did it on the left, but um, I think we learned the the color coding system
between
The first call where we did the thing on the bottom, right? Okay
And so this is the evolution of shaving. So so all i'm trying to do is um
Is just get us to the place where we have like one picture where we understand what we're looking at
Do you know what I mean?
Because i'm I I can see that as we jump around it's a little bit like well
There's this here and there's that there and it's just a little bit confusing. Um, uh,
um
Let's step back though before I continue to do that. Um, bruce
We we started where you you said that they're they're uploading transcripts
and then there's currently a a a back end process where some some the the
Um open ai api is being used
and um
Okay, and then the place where I got stuck is that somehow something is being saved to the to the to the database and it's
Summary analysis statements can do you guys have an understanding a little bit more concretely?
Of what is actually being saved to the progress database?
Do you have any sense of like what is the shape of this data or or what is this?
That's yeah, we started. Yeah.
Um, the summary of the transcript, the transcripts are like totally different from like
files. Um, it's just like, they had a meeting, you know, like through a hundred and less
and, uh, summary and they extract like claims lists. So basically you can make like a claim
for like knee injury and PTSD and whatever. Uh, and then statements is like, kind of like drafting,
like, Oh, I did a service. So like in the first person, like the vet is saying, like,
I was in Afghanistan, like there was an explosion. Now I'm afraid and blah, blah, blah. So kind of
like all the artifacts that come out of the transcript summarization. I see this.
We started sketching out the database here, which we did in another project. And as we just started
doing that now, but you can see like, there's the claims add ons, there's the meeting transcript.
Um, I see. Okay. Where, where's that sketch out of the database?
Uh, what you just pasted, uh, over there on the right. Yeah. Down here. So you see the,
in the gray, like claim add ons is a, is a table claims is a table. Um, meeting transcript is a
table and you can just, we did some of the links so we could see like a claim ID links to claim
ID, you know, uh, it's not as complete as I would like it, but there's like the general idea. Like
there's like multiple places that I see. Yeah. I see. So, um, okay. So I'm just going to order
this a little bit. Um, and, uh, now this, this is your documentation of how it works currently.
Yep. Yes. Yeah. Got it. Okay. Um, can I do a section in a section? Let's try. Um,
um, um, so these are claim at, there are, there are claims with claim add ons and there are
transcripts and there are users. How do these things, these things, do they map to what you're
saying as summary analysis and statements, or is there still more work to, to, to spell out
what the relationship is between this and summary analysis and statements?
Um, like summary goes on the, like meeting transcripts, uh, table. Uh huh. Okay. And also
on the claim add ons table, uh, analysis goes on like claim add ons and statements goes on claim
add ons. Uh huh. Okay. Uh huh. So, um, my instinct here is, um,
what I want to do is I'm just going to, I'm just going to change Postgres DB to, I'm just going
to say like, what needs to happen is like save to Postgres DB. And I'm just going to describe,
um, summary and, um, uh, okay. Summary and, um, claim claims list,
summary claims list and statements as claim add ons.
Yep. Wow. So many interesting UI elements on this Figma. Okay. Summary, summary claims list
and statements as, as claim add ons and, um, transcripts. That was another thing here. Um,
uh, summary analysis and statements and the transcripts, which we uploaded those anyway,
actually before going to the, were they anyway already? Um, um, but we just had this earlier.
Yeah. We just had the summary to the transcript that already exists.
Uh, uh huh. Okay. So we're going to, so, so we're actually saving the,
and I'm, I'm, I'm just being super informal here. You know what I mean? Like I'm saying like,
um, uh, this there's some work summarize transcripts. So this, is this a, is this a,
um, what is this meant to be saying? Is this a call to summarize? And these are arguments to
that? Or is this a list of things that are going to kind of happen over here?
Yeah. There's like a event queue, you know? Um, so it's like summarizing the transcript. And then
if it can like, um, I'm probably not saying like super accurate, what it may be the process
slightly different, but it's like, uh, first summarizing. And then there's like a thing
called analyze, which is like, get the claims list. Uh, and then for each claim, like knee,
PTSD, et cetera, it makes, uh, a summary of like the part of the conversation that was talking
about that particular issue and then make statement for each like claim. I see. I see. Okay. Um,
And these are each lined up as jobs here and then performed later, or is this
all happening just kind of as one pipeline where one thing triggers off the next thing?
Like the pipeline.
Uh-huh.
Good.
Okay.
So the way that I would be trying to clarify this in my head right now,
or what I'm trying to do to clarify this in my head right now is, I want to sort
of see all the backend pieces broken out the same way that we break out the front
end pieces.
You know, when we see the front end pieces, it's really like ding, ding, ding.
Like we, we see how all the, so what I want to do is instead of kind of
summarizing it and then it's all like in our head, you know what I mean?
I want to think that there's, there's basically, there's a, there's going to
be like a summarized transcript and I'm just going to annotate it like this to say
that this is a function that's going to happen.
There's going to be an analysis of transcript to that, that produces the
claims list.
Yeah.
As the, so I'm going to say like analysis, like analyze transcript is going to like
output like a set of claims.
This, I don't actually know what's coming out of this is the summary, like a single
block of text.
Yeah.
It just like kind of removes like arms, butts and like, how were you doing this
morning?
Like, ah, so it's basically like transcript in transcript out, but cleaned up.
Yeah.
Okay.
Um, uh, and then what we have is, um, so we should find, we should feel that we
don't need this and we don't need this when we start to come to the end of this.
Right.
We should start to feel like this is happening first.
Uh, then after we've cleaned the transcript, then we're getting claims out
of it, right?
So we're actually kind of taking the cleaned transcript as the input to the
analyze step.
Yep.
And then from there, then, um, for each claim, summarize the claim and, um, uh,
and the attachment, the, the summary is, is added to the claim as an add on.
Yeah.
As a claim add on.
Yep.
Yeah.
There's the summary and there's the statement.
Mm.
Hmm.
Mm.
Hmm.
Um, again, it's like a little bit messy, but what I'm trying to get to the thing
that's important here that I'm trying to see is, um, is the inputs and the outputs
and what the chunks of work are, right?
Cause this is going to help us to understand like where we intervene and if
we're doing all the same things or not.
And, uh, um, and we're almost to the end of this.
And then I think we can lift up our heads and see where you guys are trying to get
to next.
So there's, um, so then also for each claim, I'm also, I'm pulling these out as
two different things, just cause they're two different processing steps is I want
to, um, um, uh, well, yeah.
So I'm generating statements.
This is also funny.
So like I'm summarizing the claim.
I'm actually taking the transcript in with the claim in order to do this.
By the way, pretty much just the transcript and well, the claim too, is
like, you know, we're talking to just about me because we have to know which
claim we're talking about and then, and then in this transcript, then we're
going to come out of the book with a claim add-on, um, we're going to, we're
going to, we're going to add a, um, summary as claim add-on it's like what
comes out of this.
Yeah.
And, uh, and then the last piece here,
I think we can actually, uh,
start to start to, then the last thing we need to do is generate statements.
And so, so generating statements means, um, that we are adding claim add-ons
again, um, which were statements that had to do with that claim from the transcript.
Uh, they basically take the same input, like the claim itself and the transcript.
Mm.
As far as I remember, it's like that.
Yeah.
Mm hmm.
And again, it's outputting an add-on to the claim?
Yeah.
Okay.
And at each step,
whenever we have claim add-ons are getting created,
I mean, that's like the Postgres database
is actually getting written to all the time
as these things happen, right?
It's not like all this has to happen
and then we have to do it as a batch, right?
Yeah.
Okay.
So,
okay, so now
we have,
I think what we're gonna find is this becomes
a little bit easier to make sense out of
because now we have a workflow that's happening here.
What happens when this is done?
Let's see.
We didn't have a definition of what was done then.
So, there reaches a point where like all of this happens
and then if we summarize this,
so if I just wrap this in something
and I'm just gonna say all of this is like,
what did you guys call this?
Process transcript, maybe?
Yeah, process,
process transcript.
Like this whole thing
takes in,
it takes in a transcript
and what do we get out, right?
Like what we get out is we have
claim with add-ons for
summaries and statements, let's say.
Yep.
So, this is actually,
so before we had transcripts tab was kind of pointing to
the database.
It queries the database.
Okay.
So, a different way of saying that is that,
so this is where we're gonna read what was done here.
Is that right?
As far as I know, only the summaries,
like the main transcript summary
and the like per claim transcript summaries,
but not the statements.
That's kind of what I did.
Uh-huh, okay.
So, the place where I can end up here
and then we can take a pause is that
actually like all of this,
the shape of this data that like kind of comes out of this,
like matters as an input to this like summaries here, right?
This is like when you're trying to describe
kind of like how,
what data is shown in the summaries, right?
Sure.
So,
so let's just pause there
and say that all of this, I don't think I can,
is it gonna let me do,
oh yeah, we can do something like this.
So, more or less, we're gonna get this.
So, here, I'm just doing this very clunky trick of like,
if it's input, I'm just kind of upper lefting it
above something, do you know what I mean?
And I'm just thinking of this as a read model.
I mean, if we really wanted to color code things,
we can do that more,
but I don't wanna waste time with diagramming.
I wanna kind of like keep getting to the actual problems
that you guys have.
So, let's say that,
so now what we've got is we've got a clearer sense
of what's happening with the data and the processing steps
and where it takes us.
And then we end up being able to look at summaries
on the transcripts tab,
because we have the summaries as part of the claim data.
Is there anything left here that is a mess?
No, transcripts have summaries.
Okay, good.
So, I'm just gonna delete this, delete that.
And then the last piece is there's an edit from summaries.
Yeah.
And before we keep digging,
let's just say that we're just back up to date,
Bruce, with what you were trying to walk us through
in the beginning.
Okay, so we took a little time just to get clearer.
So, what are you trying to do?
We're still in the current world.
Nice.
This is current, yeah, current.
And I think the piece that,
so there's this whole transcript piece,
which is like one half of this.
then there's the files, um, uploading, which is still, um, important.
Cause we, um, there's a whole process that happens on the transcript.
And then I started drawing it out to the right of what you, where you're
working with the color code.
I think if you go to the, uh, to the right, uh, this is the upload files here.
Yeah.
But if you go to the right of where you were, I'd kind of dragged it over and
started fixing the colors here.
And so there's these like logic things here.
Like if PDF do this, if that, and they have these little functions
called like handle evidence, PDF handle doc, and then, um, so these
pieces are like basically based off some sort of filter logic, like
depending on what type of file is there, it'll do different things to it.
Yeah.
Now, of course we would go through and I need to figure out like what we just
did now, like what does the handle PDF do?
Oh, well, it, it looks for this, it does that it does whatever.
Um, uh, and, and I'm going to just, I'm grabbing it.
Like, so we're, we're, um, I'm pulling this out of like, uh, the code base now.
And this, so what, okay.
So if we kind of pause and kind of do our, uh, step back and talk
about the game for a second.
Yeah.
Game off.
Okay.
Game off.
Like it's, uh, so, so the unique thing here is that we are
working on someone else's app.
Like we always build our own stuff before this.
So one very big rabbit hole here is we've just never worked on this app.
And so on face value, you're like, cool.
It's the same tech stack.
It's JavaScript.
This makes sense.
It's using all the same tools we use.
Got it.
It's only the, only the beginning of the mystery.
And, uh, you know, it seems, it seems clean and organized and it's well.
Um, structured, like it's not, it doesn't look sloppy.
It doesn't look like, you know, uh, the output, like an AI generated app.
Um, so there's someone did it.
He did it by hand.
They've been tweaking it.
And to me, the, one of the most valuable things about an app is the longevity.
So if it's been in use in production in the business for four years, like that,
that means something just that they've worked out a lot of the quirky bits.
Um, it may not be right.
Like the workflow may be wrong, but it's got some, it's
got some, uh, some years on it.
So, um, so, so that's the part now we're trying to like pick apart.
What was the, what's the logic of how it currently works is clearly like, um,
and I'm not trying to like, I'm not panicking, I'm not panicking at all.
If anyone's wondering, I'm not panicking because we are two weeks into this
project that's already paid for, and we are, um, uh, trying to figure
out what all these functions are.
So we, we, one massive thing I'm seeing we missed is we just like, did not go
deep enough into the existing code to really document, like, cause we haven't
even talked about the project yet.
We're just trying to show you right now, like what the current version is.
And we're still struggling to tell you what, like what pieces are.
So I've been actually the least involved in this, uh, between Chris and, uh,
Lucian, there's another developer, MJ, who's on it.
Um, so, uh, we, we do need to go, like we need, that's one thing in our process
I'm seeing is that we just didn't go that deep into the code to figure out
we should spend as much time documenting the current version as we
should almost documenting the new.
And we kind of like,
Yeah, the new will be, the new will feel very easy to describe when the
current is well-defined because the new is just going to be this little diff of
like, oh, we, so what we're going to get to is we're going to get to like, oh,
we're going to, we're going to slice here and we're going to slice here.
And then this thing is going to be different.
You know what I mean?
And you're going to be kind of like describing the interventions, um, and
you're going to have far fewer surprises and, and, uh, you're going to know how
to react when the, uh, the things that you inevitably missed come up because
you can see where they belong.
Right.
And I, that's exactly.
So I think, um, that, that was, that was the exact perfect.
What we needed right there was yes.
It's the, it's really like if we can spend 80, 70, 80% of the time on just
getting super clear about the current, because I think when we had to explain
the DM example to you, like to us, we just knew what DMS were and we just were
like, no, we're going to add this button and we're going to do this thing.
And then we were like, whoa, whoa, whoa.
Like how does it work right now?
And we had to go through, we had to actually document and that software,
like we built ourselves and we hadn't documented the current.
So it's not, I wouldn't even say it's necessarily like our appers and someone
else's app.
Um, but I think that's very helpful mindset to say, like spend a lot of time
documenting how it is currently.
Um, and Ira and I were working on a totally different app, different clients.
Um, and we actually spent, and he was new to the whole project, which actually
was very helpful for us because we ended up having a lot of sort of dumb questions.
Uh, you know, the best questions.
Yeah.
So he was like, so what's a product, like what's a variant, like how does it relate?
And we, and then all of a sudden.
Even I've been working on it for six months.
I was like, Oh yeah.
So it's like a, you know, it looks like this.
And we had these little sketches on the, on the white board to, to make it easy.
And so we all got really clear cause we, we had to document, you
know, how the database was today.
Like what, and then what's that?
Okay.
We're going to add a new table here and we're going to
connect this table to that.
It became very clear what the diff is going to be.
And that was very helpful.
Um, that was, that was worth the first 45 minutes.
It was great.
Yeah.
That's good.
Um, so, so, uh, one more thing on the game off here is, um, um, you'll find
that, you know, like if we felt like we had to document the entire app, we
would all want to kill ourselves.
But like, if we, when we know that this is the part of app I'm supposed to be
working on, then documenting, it should feel like it shouldn't feel like
this annoying, tedious thing.
It should feel like, Oh, like I'm untangling the knot, you know?
And like, this is my knot.
I'm like supposed to be working in here.
So like, I'm getting, I'm getting clearer.
I'm getting closer.
This feels like progress.
Right.
Um,
a follow up question.
Yeah.
Uh, so speaking of like untangling the knot, it's kind of what you do with
the process transcript, because all, it seemed like a summary of that
information was in the, in the arrow.
Right.
Exactly.
Yeah.
So I, I, my question is, I'm just kind of coming from that.
Um, how do you think about like arrows and putting text over them?
And is it appropriate or when should it actually be unraveled?
So first of all, I have to tell you that, um, the, uh, this
breadboarding method started with just defining the UI.
And when we were just using it, when we just use it for UI,
we would, might do something like this.
Um, um, uh, I'm just going to make up something.
Let's say there's some other, let's say there's an upload file button.
And then I select the file and then I have like an upload button
from the modal here, you know?
And then like, let's say like, after I upload, I'm going to land on the
index of files and I'm going to see all the files I uploaded.
Let's say it's as simple as that.
Do you know what I mean?
Um, I could say here, um, store the files route to index.
And when I do this, what I'm saying is like, um, I'm only
showing the interface elements.
I like to think that like anything that's in a box is something that
I'm like actually spelling out.
And, and this is like, like a little shortcut of saying, well, like, and
along the way, this is what happens behind the scenes and there's good.
There's always going to be things that aren't worth spelling out because
you've already seen it a million times before and there's nothing interesting
there, do you know what I mean?
And so like, here's a case where I wouldn't stress about that at all.
But when I see like, go do AI magic, this, this like freaks me out.
I'm like, I don't know what's happening there.
Do you know what I mean?
And so like, that's, that's the smell that makes me think, okay, so now what I
want to do is I need to unpack this.
And if I'm unpacking, then my way of unpacking is to, is to use boxes.
And, um, um, when it comes to the UI part, we have clear rules.
The, the backend part of breadboarding is actually new.
And I don't, I wouldn't say we have a clearly settled grammar yet.
Um, uh, from like a nerdy perspective, I can tell you that everything I'm doing
with the technical breadboarding is inspired by, um, something
called data oriented programming.
There's a, you, there's even a book you can find, um, from a
friend of mine who wrote that.
It's basically like how the functional programming world deals with software
design, but the thing is we don't care about functional programming.
It's just that when it comes to explaining how stuff works,
it's the right mental model.
So you don't have to code it that way.
But, but from a, from a, um, understanding the system and
reasoning about it standpoint, it's, it's, it lets you, it
lets you clarify everything.
So then what I'm basically doing in my head is I'm thinking about, I want
to have some function that happens.
So like, this is the magic AI function now.
Right.
But I still don't know what that means.
So now how do I break this up?
You know what I mean?
So like, first there's going to be this like transcript and then there's
going to be this like add, um, statements, you know, um, but now the
thing is, so now there's going to be times when you feel like just
describing the function, like you understand enough how this system
works, that this, this is, you're like, I'm fine with that.
I know.
What that means, you know, um, for example, if this says like, send email,
you're like, I know how to do that.
We have an email service.
I don't care how it works.
Right.
But if this is something which is like starting to feel weird, or we're
going to have to work on it, we don't know how it works and that's the place
where I'm starting to think, like, I want to describe the transformation
that happens here and my first step of going deeper into detail is the shape
of the input and the shape of the output.
So I'm, I'm just using a sort of pseudo TypeScript usually for that.
Um, and that's where I'm saying something like, if I, if I have something,
so let's, let's say this is clean transcript.
If I have something that we call a transcript, which is an actual object
that means something to us as a shape, then I can say, I'm going to take in
a transcript and then just using this sort of dirty TypeScript, like I'm going
to say, I'm going to output a transcript as well.
Do you know what I mean?
So, and here's the thing where like, maybe this, we understand this, but
maybe we don't understand this.
So maybe we want to break this into five boxes.
Do you know what I mean?
That like says what happens here.
So that's just kind of, um, where I'm at so far in terms of
like, um, how systematized this is.
Cool.
Um, the one, one other thing I'll throw out is what I think, what, what, um,
what I shared on the previous, um, check-in, um, with Lucian was there's
times when you don't really care about the backend so much as in terms of
processing, but you do care about the data that's coming in and out of a UI process.
So in a case like that, we might say like, I, maybe I need to have a claim
before I upload files because that's what the files are getting attached to.
So then I think of this almost like a react view where like, I have a certain
amount of like certain props that I expect to be there, you know, and then
like at the end of this, then, um, if this upload step is happening and I have
some kind of a save upload, then, then maybe I described the data shape of like
what I got from that view, for example.
Do you know what I mean?
So this is just a little bit of like different tricks.
Okay.
Uh, game on.
So we did a little bit of untangling here.
Um, uh, where, where my head is at is like, we didn't yet talk about like what
you guys are supposed to be changing or what is supposed to be happening.
That's different from the current version.
So, um, um, Bruce, do you want to get us to that?
Yeah, this is helpful.
I mean, I think, I mean, as far as how the current process is, it seems like
this is a solid 30, 40% documented of what's current, um, there's still, and
we can kind of follow unless you want to slice it to just focus on transcripts,
like for, for what's transcripts happening currently, and then what are
transcripts in the future, but there's this other piece of like
the file management as well.
Um,
so, um, yeah, let me, so let me react to that quickly.
And then we'll, we'll either go continue your line or we'll, or we'll drill down.
So like in my mental map, there's this whole blob of what this thing is.
And we have like one section of this blob is you mentioned as transcripts
and there's some whole other, well, apparently don't even know what scale
I'm in here.
Okay.
So there's, there's some part, we're enjoying watching you
learn how to use this tool.
And, uh, and, and what, uh, uploads was another kind of chunk here.
Sure.
Um, and, um, yeah, we, we actually do have, uh, maybe we can drop the monster map.
Uh, yeah, well, this is, this is what I'm doing is this is just my, uh, uh, when I
try to especially be, be dirty with it, like to show you how dirty it is in my
head, like that, there's some different things here.
So my first thought would be like, if you, if you have in your head, uh, that
there are different things that are as part of this project, if they are
orthogonal, we can choose the one that is the gnarliest and we can drill in.
If they're not orthogonal and they actually have tangles between them, we
can, we can do an untangling between the whole, the bigger areas as well.
For example, if there's some common piece that they all depend on, or do
you know what I mean?
Or changing one influences how you change the other.
Um, so, um, uh, kind of like, uh, um, what's in your head in terms of like,
uh, is there one place for us to drill down that is clearly, um, orthogonal
and gnarly, or is it like, there's other stuff that you don't understand it.
And we need to pull it apart.
Like, do you have a sense of kind of where the trouble
is or the worst trouble is?
Yeah, I think, um, well, I don't know necessarily that, I mean, which may speak
to it, but I also know.
This is all sort of, some of the stuff
is going to sort of stay the same.
So but we're trying to figure out
where do the connection points from the new piece.
So it's not like all of this is going
to be changing in the new version.
But yeah, what would you say is the pain, Lucian or Chris?
LUCIANO FLORES-CHAVEZ Well, in terms of the new system,
where we're wanting to go with this.
Yeah, like what are the parts that
are in your head that are like, this is something
that we need to understand it better.
We're wanting to not use like the summaries
from like the database, but actually do
like a vector database system.
So it seems like for transcripts, it's like fine.
We'll just like grab the output and just
like also put it in the vector database,
like keep writing it where it's written now.
And then there's like the files.
So it's like a piece that we're going to put in place
and going to change the transcripts.
And then we're going to build like similar system for files.
But that whole thing about vector database
is a bit of like a unknown.
SIMON ULLMANN Similar system for files,
similar in the sense that you're going to upload the files,
and then you're going to chunk them out
and store pieces in a vector database?
Yeah.
OK, so you're basically kind of moving
to like a vector DB backed rag approach
to both the summaries and the file uploads.
OK.
It's exactly a rag agent for sure.
Yeah, OK.
OK.
So from an interdependency standpoint,
there's a do we need to?
OK, I'm going to we'll come back to that.
I don't want to go into it if we don't need it yet.
Let me see if we need it.
There's a tool for dealing with these kind of things,
but I don't know if we need it yet.
What are some other things that are in your head of like,
this is something else that we need to do
that feels like, is there anything on the level of this?
Do you know what I mean?
That's kind of like big level stuff that's in your head
that we're going to like other stuff
we're going to have to do that we're going to have to solve?
Yeah, we're talking about like the desired system.
I think the tool calling, like figuring out
like what tools we need for the questions
they're going to be typically asking
is like another big like, yeah, dreadful thing.
And where in the current system is this tool calling thing?
So this piece, I kind of understand where it belongs.
Where in the current system is this tool calling
thing relevant?
Right now, it's not because they have just
like preset like prompts for like background processing.
And this tool calling is like a new piece for like the chat.
Like they're going to be just chatting with in the app.
And so there's a so there if I'm coming back
to my mental map here, like there's this chat.
There's the chat is a different piece
of this that's different from the file upload
and the transcripts.
OK.
Is it safe to say like it sounds like the chat
is kind of orthogonal to the transcripts and the files,
but the transcripts and the files
feel less orthogonal because they both
have to do with kind of moving to this vector DB
approach to how you're storing stuff so that you can look it
up.
Is that fair to say?
Sounds fair, yeah.
OK.
And if transcripts and files don't happen,
then there's nothing interesting for your tools to do in chat.
Is that right?
Yeah.
OK.
So the mental picture that I just kind of went through,
this is we don't need to go far into this,
but there's a very, very short version is there's the if we
factor out vectors from both of these,
we have like how the hell do we implement a vector
DB if we haven't done it before.
So I'm going to write like vector DB as a thing.
OK.
Then there's process files with vectors.
There's move summary storage to vector.
The interdependence here in terms of the question that I'm asking is, I'm trying to identify separate pieces and then I'm asking, what helps me do what?
What's the thing that I do that gives me the other thing?
You know, so like VectorDB helps me to do...
My goodness.
We should send this to the Figma people, I think, and then they can...
They can...
Because when you try to move fast, man, everything gets harder than when you're doing a beautiful demo for your team.
Okay.
This helps me...
My goodness, please just...
Let's just...
Sorry, this is going to make more sense if we do like this.
These help me do this, this helps me do this, and then...
Like, I'm going to...
This is going to...
I'm going to feel...
I'm probably going to feel better doing these things if these things are already there for me to call upon.
Does this resonate with how you're thinking?
100%.
100%?
Okay, good.
So, this is our...
This is where the true...
This is the bad place here.
This is the...
This is like...
I mean, because if we don't do this, then...
You know what I mean?
Everything is downstream from this, basically, right?
Do you have a sense of whether one of these feels like a bigger win?
If you were to go to the client or just for yourselves to say like, oh, summaries are now in the VectorDB versus processing of files, processing of uploaded files or Vector, does it make any difference to you?
Does one of them feel like a bigger win?
Probably files, because like the project name is like case GPT chat with files.
Uh-huh.
Okay.
Uh-huh.
Well, yeah.
So, we're currently not being summarized, like transcripts are.
So, it feels like a bigger piece, like we'll...
Right.
So, this is a little bit more like a refactoring, like you're changing the implementation versus here you're doing something new that's not possible before, right?
So, I'm just chunking this out into...
This is kind of how I'm getting to the...
This is kind of the structure of what's really like in the sort of nine boxes.
You know what I mean?
And it's like, okay.
So, this is first order of business, but we can't do this in isolation because we have to have something we're trying to actually work on.
Do you know what I mean?
So, we kind of take this together.
This is funny.
What is this?
If you wave your mouse, I think...
I never noticed that right now.
I don't know how that's useful.
It's funny.
So, okay.
So, my instinct is that like our first order of business now is to dive into this question.
Yeah.
I can pause here.
Am I just telling you what you guys already knew?
No, this is great.
So, this part is super useful.
Actually, it's like clarifies a lot because I had like kind of like similar order of business.
They're like on the same like scale, like tool calling.
Like, I thought like, okay, like figuring out what tools to call.
It's going to be like, but like the vector that actually clarifies a lot.
If I just focus on vectors and like putting the files in there, like calling the database,
they'll like solve a lot of like the angst, the unknowns.
Got it.
Okay.
Yeah.
Sorry, Chris.
I was just saying, you know, I was wondering how sometimes you get to that chunk, right?
And...
Here?
Yeah, the skeleton kind of what you build on for the chunk.
That was sort of a missing piece in my brain too.
So, this is super helpful.
Yeah.
Cool.
Okay.
I'm going to give you guys the tool of how I learned to do this outside of my head.
I mean, it's also in my head, but I feel like the way that we could like explain it.
This is something that I noticed that like really badass developers were doing in their minds.
And it's kind of like, how did you do that?
And I could never explain it until I learned this.
like a little tool. So I learned this tool from Bob. He calls it the interdependency diagram
and it's... is there such a thing as transparency? I mean, how do I...
There is. If you click on the... I think if you click up
here, like on this guy, then it...
Ah, this guy. Yeah, okay.
Transparent.
No, Phil. Okay, good. Here's the principle. I have a whole bunch of... so what I do is I have
a whole bunch of things that I know I need to do, okay? And I'm just gonna... and we do it
first as a list. So I need to process files with vectors, you know? I need to do tool calling
in chat. I need to... whoops. I need to move summary storage to vector.
And then the thing that I did in my head here was I factored out the common... it's just like math,
you know what I mean? You see... wait a minute. You know what I mean? There's a piece that both
of these things use. Therefore, like, if I know how to do that, if this was like save information
to database, I'm not going to pull database out as an unknown because I know how to do that, right?
But here, figure out vector db, you know, mechanics is a separate thing. So that's...
this was the first step that I did was I just got to... here seems to be the sort of list of things
that we kind of have to solve. And then the interdependency diagram basically works like this.
The idea is you take... and you can do this on anything. You put the things around a circle
in arbitrary positions. And here we just have four. And then you simply ask for each one.
There's the algorithm is like, does this help me do any of the rest? And you do it for each one.
So does summary storage in db help me with tool calling?
And in this case, it kind of doesn't because we already have summaries, right?
So, okay, not relevant. Does it help me with figuring out the vector db? Well, actually, I need to
figure out the vector db in order to do this. So, no. If I do this, does it help me to process files?
Well, like, only if I learn about the vector db. But then I come here to vector db and it's like, yes,
this helps me do this. Yes, this helps me do this. Does this directly help me do tool calling?
Well, not directly. I wouldn't say that I'm going to be able to do tool calling when I finish that.
But we could even say that if we wanted to, yes, we could. If we say that this is like...
There's always a question of your interpretation. But then you're going to... You could just go
around the circle and say for each one of these things, does it help me or does it not?
And what you end up with is the equivalent of this graph. You know what I mean? It's how you
end up with the fact that, like, a vector is on top. It's like it's got the most connection points.
Like, exactly. It's a bottleneck, yeah. What you end up doing algorithmically and is you... So, like, this
is going to help me with tool calling. It's actually funny because now that we talked about it,
it sounds like this might not really help me with tool calling because I already have it.
So, that's an interesting... It's kind of more and more argument that this is less... This is kind of
falling off a little bit. You know what I mean? And then when you do this on a complex problem,
you will have, like, a whole web of things. And then what you can do is you can literally count
the inputs and outputs of each thing. And then you sequence by that. And the things that have
more outputs than inputs float to the top of your sequence. And it tells you what order to solve
problems in, basically. So, that's the interdependency diagram tool.
It's a little hard to see it because with these four, it's so straightforward. But when you have a
harder problem, you can do this and you get... I have one post that I could...
Let me just see if I have it here.
No. Okay. If I find it, I'll send it to you guys. I have a post that kind of walks through.
Might be...
Yeah, here we go. Interrelate.
Oh that's funny. Okay. I have a post for it somewhere. We'll find it. I'll send it to you.
Then you'll have just a better reference in case you want to be able to do that. But okay, I don't
want to take up too much time. So we arrived at this is the hard thing. So then
what I want to be able to do is find the boundaries in the current system of where
this thing needs to happen and that's going to help us to create test boundaries.
So if I need to go research VectorDB in a pure science project,
that's different than if I need to research how to do a VectorDB for this specific problem.
So I'm just kind of wondering, can we isolate where this step would happen? So where my
head goes is I'm going to go to process files with VectorDB. This is here.
And so now the question is, all I want to do is I'm comfortable with a black box of vector magic.
You know, and like, we don't know what this is, but if I can have inputs to this and outputs to
that, that's going to help us, right? Like that's going to bound this, that, this, this, this, the,
this is the, we're, we're, we're very clearly bounding the hole where the research needs to
happen. And that's going to make the research feel much more focused and small.
Right. That makes sense. And if I can, just one context, Luchin, like how much of the VectorDB
stuff, the one missing piece of this is there's Chris Luchin and there's MJ who's, who has built
these rag agents before. And so he came on specifically to help us with this. I was just
curious, like how much of this is he solving versus you? Has he solved it? I know he's been
working on a bunch of stuff, but how much of this is he solving versus you? Has he solved it? I know
he's been working on a bunch of stuff. Yeah. He gave me a template. Like he wrote
basically a bunch of scripts that take like files and like put it in the, in the VectorDB and like,
like templates for like the tool calling to actually get the data from the VectorDB.
I started translating that like from Python to JavaScript and like just kind of putting it in
place. But just with like all the different things, I just get like way more clarity now
that this is like the central piece to focus on. So I just kind of put it to the side for a bit.
Yeah. I just, I was, the red flag for me was that like, you're trying to solve a problem that we
specifically got him to help come in and not just give you a template. He was supposed to come in
and sort of help you build it. And then, yeah, this seems like this. I figured like he gave me
everything that I needed so I can just like kind of wrap up the rest. Yeah.
So here's a place where if, if we take this, this this, there's some VectorDB work.
If we can just understand where this lives and, and, and what goes in and what comes out of it,
then this will, should feel almost like a mini project.
And when it's a mini project, Bruce, you guys can have the conversation of like, almost like,
you can think of it almost like who's on that project. You know what I mean? And like you can
have a very, very clear conversation around like sort of who's responsible versus who's advising
or, you know what I mean? Like that kind of a thing. So is this, is this a simple answer
of like where this fits into the current system and where it happens and what,
and what the outputs are and, and like kind of where the seams are in the system?
I think so. Yeah. Just any file that's like uploaded, just get summary. We can like tweak
the prompt for getting something and we just save that. And then just like tool calling of like,
Hey, I need like this question answered about like using files.
So like high level, but to me, I don't feel like too concerned about that. It feels like it doesn't
need too much more resolution. Okay. So I'm, I'm I'm inclined to delete this black box. I don't
feel like we need to have this like intentionally scary unknown marked out. What I'm hearing is
there is a when does that happen?
So the S3 upload happens, right?
And where does this processing begin of this new step
where we're going to do the prompt and all of this?
I'm not sure I understand the question.
OK.
So there's a new step that needs to happen
where the uploaded file, something
needs to happen to it so it has a representation in the vector
DB.
Is that right?
Yeah.
So maybe it sounds like a stupid question,
but when does that process start?
So immediately after the file is uploaded,
is this when a job is enqueued or it just
directly something gets fired?
Yeah, I would do it that way.
OK.
So when, so here's one other thing we can do.
I'm just going to say we are going to add job to queue
to process or to even to vectorize.
And then when the vectorized job gets triggered,
so at some time this is going to get triggered,
then you just gave a high level.
What needs to happen?
So I've got an uploaded file in S3,
and I'm supposed to be vectorizing it.
What's the first thing that needs to happen?
Extract the contents of the file itself.
This could be a rabbit hole.
It's unknown how much because some of them
might be just require OCR.
Some of them are fine.
I can just use the existing libraries.
Uh-huh.
OK.
This other one, I'm not sure.
It's like seems optional, but like a clean up?
Yeah, probably not optional.
What does clean up mean?
A bunch of lines and like dash, dash, dash, dash, dash
to like signify like a just divider
or a bunch of like useless stuff.
Formatting maybe?
Uh-huh.
Um, I think one thing we said as well was,
I can't remember if it was this project or another one,
where we did say we would limit it to,
because you said like the OCR stuff.
Like I think we said we would just basically keep it to,
is it this project?
I may be getting them confused.
But there was a different formatting
where like it's basically a readable PDF.
Or is that for the, I think it's the file filling out
thing, the next project.
Yeah.
Yeah, that's like a huge rabbit hole for another project.
So this is still a valid question for this project.
Like how many of these files are like difficult to read?
Yeah, I think that's a good boundary
to draw somewhere there, because that probably
is an infinite rabbit hole.
Another thing that we're going to hear
is like putting in like a markdown format
and having like proper headings.
So knowing like which questions they'll ask
will be like I think relevant to like summarizing
or like structuring.
I think it's a better word, like structuring the contents.
So did I hear that right?
That the questions they want to ask from the chat tooling
are influencing how you're going to extract the contents?
Yeah, and we have a list of those desired prompts
that they would love to eventually ask.
Like, what's the next action to move this claim forward?
Like it would need to know a lot more.
Like what contents are missing on this claim,
or stuff like that.
OK, I'm going to just do something here real quick.
I'm going to, so I'm just making a top level thing here
for all of this.
I just pulled this out into saying this gets triggered.
And I think something that's interesting here is,
I don't know how to call this out exactly.
But what I'm just noting is the expected prompts
are an input to you when you're designing this.
Like, when you are trying to solve how to extract contents so that it works well, you're
referring to the expected prompts, okay.
Yeah, but it's not actually an input here, so okay, that's fine, that's just something
that we can keep in mind.
It's almost like I kind of have a comment in my head here, but I don't know how to put
it here, you know what I mean?
It's kind of like, yeah, that would be more for like formatting, for like extracting content
that's pretty standard thing, like we.
That's more for formatting, right?
It's the file as the input, and just the super launch, and that's the output.
Is cleanup different than formatting?
I think so, yeah, formatting you can think of like a markdown file with like different
headings and sections.
And I would add another, before it like slips my mind, but like determining like file type,
right?
Because some files are like medical evidence, some files are like.
That's this here, yeah?
Yeah, yeah.
I think they'll be at the tool calling level, like to know like which type of files to query.
This will be at the tool calling level to know which type of files to query.
When you say that, are you referring to the tool calling from the chat after this has
all happened?
Yeah.
Yeah, I'm kind of going backwards.
What are the like expected inputs to the chat?
Cool.
Yeah, yeah.
That's what I, the output from here.
Yeah.
That's interesting how thinking about chat is actually helping us to do this.
Chat is like centerpiece.
That's the whole point of doing this.
Yeah, that's interesting.
I'm kind of updating my mental model here where it's almost like I want to have a, I
want to have like a, so like in my mental model now, it's kind of like there's a, I
want to at least have an, I want to have like a, I want to at least have like an understanding
of tool calling in chat as an input to how we do this X, to how we do this processing
here.
You know what I mean?
It's like, that's another thing, like we're not doing like, we're not doing this completely
without any understanding.
Like you're actually trying to think about like what needs to be happening in the tool
calling as an input to this.
And then this is like tool calling actually works.
You know what I mean?
There's a difference between kind of, well, yeah, we need to have, we do need to have
like some sense of what tool calling means.
Sorry, this is just like, this is killing me, these, these, these arrows here.
What tool calling itself is that I feel is pretty clear, but what is not clear is like,
which tools do we need for the chat to be able to call?
Uh-huh.
Same one.
I think that it's still a valid point.
Like tool calling, I don't know how familiar you are with it.
So I'll just give you like the footnotes.
When I think of tool calling, I think of like cloud code deciding that it's going to use
grep versus it's going to run some tests or versus something like that.
No.
I think Lucian just lost connectivity for a sec.
It may.
Yeah.
I'll be back.
Um, I did grab, um, the, uh, the, just the chat interface of where that sits is sitting
in the kind of often over to the, I can bring it over to you if you want, because there's
this interface is sort of what questions you can ask in this option B, I think, uh, towards
the bottom left.
Yeah.
Um, this, this does kind of, um, this piece right here is kind of where the, this is sort
of the interface piece of what we're building.
So there's going to be a chat, um, detail page, uh, where you click on a chat, you know,
like chat GPT sort of style, um, and that, uh, you would see your past conversations.
You would see, you could start a new chat, the new chat would, you know, have these pieces
and that's where it would then, when you send the message, then it's start kicks off this
whole process.
So we started like building out, you know, what happens with this, there's sort of a
default prompt that gets included.
There's like, um, you know, there's going to be other contexts that's added.
Then we take all of that and we kick it over to, to the open AI.
Which is going to then tool it.
And then that's where you started building out these tools here, like
list files in claim, like, Oh, we have these 10 files, but we need these three
other three, so that's like one tool potentially that could be used.
I see.
Um, and this is a, um, this is like a tool, um, uh, in the sense of like, is
the tool actually just another prompt, um, that's kind of named that, that, that,
uh, okay, got it.
Um, um, so it's like, do you, do you, I'm sorry.
What was that?
Uh, I don't know if this is nitpicking or it's actually useful, but these are
like functions, like for example, if you ask Chad GPT, like, Hey, what's the
weather today, there's like a list of like, you can fetch this API to answer
this and then put it back in like human readable.
Uh, because this is actually something you're going to go via, you're going to
go back to the actual database and like fetch stuff that treachery needs to know.
I got it.
Okay.
Uh-huh.
Got it.
Okay.
Um, so, uh, the, I think the thing that, that we just came to is, um, to what
extent, so we are over, uh, we are over here to what extent do we need to
be answering questions about the tools in order to get to a place where we
understand more or less what we think the work is for, for, for this kind
of piece of the, of the, of the workflow.
Are there, are there unknowns here that, that relate to the tools?
Are there interdependencies there?
I mean, to me, it's becoming like much clearer now that the order, like the
dependency was like kind of look at the expected prompts, the case agents will
have, and then these will inform what tool, tool calls will need to answer those
questions.
And then the tool calls will inform how we're going to vectorize the uploaded
files.
Uh-huh.
Yeah.
Starting with like that, like what question is the prompt going to solve
and then kind of work backwards into that.
So, um, so I'm just gonna, I'm not gonna insist on this.
Is this was just some stuff that's in like a way of looking at what's in a way of,
you know, what's, what was in my head at the time.
But, um, what I heard was, um, look at the expected prompts.
That's going to help us understand what tools we'll need.
Yep.
That's going to help us understand how to make choices when vectorizing.
Yep.
Um,
well, it's really, okay.
Um, good.
Okay.
Um, are these, I mean, these feel like, um, things that you can
clarify, um, that aren't mysterious.
Yeah.
I feel equipped.
Yeah.
Um, uh,
okay.
Would you take each prompt?
Like, would this be a good time to take, like, almost treat the prompt like a UI
element or, or, you know, like I'm going to hit this button and then stuff's going
to happen, um, like basically breadboard each of the prompts, would that be
worthwhile exercise or is that unrealistic?
Um, uh, so we always have the freedom to decide how much detail to go into here.
Right.
And, um, and that's going to depend on how much it feels like, uh, it's going to be
fine or how much we have a little bit of this, like back of our, back of our heads
doubt, like maybe there's something we're missing there that we should be thinking
about, you know?
Um, so it can go either way.
Um, in terms of like, um, method of like how to map it out.
Totally.
Because that's actually how a tool call works.
As far as I understand, do you know what I mean?
Like there's going to be the general anatomy of a tool call is there's going
to be some kind of a prompt and then there is a, uh, uh, um, I mean, of course,
like, it's not like a, just like an if, if, if, but it kind of feels like that.
Do you know what I mean?
Like the, the, the, the LLM is going to, is going to choose from among the
available tools, like what it thinks it needs to, to use, you know?
So this is a little bit like, um.
this is like a black like from so like from submit prompt in or not even what we don't
call it a prompt but like from like well let's yeah let's just say like we submit the chat message
there's going to be some sense of um somehow it's going to it's going to
it's going to go to open ai you know what i mean uh it open open ai is going to choose
from the available tools and then we're going to get to this place where we have like
if claims list is invoked uh right um uh and so on you know and and then and and you could totally
um view this as a ui thing because it's almost like you've navigated to the claims list
you know what i mean like it's a it's a it's a thing that has a view because there's a there's
a kind of a claims list has a view model or a read model like there's certain data that it expects to
get right so if you want to think about it you could do that where you could say like um claims
list as the tool as as as the tool view and i'm just going to uh
let's say that i don't remember what we were using you guys had some color for
uh orange you were using um for uh for a place for a ui place right then that claims list is
going to have affordances in it i think in this case the tool is just the uh the um we can
literally just write here like bullet list of claims but if these are going to be linked for
example and you're going to click on them to jump to a claim that's the kind of thing that you would
specify as a ui element here right like are these just is this just a list of claims in in plain
text and it's like claim numbers or are these actually like clickable links that take you to
the detail pages for those claims uh that's the kind of thing that you are designing here right
so like what is in this and then um this claims list tool just like everything else we've talked
about it's going to expect a certain kind of data shape if we think of it from the view standpoint
here you know what i mean so like if this if this is the view in the tool that openai is going to
render then it needs to have um well you know i'm the simplest way to say that is that there's
going to be some list of claims that it's going to take in as the as the input that it needs to
know and this is giving you the now we we have a way to talk about the shape of the data that
we need for this tool to be rendered yeah um okay that was just to kind of like unpack the answer to
to to bruce to your to your to your question there um how do you guys feel um uh do you want
to um we're an hour and a half in here out of our uh you know three hours max do you feel like you
need five minutes to breathe you want to take a five minute break and then come back so let's take
five and uh um uh and uh and take a little breather and then and then come back and then we can decide
where you want to go next cool okay see you in a bit
hey bruce you know i can talk i don't know
no it's good i'm going to grab some some water real quick i'll be right
you
all right guys cool so um let's do a little check-in on um kind of where did
where did we get to versus where you started
i'll go last
i guess i'll go first well uh feeling way less worried about this uh project um i see that
the value of like
and also the level of definition we should have gone on the breadboarding the current way
um when you say you you feel less worried what like kind of like why
what's different maybe it's like your calm uh demeanor that like but um the project therapy
from ryan yeah no no no demeanor can compete with uh with a problem you don't know how to solve you
know so i think there's more to it than that um the dependency like um list really clarifies it
um i mean there's like fewer things to look at it just had like too many things to look at uh-huh
uh-huh so when so so so when you say there's fewer things to look at kind of what's in your head
uh now versus before like like like before it was kind of like all the stuff in the project
was kind of just all there like side by side
yes there's also um there was like one moment that clarified something i'll get back to like the
general question was like when you put in the like black box of like vector magic is like what goes
in the input and the output of that kind of guides what you're going to research and i'm just like a
general generally curious person so i kind of went off on like what our vector database in general
so that like reduces it a lot that's a good one okay uh-huh so bounding
bounding the problem with like in this specific case i need to get from here to here
uh shrank the research problem yeah yeah and i think a bigger piece is like just that whole like
dependency graph of like these prompts are expected and we're going to focus on these
ones and that informs what tool calls we're going to do and the tool calls what we do
is going to inform like what i'm going to research and how i'm going to do the
the vectorizing and like just then like
Piece alone, like if I use that as my north star
I feel like come back and just like slash everything that doesn't matter to this
Got it. Uh-huh
so you have a
it's kind of like you have like a
Almost like a plan of attack of like how I'm gonna make my way through this problem
Yeah, got it. Uh-huh
Cool, okay
I
Anybody else any any any to any difference from where we are now versus where we started
Yeah, my mind so, you know, I'm um
I'm not working on this specific project. So I
Just in this is my first company working
This is my first time working at a company where shape-up theory was the primary approach so
I think I'm just getting
Just more knowledge into shape-up mentality
Some mental notes on like how to
Kind of approach breadboarding which I can see now Bruce who you're saying how it felt like we weren't
Doing like the full unrolled breadboard, you know
Yeah
That's cool
Yeah, so it's nice. Um, just kind of giving that context so I can like have these notes for what I am
You know doing my own breadboarding in the future
I think for me
The last session maybe the second session we had with you, you know, it's kind of a bad connection
I wasn't quite getting
The nine grid and just kind of how we got there and Lucien helped me as we were talking through earlier
but when you were I go back to that skeleton of how you're trying to like slice the problem up because I really
Didn't know where to start on some of this because like Lucien said there's a lot of stuff right and it's very yeah
Well, let's do the whole anatomy of that and then the anatomy it's like no we don't do that
Like what am I touching right now? And I think you know Bruce when your comments was I think we're
Looking at stuff. We don't necessarily need to right now just for chatting with files, which is true
So as interesting as some of these, you know strings are to pull on we need a really, you know
Iron clad focus on this but knowing how to solve the problems along with that
What Lucien was saying about the left and right bounds of you know, constraining your uncertainty if you will
Was very helpful to me because I'm sort of someone who's very visual
I need
The figmas of the world of whiteboards to kind of sketch things out and see where they come from
But some of these are like, where do I start right? I think having that
I know you called it that the the diagramming for the the programmers thing. I need to write that down, but
Having that approach really gives you a place to start to really get a finger
Hold on not even a foothold with a finger
Hold on and then you start kind of clearing that fog of war if you will, right? Yeah
Super kind of light bulb moment for me
And
I'm gonna just intercept Bruce before you go because I wanted to push back on something
maybe we'll just like put a bookmark on that and talk about that later, but I kind of disagree about like
You put a message in the chat
Like we're out of scope of these the entire project is limited to chatting with files only and I questioned that because I think
that's not a good framing because
Part of why we named the project like this is they have their custom GPT and it's tedious to like download files from the
App and then upload to the custom GPT
but I
Think the framing it as we're trying to answer like some questions. It's like way more useful
I guess in like delivering something that's usable because like just chatting with files is like
It's cool like it removes a step but like what's the concrete value of that? So yeah, I just questioned that
I mean
Yeah
I would say sorry Bruce. I didn't mean to step on your
Response, but at the end of the day, these are all components to serve a bigger thing
The the end game is this right now. I'm a case manager
It's really painful to get a complete coherent picture of where all my clients are think of I have 30 people
I need to take care of right if I it takes me
I don't know even 15 minutes to get a complete picture of where this one client is that all this DBQ needs to fill this
SDR is missing this form is missing. I can't I got a schedule the medical appointment times 30, right?
That's an enormous amount of cognitive load that an enormous amount of pain
So this little little section of chatting with files is really important
Little section of channel files and then that extended AI about form-filling the files
Then adding in the UI improvements. All of these are three legs of a stool that overall
completely help have like this perfect assistant
with perfect memory, right?
Helping the case managers to serve more people
and get those claims in faster
because the government does not move quickly, right?
But if we can get the claims in, we start the clock running
and that's the benefit.
And I think that I am more of a process person.
So I naturally gravitate toward,
hey, this process is really, really painful.
I wanna work on the sexy thing of improving the workflow.
But to serve that, right?
The underlying support is chatting with files
and then form-filling the files.
So, you know, it's hard for me to kind of like
pull out of the operational kind of workflow thing
and just focus on the files thing just for me.
I know that about myself.
But I guess I'm trying to, in my role, Ryan,
I'm trying to provide more, not color commentary,
but exact, like I'm a champion for this me, right?
Because I work with them very closely.
I can speak for them in what they want
and how they're feeling it
and how this should kind of terminate,
the process should terminate.
So again, solving some of these problems are like,
it's not like what color is the button,
but it's more like, hey, they're doing it this way now.
Can we kill five steps and make it one step?
You know, so.
This is a juicy moment in a shaping session
where the question is coming up of actually what's in scope
and what's out of scope for this project.
And this is,
this is where, okay.
The way that I like to work on this
is to use this terminology
of the shaping versus the framing.
The shaping is like,
what are we actually doing in this time box?
So like this thing we're working on
that's gonna ship in, what is it?
Two weeks now from now or whatever.
This is the thing that we're shaping.
And then like, but if it's not doing
what it needs to be doing in order for the customer
and to actually feel like we gave them the right thing
and we're making progress,
then like, well, that's the question, right?
Like, is this thing actually doing what it needs to do
in order for it to feel like it's a win?
And so that's kind of what the framing piece is,
is like, what are the expectations?
Like, what's the thing we're actually solving here
and what are the expectations
from the sort of stakeholder point of view?
The thing that is tricky here is,
Chris, when you mentioned,
there's this bigger vision that we're trying to get to.
The thing that kind of feels a little bit fuzzy to me
as an outsider right now,
and this is often a challenge in a bigger effort,
is like, there's the framing at the level of like,
if you think of this as like,
this isn't one four-week effort,
but there's gonna be like many, many slices of work
to this thing,
then there's kind of this like monster project
where like at the end of the monster project,
they get this value and this is why we're doing it.
And every step we get closer to that is a win.
But from the standpoint of selling the weeks,
like, so kind of more from Bruce, where you sit,
there is this other dynamic of what is the rate
of the delivery of the progress?
Like, how do we make it feel like this four weeks
we got somewhere where they feel like they're seeing
the right movement toward that bigger thing?
Do you know what I mean?
So like, that to me kind of feels like the thing
that is fuzzy right now,
and maybe it's clear and it's just fuzzy to me
as an outsider,
which is like when the conversation is happening
about like, is this in scope or out of scope?
Is that a conversation about like,
well, if it's out of scope,
then we'll just get it in the next round?
Or is it like, if we leave it out,
we're gonna go back to them at the end of this four weeks
and they're gonna say, I'm not impressed.
Do you know what I mean?
That's not the expectation that you set
with what was gonna come out of this.
So what I would,
so one way to try and dig into this a little bit
is to flip it to Bruce.
And like, Bruce, like from the sales side of this,
you know what I mean?
Like, how would you describe like the we're here
and this is where we're gonna be at the end of this
in terms of like what you're solving
or the progress they're expecting to see?
Great, yeah.
So this has been a great session.
So Bruce, I think Ryan just saved this project
potentially in this last two hours.
I think this was a dumpster fire
heading off the end of a cliff.
So I didn't realize it was quite as bad as it was.
I got some hints of this
from some of the comments you guys made yesterday.
And then we said, let's prioritize, let's get clarity.
So I think the cool thing is that we're learning like
getting clear.
about any project is going to be our secret weapon for everything.
It's like, oh, this feels hard.
We feel overwhelmed.
It probably always comes down to a clarity problem.
Like, oh, we're not clear.
And so we were clear.
And so things got super fuzzy.
And I think we're coming back to clarity.
So I think, I mean, I'm going to try
to separate deeper team process and things just
to try to bring it just in the context of this project.
So the first problem I saw was just
that we have, obviously, right off the bat, scope creep.
Like, there's a screenshot of the framed document
we sent to them, which is like, we're
going to let you chat with 20 different files.
And you can ask questions about these files.
And it's going to give you responses.
Very, very simple.
We're not going out.
And we spent a bit of time talking
about how transcripts work today.
They don't affect this project, actually.
And so I had to come back to this framed document.
Now, there's probably interconnectivities and things
that we missed on this initial version.
But we also have to come back from a sales standpoint.
It's like, hey, we sold this.
And so initially, it's like, yeah,
wouldn't it be great if it did all?
That's exactly what every client is going to want,
no matter what.
So we have to be like, yep, but remember files?
Like, that should have been something
we pushed back really hard on on the very first call.
So as soon as they kind of step out of bounds,
we can remind them that, like, hey, we're.
So there's kind of a skill of managing the client's
expectations now.
So luckily, we have a call with them in a few hours.
So we'll get ready for that and prep all of this
to make more sense for that.
So I think that was my first thing, which
is like, yeah, scope creep.
Now, whether it should have been files only or not,
I think this is like a different conversation.
Maybe we wrote down this, but now it's
like other conversations have led to,
oh, what we really need is a this.
Well, that thing could be two six-week projects.
I don't know.
Like, we've just blown right past any kind of framing
documents that we put together.
And we kind of treat the framed document
kind of like our agreement.
We have an overarching legal agreement with them,
just on all code and how IP works and all that.
Right, but that's like the master of services agreement.
And then the framed document is more
like the scope of work for an individual chunk, right?
Yeah, and it needs to be clear.
And we do have some out of bounds.
I'm pulling it up now.
To me, it still feels pretty clear where it was.
But it's as if we're talking about two different projects,
from what we're talking about even in the breadboarding here.
Because I think that framed document
is there to give the client boundaries
and to give our team boundaries.
Because I think that's where it does.
You start to open the code, and you start to say, well,
this connects to that.
And then, well, really, what we need is a that.
And then it's like, all of a sudden, there's 50 projects.
Like, there's just so many ways we can go.
And there's not this clear direction.
So I think that's something I think we got way off on.
And I think just giving ourselves enough time.
So maybe when we framed it, maybe we
didn't frame it all the way we should.
The document we produced and came out that we have today
isn't accurate.
I don't know if we have the flexibility
to change framed documents.
That would be a question for you, Ryan,
like your thoughts on, can framed documents change,
or should they?
We've done other framed projects.
I did one with Aira for an e-commerce ERP platform.
And I feel like that was very tight.
It felt very concrete to me, I think to Aira,
and even the client.
He could see very specifics what this is and isn't going to be.
So I think we have done it, in other cases, a bit better.
And again, this project was started maybe four weeks ago.
The one I did with Aira was two weeks.
So we're probably just a couple of weeks ahead in our thinking.
Maybe an extra call from you or a few extra breadboarding
sessions better at it.
So it's literally like we're just doing it on,
like we're kind of learning it and also
trying to implement all the things we're learning at the same time.
So I'm not picking at anything.
If anything, I think the biggest culprit on this whole project
was me, because we started it, framed it,
and then I handed it off to you guys.
And then I sort of backed off for like a week or two.
And so this is my first re-entry into the project.
So I think that was the biggest sort of bug in the system,
was me taking a step back too quickly.
And then we also threw in another developer.
So I think any of my feedback is not Lucian or Chris,
or I feel like you guys are trying, like we all kind of get
the vision of where the client wants, but we have to kind of put it
through our process, or else we're going to end up with this.
Because what's.
This happens all the time.
We're like, Oh, we're going to add, let's just add these other two things.
And now like we missed the four week deadline.
Now we're in week six, we're burning hours faster.
Like we're over budget and we're over schedule.
And the clients like now has these crazy expectations because I'd rather like do
something very small and throw one or like, and do that really, really well.
Then now they kind of have this expectations.
Like I'm going to, like, as soon as we saw things, like, I want to ask it, um,
what, you know, some of these prompts we should just be like, Oh, that, Nope.
Can't do that one.
Oh, we'll do this one that you sent us.
We can do that.
You know, like just immediate feedback to them, like, so that they're, because
we can see, Oh, like they think they're getting something they think they're
getting, like, you know, we're doing a and B, but they think they're getting
like CD and F like, we need to just be very clear about, Hey, remember this
document now, and that's where a lot of this, this fuzziness comes from.
Um, I think so.
I just want to make sure that we, we hold that frame document as like, okay.
Like at the beginning of every meeting is like, let's read
through this document again.
Like, are we on track?
Are we like within the boundaries of this?
Because, and again, we, it would also be worth potentially looking at that
document and saying like, well, this, like we probably framed it 40% and it
could have been 80 or 90%, right?
Like it maybe isn't very clear yet.
Um, and maybe it wasn't concrete enough that when we're in a call with a client
or we're digging into the code, we can't just be like, Oh no, we've
identified this issue already.
And we have a hard boundary here.
So I want to pause there.
Cause I had a couple of questions in there for Ryan and I'll
let you answer mainly around.
So that's fantastic.
Um, um, so one thing I just want to highlight, um, uh, is like in the
contrast between Bruce, the way you answered and Chris, the way you answered.
I want to make sure that like, um, uh, I, those things often, they come out to
the team sounding like complete opposite answers, like one of them is a big
expansive vision and one of them is like, we're not doing this and we're not
doing that and we're like only scoping it down to this and they can feel very
like opposed and this can also lead to like, if we don't, if we don't have a
tool for how to like talk about those things, it can lead to a bunch of weird
misunderstandings on the, under the surface of things also.
And like those things, they totally, um, they totally coexist.
There's always the bigger vision that we want to get to.
And, and like the, the, the specific sale of a certain amount of money in a certain
amount of time to make a certain amount of progress.
So, um, I like to think of the frame as the increment of progress.
So like the frame is the agreement that we made for a specific unit of like time,
money, team, you know what I mean?
And there is a thing that's bigger than the frame that it, especially when you're
the one who's in touch with the SME, like that is like the, the reason we're doing
all of this, there is this like bigger thing.
And the word I like to use for that, just to put a word on it is strategy.
Like there's a bigger thing we're trying to do.
There's a reason why we're investing in this particular increment of progress,
you know, and, um, when, when we're, when we're doing the framing that a lot of
the, the, the coming to the agreement of the frame with the client is about talking
through that bigger thing and getting, getting aligned on what the
meaningful smaller step is.
Do you know what I mean?
So like, like how does this relate to the bigger picture is very
much, um, part of the sales process.
Yep.
And then, and then when, once we're in shaping, um, we don't want to have this
bad, we can easily get into this frustration of like, yeah, but there's
all this other stuff that we're not doing.
And so we kind of want to feel like the frame is like our, it's like our
umbrella, you know, that like, it's like our little safety zone.
Like, like if I'm in the frame, I'm fine because like when we're in the shaping
work, we're already downstream from the, you know what I mean?
Like the project's been bought, like the, the, the thing's been signed,
like the expectations have been set.
So everything that we do inside of the shaping should fee, we, we, we want
this, the frame to feel like these really solid outer walls, you know
what I mean?
Like they're the outer walls.
We're like, as long as I'm inside of these walls and I'm, and I'm, I'm
meeting, it's actually the same thing.
Um, Lucian that happened with the black box on the vector thing.
When we put the inputs and outputs on it, you're like, oh, okay.
I just have to make this box work in this specific situation.
That's exactly what a frame is. It's like, oh, we're in this moment in time and we've got these inputs and we need to produce this output and like this is where we're trying to get to together and then everything is all about the inside of the box.
So that's kind of like where those two things like fit together.
I'll pause before. I have some other things based, Bruce, what you were saying about framing, especially I want to comment on changing the frame and the feeling about is the frame detailed enough, but I just wanted to see all that monologue.
Is there anything, did that, did that make any sense? Like, yeah, like, like, like, like, like Chris, does that connect at all? Or, or, like, is there a, is there a link there between the bigger picture and the smaller picture? Or, I don't know if I was too abstract or?
No, I think that was helpful. I mean, obviously, when you said, do, do we think there, we can see that increment of progress? I mean, certainly I do. And actually Lucian, we were talking about this the other day of the SME I'm talking with, the team member, the client, he already uses his own chat GBT.
So one of my little tasks, and he kind of sent some stuff over during our conversation was, what does, what do they currently do with their chat GBT? Right, exactly. How do they use it in the workflow? And what outputs is it giving them? Because that's the baseline, right? The, my worst nightmare is, oh, hey, we spent four weeks and built this thing. And it was like the same capabilities or slightly less than what they have. Like, that's the worst thing, right? It's like, what did we spend money on? What do we buy?
You know, so to Bruce's kind of the sales side of it, I want to know what that, what the current state is. And we want to be this, because we promised this, right, whatever this is, right. And then even better, if, you know, we have more time, we do this plus one, right, which is out of scope. But, you know, that's kind of how I look at it.
So the increment of progress to me is obviously all the stuff in that document that Bruce pasted in the zoom chat. But the way that it works has to work with, I mean, the way they work, that sounds like it's a tautology, but right. So, so when we framed it, when we shaped it, when we gave them this, this list of things, obviously, we listened to them, we got the input, and we, we trusted our earlier selves to put this together intelligently, right.
And so I definitely want to make sure we don't diverge from this and give them what we promised, right. So I'm trying to fill that in by getting this, all this contextual information. So to do it right. On the other hand, I will say, I think, I don't think we wasted time in the sense of understanding this, because this is the first time we've done anything with, I've never heard of a vector database before this project, right, or a rag, right.
So I think there is some useful learning, both from an intellectual curiosity perspective, but also to inform how we move forward. I think role, to Bruce's point, role definition is important. And knowing who's doing what, when is of course important, right. If you're on the kitchen, and you're all, you know, cutting potatoes, when someone's supposed to be boiling potatoes, that's not going to make the meal go faster. Right. So there are a lot of little learnings that we've, we've, we've equipped ourselves with. And I think, again, to Bruce's point, we're getting smarter and sharper.
Two weeks ago selves are going to be smarter than four weeks ago of ourselves. And we just have to keep, you know, improving how we do this. And again, I do agree with Bruce that if we keep this as our sort of our North Star, and we don't deviate from this, like that'll help us a lot more as we say, Hey, is this in scope or out scope and be very, very stringent with ourselves and with the client about what the expectations are. So yeah, this is this is eye opening.
Um, cool. Okay. Bruce, to your question about like the detailing of the frame and changing the frame.
The
I'll make a couple comments. And, you know, framing is just like shaping in the sense that like, if we really want to be like, how would we shape this differently than we have a shaping session, and then we experience it, you know what I mean? So the framing is like something where it's its own work step where we can also dive in there, you know.
And the, the, the, the two kind of places where we, how to say this, the more that the frame describes an outcome, and the less that it describes exactly how we're going to do it, the more that we are going to feel secure that it's clear.
So like, if the frame is like,
I'm going to be able to get answers to XYZ via chat instead of
of like in terms of like almost like in terms of expected prompts, you know what I mean?
Like, I'm going to be able to get out of all the different things that take time.
Let me actually, I'll just share my screen here.
This problem definition here is, as I understand it, the problem definition of the bigger thing,
not the four weeks.
Is that right?
I mean, in my mind, it was the works, we're really speaking a lot to like large sets of PDF
documents. And then we're speaking very much to that, like, we don't mention the word
transcripts here, where I think that to me, the strategy or that bigger picture thing is, I
want to get a holistic view of everything that's that's going on with this, you know, with
this person somehow, it could be, it was kind of, it could could be both, I do see the word
transcript now, like in the second paragraph. So that could just been like, a slip of
again, maybe us not being super clear on the distinction between transcripts and files, for
example.
I'm trying to see what we can do here without fully going into a framing session.
Mm hmm.
The feedback's good. Like if that sounds to you more like the bigger vision of like, here's what
we want to do. The next three to four months, like would be this this overarching, you know,
place we're gonna land.
Yeah, so what I heard when you described the problem to me, I even I did a quick sketch,
because I didn't know if we might need it. I did a quick sketch of like, we're doing 150 claims a
month right now. Each claim involves 20 plus files, interviews and lots of data. Currently,
they're pulling files into the computer, dropping some things into chat GPT, trying to
summarize. And like, this workflow is like, it's too time consuming for this day. And it's
bottlenecking their ability to take on the amount of cases that they need to take on, they
want to be able to take on more cases. Is this kind of like what the bigger picture is about?
Yes. So yeah, that's sort of big pictures. Like, yeah, making it more efficient, make it more
accurate. And right now, there's only like one or two people that can do this, like perfect, like
really well. And that's because we can get through this company. Yeah, because they're doing it all
just out of experience, not with with any tools.
Right. So there's a story here, there's a basic, the basic framing here. And this is, so this,
this is like, this is the crash course in framing. The thing that starts to really help is to split
the world into two things. Only describing the current reality and what's bad about that. And
then describing the after state. And like, this is like an after that I want to pay for, you know, I
don't actually understand the after state here well enough yet, to to to really nail it down.
There's something where, for some reason, only one or two people being able to do this is not okay
with them to the point that they're actually willing to spend money on improving this. I don't
know if that's like, I don't know what the story is behind that. Do you know what I mean? So I don't
know if this what what what what it means. Sometimes you can look at this and you can see, obviously,
this sounds inefficient. But there's a difference between it's inefficient. And we're going to live
with it versus it's inefficient. And now is the time to invest. Yeah. So from a sales standpoint,
I would, I would want to, you know what I mean, like, I don't want to, you know, I don't want to
I would want to, you know what I mean, like, have that answered.
Then that if that's the bigger thing, I don't actually know what this is, like,
that what I would be trying to do, then is if if if this is the bigger thing, and we can't,
we can't achieve, like, in four weeks, I don't think the promise you're making is you will now
be able to scale this because other people in the org will be able to do the workflow.
Yeah, definitely.
So then the so then the question is, how do we define a segment of this? How do we define a
slice of this? But in the same way? So like, the thing the thing that that I'm that that I'm
missing. And by the way, framing is a bit of a black art, to be honest, it's it's a blacker art
than shaping is. It's so like, we hadn't even heard of it until we started working with you.
Yeah, none of so none of this is criticism.
because it's really like a dark art, you know.
What very often happens is the problem
is easier to talk about,
so we do a pretty good job of describing the problem,
but the outcome is harder to talk about.
This is how you will know that we succeeded.
And so what we tend to do,
because the outcome is hard to talk about,
we tend to describe the outcome
as a bunch of supply-side stuff that's gonna get built.
So at the end of this,
you're gonna have a Ragnology engine,
and you're gonna have a chat interface,
and you know what I mean?
There's a short list of success just at the bottom,
because I'd make sure we didn't miss it down here.
Like we had the success criteria,
so these are the checkboxes to hit.
So it says answers based only on client PDFs.
Like it was very specific.
This is great.
So here, I'm gonna hack a frame, okay?
We're not gonna care about some ideal thing.
I'm just gonna hack this. So here's what I'm gonna do.
I'm gonna ...
We're just gonna do like this.
You can do it in Miro if you want.
We can translate it.
I actually like having it in the shape as a reference,
so I'll just ...
I have a boiled-down version of it,
even a way of kind of sticking it in the corner.
Sorry, one second.
Guys, can you give me 30 seconds just to receive a key
that someone is dropping off?
That's great. I'll be right back.
Sweet. I think that's good.
So I think ...
I mean, one thing I'm seeing as well,
that success criteria is a north star.
The more I read the frame documents,
I'm more confident than ever
that we did a great job at actually framing it.
I just think we sort of glanced at it
and then forgot about it
and then didn't really treat it like the legal agreement
of what we were ...
That's exactly what happened.
Yeah.
That's nice, but it's cooler.
So that's important, because I think ...
I was thinking we were like ...
It seems confusing, but I'm glad ...
So I'm glad we were ...
That's actually one of the pros of this,
is that I'm seeing like, hey,
we did actually do a pretty good job two weeks ago
when we presented it to them.
I do think there's probably some coordination
we need to make after this call.
Because we started on the second.
There's little things, like the dates
weren't even in the project in Notion.
I'm not being picky, but I just added them now.
So we have 19 days left.
I've been texting with MJ on the side as well.
So I'm going to do a call with him.
I think we got off a little bit on a bumpy start there.
And then right now, it's like we've planned to do June 30th.
So what we do need to do is if we get super clear today
or tomorrow, either push it out a week potentially,
because that is one of the knobs we can turn
to give you guys more free.
But let's coordinate that after.
Ryan doesn't have to be dragged into that, those details.
Okay, so we left off.
I was just going to hack a frame together.
So I'm just going to take this problem definition,
and I'm going to skip the solution.
And I'm just going to go straight to success criteria.
And then I'm going to add one thing that's missing.
What this is not.
There's a, yeah, we didn't do a lot of this.
Like we did a little out of bounds section,
but it's just two things which would have been,
we did not mention transcripts specifically,
although it's not mentioned anywhere in the solution
or the success criteria.
So the way that I'm thinking when I'm framing
is all framing conversations
are very important conversations
where misunderstandings are deeply costly.
This is, if I'm going to sign a contract with a client,
You know what I mean?
Like we're kicking off a project and what I need is for it to be so boiled down that
no one will miss anything.
You know what I mean?
If I really, really boil it down, I can be sure that the things that I'm worried about
as a, to sort of protect my own skin, you know, are not going to get missed.
So this does a really good job of explaining that you understand the problem that you're
solving, that you are aligned on like why this matters and why this should happen now.
This is a perfectly, perfectly acceptable way to describe like, this is how you know
that the project worked, you know.
The more that you leave out the how, the gladder you will be because that's your latitude.
That's your freedom of movement, do you know what I mean?
And the more that you, the opposite is also true.
The more that you put in, like there's going to be a vector DB there or there's going to
be, because maybe you don't even, there's a, like, there is, I mean, even the state
of the art of rag is moving so much.
Like you could decide at the last minute, like we actually think we can just get by
with a bigger context window and blah, blah, blah, you know what I mean?
Like the more that this is boiled down to this is what you're going to be expecting
to test when we bring this to you, you know, or this is how you're going to know that we
did our job.
And then there's going to be something here that, you know, you kind of don't want to
get sucked into, or for you was important, that was out of scope, that you kind of like,
it was in the negotiation, but it kind of needs to be noted when it was signed.
And let's say that that was like, this is about files, not transcripts.
If I give this to a client and it's literally only this, and then there's going to be, there's
going to be, you know, there's going to be, if this is a, I mean, there's going to be
like a, like a sort of a time cost piece, time cost schedule piece, you know what I mean?
Appetite, yeah.
From the, from the framing perspective, you know, of like working with a client, like I
wouldn't even use the appetite language.
I would literally just be like, this is a, this is a time commitment.
These are, this is like, there's a, there's a moment in time where both of us expect to
meet up again and, and I'm, we're supposed to be delivering something.
Do you know what I mean?
And then this, um, uh, appetite is a little bit more of, um, uh, it's a little
bit more of a product team concept, you know, and it's a little hard to
translate that, um, to the client.
You know, it's, it's a good concept to have in mind when you're in the
negotiation of the frame, but the output of the frame is actually like a harder
agreement of like, this is just schedule, you know, that's at least how
I find it helpful to separate those, you know, cause for me, appetite is a
very, um, design process word.
You know what I mean?
Like, like, so when I'm early in a negotiation, trying to frame something,
I'm trying to feel out your budget.
Right.
Like, like how important is this to you?
Like how, how urgent is it?
And, and, and I'm starting to suss out appetite based on that.
And I'm like, so maybe we could do a, maybe we could do like a, let's do
three, four week projects on this and sign that as one thing.
And they're like, no, no, no, we're not ready for that.
And you're like, oh, okay.
You know what I mean?
Like, uh, it's, it's more like in the negotiation, but then the kind of
when we're done, it's, it's kind of like baked, um, anyway, that's how I think
about it, um, is this, um, uh, uh, the, the, yeah, I don't think we need
to go further than that, like, do you, let me, let me hand it back to you.
Yeah.
Um, um, do you, what do you see in this?
Like what's helpful here.
Um, and just to summarize, like you have, you're thinking of having a problem
section, the success section, like a checklist like this, and then having
a specific, like not section.
And then maybe the fourth is just this budget schedule.
Uh, like, so, yeah.
So if you're thinking of it as a document, then it's, there's, then
there's going to be something like, uh, you know, usually there's going to be
something.
like fees or cost or price or whatever.
Yeah, okay.
Usually this is kind of the package here.
And how this gets done,
I mean, that's at your discretion.
As long as these, the crispier these are,
and the better you get at spelling these out,
the more you're gonna feel like,
I'm just responsible for delivering the outcome.
And I'm gonna have five different ideas
and shaping of like how I actually do that, you know?
Right.
I think we, yeah, that's good.
And I think in every meeting, including this one,
like if Chris or Lucian,
like we're talking about anybody to the project,
like coming to this for the context,
like we probably should have started here today
because it would have directed what we did.
We learned a lot as far as like how to shape some things up,
but all of that was mostly around like transcripts
and stuff, so we can apply that to the file side.
But this would have been, you know,
just even for framing this conversation
would have been helpful to revisit this as a starting point.
Because ideally, I guess if this is,
because if you had seen like,
oh, this is not about transcripts,
and then we spent 10 minutes talking about transcripts,
we were like, wait, didn't we say we're not doing this?
Like, you know, it'd be so clear and obvious to people,
like you're saying.
So I think that's helpful
if we do spend the time to get it like that.
I do like the idea of keeping the freedom.
So actually not having the skipping,
and this is where we were very fuzzy about shaping,
like we were doing shaping so badly
that it looked closer to framing.
And even at that, it wasn't a very good frame
to summarize where we were.
So I was like, it's shaping,
this is technically our kind of shaped,
packaged document template.
I'm gonna, I'm making a new one now,
it will just be for framing,
which would be the thing we hand to the client.
And then everything else we can do,
you know, in using whatever tools we want,
because I think that's trying to pick
like the right level of abstraction,
like to show, hey, this, like I said,
general problem, here's success, super clear.
In case you're worried about this,
these are the three things we're not doing.
And then getting them to agree to that.
Because I think what I'm most concerned with,
because Chris was like, oh, the worst thing we could do
is like they spend four weeks of budget,
and then, you know, they don't get what,
no, the worst thing we can do is we said we would do A,
and then we, or A, and then they expect A, B, and C,
and then we sort of have to deliver A,
and sort of have to deliver B.
So like, the best thing we could do is like,
we said this, we held you to it for four weeks,
so the next time we,
because what I would be,
I would love for them to do the four weeks,
pay for it, get the,
deliver exactly what's on the screen now.
And then when they'd say, oh, but what about X, Y, and Z?
Oh yeah, remember when we like,
you remember that we agreed to like this thing?
Oh, okay, like we held it.
And they shouldn't be surprised
because we should be like reminding them of this weekly,
as we, every conversation we have.
Keeping this moral star.
The more that you boil this down,
the thing I aim for is literally one paragraph
of the size of this first paragraph for problem,
and not more than that for solution.
Do you know what I mean?
The more that it becomes really, really, really crispy,
like small, but extremely crispy and clear what it is,
the easier it will be to repeat it
every time you ever make a decision.
You know what I mean?
Like you want the frame to almost be like
this little like mantra of like, why did they hire me?
Do you know what I mean?
Like, why did they hire me?
Like, oh, I'm here in these four weeks to do this,
to solve this thing.
You know what I mean?
And you would say the exact same thing to the client
if they walked into the room
and they started to talk to you about some other scope,
you'd say the same thing to each other as engineers,
or you'd say the same thing internally,
like between sales and engineering or whatever.
You know what I mean?
Like, it's that very clear mission statement
kind of of what we're doing.
Can't believe I just said mission statement,
but you know, mission definition of what the mission is.
No.
Yeah, and to Chris's point,
these should feel,
I mean, in the negotiation of getting to these,
it should feel like, yes, this is valuable.
This is moving them ahead of the baseline
of what they already have, right?
Yeah, because I think part of this is also setting up,
which is not really clear here,
but it is kind of giving the foundation
to do a bunch of other things.
Like we kind of, they don't have anything like this
in their code base yet.
So setting this first piece just to do PDFs
is gonna kind of pave the way for transcripts
and doing other things and bolting on
more types of problems we can ask, you know?
So I think, but keeping it clear,
because what they don't wanna do is they,
you know open it on the first test run and start throwing in like random prompts at it
um and then just yeah like it's sort of quote not working at that point right yeah um
uh there's a there's a the i'll just share with you briefly the the so this is kind of like um
uh what a document can look like when we need when we need an agreement
um when when i'm just kind of framing for myself or or to get to this document or to have a quick
reference during shaping i use something a little more informal and that's um that's this thing that
that that we that i had in the mural which is i just have two boxes and one is context and one
is outcome but we could to use your to use your language we could just say problem and success
for example it's the same meaning um and uh here's where um i'll very often just i like to
use just little uh boxes again um and i'll kind of quick and dirty just spell out the things where
um they uh like the um they have a manual process uh involving chat gpt
not enough people can do that process um so like as an outcome of this project we are going to
um we're going to pave the way for um uh the system to offer integrated um like ai um
uh um it's kind of like um it's kind of like we're gonna we're gonna we're gonna pave the way
for the system to eventually offer all of the things our experts are our experts are doing
by hand with chat gpt
something like that so like the paving the way is part of the outcome do you know what i mean
so like this can't just be like a hack that like doesn't give you any leverage in the follow-up
project you know what i mean but then there's more to it which is like um the there's this
other thing of like um what is the what is it actually going to do at the end of this four weeks
so there's something here um
about the client pdfs
there's something where um
uh i mean i could paste these in i kind of feel like there's a bigger thing because some of these
things are more like um just requirements and some of these things are more outcomes like data
being secure isn't an outcome data being secure is a requirement uh it's not like oh we paid you
we you worked for four weeks and now our insecure data became secure do you know what i mean like
so the outcome is more around um being intuitive for example is also like more of a requirement
um so like there's a there's something that they can do here that they couldn't do before
they can ask um
they can ask questions and get context-rich answers about client pdfs i'll just put that in
as a first draft
would you spell out um that distinction of of success criteria verse uh like success first the
the uh or the outcome verse the um the requirements like those are almost the
clients are kind of like a footnote of like yeah and some of this should be pretty obvious like
the requirements the data secure like that's you know these are actually your your um fine print
right you know what i mean they're not really that it's not like what's the thing that you're
going to test and demo you know what i mean like it's it's it's uh it's more like the fine print
you know so like there's going to be these other requirements we've been using the green box that
you had with like the requirements um yeah the fit check fit check yeah that's right yeah that's
that's yeah that's i mean they they kind of fall under that right um uh but they're not like the
big thing that this is about you know um so um i'm gonna do this very roughly um um
i'm just going to mark this as red um
If I were really trying to understand this better, I would want to understand a little
bit about what the before world is of case managers not being able to ask questions about
client PDFs, like what is going wrong with that?
And I would like to say something about that here, do you know what I mean?
Because that is a little bit more specific than this and it justifies like why, because
what's going to happen is they're going to say like, well, this project is just about
case managers being able to ask client, being able to ask about PDFs and somebody is going
to say, well, that's not enough, like we want transcripts too.
And if you can say, yeah, but today this is what's going wrong because case managers can't
ask questions about PDFs, that's the thing back to like, oh, right, okay, that is a win.
Yeah.
Do you know what I mean?
So the homework I would have here is, you don't happen to have an answer for that offhand,
do you?
About like what happens in the current world without this new ability to ask questions
about files?
I do.
They spend more time looking for the files themselves.
So literally, yeah.
Can you put that into a little bit more of a specific context to...
Case managers spend more time trying to manage the case, sorry, case managers spend more
time trying to figure out how to manage the case and actually doing the work of managing
the case.
So that's good at a very, very high level.
And now the challenge is I'm trying to bring it down to this project.
So case managers spend more time trying to find the files that they need than actually
doing the work, finding the files takes more time than actually doing the work of the files.
Now here's a dumb question.
Why is this bad?
Like, I get it.
Like there's everything takes too much time.
There's a million things that are inefficient in life.
How is this?
I'd say it's hard.
One thing we did identify, and this is where we actually framed up a different project
and they put this on for later, was that the entire file interface is terrible.
They're in these weird folders.
You can't see how many files are.
You have to click on each one like two or three times just to get it to download or
to view it.
So we had like a kind of an updated two-week, we had a two-week project we did for just
fixing like how to get to files.
So I think that this problem would probably fit more under that frame.
And that's where we kind of spliced it here to say, hey, this is this problem over here.
Like we're going to solve this problem with this outcome.
Whereas the other project was really, they kind of had the same problem, but like we're
solving it in different way.
One, we're using a chat interface.
The other one is making it just a much better UI to get better context of how I see files.
So there's another project where of improving the UI for the filing upload and stuff like
that.
Yeah, and just visibility of seeing, you know, hey, there's three files here in this section
versus, you know, there should be 10 or whatever.
I don't know.
Got it.
Okay.
And we have that shaped in a similar or framed, I'll say, in a similar format.
Right.
They just saw the AI shiny object and we went with that.
Okay.
So there's another, one of the things I try to do is because we're bouncing back and forth
also between the levels that Chris introduced and both of these levels are important.
One of the things that I sometimes find is helpful when I'm trying to frame something
is Bob calls it, sometimes we think we're selling the sandwich, but we're actually selling
just like, but we're actually just selling the mayo and like, we're the mayonnaise, not
the sandwich.
And the thing is that like this project is like, it's just like mayonnaise.
There's going to be another project that is lettuce and another project that is tomato.
You know what I mean?
And so what I'm doing is like, I, in my head, I just think of it as like the sandwich, but
like it's, it's, it's, it's like the, the bigger thing that we're trying to get to.
It's also like, sometimes like what we call like the, it's like the monster, like the
big thing that all these projects add up to, you know, and sometimes it helps to be able
to pull things up and be like, oh, okay.
So like this, this is like a general part of the big problem that like they're spending,
like, like Chris said, they're spending too much time on like the management of all this
stuff instead of actually doing it.
the work. So there's another slice, there's another like Mayo project which is improving
the file UI. This project, it's like, what I'm struggling with is to see the connection, is it
that in the current world files are all over the place, it's hard to like find the files that you
need, they're not all centralized in the system, and so like in order, when they do this manual
workflow of getting the files together to upload them to, if they want to like ask questions,
okay let me ask you this, are they already asking questions about files using AI,
or is this new behavior for them? They are, they're using their own
chatgpt and chatbots to do that. So they're like uploading the file kind of like one by one and
then asking questions about it? So I don't know exactly how they're doing that because they do
have some qualms about you know privacy and all that stuff, that's why I did reach out to this
me to see, hey can you just record what you're doing? I think what it's, he's, I think
safe to say, he's using it like an assistant, like hey, so for John Smith, they have these
five forms, he's supposed to send me this next Tuesday, right, and then he has tons of
post-it notes, like literally pasted around his monitor, like the battle things. So the reason I
kind of talked about files and trying to find them is because that's a first step, but then
there's not just files, like what are they done with the files, right, what's the newest update,
are there things on the calendar, so to speak, that he said, okay, next Tuesday they're going
to do this, oh the medical provider is on vacation, they have to come back in three weeks, so I got
to schedule that. So it's like a lot of to-dos and that's really the pain point, to get a
bigger picture of what's supposed to happen now, later, and then just, you know, what has happened
in the past. How does that relate to asking questions about PDFs? Because there's a
status to each PDF, is it filled out yet, has it been approved by the medical provider, have they
been submitted, it's kind of like phase gates, right, so each document on its own has to go
through these different statuses to be approved, and are there any, and this is
now, not later, are there any things in the PDF that require other interventions, so for instance,
do they have a claim that could increase, have, like for instance, maybe a veteran has
done their own work and submitted something before, right, that might be an increase, so now
they're a client of VR, and so VR is looking at, well, we can bump that percentage from 11% to
18%, you know, if the max is 20, or something like that. Got it, yeah, okay, so I'm gonna
jump a little bit because we're running out of time, and I'm gonna do a not great job of
this, but I'm gonna just pull out a couple things. So, like, I'm just calling out two
things I heard you say that were really meaningfully concrete, you know, like, especially good was, like,
there's something about, like, status that they have to, like, look at the files one by one to, like,
figure out the status based on what's in there, and then there's also this question about, like, are
there, there's things they need to know, like, about interventions and so on that require, kind of,
like, taking a lot of time through the files, and if they can just ask questions about these files,
this is going to be a significant time savings and productivity for them. This is an example
of, like, there's this bigger problem, right, of, like, we can't scale our workflow,
and then, like, if we could just ask questions about PDFs,
you know, this problem, this, these specific problems are going to go away. There's all these
other problems that aren't going to go away, but we can point to the problems that are going to,
that are going to go away. So, this is, kind of, like, this is a defense against scope creep,
because when somebody says, yeah, but we also want to do this and this other thing, you say, yeah, but
we are making this, these problems are actually going to go away at the end of this four weeks.
Isn't that a win? And they're going to be, like, oh, yeah, that's actually a win.
That's good. That's helpful.
So, that's also, that's the kind of thing, too, where, like, sometimes it helps to be able to,
what I like about this loose format is I can, sometimes we can use language we wouldn't use
with the client that, kind of, cuts to the point a little bit more, you know, also just to remind
us about, like, what this is and stuff like that. So, very often I'll have just one of these in the
corner of a shaping document, and then if there is a bigger thing, usually we, kind of, don't need it
if we're
in a project already, this is kind of like,
but if we sometimes have to tease them apart,
this is just a quick, hacky way of doing that.
You know what I mean?
Of just kind of separating out,
what are the things that are high level
versus what are the things that are outcomes
for this specific slice here?
Yes.
Just for our context, like this would,
for our team, like would happen more
like in this, the blue, the discovery project.
Like I can see us, the like,
the outcome of that being a bunch of these little
problem outcome, problem outcome,
like stacking them and kind of separating
all these problems out.
And then working with the clients kind of figure out
like which of these do we want to even pursue?
And then turning that into like kind of packaging it
and making that our framed document.
Because I think it was, that was,
we were better because we like,
oh, we know breadboarding now,
or we know like we should use breadboarding.
So we jumped into it headfirst and try to,
and it was definitely helpful.
Like at least we were getting a lot more clear,
but it very quickly, I mean, it was,
it was very, it was such a knot of different things
that they wanted.
And then we were kind of guessing at what we think
the problem was.
So we kind of, we went a little too deep,
I think at that first dive.
Cause if I think we got it,
I think the, well, the good,
the benefits of it is that we went really deep
into like learning their process and learning the business.
And there's just some of that stuff
you just have to get the lay of the land
before you can even start to make,
you know, intelligent recommendations.
So, but I think this would have been a helpful tool.
I can see us using that a lot, you know,
just even just separating out these different things for us.
Cause we, we had them spread across different Figma boards
and different things.
It was just, it got very chaotic.
I remember that.
And then we try to distill it down to three projects.
So we didn't totally overwhelm them.
So that's my takeaway from that right there.
Bruce, did you get an answer to the question
about changing the frame?
No, but I think I'm, we're understanding, you know,
just to bring it back to that.
I think if, I think these, if we need to change it,
if we're super clear and then our team's super clear.
And when we talk to the client,
we're continuing to bring this up every single time
to remind them that's when they start to think about it.
And they start to like, oh,
but what if I asked this question, this question,
that helps us bring it back to a central point
because they may come back to us and say, no, like,
you know, well, this thing is way more important.
And at that point, like,
we also have to kind of hold the line.
Cause like we already started the project.
So we have to, that's where we have been thrown.
It's easy for clients to be like, we go 30% down one path.
And then they're like, oh no, this other thing over here.
And you end up like, you know,
sort of having this weird path up the mountain,
so to speak.
And it's just changing strategies like that
is a great way to just burn money and not see any outcome.
Oh, for sure, yeah.
So we want to kind of,
we're trying to like teach them something,
show this, the framework,
this is like the right way to do it,
but they have to kind of get on board with the process,
which they're, you know,
part of that discovery project is figuring out
if they're going to be okay
with this slightly different way of doing it.
And, but it is a more effective way
because we'll get to very meaningful, you know,
upgrades to their business and their platform as we go.
So yeah, anyway, that's, so yes,
I think we could change it if there's,
if there's like,
that means that we really understand their current phase.
We see the framed and they say,
actually we want to change this one bullet
to this other thing.
And now we have a very clear idea
of like what the current framework is.
And we're just really connecting this one thing
over here to that.
And it feels very, oh yes, that's like half a day.
We can make that one change and get them a ton more value.
I think that's where we have that as sort of the frame,
of course, for the project.
But once we get into the shaping,
we really have a grasp on every little dial.
And so that when they do want to tweak it,
we just have to be very clear.
What we can't do is be fuzzy on where all the things connect
and all the interdependencies.
And then they make it, you know, a question.
Then we jumped to like, yeah, sure we can do that.
And then now we've like expanded that frame,
but now it's not updated in the document.
So that's where expectations,
that's where I'm sensitive to.
Cause I'd rather like, hey, we did these four bullets
and we gave you the four bullets on time, on budget.
What do you want to do next?
You know, maybe those weren't the four bullets,
but the point is like the process executed
on those four bullets the right way.
Yeah.
Yeah, I've noticed that the things that we want
are very hard to keep fixed.
Like on the like sort of feature functionality side,
it's really hard to keep that under control.
But the things, the problems that we need to go away.
way, it's much easier to align on like, oh, this thing's first, or like, if that
problem goes away, that's a big success for us.
So, um, what I find is that like, there's the, there's the monster sandwich level
problem, which gets us into the relationship, but it's the, it's the piece
of the problem of like this particular piece, you know, is going away.
That's kind of the thing that really helps us to stay on the same track.
And we can keep referring back to that.
And it's very often missing, you know what I mean?
This, uh, this kind of what's wrong with the current way at this level, you
know what I mean?
So you can really see like the, the before after that's good.
Yeah.
Um, yeah, cool.
Um, uh, how are you guys feeling?
Um, so we, we, uh, we spent the first half, um, kind of detailing out the
current system and, uh, and then kind of, uh, sequencing the path through it in
terms of what problem to solve for a second, third.
Um, and then the second half, we kind of ended up a little bit
more on the framing side.
Um, yeah, any, um, uh, any takeaways from the second half
from the rest of you guys or anything outstanding also that's that, that we
didn't get to that, that's, that's a question hanging.
It's been pretty rich for me.
It's good.
Okay.
Uh, I mean, really enjoyed the framing part.
It almost makes me want to go into sales.
Almost not, not really.
The, uh, what, you know, I learned about, um, I was always
horrified by the idea of sales.
And, um, uh, I, I learned about it from, from Bob and, uh, in connection
with kind of a lot of this like sort of framing stuff, we didn't have the
word for it at the time, but, um, a lot of these like skills I learned from
Bob and, um, the thing that made it work for me is that like, he, he explains
sales, all of this stuff, like from the viewpoint of an engineer, which is why
I like it, like, cause it's like, you can, you can, you can, you can sort of
systematize like how it's working and why it works and when it breaks down.
And you know what I mean?
Like, and for me, that helps a lot to, to be able to rationalize it.
And it's not just the, uh, it's not just, um, you know, steaks, steak
dinners and stuff like that, but there's kind of like a lot of, uh,
there's a lot of, uh, function, something that you can analyze that,
that actually works in it, you know?
And, uh, Bob, are you talking about uncle Bob?
No, this is, no, this is, uh, this is Bob Mesta.
He's a, he's a really good friend and, uh, and mentor of mine.
Um, he, uh, uh, well, yeah, you, you, if you, uh, if you, if you look
them up, you can find all kinds of stuff.
Um, he's just right.
He's he's the, he's the original guy, um, behind, uh, jobs to
be done with Clay Christensen.
Yes.
I do remember that.
I think I've read some of the stuff, but he's actually a, he's a
hardcore engineer in his soul.
That's his real background and, uh, working in everything from
automotive to like NASA to, um, even like food, food production,
you know, like, uh, across industries.
And, uh, he's, he's a fascinating guy.
So yeah, one of my main mentors.
Nice.
That's great.
We'll check out.
Yeah.
Um, uh, anything, anything unclear about like next steps?
It's good.
I think, I mean, um, I think our next call is in a month.
Uh, so we'll see, we should be wrapping this up or have wrapped it up, but I
mean, I feel pretty good with all the tools, uh, you've given us all the tools
we need, we just need to go, you know, apply those again, kind of refresh it
and then have something, um, to just kind of, it's, you know, it's just resetting.
It's kind of taking this.
And once it's clear, like it's, it's very, you know, we can show that to the
client, it makes it easy, but it's really hard to, to fake clarity.
Like if we're vaguely, we're just filler busting, like making stuff up and it's
easy for any of us to do that when we don't really know what it is that we're
like, um, so we can be really clear and make it super concrete, then it's
just going to protect us.
Um, and also like, we're, this will be our second call with them.
I think the second check-in call.
So we'll just make sure we kind of course correct very quickly and like, Hey, just
bring it back.
Remember, this is the document.
Just want to reset expectation.
Now you guys are sending these things over here.
We got it, captured it.
So.
We're here on this other board, right.
Here's like where we're going to focus.
This call is like where these specific three things are or wherever.
Um, and, uh, you know, I think also once it's very clear for us, we can also kind
of see a quicker path to like test this.
And so, you know, even just grabbing, you know, 20, I think I, we talked about this
early, I don't know if this got done or not, but taking 20 files, throwing it
into a quick AI thing and just giving it as the full context window, like all
these files and just see how good it is just right off the bat without
us doing anything fancy to it.
Um, you know, that's something we could do as a proof of concept quickly.
Um, and those are things where I see like rapid prototyping kind of just to quickly,
cause we, a lot of things to see, and that would also help clarify like the vision.
Um, that's, that's even something we could probably hack before the call.
But, um, yeah, just cause I think, I think when they start to, I mean, a
demo is worth a thousand words, like showing a two minute demo, something,
um, clarifies all that stuff.
So, um, yep.
Were you gonna say something Ira?
Oh, no, I was just, um, okay.
I thought you unmuted, but that's good.
Cool.
Uh, I love, I love, I love what you said about like this, like you can't fake
clarity, like the quote I usually think of is that it's, it's, it's, uh, it's
easy to macro bullshit and hard to micro bullshit, you know, like, so it's like.
Yeah.
Um, cool.
Okay.
All right, guys.
Hey, excellent.
I mean, nice session.
Nice work.
And, uh, uh, you know, good luck with the project and we'll be in touch.
Thanks, Ryan.
Appreciate it.
All right.
See you guys.
Have a good one.
Cheers.