Episode 1: The Dangers of DIY Forensic Collections

The Dangers of DIY Collections: When "Good Enough" Can Create Risk

In the inaugural episode of Evidence Matters, digital forensics experts Jerry Bui and Steve Davis tackle a common question facing corporate legal teams, litigation professionals, and eDiscovery practitioners: When is it appropriate to handle data collections internally, and when does it make sense to bring in a forensic expert? 

The discussion explores the risks of "DIY" collections, including challenges related to chain of custody, metadata preservation, defensibility, and expert testimony. Jerry and Steve compare the roles of IT teams, law firms, and independent forensic consultants, explaining why collecting data for litigation or investigations requires a different skill set than simply accessing information. They also examine how evolving technologies, cloud platforms, mobile devices, and emerging data sources continue to reshape forensic best practices. 

Listeners will gain practical insight into:

•  The difference between IT collections and forensic collections 
•  Common risks that can arise from self-collection efforts 
•  Why defensibility matters in litigation and investigations 
•  The importance of independence, documentation, and expert testimony 
•  How forensic best practices continue to evolve alongside technology 

Whether you're an in-house counsel, litigation support professional, eDiscovery manager, or outside counsel, this episode offers a practical look at how collection decisions made early in a matter can have significant downstream consequences. 

Subscribe to Evidence Matters for candid conversations on digital forensics, investigations, eDiscovery, expert testimony, and the technology shaping modern legal matters.

Evidence Matters with Bui & Davis is a podcast for litigators, in-house counsel, legal operations professionals, and eDiscovery teams navigating today's complex data landscape. Hosted by digital forensics experts Jerry Bui and Steve Davis, the show explores digital evidence, investigations, mobile devices, AI, defensibility, data governance, and the evolving challenges shaping modern litigation and discovery. Practical insights, real-world experience, and conversations that help legal teams make better decisions when evidence matters most. 

Visit www.purposelegal.io for more information

Transcript

Jerry Bui (00:01.26)
Hi everyone, and welcome to the Evidence Matters podcast with Jerry Bui and Steve Davis, because both of us do agree that evidence matters. Welcome to our inaugural podcast, and we'll be talking about DIY collections. We are both digital forensics practitioners and want to talk about the hot topics of the day as it relates to our field. Steve, what how do you define evidence, D I Y when it comes to collections.

Steve Davis (00:33.489)
Yeah, it's funny. You just you said something in the intro and it totally just got me storytelling. And I know we have a finite amount of time, so I'll make it a quick story. But my wife is convinced, I don't know where she got the evidence that I'm actually good at fixing things. So she had identified one of the commodes need to be fixed across the house. So she said, Well, there's obviously something you can do, no reason to spend money on it.

And I did a pretty good job, took the commode out, put the new one back in, and I was securing it to the ground, I ran out of washers, so I grabbed one from the garage. Unfortunately, all the washers I was utilizing were rubber and this one was metal. You can see where it's going. Trying to affix the porcelain commode to the ground as I stood up to kind of revel in my abilities, it started cracking down the middle.

And cost me, I went ahead and hired somebody in the new theater doing, and and it cost me, you know, a thousand instead of the hundred dollars I thought I was gonna, you know, incur in expenses. And and the parallel there for the fact that I'm not very good at fixing things is that sometimes we think we can do things and you really can't. And there's a whole variety of reasons why that happens. And it's not that anyone's evil or nefarious or bad or uneducated, it's just that there's very it's very much of an idiosyncratic

to be able to do what you do and I've done. And that is go in and be able to utilize the right tools, the right techniques with the right knowledge base, recording things contemporaneously so that you have evidence of it later. And I think if you don't have all those belts and suspenders on and you don't hire a forensic practitioner that's licensed to do what they're doing, sometimes you fall short. And maybe it's not a crack in porcelain, but maybe it's an argument in court someday about authenticity or spoliation.

Jerry Bui (02:30.094)
Yeah, I love the analogy because like trying to fix a commode, it can be dirty work. And two, it's really us getting into the plumbing of the data. Right. And you know, you might think instinctively that an IT department at the corporation who has access to these systems, who might know the data well, could be fully capable of a DYI forensic collection. Help us distinguish, Steve, or

understand why a forensic lens on a data collection might be different from an IT lens on the same task.

Steve Davis (03:10.621)
Yeah, what a great, great intro and great argument about this because that's probably the most common thing people ask us who our competitors are. I typically say it's it's do it yourself. Like it's typically not that there's a hundred other companies that do what we do in our lane. It's more likely that there are people that have some aptitude from IT. And you gotta remember that IT people are really about testing solutions, ensuring continuity.

Jerry Bui (03:21.379)
Yeah.

Steve Davis (03:37.669)
And availability of systems for practitioners internally to be able to do their jobs. So their job is very much of keeping things up and running and not so much about forensics, which is going in and like you say, digging below the surface. I think of it as Stranger Things, the show on TV, where you have the upside down, you know, the upside-down world, which is where all the creatures live. Well, that's kind of where our forensic artifacts live.

Jerry Bui (03:59.958)
Yeah.

Steve Davis (04:04.687)
Is the upside down for us. And so you've got to have the right toolkit, which there's no reason an IT person would have that to be able to get the granularity and visibility into those artifacts. And then the training. And this is where kind of your experience, you and I talked off camera not too long ago about a project. And what I said was, I need somebody that's been there, done that before, and has experience utilizing both the tools and understanding.

what data sources they're up against and what they might not be able to see on the surface that they have to dig further for.

Jerry Bui (04:41.154)
Yeah, that's a really good point. You mentioned forensic, right? I mean, we add that to we call it a forensic collection because we are collecting forensic evidence. And so we do inherit some practices from law enforcement, which an IT person doesn't necessarily have exposure to. It's all of that chain of custody documentation and treatment of the metadata belonging to the files. And more importantly, like you mentioned, there are artifacts that are not obvious to collect or to preserve.

As part of a litigation or investigation or regulatory response scenario. So there are some different practice areas and knowledge areas that a forensic practitioner might have. But other than kind of the appropriateness and familiarity with industry best practices, there's also for the corporation who might be the end client for our law firm clients, for example, and we might engage with them directly, there is a risk mitigation component.

Opponent, right, Steve? You don't necessarily want your IT personnel who have no past experience in testifying to defend a collection methodology or stand up to the scrutiny of cross-examination on the stand, defending your company and defending the way that you preserve this data. Is that right?

Steve Davis (06:02.726)
Yeah, I mean that's a great point. The the independence and objectivity that you bring to the table and that ability to communicate it, because you're gonna need the the words and the experience to communicate to stakeholders. Those stakeholders may be partners at law firms, they may be C level individuals, they may be IT or compliance individuals at a corporation, and you've gotta speak the vernacular of all those different parties and be able to communicate in in a

finite period of time, what we've done, why we did it, and what the end analysis shows. And so it's very crucial that although people may have familiarity with parts of it, they probably don't have that general contractor's ideas in terms of being able to talk from cradle to grave about acquisition, identification, triage, parsing, normalization, downstream, and then doing correlation analyses and things about

Data so that you can go ahead and answer the questions. I think that IT people are very good, but you have the whole argument about the fox and the hen house. Not that they're evil, but when you work for a C-level person, they tell you, listen, grab my last week's email. I don't need you going back further. Just do what I asked, please. I mean, that's that's problematic because we need independence to say.

Well, was data migrated two years ago? Did you retain this data? Has it been rolling off through some type of rule over time? Has there been the appropriate legal holes? There's a lot of predicate questions that have to be asked in order to ensure that we're doing a fulsome collection and then an appropriate investigation or analysis. And that's just not what they're educated in, which is why it's so important to have somebody who comes in.

And and many times, I mean, when we were younger, I'm sure you've run into this as well. I had people give me a script of what they wanted me to say under oath. And of course I was like, fascinating, but I'm not doing that. Like, I I'll tell you what I'm gonna say, but you're not gonna direct you know, five bullet points that you want me to get across an examination. If it's within the confines of my analysis, great, but we've got to be

Steve Davis (08:16.252)
People that bring the truth to our side so they can underwrite it as well as the opposing side so they can critique it.

Jerry Bui (08:23.992)
Yeah, it's a different skill set entirely. Even with something that seems to be a vanilla run-of-the-mill data collection effort, could go sideways real fast. Right. The challenge around procedure could come up. you know, that's that's always a chance that the procedure will be scrutinized and challenged. and you really have to have the experience and the wherewithal to articulate those things. So practice makes perfect, and it's you know, the

Steve Davis (08:35.889)
Yeah.

Jerry Bui (08:53.688)
Testifying expert witness skill set is something that's going to be foreign to a lot of IT personnel. So that DIY collection could really shift quickly depending on the evolution of the case. And so just I think the strategy should always kind of kind of assume that you might have to testify and defend the methodology. So that arm's length distance.

Steve Davis (09:07.836)
Yeah.

Jerry Bui (09:22.892)
really helps in addition to the experience of being on the stand and y you know, really standing up to the pressure that that that's involved.

Steve Davis (09:27.334)
Yeah.

Steve Davis (09:34.471)
You you grow some some thick skin over time. you know, we've learned and and you know, lawyers are going to probe and that's their job, right? To to get to the the base analysis and they do that typically through depositions and then through, you know, examination when we get brought into a a tribunal or or a court. And you've got to, like you say, a articulate what you've done. You also have to be

A thick skin so it doesn't become an emotional thing. All we are is scientists. We're just trying to basically outline facts. We're not advocates. We're not taking sides. I mean, to you and I, it obviously clients like when we find the proverbial smoking gun. Sometimes we do, sometimes we don't, right? It kinda is what it is. But I think that when you can tear down

that advocacy and they see that you're really kind of walking down the middle of the road and you're just telling them the truth the very best you can based on what you've examined and the the evidence you've been able to gather and then underwrite. I think it kind of softens some of that blow and some of the angst and anxiety that goes on. And people don't necessarily try to attack conclusions so much because they know you're really just trying to do your job and disseminate to the best of your ability what you have uncovered.

Jerry Bui (10:56.386)
Let's sh shift gears a little bit, Steve. There is the IT person that's often asked to do a self-collection for the end client at the corporation. But the law firm is also involved in DIY collections. There are e-discovery or forensic level practitioners inside the law firm. And so what do you think about the law firm personnel performing DIY collections?

Steve Davis (11:24.21)
Great. I I call it the doctor doing his own investing. Doctors make a lot of money and over time they kind of figure, listen, I've got a medical degree, very bright person. I understand oncology. There's no reason I can't understand options or buying stocks and bonds. And and the reality is it's not.

Cross-functional. It's not that because you're smart in one area, you necessarily can do it all. With a law firm, yes, they have people that are probably as bright, if not brighter, than you and I, and probably have a lot of the same toolkit. But those those entities, those you know, parts of of enterprises that attempt to do this, a lot of times they don't have the basics in terms of an evidence vault where you're gonna store a bunch of data.

Preservation's a huge part of what we do. Underwriting analysis investigation is too, but a lot of times they don't know what's going to happen. So early on in the case, we'll preserve hundreds of images, 50 phones, Slack environments, you name it. And you know, we're talking about cases every hour, every couple hours that our business development people bring in. So that's a lot of data. And law firms aren't really created.

To house all that data, monitor that data, intake the data, ensure it remains there, making sure that you have backups and disaster recovery associated with it. Their job at law firms, a lot of times, is to minimize the amount of data they hold on, not maximize it. So it creates an issue over time in every single case, right? As you get older, you learn the cases, some of them last weeks, some last years. And I have no shortage of cases are

Three, five, seven years ago that we still have to get back to the original evidence. And so it's really not practical with law firms that sometimes are merging and moving and changing and acquiring people, or the individual lawyers are leaving their practice and taking that evidence with them. Nothing's really built in and baked into the pie to allow them to take that data and transition it with them. So it really requires somebody that does this day in and day out.

Jerry Bui (13:37.1)
Yeah, and you know, I think that the best practices are actually evolving practices. So best practice one might day might not be the best practice the next. I think law firms do actually a responsible job in limiting the types of D UIY collections that they do. I often see them do like M365 collections, which can be very tricky, but they do, you know, understand the limits.

Steve Davis (14:02.117)
Yeah.

Jerry Bui (14:05.266)
Of of what they can do, especially if it doesn't involve modern attachments or any like real complexity, they do know to reach out to their vendors when needed and outsource things like mobile phone collections, which are very volatile in terms of best practices. So those industry practices, depending on the stability and kind of the known quantities around certain data sources, I think if you are aware of those limitations.

and knowing and drawing really clear lines around what you can handle or not handle, then I think that you know, that that's possible, right? But it all requires a constant education and some upfront disclosures around what what you can and can't do. and I think law firms who have eDiscovery practitioners working behind very you know well informed partners will make the appropriate decision there for sure.

Steve Davis (15:04.687)
Yeah. And I and I think I was just like to you know, to kind of tie the bow on that, when when you come to this, what you're highlighting and illuminating is this necessity to converse about it, right? Like is this a case where it matters or it doesn't? And I always tell people, like, listen, it doesn't cost any money to pick up the phone. Pick up the phone and call us. We'll call you back as soon as we can. And then we'll talk about the merits of of how you, you know.

Jerry Bui (15:05.624)
How how did Yeah. Go ahead, Steve.

Steve Davis (15:31.237)
consume the elephant one bite at a time. And it may be that you're right, the people in-house can do it and it suffices given the fact situation and what they're up against and and some of the implications of not hiring a third party independent forensic entity. And then there's other cases where people will fight that battle because maybe they don't have the facts or the law on their side. So they're going to attack the the methodology by which you collected the evidence.

So it is not a one size fits all, but I think what this whole convo illuminates is that you've got to have a conversation talking through the merits, talking through the risks, and making sure everyone appreciates that both the people that are engaging us and then the end client on on whose behalf we're working, you gotta make sure everyone understands what we're doing, why we're doing it, and what the risks are for not doing it, and that'll help guide you.

Jerry Bui (15:57.87)
Precisely.

Steve Davis (16:26.857)
in in which way you want to attack a given project in term of access to information and whether you can use do it yourself.

Jerry Bui (16:37.112)
Yeah, corporations should seek advice from outside council. Outside counselors should seek advice from their trusted consultants. And as long as there's an upfront conversation about the appropriateness of one approach versus the other, I think that handles the risk really well. I want to also talk about us as the consultants, as the service providers, and not to sound arrogant or overconfident. not everything in our world has a well established best practice, right? We are also involved in DIY collections or DIY solution making, right? Let's talk about a little bit about the kind of speed of change or rate of change when it comes to devices and data sources and how we might also DIY an approach when appropriate and when ni necessary.

Steve Davis (17:32.155)
Yeah, so this is very common in the mobile device area where we're having to capture phones and we're up against Apple security and others that are constantly making things I don't want to call them anti forensics, but making our lives a little more difficult. When I first got into the business and I was young, I w I I was under this impression that everything was totally scientific. And I I'm come from a scientific background of statistics, so Empirical evidence is big to me. Like and to me, it's not a guesstimate. Like you can go ahead and and create solve the matrices or solve the algorithm or do the math behind it to get an answer. Well, that's not really true. I found out that forensics is as much an art as it is a science. And that's because one day the APIs may work for you utilizing a certain program software off the shelf that you have. Next day it may not. And so we're thrust into this, no good example was WhatsApp, right? For a long time, WhatsApp, the best thing out there was a a singular software program that got you of the past year of the data. Now that may be great if the action that you're underwriting was two months ago and it may be horrific if you're looking for something that may have occurred two years ago and you don't have access through it through that piece of software. So we're constantly having to test what's available, what's the best solution for it.

And understanding that an Android phone is nothing like an iOS or an iPhone, the same way that a Mac OS is different than a PC. And we talk about, you know, certain forensic artifacts that exist with with the Windows operating systems. And when you go to Mac, it's a totally different, you know, kettle of fish that you're trying to attack. So I think we're often
having to spend as much time planning to do the collection as we do the collection because it's not just an easy button and it's not self-evident. When someone says, I have a phone, that's the start of a conversation, not the end of it.

Jerry Bui (19:40.706)
Yeah, and all of that work that we do, you know, addressing the challenges and coming up with solutions really puts us in a position to pivot really well because our baseline experience level gives us access to things that work and don't don't work, you know, based on past experience. That trial and error really helps us in the long run as that experience cu accumulates in one examiner or a team of examiners, all of that becomes institutional and then ultimately in this industry knowledge and we and we share this this information, we share these solutions until they become best practices. And of course, in the litigation field it's bolstered by case law. So if some of these techniques and these approach approaches are actually mentioned in rulings and have some basis of case law, that also helps and establishing industry best practice. There are some tools that are very well known to the court, you know, and w, you know, will be de facto you know, best practices. But it takes time for those things to develop and it takes time for the consensus to be developed. and so, you know, doing it day in and day out makes a difference and it allows us to pivot and invent new solutions as they're needed, especially with cloud sources, with apps and especially now with AI. So I do kind of cheer on those brave souls, those brave forensic experts that are in court and establishing best practices, but the risk is inherent, especially or the risk is manifest, especially with some of the testifying forensic experts who don't do so well on the stand. and and that's because they're, you know, venturing into new frontiers, and they could get it wrong.

Steve Davis (21:36.453)
Yeah.

Jerry Bui (21:38.424)
So we are at our time here for this episode. We do hope that this episode was informative. Any closing remarks, Steve, on DIY collections before we end this episode and encourage the audience to join us in future episodes.

Steve Davis (21:53.317)
Yeah, I think, you know, what you're learning is that there are a lot of takeaways to experience and in how you do things. I I usually learn as much as I, you know, hand out in terms of knowledge base when I participate in these. I love talking with you because you know, you're a very straight shooter. you and I, when we were across the table, we weren't at the same firm. I I feel like it's the same relationship that when we're on the same side of the fence, just because it is fact-based.

And and there's this yearning to do things correctly, not a yearning to stroke your ego. So I think it it's good. I can't wait to talk about other topics because to me there seems like a myriad of them that are out there that a podcast, you know, with a sound bite of fifteen, twenty, twenty five minutes makes a lot of sense. So I I would just say that, you know, you got to talk to your consultants. And I'd I'd do the same thing when you're buying a car or checking out restaurants, you know, test it and and talk to some people.

And the people that are looking to secure a master services agreement to you the first time they meet you, probably steer clear until you've done a little bit of work with them and you've tested it and you see how the car drives and say, okay, I actually like that. And there is no blind spot here. And the same blind spots exist in do-it-yourself forensics as they do in buying an automobile or a truck. So I I just think that it's worthy of conversation and it's worthy of before you get there.

Right. You've got to talk about how are we going, where are we going, so I know the right path. The same way that we'll utilize Waze and app to get to a location we're going. You gotta do the same thing when you're attacking a forensic project and you've got to think forward of what bridges we have to cross and mentally you cross those before you ever get to them.

Jerry Bui (23:41.154)
Yeah, and I think a lot of our clients are aware that this is an apprenticeship discipline. You know, look for those teams that have you know, experts and experts that are willing to train juniors. and also for our clients, give some of those juniors a chance. Everyone really needs their first testifying expert job. So if they're if they're doing the forensic analysis, they might be ready to write the declaration. They might be ready to be deposed.

Steve Davis (24:03.417)
Mm-hmm.

Steve Davis (24:08.625)
Sure. Yeah.

Jerry Bui (24:10.681)
And it is a virtuous cycle. The teams that do foster that kind of apprenticeship model, I think, go a long way in demonstrating value to our clients. And those experts also experience new things in in terms of devices and data sources, and they are on the hot seat. And so that experience alone helps them see around corners. So it informs the collection process to shore up forensic defensibility around.

what we do and the methodologies that we have and to strengthen and bolster any gaps or vulnerabilities in that process. And hopefully those teams are also willing to share information by going out to conferences and speaking. And really I I do believe that a rising tide lifts all ships because it moves the entire industry forward because there are always new frontiers to explore.

Steve Davis (25:03.321)
Agreed.

Jerry Bui (25:04.994)
Thank you very much and catch us on our next episode. Subscribe on your favorite podcast platform and remember to follow Purpose Legals, Evidence Matters with Jerry Bui and Steve Davis. Thank you, Steve.

Steve Davis (25:19.908)
Thank you, Jerry.