Video: From Collection to Clarity: Collect, Cull & Find Responsive Records | Duration: 1848s | Summary: From Collection to Clarity: Collect, Cull & Find Responsive Records | Chapters: Welcome and Introduction (26.27s), The Data Challenge (118.91s), Six Core Pillars (261s), Data Intake Methods (386.885s), Data Transformation Process (543.56s), Filtering and Searching (616.075s), Deduplication & Auto Tags (737.83s), Sensitive Data Filtering (826.01s), AI-Powered Email Search (914.73s), AI Search Capabilities (1167.655s), Role-Based Access Control (1430.455s), Keyword Filtering (1531.56s), Next Session Preview (1669.31s), Wrap-Up and Resources (1712.465s)
Transcript for "From Collection to Clarity: Collect, Cull & Find Responsive Records": Hi, everyone. You've probably seen us testing a presentation, making sure that every... Everything goes well. Welcome to the second session of our four four part webinar series on Logikcull for public records. I'm Laura from Reveal, and I'll be your host for today's session. So, if you were in our first session, we looked at how agencies can simplify the intake and management of public records requests. And if you miss that one, you can catch up on demand using the link that we have included in the docs section. It's the last one on the list. And today, we are picking up where we left off last time because receiving and organizing your records requests is one thing, but finding the right information across potentially thousands of records and multiple data sources is another challenge entirely. And that's what today's session is all about, collecting, calling, and finding responsive records faster. And to walk us through that process, we are joined again by Daniel Schuuring, our principal product manager, and Jason Christian, our principal, strategic sales engineer. They are going to show you how it all works in a live demonstration in a second here, and we'll also have some time for q and a. So please feel free to submit your questions in the chat at any time. Daniel is gonna be responding them as they come. So with that, I'll hand it over to Jason to give us some more context around the demo, that we will have today, which he will, later drive. So, Jason Christian, over to you. Laura, thank you very much for handing that over. And as she said, Daniel Schuuring and I are gonna be presenting today. So I'm gonna be walking through a few slides, kinda talking about the challenge, the problem, the solution, and everything that we have here. My friend Daniel Schuuring is going to be answering any questions. So should you have any questions, please place them in the chat, and he will answer them as promptly as he possibly can. But as we mentioned, this is session two of four sessions. So a couple weeks ago, we talked about the portal. We talked about records requests and foias and d sars and the intake process. Someone puts in a request. How do we track that request and how do we take that request all the way up to the point of collecting data and putting it into Logikcull to process. And that's what we're gonna talk about today is how do we get data into the platform, things we do, transformation to the data, and then how do we organize it, how do we collect it, cull it down, search it so we're not putting eyes on absolutely everything? To set the stage for session three, which is next week, and we'll talk about, sensitive data and the redaction of sensitive data. So what is the challenge? The challenge is just just piles and dump trucks and mountains and records of data. You know, for our DSARs and our public records and our FOIA friends out there, a lot of the workflow sometimes is manual when all of this data is coming in. You have to put visual eyes on absolutely everything. And if there are deadlines, and we talked about deadlines, time is always of the essence. If you have to look at everything, it takes a lot of time away. Also, we're gonna talk about data collection and going back and forth and finding the right data for everything that's up here. Right? And what we're also gonna try and do is how do we call this down? How do we filter it down so you put your eyes on only exactly what you need? There's duplicate emails. There is emails that have the same content in there. There are email threads where conversations go back and forth, back and forth, and you're looking at all these all the time. And sometimes the data is not clean. We like to call it dark data. PDFs, sometimes they're your friend, and sometimes they're your enemy. And it depends what's inside those types of things, what's been scanned in, paper documents, handwritten documents. Is all my email in a PDF, or do I have the original file format? All of those things, and we're going to talk about that today. So the six pillars that we're in, the six topics we're gonna talk about is getting data into the platform, connecting directly to source. Right? Not everyone has data just sitting on their platforms or sitting on their desktop. Right? We now have cloud collaboration tools, and how can we, mitigate the time to get that data in there? We're gonna talk about processing and this transformation process we do of the data, get it into the platform. Then we're gonna talk about how we can eliminate duplicate data, how we can look at the most inclusive emails so that you're putting your eyes on exactly what you need to. Right? Well, we'll call potentially responsive from our friends in the legal world. But, right, but not wasting your time. Let's look at it once, and let's make sure we're looking at what we need to. Right? And we're also gonna have data formats. I don't have what we call extractable text. Right? The ability to process data, we build indexes. We wanna make things searchable. So we do automatic OCR to look for characters so we can make things searchable and put that in there. And then we're gonna call this data down, and then we're gonna talk about some different types of ways to search the data. And we're gonna talk about generative AI search and applying this to all your workflows today. So in here, one of the big pieces here that we're gonna focus on is what we call our auto tags and then what we call our generative AI search feature called ask. Our auto tags simply are tags that where we find certain aspects of the data, we call out those features that might be important to you. Does my data have sensitive information? Is there an email thread here? Is there an inclusive email? And we'll take a look at all that different type of stuff. And then for a lot of us out there who don't know how to come up with the most complex and all these complicated ways of searching the data, let's ask a natural language question, right, of the data because we all know how to ask questions. So I'm gonna jump into the demo today. I'm gonna go through all of this and kinda show you what we just talked about, and then we'll finish up with a few slides, and we'll finish up with a few questions. So jumping over in here, let's talk about the data intake. So if you're on our last session, when we jumped into the request portal, and if you were not on our last session, what we did is we have an intake portal for public requests, for foyers, for DSARs. It integrates with the logical platform, and it also has the ability to create a project. So we're gonna pretend we've gone through that whole intake process, and now our project is ready for putting data into this. Now for our friends that are... Have cloud collaborations tools today, look, a lot of people have Microsoft three sixty five that has just been an enterprise solution for a long time, but we have a lot of people using Google Workstation, using Google Vault. Also, people... Some people use Teams for communications. Some people use Slack. We now have the ability to dig directly into these and pull data into these. Right? So what we're trying to do is eliminate the back and forth with IT. They have the keys to the kingdom. They're going into Purview. They're running these collections for you. And sometimes the scope is too broad. Sometimes it's not enough. Right? But we give you the ability to go in and pinpoint who you're looking for, apply some date ranges. You can bring this data directly into the project and eliminate some of the back and forth and the time to get the data to start collecting in here. Again, time is of the essence. We wanna make sure that you're meeting deadlines because a lot of this stuff has deadlines. So we have the ability to bring this directly in here, and we can come in and select and configure this over here. And once we authenticate into it, all we need to do is supply an actual target individual and actually put a date range that's in here. And so right now, it's searching and building onto my connection right here really quick. I'm gonna take a quick step back out, but that's really kind of the next step is who's my target? What are my date ranges? Let's go in and collect. Now we also know that not everyone has the ability to do that or connect to that. So, obviously, we have traditional ways of bringing data into the platform, and we talked about it last time, but the good old drag and drop. I can come in here and just drag and drop my data from anywhere. Inside this zip file is a PST, but inside the zip file could be 900 plus different file types, and that's really what we support in this platform. The big piece of this is coming in here and adding a custodian name to this. So I'm gonna say that this data belongs to Philip Allen. Now not everyone's data belongs to one individual. This doesn't have to be a name. This can be an entity. This can be a basic description. Right? But we do require the custodian, and I'm gonna hit start upload. It's really that easy to get data into the platform. So just really quickly, what's happening here? Number one, we're a cloud based solution. We're AWS SaaS. We're actually pushing data up to the cloud right now. As you can see this nice little yellow, it says don't close this browser. We want data to get into it. But the next step, and a great segue into this, is our transformation process. We are now processing data. Right? And so this data is gonna go through 300 plus steps in order to get us to start calling down and searching through this data in here. We're going to apply OCR. We're going to automatically create PDFs so we can do redactions and annotations, and we can collaborate on that. And we're gonna show you more of that in the next, two sessions that are coming up after this one. So we're not gonna watch the paint dry for this one. I'm actually jumping to a project where we've actually gotten data processed into it. So I'm gonna come back here, hit projects. I'm gonna jump into a completely different project, and here we go. So my last screen, you saw data processing. Once it's gone through its three different steps and taken that data transformation process, this is what a complete data source, process data source is going to look like in here. Our next step going into this here is going to search through that. I talked about these 300 plus, transformation steps, that the data goes through. That's what drives our filter carousel, and that's essentially what you're seeing here. And this is where a lot of our customers are taking their first steps. We talk about reducing the noise, calling out the noise, or calling down the data, or filtering. Right? Whatever terms, you you know, are going to make more sense to you here. But there's a lot going on. So during processing, we drive all these things for you. So a lot of our customers look at the different document types, right, that we have processed. Maybe I care about the communications. I can filter on email. Right? If you have Slack data, if you have Teams data, I can come down and look at various conversations and filter on those as well. I wanna look at my communications. Right? Now remember, we assign custodians to the data. So if I want to pick a per... Particular custodian entity or how I organize that processing, I certainly can. If I have more than one upload into here, I can organize it this way too. But there's a lot going on in here. If I'm really focused on email communications or any type of chat communications, maybe I wanna jump over here and now start looking at some of the email data. Right? So this all comes from metadata as part of processing. We pull all that stuff out to make it searchable. So maybe I care about from and to domains. Who's it from? Who's receiving it? And the particular email addresses, I can filter on who it's from and who it's to. If chat data, Slack, Teams data is important to you, we're going to log out all the various conversations that are going on in there. Who are these participants? I could come back over and see who the senders are. And if we're gonna look at any types of emojis or anything like that in there, if that's important, we extract those out and we can filter on all of those too. What we're trying to do is reduce the noise. Now one thing I did not talk about really quick, and I probably should have, let me clear out of this. Up top here, I actually processed 1,200,000 documents in here. That is my complete universe. Now we talked about deduplication. Deduplication is getting rid of the duplicate data. If you have this much data, what I'm really looking at here when I run these searches is across 470,000. I've eliminated that much data because the rest of it is all duplicative through here. So that is one thing that we do for you. When we run these searches, when we run all of these filters, it's being... It's reducing the actual amount of data that we're looking at. So, technically, we're dealing with 359,699 documents in here. Now I wanna go to these auto tags really quick because we talked about these a little bit. So the auto tags, again, taking a step back, this is finding features of the data, things that we have found that of interest in here. So a couple different things. Our customer base are finding a lot of interest in here. Number one, the most inclusive email. Right? And so they look at the inclusive emails, and they look at emails that have threads. So imagine an email conversation that has just gone back and forth, back and forth, back and forth. Right? The old process is to look at every conversation back and forth. Well, we have the ability to come in here, filter this by the actual thread, and then look at one inclusive email where all the conversation is as one. So I'm now looking at one record, one document, one email with the entire conversation in it. But on top of this, sometimes we have to go through and see if there's sensitive data. We call that PII. So what I'm... What we're telling you here is we have a 125,100 documents, records in here, communications, that's what we filtered on down here, that have some sort of sensitive data in it. Right? In the next session, we're gonna talk about redacting that because you're gonna have to go through and redact sensitive data. But the next natural question is, what sensitive data is in my project or in these documents? Filter carousel. I can go to PII type. We're gonna come back all the way over, and you're gonna start to be able to see what... How much sensitive data is across those 125,000 documents, and we're going to call it all out here. Right? Maybe people's names have to be redacted out. Maybe there are personal personal physical addresses here. We have to redact that out. But take a look. Maybe I'm a financial institution. Maybe bank routing numbers, bank account numbers, credit card numbers, Social Security. This is all potential sensitive data. Well, it is sensitive data, and you're probably gonna have to redact this out. We can filter down and get down to these documents if that's really what the next step here in this entire process is. The whole ability to be able to filter down your data, get down to it, and really start searching and going through all of this different type of stuff. And we haven't even ran a search yet at this point. So we can come up to our search, and if you have search terms or phrases or anything like that, we can come in here and put in basic stuff. I can be more complex and say banking or credit or finance. Right? Or if you have a ton of search terms going, that's nice, Jason, but I have about 50 of these. I'm not typing this in 50 times. That's fantastic. So we can come to our advanced search builder, open the search builder. I can come into my bulk keyword search, copy and paste your searches in there. We can come in here and go back and hit search all. I've done my filtering down to my communications. I have search terms I wanna hit on. Look. I now have 1,803 documents that are in here. So I'm gonna put my eyes on this, not the whole 470,000 or the 1.2. That's what's responsive to me right now. So now what sensitive data is in here? Right? So we can come through and maybe focus on getting rid of this sensitive data across 1,803 and not 470,000. Now another exercise in here is going down and maybe looking at conversations and going in a little bit further. Right? So I'm gonna come into email. I'm gonna filter on this email. I'm gonna come back to our auto tags here when this is populated over for me. Right? We talked about that inclusive email. I don't wanna read every thread. Well, are there email threads here? There are. There's 78 email threads that are in here. So I can come into and say, I'm gonna start with this particular email address over here. Now jumping into the document view, a couple of things that might be helpful here to the right is if the activity is to see if there's sensitive data, we're also gonna call out sensitive data that's in at the document level. Level. So we showed you at the project level. Now we're gonna show you what's in this particular document. Right? So I can select all. I can come in and redact all, and we'll cover that more. But, also, I wanna look at the email thread that's in here. So... And I wanna make sure I'm keeping my eyes on the most inclusive email in here. And so, basically, this is the entire email thread, but we have found three inclusive conversations that are in here. I can come in here, click on this inclusive email, and now I know I'm going to read this one document with the entire conversation thread, and I can Google all the way through and read this. Now imagine this being very long, right, and having to go through and read this. Right? So the next generation of search is what we call generative AI search. So a lot of us are familiar with the CLODs, the ChatGPTs, the OpenAIs, all that stuff out there. Well, we have a feature called Ask AI, and that is ours. We can actually open this up on a particular document, and I can come here and say, summarize this email. Right? Maybe this thing spans 50 different conversations. Well, let's see if we can make light work of going through all of these 50 conversations. So let me kind of explain what we're doing here a little bit. So what we're doing is we're going through this particular document. We've asked we've asked our generative AI to summarize this email. It's gonna go through all of the content that's in here, and what it's going to do is go through it and provide us a narrative, provide us a summary, provide us a scope of the conversation that's going on in here. And we'll give this a couple minutes to kinda run really quick. What's happening now is our AI agents are going through so it can come through and generate this for us. And this is what it's brought back. Right? So it's gonna bring back this narrative and summarize this entire email for you. So if you need to interrogate this further, if you're pulling out particular dates, particular timelines, particular names, individuals you didn't know were in here. If I need to build on this, build on my search, we can certainly do that and be able to go all the way through. You know? And that's going through and being able to really make light work of all of this stuff. So what did we do? We got down to our communications. We filtered certain things out. We found some of our email threads. We ran some keyword searches. I have this one completely summarized for us, so I didn't have to go through this and spend hours going through and writing notes and writing notes. Now I could go through and look at the individual citations coming from specific documents that also help help us generate the summary through this particular email thread. Very helpful. We could also take a different approach of this, and I'm gonna clear out of this. That's just jumping into one particular document. But keeping on the key theme of using our generative AI search features, I could actually come in here and now search across my entire project. Right? Maybe I'm looking for something specific in here. So what I could do is come across and say, hey. Are there communications from Ariana Akers? And so very similar to doing the search on the document level, this is now doing this across the project level. So I'm now searching across 470,000 documents right now to be able to go through and say, hey. Does my data contain communications from this individual? So I've asked the question. The question's being asked against the data. We're gonna go through, and, essentially, this is a Symantec, index is what this is. We're going to score everything. We're going to find what we call very high semantic similarity values. We'll take that to the CERT service. What gets returned to us from this CERT service and the agents going all the way through is a summary, a narrative of everything that has come back, and that's what we're through. And, yes, there are communications from Arianna Akers in this document. Now some of you might be asking this question in the background. I don't know because my friend here, Daniel, is answering these for you. Is this secure? Is my data living out in the universe? The answer is no. Okay? So we have basically safety guidelines around this, protocols and stuff. A, we don't allow data to go out to the search service and stay in the search service. And the large language models that are used, which is AWS Bedrock, they don't allow it either. So the data stays secure within Logikcull. The questions are being asked against it. And then, essentially, we take we take what is the text, the content, and we just ask the search service, can you provide us a narrative? Can you provide us a summary on all of this? And that's essentially what ask is doing. Going back to our topic of dark data, think about emails that are in PDFs. Right? We talked about this filter carousel underneath here really quick. This only works if we're able to get through all the metadata. Right? All that stuff. Well, if you process a PDF with email, you're not gonna have email metadata in there. All Ask cares about is content. So if the content is inside the email, this is how we get around the dark data. This is how we get around the troublesome and burdensome PDFs where some of that might be missing. As long as the text is there, the content is there, we can come in and search that on top of it. I am going to switch back over. I am at the tail end of the demo presentation piece of this. Oops. So what did we cover today? So we talked about direct source connections. Right? We can either do the drag and drop or we can actually start building, integrations into the cloud collaboration tools, Microsoft three sixty five, Google Workstation being in there. Being able to reduce and reduce. and what really, have... Yeah. one one quick remark on that, Yeah. because in your story, you mentioned something about, okay, we can, set credentials in the configuration of logical system, and, the legal team, the FOIA team can do their own collection of data. Now I know that a lot of organizations have, let's say, kind of borders, boundaries between. departments, and IT not always is very happy with other system accessing other systems. And I just want to mention that, for example, what we can do is that we can also create what we call upload users within Logikcull. So an IT person can be an upload user in our system, and they can do the collection from our system. with their credentials so that they don't have to hand over to the FOIA team all the credentials that. they need to collect the data. So then this is... This is a little bit more secure. Just wanted to mention that in this, on this topic of direct source connections, Yep. No. because I know from experience that a lot of organizations have issues, with everybody accessing. every... Everything. Yep. And to that point, that's a role based access control, and it is called upload only. So you can give other people permissions to just drag and drop and put data in there or IT organizations to build the integration or do the drag and drop for you as well to eliminating the time that data has to transfer from one hands to the other to get it into a logical. Yeah. I think it's talk... also... Sorry. No. I think it's also useful. We have a few minutes. So... And and I didn't see any questions yet. So if people have any questions, please put them in the, in the system. I think the Airflow's only user is also really useful if you ask for information inside your company because, normally, you put out a question of, yeah, who's responsible for this project where we have, requests for, that these people, it gives them an upload user role, and they can upload their own data to the system. So you don't have to go around collecting data from various people doing it yourself, but you can just hand out that. task to the people that actually own the data and still protect all the data. Yeah. So... Abs... But that. was all going in the, in the flow. Absolutely. We we got a, almost like a product request from Leonid. Maybe we can address that. It's around filtering by keyword during collection of of data. Is there any way we can do that today? Filtering on the keywords. So out... So outside of outside of running the searches. So we can run the searches. We can also test the searches. So when we say filtering on the keyword, would would it be a During collection, he mentions. During, collection. is... Yeah. Yeah. So currently, we filter for date ranges primarily, Yep. Yep. And it will be nice to filter also by certain keywords, Yep. because I understand what... And I've seen this before that people don't want to collect the entire mailbox, of people. I know a few clients that we have that actually do collect the full mailbox and then quickly call down the information that they don't need and keep to the information because the processing part in Logikcull is very important to make sure that you have, let's say, a complete set of data, that you. can search all information. And if you do the keyword filtering during collection, I know that you might miss something because not everything is like, for example, you mentioned the OCR process. No. Every... All the data is always searchable on keywords. But. I know this feature request, That's a good point. know it's on the top of the, the backlog that we have for logical. So there will be more news on this soon. Yeah. Yeah. Absolutely. There will be ways to do that through ONA, but I don't know if this is the right forum to talk about that. Yeah. That that is true. So we do have another tool called ONA, that, keys into keys into 24 plus different cloud collaboration tools. And one of one of the next steps for that, that also imports directly into logical, is going to allow to run searches into Purview and the Microsoft three sixty five as part of the criteria before the collection and review starts. Yeah. They only let us know if you if you would like to look into that at all. Awesome. So how does this session connect? Well, obviously, in session one, we talked about the intake all the way up from the public request all the way through to getting the data collected, putting into a logical project, processing it, calling it, getting down to it... Getting it down to a responsive set. In the next session, we're gonna talk about redactions, what need to be redacted, different types of sensitive data. Right? Sensitive data that we know of that we can auto redact or redact ahead of time and things that we may not know that's in there and how we can redact and do that as as well. And so in our session next Tuesday, we're going to focus more on the redaction piece and redacting out sensitive data. Perfect. Thank you so much, Daniel and Jason, and thanks to everyone who joined us today. I hope that you found the demonstration helpful and you came away with ideas for making your own, process more efficient. Before we fully wrap up, just a quick reminder that, again, we're only halfway through this four part series. So next session, as Jason was mentioning, is next Tuesday, November 17, all around reductions, which is obviously a super important topic in the context of records requests. And then on December 8, we will close out the series with a production and disclosure, including how to get responsive records into the hands of requesters through a public reading room. You can find the link to register to both sessions in the Docs section again. There, you will also find a few additional resources, including our public record software RFP template that we've created based on a myriad of RFPs that we've ourselves filled out throughout the year. Hopefully, this is helpful for agencies soon going into an RFP stage. You'll also find a comparison guide for agencies that are evaluating different solutions like, COF QA or, NextRequest and how those compare to logical for public records, and even how to use them in tandem because we know that many agencies, prefer to do it that way. And then a a few customer stories from, agencies like the Poudre School District, the City Of Corona in California, and Fairfax County in Virginia. And, of course, if you would like to explore how Logical for Public Records could work for your own agency, you can always request a more personalized demo using the bottom up above that says Request Logical Demo. So thanks again, everyone, for joining us, and we will hope... We hope to see you again next Tuesday for the next session. Bye bye. Thank you. Thanks, Ivan.