▶ 0:06:54This. I know the quorum is present. Without objection, the chair may declare a recess at any time. Also, without objection, the hearing record will remain open for five legislative days, so members may submit any materials they wish to be included therein. We're here to discuss the congressional research service and the critical role it plays in helping congress carry out effective, informed oversight. The world is in the middle of a transformational moment. Rapid advances in artificial intelligence and other emerging technologies are changing how most sectors operate.
▶ 0:07:23Capitol hill is no exception. Major technology companies, startups and think tanks are rushing to develop tools that promise ai enabled policy synthesis, legislative analysis, or new oversight capabilities. These efforts can have on net a positive impact. Congressional staff are beginning to expect richer, more tailored analysis to support their work and a members offices. We're seeing something worth noting using the ai.
▶ 0:07:51Using the tools and process this committee on house administration has put in place over the last three years. Staff are beginning to build their own internal tools to support their daily responsibilities. Let's go through an example of this one example. Representative keith self office. His team actually built an ai based tool that provides summaries of introduced bills, outlines parliamentary consideration, recommends floor staff, floor strategies, and even drafts a regulatory impact statement.
▶ 0:08:18Uh, and because they prioritize protecting taxpayer funds, it's keith self, after all. Uh, they did this in a, in a cost effective manner. And, um, it, it costs less than a dollar, uh, to run the analysis. This is an example of what happens when offices plan to execute on new technologies. It shows that the legislative branch can be a leader in responsible technological adaptation.
▶ 0:08:42And it offers early proof that ai can help members carry out more effective oversight and legislative analysis. This is the landscape crs finds itself in today, one that is changing quickly and where expectations are rising just as fast. Cr crs has an opportunity, uh, to complement and even shape this emerging ecosystem, bringing it, bringing to it the values that have defined the service for decades, authoritative, trusted and transparent analysis.
▶ 0:09:10But maintaining that role of thought leadership requires crs to keep pace with the moment, even as it balances the significant technology investments it's made over the past decade. The question before us today is how can crs innovate boldly while remaining grounded in its core mission, serving every member of congress and every staffer impartially?
▶ 0:09:34How can crs approach this challenge while remaining responsible stewards of taxpayer dollars, where every dollar invested in an it system counts, and where every crs staff member is fully engaged? We look forward to hearing from crs, uh, and from you, doctor, about how you see this moment, how you understand the art of the possible, and how you plan to execute on a vision that allows service to step forward responsibly and confidently on behalf of congress.
▶ 0:10:01Uh, ranking member morelli, uh, is, uh, is tied up for a couple of minutes. He's coming in. I'm going to allow him to give his opening remarks, uh, in just a few minutes. Uh, all other members will have until 5 p.m. Today to submit their opening remarks. Uh, and today we obviously, we have one witness. Uh, we have, uh, doctor karen donfried, uh, director of congressional research services, uh, director donfried, you're now recognized for five minutes to provide your opening statement.
▶ 0:10:31>> chairman steele, members of the committee, thank you for this opportunity to discuss crs and the future of ai enabled policy analysis. Ai is advancing rapidly, and crs sees its potential to streamline our workflow and enhance our service to congress. The use of ai, however, also carries risks of outdated information, hallucinations, bias, and distortions.
▶ 0:10:57In this age of ai, I believe the role of crs is highly skilled. Workforce will become even more important to congress as you look for sources you can trust in an increasingly unreliable information landscape.
▶ 0:11:12I know the committee will have valuable insights regarding how crs can best support congress by utilizing these new tools, while preserving our five core values of confidentiality, objectivity, nonpartizanship, authoritativeness, and timeliness. What is crs ai journey been like?
▶ 0:11:33In close partnership with the library of congress's office of the chief information officer, or crs, has been actively exploring ai's potential. We are developing not only principles to govern the responsible integration of ai tools into our workflow, but also an implementation plan to identify priority use cases for development in evaluating any ai enhanced tool, crs focuses on the extent to which these
▶ 0:12:03Applications. One increase productivity in developing products for congress and two align with crs values. Crs has tested several ai applications. For example, we experimented with ai to produce bill summaries. Crs tested six different large language models over a two year period. Approximately 1000 bills were fed into the models, yielding 3000 ai generated bill summaries.
▶ 0:12:31Less than 3% of the summaries met crs standards for accuracy, coherence, relevance, and objectivity. While that result was disappointing, crs has applied the lessons we learned to develop a plan for several use cases across the bill. Summary workflow that, if fully implemented, would increase our production of bill summaries. Our testing of ai applications has made clear the importance of training crs staff.
▶ 0:13:00We've offered training on topics ranging from ai ethics to generative ai and prompt engineering in fiscal 2027, in collaboration with ocio, we are planning a 12 month evaluation to compare five ai tools chatgpt, claude, google, ai, perplexity, and copilot.
▶ 0:13:19As part of our fiscal 2027 budget proposal, crs requested an increase of $1.6 million, including five permanent staff members with data science and ai development experience, which supplements a request by ocio to support the development of an ai platform that will be critical to our testing and deployment of ai applications.
▶ 0:13:44This platform would provide a dedicated and secure cloud environment to develop ai solutions tailored to crs unique requirements. This funding is essential to realize the most transformational impacts and productivity gains that ai offers, along with understanding the promise of ai, crs has gained valuable insight into its limitations.
▶ 0:14:08Ai enhanced technology excels at processing lots of information quickly, and thus can serve as a valuable tool to assist crs experts. That said, many of the applications we tested produced results that fall well short of crs standards for accuracy and objectivity. Quality control by crs staff will be required to ensure that the end product is authoritative.
▶ 0:14:36Having that highly skilled human in the loop is essential to crs. Building an ai augmented workflow consistent with our values. What's the bottom line? Crs exists for one purpose to support congress with objective, nonpartisan, authoritative, confidential, and timely research and analysis.
▶ 0:14:57Our most valuable assets in meeting that goal are the breadth and depth of expertise, knowledge of the legislative process and institutional memory that our staff bring to bear on every issue of interest to congress. Ai increases the value of that high quality crs expertise.
▶ 0:15:16I appreciate the excellent relationship crs has with you and with your staff, and would like to thank you sincerely for your continued support of and trust in crs. I welcome your questions. >> thank you very much, doctor donfried. I'll begin with questions followed by, uh, the, the, the minority. I'll recognize myself now for five minutes for the purpose of asking questions.
▶ 0:15:42Uh, director donfried uh, see, I, let me put this in about three different buckets of questions here in five minutes, but, uh, crs is authorizing statute guarantees, complete research independence to crs while requiring the most effective and efficient service to congress. Um, balancing accurate. You, you brought this up a little bit in, in your comments about ai. It's kind of balancing the speed of response, the accuracy, the depth, how ai, uh, assists, but also challenges this.
▶ 0:16:10You commented on the oversight, the accuracy of these ai models coming in. Can you provide some color as to how you and the team at crs are thinking about leveraging this new technology while maintaining these standards and the speed at which you're able to deliver that product to members and staff? >> it's a great question. And I talked about the five core values that crs has, and those values are really our promise to you. So we are promising you what we send.
▶ 0:16:38You will be nonpartisan, will be objective, will be authoritative, and we will do that research, analysis in a confidential setting. No one is ever going to know what any of you asked. Crs. And we also pledge that we'll get it to you in a timely fashion, because we know that if you give us a deadline, if we get it to you after that deadline, it's not going to be useful. So that's the promise. But yes, there is a tension across those values.
▶ 0:17:04So when we say we want to give you something authoritative, let's say you give us a very short time frame, but you want to fully sighted memo. We might call the staffer who put in that request and say, okay, we want to help. We know the timeline you have. We may not be able to give you that memo, but what is the urgent thing that your boss needs to know? And we will do our best to get you that information. But the promise is in getting it to you in that time frame.
▶ 0:17:33Whatever we give you is going to be accurate. So we always marry those values. Now, it's not going to be as deep an answer as if you gave us a week. >> does ai just interject here? Does ai speed that up in the sense of some of these ai tools can get you a quick, accurate answer, obviously, then, in the current tools require human oversight to review, double check, confirm those sources, although they may be able to pull up that source, um, almost
▶ 0:18:04In real time. >> so the challenge that we're seeing with ai when you come to research and analysis is these chatbots are pulling from the internet. So some of them perplexity will give you sites, many of them don't give you citations. So we are having to verify any analysis that ai is producing. So that's challenging. And we're increasingly getting requests from congressional staff where they're saying, chatgpt. Chatgpt gave me this answer. Is it true?
▶ 0:18:33And oftentimes it's not accurate. So today we are not finding that that analytical or research function is speeding things up. Where it's really helping is it's very good at summarizing meetings and giving you meeting notes. It's very good at drafting emails. It's very good at coding. So we can use those tools to provide code to put graphics into our reports.
▶ 0:19:00So there are some things it is excelling at and that saves us time and increases productivity. But we still think that verifying any of the research and analytical outputs is essential. >> how do you think about. I'm going to totally shift topics for our final 90s. Um. Siloing inside crs. And how do you build those collaborative teams where people have depth of expertise in certain certain areas? You also have a separate it team.
▶ 0:19:29How do you break through that at a period of time where technology may be able to level the playing field in that space? >> great question. And I would give you two answers to it. First, one of the real benefits of crs is that you have people who have subject matter expertise and also functional expertise. Many of the complex policy issues you're working on, there's a policy element. There may also be a legal element.
▶ 0:19:56There may also be a question about legislative procedure to get that thing done. And we can bring all of that to bear on that question. I actually think in that context, we do a very good job of not siloing the research and analysis we're producing to you.
▶ 0:20:10The other piece of it relates to a realignment that I put in place last year, directly connected to technology, because when I started as director of crs, I thought in looking at the org chart that there were people performing like functions in different parts of the organization. And so we created a new division that is focused on the role technology can play in the legislative work that we do. >> thank colleagues at a time. Thank you very much.
▶ 0:20:40I should have said in my opening, because I know there's a lot of men and women that work at crs with you. Uh, we appreciate all their work as well. Uh, I know you work really hard, but I know there's a handful of people sitting behind you, but a lot of people, uh, at a building just next door, uh, that work incredibly hard on behalf of this institution. We appreciate all of their hard work. Um, I know you'll be back, and I'll recognize the ranking member, Mr. morelli, for a twofer, both for an opening statement, uh, followed immediately by his questions. >> well, thank you.
▶ 0:21:09I do apologize, doctor, to Mr. chairman for being late. Miss bice and I were at the congressional art competition thanking all the awardees. So we were. It got started a little late. Our apologies, but we're glad to be here. Thank you, Mr. chairman, for convening this. And doctor donfried, always a pleasure to see you and have you in front of the committee today. Uh.
▶ 0:21:31For more than a century, congress has relied on the crs for accurate, authoritative and nonpartisan information so that we can better serve our constituents and carry out our constitutional obligations, responsibilities, and duties. To say crs is busy would be an understatement. I'm sure that, um, you'll be the last one to argue that point. Um, uh, in most recent year for which we have public data and this is probably already been disclosed. So I'm repeating what you've already said.
▶ 0:21:59As I understand it, 75000 custom responses to member requests, 2700 new and updated reports, 6500 bill summaries, 2500 confidential memos, and 1850 in-person consultations. I feel like I'm 1800 of those 1850. So thank you to you and your staff for all of that. And the work I understand has not gotten easier.
▶ 0:22:22Like so many of their fellow legislative branch support offices, crs has been asked to do more with less, even as the policy landscape grows more complex and the volume of legislation requiring analysis climbs year after year. Um, doctor, I want to recognize both you and your capable predecessor. Now, the acting librarian, uh, robert newland, for the steadiness with which you've brought the service through a period of real transition.
▶ 0:22:51And so thank you for that. Looking forward. Want to ensure crs is making sound investments in the tools that will get the most out of the service and its brilliant employees. Um, I know in previous testimony, you said the adoption of any I ai. Technology, etc. Crs must enable adherence to crs core values and deliver on the potential to create efficiencies in the development and publication of products for congress. I completely agree and appreciate that. Uh, I'd like to underscore an important point.
▶ 0:23:19The most valuable part of crs is not its technology, however, it's its people and the judgment they bring, the rigor, the expertise, professionalism that lets your analysts and attorneys break down any question for any member on any issue is why crs is the gold standard. Uh, and I will also say that we've been, and I've traveled with my colleagues, different places to talk about the types of institutions that help other legislatures around the world.
▶ 0:23:47And it's clear you have no peer in that regard. You truly are the gold standard. So I appreciate that. Um, and I will say in this age of ai, the scale and speed of legislative work only make that standard more important, not less. And I know I see this, that I'm sure members will continue to use ai to draft legislation and which means the volume is going to go up because instead of 2 or 3 weeks for staff to sometimes develop legislation, ai can do it in a matter of minutes.
▶ 0:24:16Um, I'm not sure that that's good necessarily, but that's the world we're living in and we cannot make the mistake of thinking technology can be a substitute for skilled and dedicated workforce. So crs investment in ai, as I'm sure you agree, should make analysts more efficient, free them to do what only they can do and they should never, ever replace them. So I want to thank you again for your service for being here today. And thank you, Mr. chair, for for holding this. And I'll if you continue, I'll go right into my questions.
▶ 0:24:45This is great. I get to reset the clock. That never happens. Um, but, um, I do want to maybe I can just then transition into. And I apologize if the chair asks this if I'm asking you to repeat anything, but, um, talk about your experience using I, I, I to produce bill summaries. What have you learned? What lessons should we all be mindful of as, as we not only work with you, but work with our own staffs and develop standards about the use of ai?
▶ 0:25:14>> thank you, ranking member morelli, and I just want to thank both of you for sharing your support of the 595 people I work with at the congressional research service. Because I am here representing them. I hope I will do it and make them proud. I'm here to answer all your questions, but those are really the heroes of all of the work that comes out of crs. Um, thank you for asking me about bill summaries. I think you were talking about how ai can produce bills that are really quick speed.
▶ 0:25:42Turns out you all can produce bills that are really quick speed. I think you're up to almost 17000 in this congress. And we have. >> which I will just I pause, and I don't know if the other members have this experience. When you're at an airport and someone says, what do you think about h.r. Six five, seven one? Like, I don't know. Uh, so I, I appreciate you highlighting how many bills are introduced. >> and so, and we got 12 people who were summarizing those bills. So technology is our friend.
▶ 0:26:10We would love to figure out how technology can help us summarize that large number of bills. So there was enthusiasm about an experiment to test the role of ai. And I, I shared in my opening comments that we were a bit disappointed because those six lms that we tested out of the 3000 bill summaries that came out of it, there was an accuracy rate of 3%.
▶ 0:26:35So it was really good at summarizing bills to rename a post office. But when you got to any level of complexity, it was really challenging for the lms. >> so I'm sorry I didn't I misunderstood you say that again about the accuracy level. >> so we tested six large language models. We fed in a thousand bills.
▶ 0:26:57We got 3000 bill summaries at the end of that process when our analysts reviewed them for accuracy objectivity, they assessed that there was a 3% accuracy rate. >> so 97%. >> of those bills would never have met our standard for sharing them with you. Now, what are our lessons from that?
▶ 0:27:23I mean, one lesson, which was your question was, oh, writing a bill summary is more complicated than we might think at first blush, because you're not actually summarizing the bill. What we're doing is analyzing how that bill would change existing law. And it turns out that's a little more complicated. And, you know, when those lms are drawing from the internet, they're they're coming up with some unusual things. Okay, so more complicated. Okay, so how could we use technology?
▶ 0:27:50Second lesson learned is, well, don't throw the baby out with the bathwater. Maybe ai wasn't great at producing those bill summaries, but let's break down the bill summary workflow and see if ai tools can help us along that process. We were able to use end of year money to work with a contractor, and we now have an implementation plan for discrete places in that workflow where ai could help us prioritize bills going to the floor.
▶ 0:28:18I won't go through all the examples, but that is our next step of how we think ai can allow our analysts to be much more productive in producing the bill summaries. Third lesson learned is unfortunately, you need resources to do that. So that's connected to the fy 27 budget request. And I realize it's a hard year. I realize there's not a lot of money. I'm eyes wide open. We may not get that request.
▶ 0:28:43So we will continue to move on that path, but will be much slower and much more limited in what we can achieve. So thank you for the question. >> is that, um, and I note in your budget proposal for 2027 includes a request for five staff members with data science and ai development experience. Is that too in line with what, what your request that you were just referencing? >> yes.
▶ 0:29:06So when we talk about integrating ai tools into that bill summary workflow, we will need to customize tools for that. And I work with really smart people at crs, and many of them are integrating ai in important ways in the work that they're doing. We have a couple of examples, but they have full time jobs. They're not ai developers.
▶ 0:29:30So we really need to buttress our workforce with, we think five people is the reasonable estimate to achieve some of these goals. >> so you actually want them to help develop the ai tool. You're not going to outsource that. You think it's. >> we have various uses for them. But yes, that would be one. >> thank you for all you do. Thank you and your team. And with that I yield back. >> gentleman yields back. Uh, representative lee is recognized for five minutes. >> thank you, Mr. chairman.
▶ 0:29:58And thank you, doctor donfried, for being here with us today. Uh, the congressional research service plays a unique and important role in helping congress understand complex policy issues and serve the american people.
▶ 0:30:13And as technology continues to transform how information is gathered and analyzed and shared, congress must do our part to ensure that crs remains both a trusted source of nonpartisan expertise and an institution prepared to meet the demands of this rapidly challenging environment.
▶ 0:30:35So I appreciate your testimony about your human resources and the staff and expertise that you currently have, and also would like to have to evolve with that environment. I am particularly interested in how emerging technologies, including ai, can help crs better equip members and staff to respond to constituents, while still maintaining our necessary level of accuracy and rigor.
▶ 0:31:05Uh, on that note, we are, um, we've had a great discussion already about the specific aspect of bill summaries and the potential use, and in fact, the unsatisfactory response of the, uh, large language models in creating those bill summaries.
▶ 0:31:24Can you provide us with any specific examples where crs is already using ai or advanced data tools to help analysts deliver information to congressional offices that have been successful? >> mhm. Sure. Um, let me give you a couple. So one I mentioned in passing, which is a I tools are proving excellent at coding.
▶ 0:31:51And so we found that our colleagues who were producing graphics, we have interactive interactive graphics in many of our reports. We also have infographics. I brought you our ai taxonomy given the title of the hearing. But these graphics, you need really quite a bit of coding to produce that result. So we found ai is very good in speeding up that process and freeing up our visualization and graphics experts to do the work that ai can't.
▶ 0:32:19And I think these are helpful for you, for your staff, but also for your constituents. We have them on a whole range of issues, and graphics can be a great way to capture the nub of a complex issue. So that's one example. Another example is at congressional direction. Crs, starting in 2023, was asked to focus on data analytics.
▶ 0:32:39And so to date, my colleagues have created at this point five different data analytic models that we're using from everything from health insurance pricing to a student loan calculator. And one of my colleagues has developed a tool that looks at the regulations.gov website where they're pulling in comments from the public.
▶ 0:33:05And this data analytical tool can look at those comments, synthesize and verify the results, and give you an understanding of how many comments are positive, neutral, negative. And he is using an ai tool with that data analysis, and he's continuing to develop it.
▶ 0:33:29But it's another concrete example of where we're finding ai being very useful in helping us produce a policy relevant result. >> and you just touched on something that that I think is also very important for each of our offices. So we are increasingly asked to respond to constituent questions in real time. And, and they involve, you know, oftentimes many complex issues, some of which you just touched on.
▶ 0:33:56It could be anything from a specific policy area to disaster assistance or help with a federal program or benefit, like veterans benefit or another federal benefit. Uh, and are there uses that you have found in crs? You just touched on a couple with the data analytics and the graphics that you think would help our member offices in responding quickly and accurately to those constituent needs.
▶ 0:34:25I'm interested in the use cases you've identified and how you think those ai tools could be helping us serve our constituents better. >> my advice to you today is go to crs first. We are here to help you and your staff. With that, we serve you in your legislative, in your oversight and in your representational duties.
▶ 0:34:48We do a lot with your constituent requests, whether it's helping your staff understand what grants might be available. We have, uh, one of our most popular reports is actually a liaison for who can your staff contact in your district or in the federal government to help constituents?
▶ 0:35:07Uh, and because I am not confident in the accuracy of ai models to produce analysis for you, I would prefer you come to us and we will do our very best to meet. >> thank you, doctor donfried. We are out of time. Mr. chairman, I yield back. >> gentleman yields. Representative sewell is recognized for five minutes. >> thank you, Mr. chairman. And I wanted to thank you. Um, I have used this crs a lot in my, um role as a member of congress.
▶ 0:35:34But also just to understand, uh, you know, subject matters, uh, expertise in my committee. So I wanted to, again, echo what my colleagues have said about yourself and your amazing staff. Um, and, uh, as the ranking member was talking about, um, you all being the gold standard, I just got back from the british parliament and I met with your counterpart in england and, um, they raved about how you all were the model that they wanted to follow.
▶ 0:36:04So I just want you to know that you are that crs has really gotten a great reputation, not just within, uh, serving us, but with other parliaments. So that's a great thing. Um, now I wanted to know what you see as the future use of ai. You talked a lot about the study and only 3% was accurate. Um, but are there, I mean, ai is improving daily.
▶ 0:36:28Uh, and there are so many different language, uh, you know, models that, that, that are being tested. And so my question is, what do you see as the future use? And then what is the best use currently of ai if not in analytics? Is it in those graphics or is it you just explain a little bit more if you don't mind. >> right. So our view of the future is, okay, we're going to integrate ai into the bill summary workflow.
▶ 0:36:57And that's going to allow us to produce a much, much more significant number of bill summaries for you. We think that on the research and analysis side, it's how do we take advantage of the power of ai, the speed of ai to process information that is more reliable, that can almost be like a research assistant to crs analysts, and that's going to take some time.
▶ 0:37:25I don't think that's a tomorrow thing, but that's why we want to be testing these five different chatbots, because they all are a little bit different and excel at different things. So it might be that we use x ai model for coding, y ai model for that research assistance. But we think there's a real opportunity there. And first we test it for our own use. But then we would imagine you all could benefit from it as well. And if it's coming through us, you know, it's meeting those standards of accuracy and objectivity.
▶ 0:37:54Uh, so there are these very important use cases that we are moving toward, and we'll have to see how quickly we move there. You're right. The technology is changing with every passing year. So that's and I would.
▶ 0:38:07>> think that there's a way that you could work with some of the, you know, some of these, uh, model that are being tested by talking about the expertise and giving the analytics that is required so that the input into these ai models come back better quality. Is that, is that even a possibility? >> well, you know, if you look at. So I mentioned our budget request, which complements the library's request.
▶ 0:38:33So in an ideal world, we would like at the library of congress to build an ai platform that we could train a large language model on so that we would train it on legislative data. And then you have circumscribed. >> your value. >> is pulling from. Yeah. And that would be fantastic. So yes, you know, we, we have a lot of ideas about where we can go here. And I think it is transformational.
▶ 0:39:03>> um, and so the best use of us, I think is an appropriation. So I wanted to know how much more you've asked that relates to the crs being, um, technologically savvy and advanced. >> well, thank you for that. And so the library has a request and the library is open to that, right. The crs request is for an additional $1.6 million a year. That is primarily going to those five individuals.
▶ 0:39:30I mentioned data analytic and ai development experts. It also would allow us, when we're testing those five additional chatbots, to not only use the free models, but to have. I think, as we all know, there are multiple levels on these models. And the more sophisticated levels do perform better. So it is for other things as well. But it's 1.6 million and that is a recurring expense. But we think that puts us in a very good position over the coming years.
▶ 0:39:59We don't anticipate next year or the following year having an increase. >> and I think it also will save us money in the long run. I would think in terms of quality of productivity and and the like. Thank you again for what you do for us and our constituents. Thanks. >> Mr. griffith is recognized for five minutes. >> thank you very much. Let me join in the. Thanks. Uh, crs has done a great job during my time here in congress. And it gets complex sometimes. And and I greatly appreciate it.
▶ 0:40:28Miss sewell mentioned, you know, getting help on, uh, subject matter, uh, particularities or stuff. And, and last year, uh, I had an experience where we had that. And for the folks watching back home, you know, it and we know it, but the committees up here don't always have clear cut jurisdiction. And so trying to figure out what the subject matter is of your particular committee or subcommittee can sometimes be interesting.
▶ 0:40:53I remember I was first here, and there was an issue I was serving on the energy and commerce committee, and there was an issue about, uh, water outside of the min<u>%</u>, but that was affected by mine in west virginia. And I'm like, oh, we gotta get on this right away, only to find out it was natural resources and not, uh, the committee that had primary jurisdiction over the epa. But if the water was in the mine, it was ours. But if it was outside on the ground, it was somebody else's.
▶ 0:41:19So, uh, I took on a new task, uh, at the beginning of 2025, taking over the environment, uh, chair of the environment subcommittee of energy and commerce. And I'm like, okay, what exactly can I go after? Or what do I have to keep my hands off of because it's not mine? And crs did a great job of putting together a paper explaining all the jurisdictions and some of the history of the subcommittee. So I'm greatly appreciative for that. I'm also appreciative, uh, for your skepticism on ai.
▶ 0:41:47Now, I love ai, and I think it has great potential. And I loved your example that you gave a minute ago about using it as a research assistant. But in a timely article yesterday, and you probably saw it in the washington post, they did basically what you had done, trying to figure out whether it would work for you, and it was only 3% effective on bill summaries they went through and they specifically asked political questions. So a lot of times when I'm using ai, I'm asking, you know, what year did a particular battle happen?
▶ 0:42:15If I'm doing a history or, you know, what was the time sequence on something? And it was pretty good on that. Uh, but the causes of the conflict and so forth are more political in nature and the study. And Mr. chairman, if I might, I'm going to offer the washington post article of yesterday by kevin shaw. Uh, our chat, uh, tea and other ai chatbots, politically biased. Uh, they tested them and the answer was yes.
▶ 0:42:41Uh, if you ask a political question, open ai came back 80% of the time with a left leaning argument, and only 3% of the time with a right leaning argument. Uh, deep seek came back with 70% left leaning and 7% right. Uh, gab came back 50%, uh, left, 3% right.
▶ 0:43:02The other the, the rest of the percentages were in the middle, uh, anthropic 43% on the, the left, none on the right. Uh, xai grok, uh, came back 40% on the left and 33% on the right. And google was 7% on the left. But the rest of it was, uh, they presented both sides, which was kind of interesting. Now, whether they got it right or not, the post didn't go into.
▶ 0:43:32And of course, the companies all argued they did the test wrong. Uh, but this was their study. They did quite an extensive study. And so I'm going to ask that we submit that for the record, uh, objection in support in support of, uh, uh, doctor don fritz, uh, uh, statements. We should be skeptical. It is a tool we can use. And you also mentioned in your comments, and I just want you to talk about that a little bit more. If you have time, uh, that you want these, this extra money to train the ai. And I think that's the real problem.
▶ 0:44:02If you have people, I don't think any of these companies set out to give left leaning answers, but the people they hired to do the training of the ai were probably all leaning left. And to them, this is the right answer. Um, and it was interesting that the google one mostly came back with both sides. Here's this side, here's that side. Uh, I would think that, uh, that would be helpful in your training and why you need those five people. Am I correct in that? >> yes, yes. I mean, I really appreciate, I was wondering if that article was going to come up about bias.
▶ 0:44:31It's so interesting. Well, is it okay if I speak? >> yeah, we got 45 seconds. >> um, there was a I've talked to many members of congress who I would say are tech forward on both sides of the aisle. And one of them we were talking about crs, will it be replaced by ai? And he put that prompt into perplexity. And I got this two and a half page letter from him, and I thought it was so interesting. It's the central challenge is not that ai will make research obsolete.
▶ 0:44:59It is that ai will make plausible looking research cheap, fast and ubiquitous so that it's not that our eyes are closed and we don't want to embrace this technology. It is a tool. It's a powerful tool, but it needs to be marrying that human expertise and experience with the technology. >> and if and to summarize, if you want c minus to f work, you at this point in history, use the ai.
▶ 0:45:27If you want a work which the crs is always given us, you need the human factor. Correct? >> I appreciate that great. >> yield back. >> gentleman yields back. Representative torres is recognized for five minutes. >> and thank you, chairman. And thank you so much for being here, doctor friedman. Um donfried I'm sorry. Um, thank you for being here. And I, I want you to also share my gratitude, um, with your workforce. They do an incredible job on so many different topics at every level.
▶ 0:45:58So I've used them, you know, constantly. And I, I find their work, um, very useful on the issue of, uh, of the ai and how it could help us as members of congress, um, be more transparent and more accountable.
▶ 0:46:14Uh, the inland empire, the area that I represent is poor working class community, some of the fastest growing communities, um, in the country during the good times, during the bad times, you know, we are the first ones to feel the hurt. So oftentimes my constituents are confused as to where do I go to find information about what congress is voting on? How are they addressing the issues on the economy?
▶ 0:46:44Um, you know, the price of groceries that something so basic, um, what are you working on? How are you voting on the issues that are important to me? And I think that that is a place that, you know, this committee really has a huge responsibility to try to focus on providing some of some of that background work. And I know, um, the chair of the modernization subcommittee, rep lee, will be hosting a meeting next week.
▶ 0:47:13And I look forward to, um, having you in, in getting more information on how we can modernize congress more. But I think a very simple tool that could help us be more transparent. Um, for example, why are the vote tallies, how each member is voting in the moment on the house floor? Um, why isn't that public?
▶ 0:47:43Congress.gov can have amendment text up right away. I mean, I don't, you know, putting something on the website, I don't think that that is, is such a hurdle. Uh, so the american people know exactly what congress is debating real time, not just the end result. Uh, and then how our constituents are able to comment back whether it is about a policy that congress is reviewing.
▶ 0:48:11How, how are we measuring if it's bots that are, are, are providing feedback, or if it's real humans, uh, providing that input. Um, those are some of the things that I'm curious about how you're working through some of that and how you're, um, you'll be helping us modernize, be more transparent.
▶ 0:48:37>> so as I hear what you're saying, you're suggesting that congress.gov could play an even larger role in leading to transparency of the workings of congress. And the library of congress is the owner of congress.gov. And we curate and put certain data on congress.gov, like the bill summaries and and other things.
▶ 0:48:59And I think what you may be referring to is I believe the deputy cao is meeting next week to talk about congress.gov, and I think he's very well placed to talk about this. Um, congress.gov. We also work with many other stakeholders on the hill, and I think there'd be real interest in pursuing the ideas that you have.
▶ 0:49:20My understanding is that currently those kind of the data about floor votes goes onto congress.gov, I think, within 24 hours. And it is likely that that could happen more quickly. It sounds like you're asking for that to happen. Is it. >> possible to have those votes, um, that are happening real time online somewhere? >> I, as a non technologist, imagine it's possible. Um, but I think that is something very much worth exploring.
▶ 0:49:49I am happy to take it back and, and share it with my colleagues in ocio. And I think in that conversation next week, it's also a really good thing to explore. But I know the library and crs are very open to responses about how helpful is congress.gov in meeting your needs. I mean, all of your constituents can pull up any crs.
▶ 0:50:13>> and utilizing the ai to, um, sift through those, uh, public comments to ensure that those comments are actually coming from real life people. >> and again, there, I mean, I do think there is a way to the example I know, which may not be analogous, is there's been a huge spike in requests of congress.gov.
▶ 0:50:37And I know ocio is able to differentiate which of those are coming from machines and which are coming from humans. And overwhelmingly it's coming from machines because a lot of folks are trying to scrape congress.gov for the data. Amazon, does it go deeper than. >> you'd be able to do it too? Thank you. And I yield back to the chair. >> gentleman yields back. Doctor murphy is recognized for five minutes. >> thank you, Mr. chairman. I need ai to turn on the microphone. Um thank you, Mr. chairman. Good to see you again. >> good to see you.
▶ 0:51:08>> so, uh, so I I'm sitting here trying to think, which is always a challenge. And I asked ai, I said, should congress replace ai with should replace should congress replace the congressional research service with ai? What did it say? It said probably not. Probably not. So I know there's a little bit of of angst and everything.
▶ 0:51:31And this is really a fascinating conversation, just living in the world of medicine and just seeing the takeover of ai and how we are doing things so radically different, literally in the last three years than we were three years, uh, just in the last three years and how the next three years are going to be an explosion. This is going to change our entire way of life, hopefully for the better. I think there are some, um, I don't want to say warmongers, but people that think, oh, the world is going to crash to an end.
▶ 0:51:59But to my good friend, uh, representative griffith point is, uh, it is what we put into it. And this is where you guys, I have to applaud. You guys are a nonpartisan service. You give us the data, you give us the facts. This is what it is. Therefore, and the fact that if you do certain searches on certain things, you will get answers that are have a proclivity to move in a different, a different political direction. So, uh, so I'm going to say probably not. I'm going to say definitely no to, to, to ai.
▶ 0:52:27When we're doing this, I think to the point of constituent services, I was just talking with miss bice next to us. Um, I think it's a great idea, in part because it right now in the field of medicine, we have ai. If a patient calls up with a particular problem or maybe a post op thing, you have a bot, as it were, that will sit there with the patience of job and answer every question. The problem is we get a lot of questions about the irs.
▶ 0:52:53We get a lot of passport things that are all confidential, and they're not able to be queried. So I just I don't see that being an everyday part of with this. And so, you know, we're going through this. I'm sitting here thinking, what can I ask you? Because there's so many different points here. Um, I would just hope that you are embracing this and not running from this because we're not going to be able to run from this.
▶ 0:53:18I had a fellow at a recent convention that said he didn't want ai in his life at all. You know, not a bad thing to take a fishing pole and go to montana. But we can't really do this. And so as we move forward, what do you see the next five years looking like this? How is it going to change your workflow? How how are you going to adapt to it? Um, are you guys fully committed that this is happening and that that crs is going to have to change the way it does business, but also still be able to deliver the excellent care as you move forward.
▶ 0:53:49>> right. So thank you for the question. And I hope there is nothing I have said this morning that suggests we are running away from ai. I just want to repeat, I think ai is a powerful technology that crs wants to and needs to harness, and the experimentation exploration we've done to date has taught us a lot, both about the benefits ai has to bring, but also some of the limitations. I mean, my crystal ball is cloudy.
▶ 0:54:16I don't know how ai is going to change improve over the next three years, but what I want is for crs to be experimenting with this tool and understanding the ways ai can best help us be more productive in the work that we do. For all of you. Yes, we want to maintain those standards. Again, that's our promise to you that we're giving you something authoritative.
▶ 0:54:42We're giving to you something that's not biased, that's nonpartisan, that's timely, and it's confidential. We want to meet all those markers. But of course, we want to harness technology that allows us to be more effective in what we do for you. So, you know, the things I've shared with other members about what we have planned. You know, could ai serve as a research assistant?
▶ 0:55:03How will this allow us to not only look at the bills that are going to the floor, which is how we're prioritizing things now, but look at, if not all 17,000, the majority of those bills. So we have a lot of ideas about how we could use ai. And if we can train an ai model on legislative data, that opens a. >> whole new. I think to your.
▶ 0:55:26>> point, I don't mean to cut you off, but to your point that you guys need to train the model, not take something off the shelf, which is just inherently sterile, not not the right thing. Um, it's really going to be, it's going to be the best delivery of a product. Now that will obviously take a lot of time. That will be intensive and require constant monitoring, constant adaptation as we, you know, embark upon this thing. But it's a, it's a brave new world and a lot of, I think good ways. Um, but it is what we put into it.
▶ 0:55:52And so I appreciate the fact that you guys are going to be able to put in nonpartisan, just accurate information so that we can, we can, we can make our judgments. The last thing, good lord, if we're putting out 15, 17,000 bills, the last thing we need is ai to spit out another 25,000. So the senate. >> just and we will need your help to do this in real time and and meet the challenge. We will need your support and we will need some additional resources. Great. Thank you. >> oh! >> you're back. The knocking is me. The gentleman yields back.
▶ 0:56:23Representative johnson is recognized for five minutes. >> thank you, Mr. chairman. It's such a treat to see you. Um, I love crs, you know that, um. I think you guys do amazing work, um, and make it possible for us to be as thoughtful as we can in this important role that we all serve.
▶ 0:56:42Um, so many of my colleagues have already asked a lot of the initial questions that I had, but I want to dig into, um, kind of your comment about the 3% accuracy rate. I think that's alarming. I'm stunned by that number. I bet many of my colleagues up here are stunned by that number.
▶ 0:57:04Um, and I think it, it really highlights that we as policymakers can't have a false sense of security of the accuracy of ai. And we have to really put guardrails on that. Um, and, and be vigilant about the data sourcing into these models. Um, and your $1.6 million request, does that contemplate using public available models now or not?
▶ 0:57:32That does not include or contemplate you having your own model? Is that right? >> so only if the library portion of the request were also appropriated, would we have the ability to train an llm? So we're complementing the library's request. And our portion of it is specifically for five humans, uh, allowing us to buy licenses for the more sophisticated level. So it would need it would need to be both of those. >> yeah.
▶ 0:58:01So it's my understanding that, you know, the really the ai models work. They can't process all the data in the universe. And so they work at their best when they're narrowly tailored to a specific function or within a specific data set or within a specific, um, defined parameter of information and content that they can really perform at their best level. So it is incumbent upon us to fund the library's request.
▶ 0:58:28And in tandem with this, to make sure that we can have our own internal ai model that's designed for maximum productivity for crs. >> and two quick comments. I think if the library were to develop that ai enterprise platform, it would serve the entire legislative branch. I mean, the library is the repository of that data. So I think it the implications are far beyond the library and crs, a and b.
▶ 0:58:56The other important piece of this is if crs uses the free versions of these chat bots of these ai models, those are not closed systems. So we couldn't put any confidential material in there. We wouldn't do that. So it's we can only create a closed system with those higher levels of these ai models. And again, that's a pledge we have to you.
▶ 0:59:21We will always treat any congressional information in a confidential manner. So that's where some of these resource questions become quite important. >> right. It seems troubling to me that the government of the united states, the congress, would rely on chatgpt for all of our data. You know, I joke with my mother, she's, you know, 85. And chatgpt is her best friend.
▶ 0:59:48She puts us a query into chatgpt and it says, oh, that's a brilliant question. You are so insightful. And then she'll ask a follow up. That is a great clarification. You know, it. It's so affirmatory of of my mother's inquiries and they may not be the best ones, you know, they may not be the most clever, insightful question.
▶ 1:00:11And so I have, I have, I'm becoming more skeptical, um, of these, of these, of these giant ai structures, um, that the federal government will rely on for its data sourcing and, and information. And I think to your point that you've made in your testimony, there's no substitute for the breadth of the expertise of the crs staff. Um, it's just robust.
▶ 1:00:35It's impressive the, the quality of expertise that you have, um, there to bring to bear that at, at least at this stage in ai development, it cannot be matched. Um, and so I think you're right to, um, assess the capabilities that ai can enhance your productivity, but certainly not replace it.
▶ 1:00:57Um, but I'd be interested in seeing how congress can help develop our own internal ai system to your point of confidentiality and resource and targeted usage. I think that would be an example of work this committee should continue to, to pursue and I think would be very worthy going forward. So thank you so much for being here. >> I appreciate it. I'm going to use the five seconds to say my favorite thing about crs is all my colleagues.
▶ 1:01:25My second favorite thing is new member seminar. And so I know congresswoman johnson from new member seminar. And it is something we do together with the committee on house administration. And I'm really proud of what we do. So I'm looking forward to january. Yeah. >> the gentleman yields back. The gentlewoman from oklahoma, the chair of the modernization subcommittee, representative bice, is recognized for five minutes. >> thank you, Mr. chairman. And yes, uh, first let me say that I want to piggyback off of my colleague from texas. Comments.
▶ 1:01:54In your original statement, you mentioned that you had tested six models to try to see if they were able to accurately summarize a bill, and that was they were not successful. So I guess my first question to you would be, are we looking at potentially building a custom llm that would be specific to crs to be able to do bill summaries, or is that something that would be cost prohibitive? >> so what we're looking at doing now is we're saying, okay, maybe ai can't help us produce a bill summary out of the gate.
▶ 1:02:26And that's why we then said, let's break down the bill summary workflow. What are all the discrete steps that our colleagues undertake to produce that bill? Summary. So one of the first steps is, right now we're prioritizing by saying, what are the bills that are going to the floor? We think an ai tool could be very helpful at doing that task and then telling the analyst, here are the bills that are going to go to the floor, focus on those.
▶ 1:02:52>> so I guess what you're saying, though, is you're not looking at building an internal llm that's specific to congressional sort of language or bill summary or bill language. >> we may get there, but I'm. >> saying right now we're not. >> though, but right. What we're trying to do right now is build that workflow tool that can allow us to produce more summaries if we actually get funding, if the library gets funding to build that ai platform and we can train an llm on legislative data, it sets us up for then producing bill summaries.
▶ 1:03:21>> so, um, I've enjoyed working with you over these last, you know, months and years. So, uh, as the chairwoman, I believe, of the modernization and innovation subcommittee, along with the ranking member, miss torres, um, we have been able to really, I think, fine tune, um, the, the requests that we have coming, uh, from members, but also to figure out how can we improve that interaction with crs.
▶ 1:03:46So in your fy 2011 budget request, it includes 3.5 million, um, programmatic increase of $3.5 million to enhance ai and data analytic capabilities. And you've signaled the funds would support expanded text and analytics tools, ai enabled research, and broader it modernization.
▶ 1:04:06My first question is, many external stakeholders have told us that there's a need for an api that will help them build new tools, um, using public crs information. Can you commit to making that tool available? >> so first, I just want to say that that increase in our appropriations ask isn't all for ai. Uh, a good portion of it is for staff. So 94% of crs budget is our people.
▶ 1:04:36>> and but don't let me let me ask, let me stop and ask a question, though. Don't you think if we were able to utilize technology, you wouldn't necessarily need additional staff? >> well, okay, so first let me say there is an api on congress.gov. So we have done that and it's being used very actively. So we have done that on congress.gov. But I'm not ready to say that ai will allow us to have fewer staff. That may be right.
▶ 1:05:01Um, but what we're seeing today is ai is not doing the job crs people are doing. So if we talk about bill summaries, I don't think we're overstaffed on bill summaries. I think the 12 people we have writing those bill summaries, there are challenges. There's no way they can keep up with the 17,000 bills. So what ai is going to enable us to do is not have the >> incredible back load we have right now. So I think it's enabling us to be more productive and therefore meet your needs more effectively.
▶ 1:05:32But it's not, at least today, obvious to me that we need fewer humans at crs. We may get there, but you know, we're not big. We're 596 people. And as I believe the ranking member mentioned, they're producing 75000 custom responses to you every year. >> so how far behind are you on bill summaries currently? >> so what. >> do you prioritize them?
▶ 1:05:59>> so the way we prioritize bill summaries now, and this was in consultation with you, was by saying, we want to produce a bill summary on every bill that goes to the floor. So now we're meeting 97% of those bills that are going to the floor. We're getting you those bill summaries 24 hours before they go to the floor. So yes, we have to prioritize, but we do think ai will help us obviously, create bill summaries that are not going to the floor. Okay.
▶ 1:06:28That's where we are today. >> last quick question, um of the increase. How much is earmarked for concrete near term upgrades like text analytics or workflow and automation tools that congressional staff can benefit from this year. >> so the 1.6 million that we're asking for, for the ai part of the request. >> so is that is that 1.6 of the three and a half? >> exactly, exactly. >> so the majority of the dollars are actually for additional staff, right.
▶ 1:06:57>> because as you know, we've had a flat budget for the past three years. Our costs have gone up 8%. So our the part of our budget that isn't staff, the biggest part of it is research materials, which we've cut quite dramatically. But we can't cut more because we need research materials to do that work in any case.
▶ 1:07:14So yes, the 1.6 million that is primarily for those five humans with those special ai development and data analytical skills, and to allow us to buy licenses for chatbots that are not the free versions. We are not in a place now where we have ai tools that we can give congressional staff to use. We would like to get there. >> Mr. chairman, thank you, I yield back. >> gentleman yields back. We appreciate your testimony today. I want to thank you for joining us.
▶ 1:07:43Appreciate all the work you and everyone at crs does in supporting congress. Members of the committee may have some additional questions for you, and we ask that you please respond to those questions in writing. Without objection, each member will have five legislative days to assert additional material into the record or revise and extend the remark. If there's no further business. I want to thank the members for their participation. Without objection, the committee on house administration stands adjourned.