WEBVTT 1 00:00:11.730 --> 00:00:15.269 Wodarczyk, Jennifer: Welcome everyone to the second session for T cal. 2 00:00:17.400 --> 00:00:30.870 Wodarczyk, Jennifer: Without further ado, in an effort to save time i'm going to let our presenters introduce themselves and our current tikal session Thank you so young and Lisa. 3 00:00:33.630 --> 00:00:38.730 Hwang, Soo-yeon: So, shall I wait one more minute and start on the hour or just go and. 4 00:00:39.060 --> 00:00:40.260 Wodarczyk, Jennifer: You can you can. 5 00:00:44.790 --> 00:00:49.140 Hwang, Soo-yeon: Okay we're just go ahead, then we're just going to go ahead. 6 00:00:50.370 --> 00:01:03.240 Hwang, Soo-yeon: So, welcome to our presentation chatbots in US libraries history and prospects, my name is so young, I am the assessment and analytics librarian at towson university. 7 00:01:03.690 --> 00:01:15.660 Hwang, Soo-yeon: And until last year I worked as a musician State University and Lisa she needs my former colleague from simonson state she's the director of library public services there. 8 00:01:20.880 --> 00:01:22.650 Hwang, Soo-yeon: So what is a chat Bot. 9 00:01:23.100 --> 00:01:45.570 Hwang, Soo-yeon: A these are basically robots made for chatting they are programmed to carry on intelligent conversations with people and according to Amazon these automated agents follow some rules in terms of conversations like if if they. 10 00:01:46.740 --> 00:02:01.260 Hwang, Soo-yeon: are given certain questions they give out certain answers, so they provide these ready made responses, based on what the customer inputs into the system. 11 00:02:02.550 --> 00:02:14.760 Hwang, Soo-yeon: And if they are unable to provide right answers they may refer the patron or the customer to a human representative. 12 00:02:19.230 --> 00:02:24.450 Hwang, Soo-yeon: So we are going to show you a quick DEMO of what. 13 00:02:25.980 --> 00:02:41.160 Hwang, Soo-yeon: How the library chatbots work so we're going to show you later about the current status of US library tripods, this is one of them, and this happens to be not one of our. 14 00:02:42.180 --> 00:02:54.720 Hwang, Soo-yeon: interview participants, so we felt more they feel better in terms of using them as our guinea pig because we didn't interview them, so let me. 15 00:02:57.330 --> 00:02:58.500 Hwang, Soo-yeon: get out of the. 16 00:03:01.230 --> 00:03:24.690 Hwang, Soo-yeon: presentation, and so this is the University of Oklahoma and they're using a chap a system called BC, which is the services provided by a third party company called IV that Ai so let's try a couple questions too busy. 17 00:03:25.740 --> 00:03:27.570 Hwang, Soo-yeon: So if you right. 18 00:03:28.710 --> 00:03:30.180 Hwang, Soo-yeon: Like are you open. 19 00:03:31.590 --> 00:03:35.550 Hwang, Soo-yeon: Your right are there are a couple of. 20 00:03:36.780 --> 00:03:42.270 Hwang, Soo-yeon: Pre filled positions are you located our jobs posted here. 21 00:03:44.640 --> 00:03:45.960 Hwang, Soo-yeon: Regiment hell. 22 00:03:47.070 --> 00:03:54.540 Hwang, Soo-yeon: Well, we see are you open now and we also see are your computer well let's say let's try, are you open now. 23 00:03:57.810 --> 00:04:04.470 Hwang, Soo-yeon: They give you the template answers about the hours a link to a full schedule. 24 00:04:10.290 --> 00:04:12.210 Hwang, Soo-yeon: So i'll see it did hell. 25 00:04:15.540 --> 00:04:25.620 Hwang, Soo-yeon: My next question be for me just for fun let's see what it answers to the prompt are your computer. 26 00:04:27.300 --> 00:04:30.150 Hwang, Soo-yeon: says, I am okay it's a computer. 27 00:04:32.790 --> 00:04:42.720 Hwang, Soo-yeon: i'm curious about their like let's say I must do it i'm looking for a textbook let's see what questions I can address. 28 00:04:43.950 --> 00:04:50.070 Hwang, Soo-yeon: and looking for Expo for. 29 00:04:52.770 --> 00:04:56.310 Hwang, Soo-yeon: 300 i'm just making up the course name. 30 00:05:03.690 --> 00:05:09.120 Hwang, Soo-yeon: So, here are some generic answers about textbooks textbooks and calculators. 31 00:05:19.560 --> 00:05:22.980 Hwang, Soo-yeon: And then let's see. 32 00:05:24.900 --> 00:05:32.430 Hwang, Soo-yeon: So textbooks it gives us some information about their textbook reserve Program. 33 00:05:34.080 --> 00:05:45.840 Hwang, Soo-yeon: let's say i'm writing on assignment and I need to find some peer reviewed articles so i'm going to just say purely do. 34 00:05:48.420 --> 00:05:50.010 Hwang, Soo-yeon: This thinking. 35 00:05:57.210 --> 00:05:57.840 Hwang, Soo-yeon: still think. 36 00:06:06.390 --> 00:06:10.680 Hwang, Soo-yeon: Oh, he says, I found several topics that may answer your question. 37 00:06:11.700 --> 00:06:21.330 Hwang, Soo-yeon: So here if you articles, how to find journals retrieve the moment so let's say how to find. 38 00:06:22.590 --> 00:06:24.840 Hwang, Soo-yeon: keep sending me to Google scholar. 39 00:06:26.490 --> 00:06:34.650 Hwang, Soo-yeon: let's go back and look how to find journals it's sending it to general search. 40 00:06:38.820 --> 00:06:39.540 Hwang, Soo-yeon: The tree. 41 00:06:42.960 --> 00:06:45.360 Hwang, Soo-yeon: So this library has learning. 42 00:06:49.980 --> 00:06:53.760 Hwang, Soo-yeon: where you can learn about the terms like peer reviewed and squarely. 43 00:06:59.880 --> 00:07:00.930 Hwang, Soo-yeon: limits. 44 00:07:02.130 --> 00:07:07.410 Hwang, Soo-yeon: Well, the same thing it's it's giving me the answer about learning lab. 45 00:07:12.270 --> 00:07:18.450 Hwang, Soo-yeon: What about unknown items search, so let me see do you at the. 46 00:07:19.860 --> 00:07:20.610 Hwang, Soo-yeon: mall today. 47 00:07:26.280 --> 00:07:29.550 Hwang, Soo-yeon: And it's giving me an answer about their catalog search. 48 00:07:30.630 --> 00:07:32.280 Hwang, Soo-yeon: And what about. 49 00:07:33.540 --> 00:07:34.140 Hwang, Soo-yeon: to you. 50 00:07:42.150 --> 00:07:57.270 Hwang, Soo-yeon: Okay it's giving me a non answer basically so when you have the word the book in the question, it detects that i'm asking about apple and it's giving me a link to the catalog to go and search. 51 00:07:58.650 --> 00:08:11.610 Hwang, Soo-yeon: notice that throughout this entire transaction it's not sending me to a live person, so I wonder whether they're just offline or just what's happening. 52 00:08:13.230 --> 00:08:20.490 Hwang, Soo-yeon: The system or maybe the system does is a program to send you to the by person. 53 00:08:21.570 --> 00:08:27.060 Hwang, Soo-yeon: So anyway, this, this is a quick DEMO of our working chat Bot on. 54 00:08:28.350 --> 00:08:30.360 Hwang, Soo-yeon: Its library. 55 00:08:31.590 --> 00:08:33.600 Hwang, Soo-yeon: So let's go back to our presentation. 56 00:08:38.940 --> 00:08:48.000 Lisa Shen (she/her): Alright, so now that we've kind of established when we report refer to a little library chat Bot what that looks like and seeing one sort of being action both what. 57 00:08:49.020 --> 00:09:00.270 Lisa Shen (she/her): yeah what it can do, and perhaps some limitations that I like to provide a little bit more background story about how, how does like our research kind of came to be or or how to study came to be. 58 00:09:00.570 --> 00:09:11.400 Lisa Shen (she/her): So over also ran a grander scheme of like the technology landscape, if we would as information technology kind of events is during the last past few decades now really user experience and expectation have. 59 00:09:11.400 --> 00:09:17.460 Lisa Shen (she/her): evolved and then Ai artificial intelligence based tools such as chat bots have really been gaining prominence. 60 00:09:17.850 --> 00:09:21.150 Lisa Shen (she/her): Definitely just overall outside or library spheres already. 61 00:09:21.510 --> 00:09:36.360 Lisa Shen (she/her): And then in a more local context, and this is really credit to some young because, like within kind of her expertise, both for information technology, what services and also user experience I think she's just been kind of seeing these type of. 62 00:09:37.650 --> 00:09:51.360 Lisa Shen (she/her): Service services kind of prop up, I should also mention think a little bit both of us are from originally from other countries, so with three rounds case, also in South Korea she's also seeing there's also a prominence of library chat bots that are. 63 00:09:52.800 --> 00:10:04.560 Lisa Shen (she/her): that have been introduced, over time, as well, so we were kind of interested in looking at it in that way, and then I also come from the perspective of asked a person kind of primary responsible for. 64 00:10:05.070 --> 00:10:13.800 Lisa Shen (she/her): Coordinating public services kind of during dependent and during the pandemic as services shifts virtual or kind of sensing an overall. 65 00:10:14.580 --> 00:10:24.990 Lisa Shen (she/her): While the number of kind of research reference request kind of hard to remain perhaps dropped a little bit initially but kind of remaining steady sort of we're definitely also kind of seeing that. 66 00:10:25.350 --> 00:10:40.530 Lisa Shen (she/her): But for our patrons sort of the hour of time of day, where folks may be expecting media services are in need of immediate services have changed so that's also part of the reason that got me kind of more curious about the use of chatbots. 67 00:10:41.730 --> 00:10:46.170 Lisa Shen (she/her): By libraries in terms of providing virtual services as well. 68 00:10:46.740 --> 00:10:58.710 Lisa Shen (she/her): So, but, that being said in our initial kind of dive into the current library literature, at least for the US, and also in large part for Canada, we really found quite limited, we were. 69 00:10:59.460 --> 00:11:02.610 Lisa Shen (she/her): I was kind of surprised by the limited research that's available. 70 00:11:03.180 --> 00:11:13.680 Lisa Shen (she/her): In both academic and public library contexts, there are definitely some presentations and case studies here and there, but in terms of more comprehensive kind of survey and also just instances of. 71 00:11:14.100 --> 00:11:23.460 Lisa Shen (she/her): library chat bots were very limited That being said, though, there were some advantages that were kind of the existing literature Doo. 72 00:11:25.560 --> 00:11:31.980 Lisa Shen (she/her): Doo do agree with do kind of have some consensus on and now also kind of matches some of our assumptions as well. 73 00:11:32.280 --> 00:11:42.690 Lisa Shen (she/her): So, for the first part is certainly chat Bot can provide services around the clock 24 seven so for our patrons they can get immediate responses, even when the library is closed. 74 00:11:43.350 --> 00:11:51.540 Lisa Shen (she/her): And then also perhaps more importantly, because if we're doing instruction session talking or different sessions talking about a highlighting library chat. 75 00:11:51.960 --> 00:12:05.790 Lisa Shen (she/her): library virtual services casing point i'd probably would be a little bit more reluctant to share the URL to the live chat Bot, or to the live chat service right away, as I did in the presentation here. 76 00:12:06.300 --> 00:12:17.610 Lisa Shen (she/her): that's because multiple chats or you will have to come with a disclaimer not everybody all click at once, because that would overwhelm the person monitoring the chat, on the other end, however, with the chat Bot. 77 00:12:18.150 --> 00:12:29.370 Lisa Shen (she/her): The computer program can handle simultaneously handle multiple users, so that also allows for there's no queuing issues that can that will be prompt and continue services for anybody that. 78 00:12:30.570 --> 00:12:34.080 Lisa Shen (she/her): yeah for mobile users chap chatting with the chat Bot all at once. 79 00:12:34.890 --> 00:12:49.590 Lisa Shen (she/her): And also, perhaps this is always always always a concern anyhow, so this may also help the bill via the chat Bot meals will help you leave some of the library use anxiety for asking questions that perhaps folks may be reluctant to ask like human. 80 00:12:50.880 --> 00:12:59.220 Lisa Shen (she/her): And then also, this is the hope anyhow it's both with expensive services but also perhaps we chat bots as the first providing the first layer of. 81 00:12:59.520 --> 00:13:12.240 Lisa Shen (she/her): More routine basic questions, then that will free up reference library as a reference personnel to focus on focus their energy on more complex console research questions that demands their attention. 82 00:13:15.420 --> 00:13:23.310 Lisa Shen (she/her): Alright, so since we kind of dive into the library literature, to now find a whole lot within also shifted, then we would like to then kind of. 83 00:13:23.610 --> 00:13:33.720 Lisa Shen (she/her): do some research ourselves so them by perhaps looking at existing library chat bots and learn from their experiences and what we quickly discovered. 84 00:13:34.200 --> 00:13:46.680 Lisa Shen (she/her): Was that there really wasn't a whole lot instances, with a disclaimer that this is not I can't say 100%, this is a comprehensive list, but I would also share that we did certainly dive into the Google. 85 00:13:47.520 --> 00:13:55.500 Lisa Shen (she/her): spiral of going down to search the multiple search page results in terms of looking for instances of specificity library chat bots. 86 00:13:55.830 --> 00:14:04.590 Lisa Shen (she/her): When ultimately only identified six and, at the time of the research when we did our initial search actually earlier last year. 87 00:14:05.010 --> 00:14:08.400 Lisa Shen (she/her): We actually started with five and now, this is a list of six. 88 00:14:08.850 --> 00:14:18.600 Lisa Shen (she/her): And then off the six identify library chat bots five of them are currently in operation, and this is covering both academic and public like public library environment. 89 00:14:18.990 --> 00:14:26.730 Lisa Shen (she/her): And then at least have we can confirm that three of the Opera through the chat box are using open source technologies. 90 00:14:27.030 --> 00:14:36.240 Lisa Shen (she/her): And particularly program open dialogue flow flow, although with the also with the caveat that program oh has now reached its end of life it's an open source. 91 00:14:36.690 --> 00:14:41.880 Lisa Shen (she/her): that's no longer technology that's no longer being updated so that also means that the library that are. 92 00:14:42.510 --> 00:14:53.490 Lisa Shen (she/her): Using it would need to migrate to another program that would be compatible with it's like it's portable thankfully its content database with a question and answers. 93 00:14:54.270 --> 00:15:06.840 Lisa Shen (she/her): party and then so ultimately so we reached out to the five libraries, that we have identified at the time, and to ask them to kind of share share their experience and. 94 00:15:08.820 --> 00:15:25.410 Lisa Shen (she/her): with us and then ultimately to responded so a little bit later than, so we are going to talk about sort of what we learned from speaking with those libraries in through their chat Bot adoption and also kind of operating process there. 95 00:15:26.910 --> 00:15:32.580 Lisa Shen (she/her): That being said, though, before we go into this just a little bit more providing the picture internationally. 96 00:15:32.850 --> 00:15:44.250 Lisa Shen (she/her): Of of why we believe this is so, without this is the important topic is also that internationally chip, I really is quite prevalent in the not just commercially but also in the library sphere i'm using. 97 00:15:45.450 --> 00:16:00.810 Lisa Shen (she/her): South Korea as an example, and this is by no means a comprehensive list, but a similar quick search that we did with us chat bots on the South Korea inside came up with multiple examples, this is just a list of 16 libraries eight on the public library side a on the academic side. 98 00:16:02.400 --> 00:16:13.920 Lisa Shen (she/her): that currently have operating chat bots are also mentioned that what's also kind of interesting is that, while the US library chat bots that we have found all use web based programs. 99 00:16:14.550 --> 00:16:23.640 Lisa Shen (she/her): A number of the South Korean library chatbots actually use also adopted messenger based interfaces and, and this may have to do with just kind of. 100 00:16:24.180 --> 00:16:34.890 Lisa Shen (she/her): Geographic regional kind of technology preferences as well, because one of the more popular ones with the most popular ones being compelled talk and then that is also. 101 00:16:35.730 --> 00:16:52.200 Lisa Shen (she/her): Sorry, this is three and correct me if i'm wrong, but like this is also being used, also in quite popular just just within the country for other purposes as well, and also it's adopted by certain government agencies as the official one of the official communication channels so. 102 00:16:56.700 --> 00:17:14.820 Lisa Shen (she/her): And then also definitely third party ability of third party vendors is also quite prevalent and then in terms of third party rather than Open Source type of chatbot technologies and also mentioned that there's also definitely quite a bit of them. 103 00:17:16.110 --> 00:17:23.580 Lisa Shen (she/her): ended oliva quite a bit of increasing adoption now we're seeing in the higher education space outside of library generous well. 104 00:17:23.910 --> 00:17:34.740 Lisa Shen (she/her): Nationally many of the higher education academic units are adopting chatbots of at various service points so case in point, and maybe a mission career services financial aid. 105 00:17:35.910 --> 00:17:48.210 Lisa Shen (she/her): University it and stuff like that, and one of the example Actually, this is the puppy dogs blinking at us as tripod, this is actually my institutions. 106 00:17:49.440 --> 00:17:51.060 Lisa Shen (she/her): enrollment management. 107 00:17:53.250 --> 00:18:08.910 Lisa Shen (she/her): Division based chat box, so this is, this is a third party with the caveat that feel free to play with chat box feel free to play with tripod but he is a puppy right now so um options may be limited, but he can direct you to the library. 108 00:18:10.170 --> 00:18:18.570 Lisa Shen (she/her): And, but what's also interesting is that some of these out of out of box kind of commercial vendor third party vendor services. 109 00:18:19.320 --> 00:18:29.700 Lisa Shen (she/her): Is kind of fully supported and still need to do more research into it, but currently, this is actually independent of our university it, so it is the support is coming. 110 00:18:31.080 --> 00:18:36.210 Lisa Shen (she/her): Directly all the technology support is coming directly from from those service provider. 111 00:18:39.480 --> 00:18:47.940 Lisa Shen (she/her): Alright, so and for folks with more perhaps modest budget, though there is definitely also more open. 112 00:18:48.750 --> 00:18:55.050 Lisa Shen (she/her): Open Source or potentially low cost to free chatbot technologies that are commonly available. 113 00:18:55.500 --> 00:19:11.460 Lisa Shen (she/her): And we'll talk about or soon we'll talk about this a little bit more detail, but also with the caveat that sort of low finance initial financial investment doesn't necessarily equate to zero cost, because the cost will come in other formats as well. 114 00:19:21.570 --> 00:19:37.590 Hwang, Soo-yeon: Okay, so the big question in our study was how are the US libraries different in terms of adoption in comparison to other units in the US higher education institutions. 115 00:19:38.040 --> 00:19:56.850 Hwang, Soo-yeon: such as the admissions in order management constantly and the places it that the places that we saw earlier, and also in comparison to other libraries internationally, we saw lots more adoption examples previously, so why is it, I mean us is. 116 00:19:58.260 --> 00:20:12.480 Hwang, Soo-yeon: South Korea is very small country geographically speaking so us is a much larger example, but we don't see as many libraries that have put up the chat box, so why is that. 117 00:20:15.990 --> 00:20:20.310 Hwang, Soo-yeon: So, in terms of successful chapter adoption we uncovered. 118 00:20:21.510 --> 00:20:35.430 Hwang, Soo-yeon: two biggest factors, the first is closed, and the second is quality of intelligence so cost in terms of not only building and implementing but also running and maintaining the system. 119 00:20:36.360 --> 00:20:55.950 Hwang, Soo-yeon: And the quality of intelligence in terms of the algorithms of figuring out the questions and also the quality of insert data that we are providing to the patrons so basically how accurately the chat Bot answers customer questions. 120 00:21:00.690 --> 00:21:03.450 Hwang, Soo-yeon: So let's talk about the cost issues first. 121 00:21:04.560 --> 00:21:15.930 Hwang, Soo-yeon: So there are two routes versus in house development and the second is third party vendor route, when you. 122 00:21:16.980 --> 00:21:30.120 Hwang, Soo-yeon: Do the in house route we've seen earlier several examples of free and open source technologies, so the cost can be either free or very nominal. 123 00:21:30.930 --> 00:21:45.540 Hwang, Soo-yeon: The thing is human resource support that can be very costly so basically you need someone with the technical knowledge to set up run and maintain the system within the library. 124 00:21:48.720 --> 00:22:09.870 Hwang, Soo-yeon: One of the libraries that we interviewed a head on head up while they use an open source software and they were lucky to have a an in house programmer was able to set up the systems that have the databases and all the technical stuff and without that help it may not have been feasible. 125 00:22:11.250 --> 00:22:13.410 Hwang, Soo-yeon: So if you go to third party route. 126 00:22:15.630 --> 00:22:29.070 Hwang, Soo-yeon: Well, obviously there's ongoing costs for the service, because you have to subscribe to, you have to either pay for the initial installation set up and then ongoing maintenance costs. 127 00:22:31.200 --> 00:22:37.800 Hwang, Soo-yeon: But you don't need a programmer in house, so you have to think of the cost in that sense. 128 00:22:38.910 --> 00:22:48.840 Hwang, Soo-yeon: Although you still need a point person in the library to check the data quality and make sure the local installation runs smoothly, etc. 129 00:22:51.300 --> 00:23:01.350 Hwang, Soo-yeon: But the level of technical knowledge wouldn't be as high as you doing all the installation setup and upkeep yourself. 130 00:23:04.770 --> 00:23:12.030 Hwang, Soo-yeon: And the second challenge is the quality of intelligence building good data takes time. 131 00:23:13.290 --> 00:23:26.250 Hwang, Soo-yeon: more time than you had originally planned that was what our interviewees said it took one library and have full year to develop the database that when underneath the chat bar system. 132 00:23:30.240 --> 00:23:34.080 Hwang, Soo-yeon: And there is also some user interface limitations. 133 00:23:35.160 --> 00:23:52.440 Hwang, Soo-yeon: As we saw earlier in the DEMO the chat windows are especially the Web based well if if you do the smartphone messenger route to because the smartphone screen real estate, as long as those are small pop up window real estate. 134 00:23:53.280 --> 00:24:09.600 Hwang, Soo-yeon: You cannot be very long wordy answers in there, so you cannot pull your answers directly from faqs you have to write like separate smaller bite sized chunks of answers so. 135 00:24:10.680 --> 00:24:20.370 Hwang, Soo-yeon: you're basically building your database that's geared towards chatbots it's very hard to be used materials, as is. 136 00:24:21.810 --> 00:24:36.690 Hwang, Soo-yeon: Also, there are some algorithmic bias issues like in the DEMO before at first when I wrote are in the list the pre populated questions included like. 137 00:24:37.680 --> 00:24:51.990 Hwang, Soo-yeon: Job questions or originally hell, they may not be 100% relevant to an academic library environment, but they are the kind of questions that the developer. 138 00:24:52.530 --> 00:25:07.770 Hwang, Soo-yeon: In this case, the third party vendor developer provided like they thought well it's libraries, so people must be coming for like job search questions or Regiment questions they may be relevant for. 139 00:25:08.340 --> 00:25:21.300 Hwang, Soo-yeon: public libraries, but not as much so, for academic libraries So these are the kinds of our way to meet by small algorithmic buyers are still some buyers that are permeating into the education sector. 140 00:25:26.220 --> 00:25:42.720 Hwang, Soo-yeon: and some are interviews, we found some additional challenges to check out that option to the first is getting internal buy in at the library, basically, the question is, will the chat Bot replace my job. 141 00:25:44.070 --> 00:25:52.470 Hwang, Soo-yeon: So they had to go extra miles to ease people's minds and getting tunnel buy in at the library. 142 00:25:53.610 --> 00:26:00.630 Hwang, Soo-yeon: So, having a chat Bot enabled only after hours when the Librarians are not working well. 143 00:26:01.710 --> 00:26:12.720 Hwang, Soo-yeon: As well as running things like check out naming contest in the library, which created a sort of sense of ownership in the library. 144 00:26:15.420 --> 00:26:27.600 Hwang, Soo-yeon: And some additional challenges included expanding its search capability so, for instance in the DEMO before you so when I asked about a book. 145 00:26:28.260 --> 00:26:39.780 Hwang, Soo-yeon: On known item search he gave me a link to the catalog instead of sending or patron to another webpage to do the search again. 146 00:26:40.530 --> 00:27:01.770 Hwang, Soo-yeon: Actually, if you were to do the same transaction with a person that person will probably do a catalog search in the back end, you know behind the chat and gift a patron answer of either yes or no, yes, we have double or no, we don't have the book. 147 00:27:02.820 --> 00:27:17.040 Hwang, Soo-yeon: So same thing could be possible with chat bots if you connect the chat Bot to like library discovery service API or. 148 00:27:18.150 --> 00:27:30.570 Hwang, Soo-yeon: catalog API volume the catalog has an API or institutional repositories and so on, so the API will go out and do the search and return the answer. 149 00:27:30.960 --> 00:27:44.250 Hwang, Soo-yeon: And it will be out put it into the chat bar, so this is not possible, but something that our interviews we're hoping to be able to do in the future. 150 00:27:50.220 --> 00:27:54.600 Hwang, Soo-yeon: And they'll write some assessment issues too so basically. 151 00:27:56.190 --> 00:28:00.600 Hwang, Soo-yeon: They want to prove the value of chat bots to their libraries right. 152 00:28:01.740 --> 00:28:08.520 Hwang, Soo-yeon: And in order to do that, they want some better analytics well, they have some. 153 00:28:09.840 --> 00:28:17.430 Hwang, Soo-yeon: Quantitative metrics such as well the chat logs are not exactly quantitative but the chat logs that are there. 154 00:28:18.000 --> 00:28:30.330 Hwang, Soo-yeon: And in addition to that they have some more measures like the total questions asked and answered the day in time so basically when it's being used more heavily. 155 00:28:31.020 --> 00:28:50.820 Hwang, Soo-yeon: And how many people went beyond the initial welcome screen APP actually asked questions and what are the top queries and so on, but it's very hard to measure user satisfaction there's no easy user friendly and less costly solution at the moment. 156 00:28:51.870 --> 00:29:03.060 Hwang, Soo-yeon: And without being able to ask dangers how satisfied, are you with the chat bots it's very hard to gauge whether the chat bodies are successful or not. 157 00:29:04.110 --> 00:29:13.050 Hwang, Soo-yeon: Also, the interviewees expressed interest in learning what links people actually click because. 158 00:29:14.220 --> 00:29:35.340 Hwang, Soo-yeon: As, as we said earlier, without the help of api's and so on you're bound to give out lots of links back as an answer and there's currently no metric in terms of whether those links are being clicked so basically whether those patrons are going the extra mile to find the answers themselves. 159 00:29:37.200 --> 00:29:51.840 Hwang, Soo-yeon: And Amazon offered some more assessment ideas such as whether the chat box was able to complete a transaction without being referred to a live person. 160 00:29:54.060 --> 00:30:17.010 Hwang, Soo-yeon: And whether the patron the transfer assistant within 24 hours While this may or may not be useful in the library context, but the the gist of this question is whether the patron was satisfied with the chat Bot answers, so that they don't have to return within a day. 161 00:30:22.410 --> 00:30:30.900 Lisa Shen (she/her): Alright, so kind of a quick recap here just overall based on kind of our in speaking with the chat Bot adopting or. 162 00:30:31.440 --> 00:30:45.840 Lisa Shen (she/her): Managing Librarians and also in playing kind of with the available chatbots that we have identified out there, right now, are sort of tend to completion, is really bad chat Bot us in the US, for us, libraries right now chat Bot is really. 163 00:30:47.370 --> 00:30:57.180 Lisa Shen (she/her): more of a tool to enhance or complement existing reference services, but not necessarily to replace just yet. 164 00:30:58.290 --> 00:31:06.150 Lisa Shen (she/her): That being said, it does provide an added value, providing that immediate engagement that users or patrons are definitely growing to expect. 165 00:31:08.040 --> 00:31:16.440 Lisa Shen (she/her): And then, it is also good for answering with you know intentional design, it would be good for answering basic routine simple questions. 166 00:31:18.750 --> 00:31:28.560 Lisa Shen (she/her): Even without some of the some of the more advanced technologies, especially if we're while we're coming is also you will be able to kind of defer recommend to. 167 00:31:29.100 --> 00:31:38.580 Lisa Shen (she/her): Two additional resources and also to the actual reference services when more complex conversation, or perhaps or are needed and, that being said. 168 00:31:39.150 --> 00:31:46.650 Lisa Shen (she/her): For folks that may be looking to provide more 24 seven kind of around the clock in depth comprehensive reference or resource consultations. 169 00:31:47.010 --> 00:31:55.140 Lisa Shen (she/her): Probably the currently the opportunity to or the resources for doing that would be more consortium based reference monitoring still. 170 00:31:55.590 --> 00:32:09.150 Lisa Shen (she/her): And, and we are you spring share a global co op so one of the options, but as a sort of quickly segue to or a quick add on a random is that screen share actually has also announced. 171 00:32:10.620 --> 00:32:22.950 Lisa Shen (she/her): That it is going to be launching a chat chat Bot in fall of this year fall 2022 and, that being said, sort of currently based on the available information we have, it sounds like it would be. 172 00:32:23.820 --> 00:32:36.960 Lisa Shen (she/her): The chat Bot would be more rule based kind of like the ones that were done we had download and we're seeing right now, rather than an Ai based kind of sanity and chat button not quite there yet, and then the chat Bot kind of. 173 00:32:37.530 --> 00:32:46.200 Lisa Shen (she/her): Content bank would be kind of based off probably a shortened version of the live answers faq module. 174 00:32:47.220 --> 00:33:07.710 Lisa Shen (she/her): kind of the building questions and answers so more likely than not for existing spring share library customers, this will likely be kind of a nominal cost or economical option to, or perhaps to be able to test out that test out a chatbot kind of service, that being said. 175 00:33:09.210 --> 00:33:12.510 Lisa Shen (she/her): It would it's really meant for at least currently for. 176 00:33:12.990 --> 00:33:25.320 Lisa Shen (she/her): Their kind of their routine basically addressing the questions and not for advancing directions and also more likely based on our interviewees experiences so far that building out or perfecting that. 177 00:33:26.130 --> 00:33:38.520 Lisa Shen (she/her): That content bank would likely take some time and i'm also using my own library as an example, we are currently a screen share customer bar our faqs are really novels sometimes. 178 00:33:38.790 --> 00:33:48.330 Lisa Shen (she/her): runs into as for Librarians that we do provide really long answers to kind of hoping that ensuring that patron would have answers to like addressing all the questions they might have. 179 00:33:48.630 --> 00:34:05.430 Lisa Shen (she/her): But in the chat Bot kind of context, these type of answers would not would definitely not be optimal so they will that will essentially basically if we were to offering, then that would require kind of extensive editing and kind of rethinking about the type of questions and responses. 180 00:34:05.580 --> 00:34:06.360 That will be needed. 181 00:34:08.400 --> 00:34:20.100 Lisa Shen (she/her): and also all That being said, though, we also kind of discovered a surprise, which was noted also about both of our interviewees is more the likelihood for library patrons who enter multiple size. 182 00:34:21.480 --> 00:34:26.640 Lisa Shen (she/her): The chat bots they have even knowing that it's after knowing that his computer program. 183 00:34:26.970 --> 00:34:43.170 Lisa Shen (she/her): So, so these type of social chatters or mentioned definitely in the earlier chatbot literature, but what's also interesting is that even nowadays like currently that's still seems to be happening, maybe also kind of association with like library chat Bot, in particular. 184 00:34:44.370 --> 00:34:46.140 Lisa Shen (she/her): anecdotally i've never had the. 185 00:34:47.400 --> 00:34:59.940 Lisa Shen (she/her): That the sort of the inkling to you know social chat with my Internet providers chatbot, for example, however, both interviews are observing that these type of social chatters, what do you eat. 186 00:35:01.560 --> 00:35:11.040 Lisa Shen (she/her): yeah that kind of stuff those still come up with with students so, so this is not necessarily a finding that will help with answering sort of. 187 00:35:11.370 --> 00:35:19.020 Lisa Shen (she/her): Academic research questions per se, but perhaps chatbot do provide a unique or a different way for in perhaps cheap way of. 188 00:35:19.500 --> 00:35:37.260 Lisa Shen (she/her): providing additional ways to engage students and also remarketing library services or just for branding in general as how to to get patient buying into your university communities buying into the service or then subsequently into the library as well. 189 00:35:39.090 --> 00:35:54.840 Lisa Shen (she/her): But that also means, on the other side is a quick note to kind of remember or for folks that will be involved in building chatbot content banks to be mindful of where to consider developing responses to these type of personal, social questions as well for the chat Bot. 190 00:35:55.170 --> 00:35:56.760 Lisa Shen (she/her): Potentially this so you're, the better. 191 00:35:58.170 --> 00:35:58.770 Lisa Shen (she/her): And then. 192 00:35:59.790 --> 00:36:06.420 Lisa Shen (she/her): And, forgive me for a little bit i'll put the link this is kind of for your own bring pleasure and stuff. 193 00:36:07.020 --> 00:36:11.040 Lisa Shen (she/her): There also seems to be an unnecessary library space you just given to library. 194 00:36:11.430 --> 00:36:24.510 Lisa Shen (she/her): Given a chat Bot adoption rate but we're also seeing some this is fairly recent a different type of anthropomorphism in the sense of i'm building avatars for chat bots that are happening higher education space i'm. 195 00:36:25.140 --> 00:36:41.010 Lisa Shen (she/her): Just something to keep note of that being said, the example that i'm sharing here is the University of Nevada Las Vegas you know these President actually introduced a chatbot that is based on the President, the university Presidents likeness. 196 00:36:42.720 --> 00:36:48.060 Lisa Shen (she/her): I have some personal mixed feelings about that part of it is also the sustainability of it just. 197 00:36:49.140 --> 00:36:53.100 Lisa Shen (she/her): For practically speaking the likely. 198 00:36:54.660 --> 00:37:06.480 Lisa Shen (she/her): To retain your for university high university administrative officers is not really it's not really like there's a limited term kind of expected it's not a lifelong position, which means that if it's. 199 00:37:07.110 --> 00:37:17.310 Lisa Shen (she/her): sort of the effort and energy built into perfecting a more lifelike Avatar that may be better spent elsewhere, and also for university branding as. 200 00:37:17.790 --> 00:37:23.010 Lisa Shen (she/her): personality change, then, then the identity of the Bot, is going to change and also. 201 00:37:23.970 --> 00:37:37.380 Lisa Shen (she/her): This is kind of in connection of if students are are we seeing patrons interested in social chatting with a Bot, it may be pro or con whether or not they would feel the same about chatting with a high university administrator. 202 00:37:38.730 --> 00:37:41.670 Lisa Shen (she/her): The same way, maybe something to consider. 203 00:37:43.890 --> 00:37:55.410 Lisa Shen (she/her): But all That being said, though, the technology what was kind of also unique and particularly surprising to me as a non tech non sort of web IP information technology kind of IP person. 204 00:37:55.740 --> 00:38:12.390 Lisa Shen (she/her): is really bad it just seemed like technologies are getting easier easier to implement so while certainly to support of programming and it expert is necessary, it may not need to have a full time professional to necessarily need to. 205 00:38:13.440 --> 00:38:25.740 Lisa Shen (she/her): to manage the technology side, regardless of whichever chat box technology library intend to adopt that's, not to say there's not a lot of work that still needs to go into it in terms of maintaining to. 206 00:38:27.810 --> 00:38:39.540 Lisa Shen (she/her): Maintaining the login monitoring kind of the transcript to to improve or revise the content bank and stuff like that, and certainly do assessment questions still certainly lingers so. 207 00:38:40.380 --> 00:38:50.310 Lisa Shen (she/her): In summary, chat bots are not quite ready for to replace for reference functions, yet, but perhaps within reason, and if the opportunity arises, they do. 208 00:38:51.900 --> 00:38:57.840 Lisa Shen (she/her): They do provide come, they would be a sort of a helpful positive complimentary tool to human reference services. 209 00:39:05.910 --> 00:39:06.420 Lisa Shen (she/her): and 210 00:39:10.890 --> 00:39:11.820 Lisa Shen (she/her): here's the end of our. 211 00:39:12.930 --> 00:39:16.380 LuAnn Fisher: Thank you very much, it was lovely I have a question. 212 00:39:17.880 --> 00:39:23.910 LuAnn Fisher: Now my school has a chat box, but the Jews, I think, with reggie registration. 213 00:39:24.960 --> 00:39:31.920 LuAnn Fisher: Do you know of schools that use a chat Bot across their various functions, including libraries. 214 00:39:33.240 --> 00:39:42.630 LuAnn Fisher: And what's your what's your thoughts on that with with having the library kind of get in there on a chatbot would that be something that would be worthwhile. 215 00:39:45.210 --> 00:39:52.050 Hwang, Soo-yeon: Well, let me, so I think that the Christian ease well so. 216 00:39:53.490 --> 00:40:17.010 Hwang, Soo-yeon: yeah the question becomes how much how much of the questions we get in the library our routine questions versus specialized research questions because chatbots as we found what we found out was the templates are good for answering small people routine questions. 217 00:40:18.210 --> 00:40:35.580 Hwang, Soo-yeon: And maybe, like other units in higher education may have more types of those questions than the other in depth personalized questions but libraries tend to get lots of research questions. 218 00:40:36.180 --> 00:40:49.800 Hwang, Soo-yeon: which may or may not be feasible, with the crunch about technology, so I think that there is one of the reasons why it hasn't been adopted that much in the US, especially. 219 00:40:50.370 --> 00:41:00.870 Hwang, Soo-yeon: But the interesting thing is that internationally it's being adopted a lot, so I wonder if it's the technology issue or. 220 00:41:02.010 --> 00:41:03.030 Hwang, Soo-yeon: What is the shipping. 221 00:41:05.280 --> 00:41:06.510 Hwang, Soo-yeon: So do you have any idea. 222 00:41:06.840 --> 00:41:12.210 Lisa Shen (she/her): So not quite, but I think think that that's a really good question i'm actually just kind of thinking. 223 00:41:12.570 --> 00:41:24.630 Lisa Shen (she/her): about it a little bit for my library so folks, and this is kind of my personal opinion, because now quite there's no quite enough, we haven't found enough to really back it up, I can't say like evidence base, this is what would but um so. 224 00:41:25.650 --> 00:41:35.370 Lisa Shen (she/her): It will be interesting to see the control part of because, having to share the burden or the cost or have someone else that can support a technology Ai interface, I definitely would be helpful. 225 00:41:36.480 --> 00:41:39.480 Lisa Shen (she/her): But then the other, the other part of me, then, but then the. 226 00:41:39.930 --> 00:41:50.850 Lisa Shen (she/her): sort of the management of the actual content and I may be thinking about this in a very simplified way but, like the content database for the actual answers and questions and being able to. 227 00:41:51.750 --> 00:42:07.470 Lisa Shen (she/her): sort of build that maintain maintain develop and revise that part of the cost or as long as we can have access to that because what's not clear to me with some of the third party pieces of how much we can control or not i'm with my save my libraries quran. 228 00:42:10.140 --> 00:42:18.810 Lisa Shen (she/her): But that my library my institutions current chat Bot, because it seems to be building, but like so that may be part of is just how much of it but yeah. 229 00:42:19.350 --> 00:42:21.750 LuAnn Fisher: If your institution had a chat Bot. 230 00:42:22.860 --> 00:42:29.910 LuAnn Fisher: Would you want there to be library questions available to it and a presence of the chat Bot, on your library website. 231 00:42:33.330 --> 00:42:38.100 Lisa Shen (she/her): I would, I think it depends on the level of involvement, because right now, where I think Goldman have. 232 00:42:38.940 --> 00:42:41.910 Lisa Shen (she/her): Hopefully, their keywords identified, and it will be users will be. 233 00:42:42.450 --> 00:42:53.250 Lisa Shen (she/her): redirected to the library website, but it's more do we provide additional content to what extent I would feel like if we're going to start providing extensive content and we definitely want to feature it on the library website. 234 00:42:53.580 --> 00:42:54.240 LuAnn Fisher: Yes, I mean. 235 00:42:54.300 --> 00:42:56.370 LuAnn Fisher: directional questions or you know. 236 00:42:56.400 --> 00:43:05.070 LuAnn Fisher: Hours via that would you start, I mean if that's where you'd need to start, because of course in depth reference really isn't something you can do there but but. 237 00:43:06.360 --> 00:43:16.770 LuAnn Fisher: If your institution had one would you think it would be worthwhile, even for those small questions or there's simple questions to have the presence on your website. 238 00:43:17.640 --> 00:43:19.380 LuAnn Fisher: So on the library website. 239 00:43:23.670 --> 00:43:25.200 LuAnn Fisher: If somebody else's managing it. 240 00:43:25.800 --> 00:43:26.220 Lisa Shen (she/her): yeah. 241 00:43:26.280 --> 00:43:26.700 LuAnn Fisher: And I think. 242 00:43:27.900 --> 00:43:32.790 Lisa Shen (she/her): It would just be maybe it's its ability to refer out or refer back because. 243 00:43:33.630 --> 00:43:44.910 Lisa Shen (she/her): I would like to like right now owning if it's exist elsewhere, I would like it to be able to point to the library website, but if a picture on the library website, and I would like it to be able to do a little bit more beyond that, because even for after hours. 244 00:43:45.510 --> 00:43:57.030 Lisa Shen (she/her): boy, to avoid user frustration, because if i'm seeing it on the on the library website, I may have slightly higher expectation not, not that I can answer all the questions but Richard you a little bit more. 245 00:43:57.450 --> 00:44:00.090 LuAnn Fisher: that's really great point the higher expectation. 246 00:44:01.290 --> 00:44:01.980 LuAnn Fisher: to think about. 247 00:44:02.760 --> 00:44:08.100 Lisa Shen (she/her): I will mention, though, one of the one of the interviewee or not looking to read, one of the. 248 00:44:08.820 --> 00:44:18.960 Lisa Shen (she/her): The folks the current US library tab and we had identified actually that particular library, we were unfortunately not able to talk to them, but. 249 00:44:19.560 --> 00:44:27.090 Lisa Shen (she/her): That particular there was a library chat Bot, that was actually the offset of the organization or chat bots so so we were able to kind of discern like the. 250 00:44:27.330 --> 00:44:38.160 Lisa Shen (she/her): sort of the whole for the whole organization at that a general chat box lunch first and then later on, I think that there is a sense of utility but a more intentional specifically library Brent is. 251 00:44:38.730 --> 00:44:46.680 Lisa Shen (she/her): sort of I guess chat Bot child was or cousin family member was introduced so that's a standalone so. 252 00:44:47.010 --> 00:45:00.120 Lisa Shen (she/her): i'm granted it didn't seem like there's sort of API linking into the actual catalog and stuff like that, but, but the branding is definitely there and that may be down the road where those two chat bots would have different more different kind of can answer different questions. 253 00:45:00.390 --> 00:45:01.860 LuAnn Fisher: that's great Thank you. 254 00:45:05.430 --> 00:45:13.410 Wodarczyk, Jennifer: Have a question, is there a way to have like when a chat Bot, can you have staff that are kind of in the background and then seeing that maybe. 255 00:45:13.860 --> 00:45:24.660 Wodarczyk, Jennifer: Knowing that it's a that is question that a chat Bot not it just doesn't have the programming in it to answer at the time that could then jump in is that an option that's that you've seen out there. 256 00:45:27.810 --> 00:45:34.200 Hwang, Soo-yeon: there'll be a good option to have I don't know whether it's feasible or not, for instance. 257 00:45:34.740 --> 00:45:54.030 Hwang, Soo-yeon: One of the benefits of having a chat Bot, is that he can handle multiple questions simultaneously and do you want to be able to monitor all those, like several instances of chat bots all at the same time there'll be more than a person can handle. 258 00:45:56.760 --> 00:46:07.020 Wodarczyk, Jennifer: I can see that, and I do like the idea Lisa had for the different like you know, maybe like a chat Bot pens or to another chat buttons like in a foot in the chat box like I can't answer it, let me send you to a real person. 259 00:46:08.220 --> 00:46:13.230 Wodarczyk, Jennifer: But that's that I love the ideas are spinning, so I love this anybody else have any questions. 260 00:46:23.640 --> 00:46:28.830 Wodarczyk, Jennifer: Or would like to add questions to the chat if you have any questions if you don't feel comfortable. 261 00:46:43.560 --> 00:46:47.190 Wodarczyk, Jennifer: Anything you both would love to add that you know. 262 00:46:48.660 --> 00:46:51.600 Wodarczyk, Jennifer: As thoughts have been coming to mind or anything. 263 00:46:53.100 --> 00:46:55.320 Wodarczyk, Jennifer: That may spark some more questions. 264 00:46:58.500 --> 00:47:03.120 Lisa Shen (she/her): i'm good for now, and this is not sponsored or, this is not a spring share advertisement. 265 00:47:07.170 --> 00:47:15.360 Lisa Shen (she/her): For for folks that if you are your institution is already a screen share customer and you are using the live answers module and have chat and stuff already. 266 00:47:16.140 --> 00:47:25.380 Lisa Shen (she/her): My understanding is that there is a beta group that will be happening with the for the chat box, so if you're interested and by default of being at. 267 00:47:26.130 --> 00:47:38.550 Lisa Shen (she/her): The session grabs you will be interested that may be one quick one way too easy way to kind of test one out and see if if that it may be something useful or not. 268 00:47:40.110 --> 00:47:45.750 Lisa Shen (she/her): yeah I, like my institution, we are going to be participating and i'm kind of curious. 269 00:47:46.890 --> 00:47:54.690 Lisa Shen (she/her): But yeah but their first challenge would just be like I don't think we realistically we don't have much sort of ready content that can be plugged into. 270 00:47:55.650 --> 00:48:06.510 Lisa Shen (she/her): into into the database like it will have to be more or less kind of built from if now scratch from like definitely with a lot of modification editing well. 271 00:48:08.730 --> 00:48:17.850 Hwang, Soo-yeon: yeah there's pressure system is using the signature faq database as the answer database and. 272 00:48:19.050 --> 00:48:21.570 Hwang, Soo-yeon: According to what the representative says. 273 00:48:24.060 --> 00:48:27.990 Hwang, Soo-yeon: You have to write the short version of the answers for the chat Bot. 274 00:48:29.610 --> 00:48:42.540 Hwang, Soo-yeon: which will it seems like the chat Bot, will give you the short version of the answers that you write into the database and then with the link to the full version of the faq or something like that. 275 00:48:44.550 --> 00:48:59.160 Hwang, Soo-yeon: I think it's interesting the exposure is going for the again the rule based conversation route because the artificial intelligence technology development today is such that. 276 00:49:00.180 --> 00:49:11.040 Hwang, Soo-yeon: It can well, there are some type of systems, not necessarily institutional chatbots but some like general purpose chat bots they will learn from. 277 00:49:12.270 --> 00:49:33.300 Hwang, Soo-yeon: The transactions and add to their database more and more, depending on what kinds of answers and questions is receiving and giving how the thing is research has shown that many of those chatbots tend to. 278 00:49:34.380 --> 00:49:39.840 Hwang, Soo-yeon: lean towards like these two peon tendency. 279 00:49:40.860 --> 00:49:41.610 Hwang, Soo-yeon: So. 280 00:49:42.750 --> 00:49:45.810 Hwang, Soo-yeon: Like it's learning that language is basically. 281 00:49:49.170 --> 00:49:50.520 Hwang, Soo-yeon: So the question is. 282 00:49:51.600 --> 00:50:00.360 Hwang, Soo-yeon: Do we won things like that, for the library or the institution, because we have certain image to uphold. 283 00:50:02.070 --> 00:50:14.310 Hwang, Soo-yeon: And we don't want chatbots to be spewing out that languages to the patrons so those artificial intelligence more like technologically advanced. 284 00:50:15.720 --> 00:50:26.610 Hwang, Soo-yeon: chatbots may be able to give more like natural language answers, but they may not be adequate for the purpose of. 285 00:50:27.930 --> 00:50:29.670 Hwang, Soo-yeon: Like institutional goals. 286 00:50:32.400 --> 00:50:49.830 Hwang, Soo-yeon: So when I when I first heard that spring share is adopting a checkbox neurology I was wondering whether they'll be one for the more a round or more will base route and tell us that they're going for the war room base round. 287 00:51:01.890 --> 00:51:03.750 Wodarczyk, Jennifer: Two This is all very intriguing. 288 00:51:05.220 --> 00:51:06.480 Wodarczyk, Jennifer: This is all very intriguing. 289 00:51:10.350 --> 00:51:11.790 LuAnn Fisher: It is it's very interesting. 290 00:51:14.160 --> 00:51:15.780 LuAnn Fisher: Well, thank you very much, ladies. 291 00:51:17.010 --> 00:51:25.050 Lisa Shen (she/her): problem, thank you for hosting this and yeah I think we have like 10 minutes to spare for the next until the next session so should this be a. 292 00:51:26.460 --> 00:51:27.510 Lisa Shen (she/her): Great see you soon. 293 00:51:29.010 --> 00:51:47.610 Wodarczyk, Jennifer: i'm going to add some links to the chat if anyone is interested for the next sessions, and also a pamphlet if you want to maybe continue the question you know, maybe add some questions there or just say hi and talk about maybe where you you're and you're logging into view from so yeah. 294 00:51:51.000 --> 00:51:57.330 Wodarczyk, Jennifer: Well, thank you Lisa and young, I appreciate it Thank you lewin for co hosting in a way, early. 295 00:51:57.450 --> 00:51:58.140 LuAnn Fisher: You know, for. 296 00:51:58.380 --> 00:51:59.910 Wodarczyk, Jennifer: Presentation thank fart sorry. 297 00:52:01.350 --> 00:52:01.920 Wodarczyk, Jennifer: I am. 298 00:52:03.000 --> 00:52:08.670 Wodarczyk, Jennifer: thankful that everybody came to town, thank you for coming so i'll add the pause the recording.