ChatGPT takes Steven Schwartz to Court

Mark Miller: 

There was a historical case presented in federal court in New York City last week. Lawyer Steven Schwartz was hauled into court for using unresearched, unsubstantiated output from ChatGPT to defend a client. For those of us following the case, it’s a chance to get a firsthand look at a judge who will be setting precedents around the country, if not the world, when it comes to using AI output in a courtroom. 

Joel MacMull has presented before Judge Castel, who is presiding over the case, a few times. So he and I planned on meeting in the courtroom that day. Yeah, life got in the way. Instead of few days after the court date we got on a call with our friend and technologist, Shannon Leitz, to discuss the legal aspects of the case, as well as the future of using this type of technology to do research. 

How can we trust output from an AI engine? When is it appropriate to use an AI engine to do professional research? Should Schwartz have known better or do we believe him when he says he had no idea that ChatGPT could just make things up? 

Should we have sympathy, empathy, or disdain for what he did. Joel, Shannon, and I will give you our opinions and insights, but then it’s up to you to decide. What are the legitimate uses of AI in its current state? Is it an experiment? Is it really changing the course of history? Does it have a legitimate use on the open unfiltered market? 

Like I said, you decide. But get the facts first. That’s what we’re about to do. Stay with us. 

You have pleaded before this judge, right, Joel? 

Joel MacMull: 

I have appeared before this judge. He’s a George W. Bush appointee, is a no nonsense guy. And not surprisingly is one of these judges that often when they take the bench has really made up his mind before he even hears counsel open their mouths.

 From what I’m reading about this case, again, I don’t have a transcript and I didn’t have the benefit of being there, but you can appreciate all the legal rags had more than something to say about these proceedings.

He came to the bench mindful that this is a seminal case in many respects, and that he has an opportunity to really strike a precedent in terms of what’s going to be acceptable, not only in his court, but really for lawyers across the land.

Mark Miller: 

Where was the interest level for you as a technologist? 

Shannon Lietz: 

It was really interesting to me, both as technologist and security professional, to see someone who’s got a major license to practice law, leverage technology that they were unaware of. 

Mark Miller: 

I don’t believe him. How could you not be aware? 

Shannon Lietz: 

How could you not be aware of the hallucinations? Yeah. Look, I’ll tell you, I’ve been studying ChatGPT for months now, and probably prior when I was doing ML, some of the work behind it…. ChatGPT is really good at explaining things in a way that makes you believe it. 

What’s really challenging about this technology from my perspective is that it’s super believable and it’s super believable in a lot of ways.  There’s ChatGPT created articles that are out there that I’m sure you’re reading at this point. Those articles are extremely believable and could be factually incorrect, but who’s actually fact checking them. Unless you’re actually fact checking, you might actually have a problem. 

Mark Miller: 

Joel, obviously you are a lawyer, that’s what we’ve been premising this whole show on for three months. You and I have talked about the use of ChatGPT like this. Were you surprised at what happened? 

Joel MacMull: 

Am I surprised that a lawyer has relied on the tool? No. Am I surprised that the lawyer has seemingly relied exclusively on the tool and for the reasons , not the least of which is, we are licensed professionals and that our rules in federal courts specifically provide that every document we sign and file, we are essentially attesting to separately. That each separate filing we’re saying in essence, we’ve conducted our due diligence. We believe that which we’ve put before the court to be not only truthful, but to the extent that it doesn’t necessarily comport with the law as drafted. It nevertheless must be a good faith basis that the law should be extended for some reason because of the particular facts and circumstances.

That is something not only that I take, and then every lawyer that I know takes very seriously, but in reading what has gone on here it’s not even close to what should have been satisfied by Schwartz and LoDuca. That’s what surprises me. 

 Now I’ll go one step further and tell you why it doesn’t matter. Because there are many areas of the law where willfulness is not a factor in the equation. They are rightfully throwing themselves at the court’s mercy, as they absolutely must. But what I suspect you’re going to see in the judge’s written opinion, which, and I have not checked today, but I believe I did yesterday, has not issued yet, is this issue of intent being put to the side and the judge saying, invariably under Rule 11, which is the rule that we as lawyers must comport with when we make a filing or the broader judicial powers that are afforded to every federal judge under a federal statute, is you’re going to see the judge put that to the side and say, intent is not the issue here. Intent is not the issue. 

Mark Miller: 

Shannon, your area of expertise, this idea of trust, who do you trust and why? And that’s the foundation of what we’re discussing 

Shannon Lietz: 

That’s what I’m really getting at is what can we trust ChatGPT for? Do you have to have a prompt engineering license to leverage it to truly understand what it’s going to produce?

But more importantly, what I’m really pushing on is if we fed it case law, I could imagine what it could generate, which might be jumps in logic around how you might best use that case law, but why on earth would it generate and create case law that’s fictional that then is being leveraged.

I’m in that weird 50/50 zone, I’m right split in the middle of Steven Schwartz and on his side I’ve leveraged the technology. I was deeply concerned when I saw it generate URLs, fake URLs, but not fully fake. That’s the most interesting part. It got some part of the URL and then a whole bunch of it wrong. That was the most interesting piece to me, is that it’s actually getting some things right and some things wrong.

 Is it really explaining back what it heard from you? Is the technology going far enough for you to trust it? Is it giving you a sense of how to really embrace it? Because we’re talking about intent, poor Steven Schwartz. I think he intended well, and I am interested in how this turns out from that perspective of intent, because I do think he was really trying to do right by his clients. When we think about what he was trying to do is he was trying to provide them with the best information in the legal briefing he was putting together. 

Mark Miller: 

I would argue not. 

Joel MacMull: 

I gotta disagree with you there. 

Shannon Lietz: 

Okay, great. But something here has to be argued right, about why he used the technology. Was it to save him time? It could be that, yes. and everybody out there is talking about productivity. 

My curiosity is, when it gets good enough, could it actually provide better than human connectivity and knowledge? I still go back to why are we allowing it to generate case law? That should have been like a no-no by the ChatGPT folks. Why is it creating information that’s fully fictional. All of a sudden, the prompts I was using before no longer work. It’s actually now backing up and saying, Hey, we don’t do that. I don’t do that. I don’t have access to the internet. 

But, not only a couple of weeks ago was it actually spitting out a whole bunch of profuse fictional information. If I had not checked it, if I actually trusted it, I probably wouldn’t have known that so quickly. But I actually took the time to click on the URL and follow it somewhere and find out that there’s a whole bunch of 404s. 404s in the technology industry means you’re going nowhere and it didn’t exist in the first place.

I went one step further to say, okay, maybe they don’t actually keep this kind of thing online for too long. So I asked it, can you create recent information? It did a whole bunch of that too. I was like, wow, I’m really bad at Googling, I think cuz I didn’t find any of this. 

And then on top of it, I went even further and said, all right, let me go figure out maybe it’s in the archives, go to archive.org and see if Hey, it’s just not online any longer. People are actually doing a great job of getting rid of bad articles that have been around for too long. 

But the truth is that it actually was generating to real periodicals, fictional parts of the url, and I’m wondering now, are those periodicals actually seeing a whole lot of 404 spikes since ChatGPT? That’d be very interesting to know. 

Joel MacMull: 

The other industry that I think is also at risk for the misuse of ChatGPT, or at least with licensed professionals, are accountants. The reason why is, accountants all day deal with what does the IRS have to say about A, B, and C. 

 I have every expectation that something analogous to this is going on in the accounting industry. Then in the not too distant future, it’s actually going to find itself in a court because some bookkeeper somewhere gave their client bad advice, they got audited by the IRS, and the next thing you know, there’s going to be the functional equivalent of a malpractice claim coming back to that accounting practitioner because they relied on misinformation from ChatGPT. 

Mark Miller: 

Shannon, with the background that you have, how likely is that scenario? Will AI be built into accounting software anytime soon? 

Shannon Lietz: 

Oh, it’s happening already. we’re seeing this technology now.

It may not be ChatGPT specifically could be some other generative AI technology ML out there. it’s really highlighted this need for us to maybe not be moving quite as fast, but actually starting to prove things out. I’ve noticed that in the last couple weeks we’re now seeing big companies start to create some sort of learning account so that their technologists can learn about generative AI before they start to leverage it. That, I think, is a wonderful way to move forward with this technology. 

Why are we, seeing something come out, signing up for it and immediately leveraging it for licensed professionals is why I thought this was a really interesting article in the first place.

 Like I said, I feel like. Steven Schwartz is just the beginning of the things that we’re going to see in the legal atmosphere. 

 One other thing we need to be concerned about, not just accounting and the possibility of it being these generative AI technologies being included in accounting, but medical. That’s the one why I think is super scary, is all of a sudden what happens if this technology creates some sort of treatment plan that’s fictional for a disease. That could happen. There’s a lot of things that if you really step back from this and understand how it works, truly that I don’t fully believe everyone who’s using it understands. I am seeing so many articles about ChatGPT, about OpenAI, about including the stuff into software very quickly. That is my core expertise. 

Why are y’all including this in so fast and being so hot, heavy to use it? It’s because there’s a lure. People really want to be close to this technology. It’s probably the first time in the industry where I’ve seen this many people running after something so quickly and getting as many hits off of it and seeing revenue aspects associated with it. ‘

But there’s a lot to be learned.  The learning atmospheres that are being created, that’s fantastic. Thanks to lots of companies out there. Walmart was one of the recent ones who actually created an interesting way for their technologists to be able to learn about generative AI. I think we should see more of that sooner.

I do believe that if a company is going to include it in their technology, that they should understand fully what they’re actually doing with this as a technology add-on. 

 It should be that when somebody signs up for using it, they should know that hallucinations is a possibility and especially for licensed practitioners, that they should think twice.

Joel MacMull: 

First of all, you’re absolutely right. The speed with which this has taken hold, we’re talking about a software application that was released last November. We’re not even, what are we, six months into this? And it is literally a game changer overnight.

I think you’re absolutely right when you say that Steven Schwartz is not the beginning. And in fact, if you read some of the fallout from the hearing before Judge Castel, he says exactly that. He says, this is not the end of the narrative. This is the beginning.

Which is why I think he’s deliberating in writing a written opinion, because he knows he is fundamentally, at least for lawyers, setting a precedent in terms of what’s going to be acceptable, effectively in every courthouse in this country. At least from us as licensed practitioners. Yeah, I agree with you.

The healthcare point you made is an interesting one. I suppose somewhat naively. I was thinking about if ChatGPT creates a treatment plan, as you said, I was thinking, no doctor would rely on that. But I guess I’m being naive. I acknowledge that in thinking about it for 10 seconds and also, the danger becomes not so much, even if it’s doctor authorized, but if you remember when WebMD came out, we were all giving ourselves self diagnoses. 

 I think there’s real dangers there. Correct me if I’m wrong, when you open your account at least does it not bombard you with these sorts of disclaimers. 

Shannon Lietz: 

It bombards you with lots of. Information. Even when you generate, a chat with it, it’ll come back and actually do some work to review and disclose differently. 

It’ll tell you, I don’t have access to the internet. It will tell you I, don’t have this information. I’m a bot and I shouldn’t be opining on these types of things. So it’s fascinating. 

If I think about the biggest use case for this technology as a technologist that I could imagine it being used for, it’s for testing software. It is really good at creating fake stuff. So we could be really useful at creating fake data that we can leverage instead of real data and databases. it’s got a bunch of applicable use cases where it could be super valuable fast. 

But we unleashed it on the world as this thing is amazing. It can generate a whole bunch of stuff. They have DALL-E who, can take in images and generate new images that never existed. All these really wonderful applications that are being created. It is definitely a creative wonder.

 These technologies have a lot of amazing potential. They are going to be the beginning of a new start for humanity in not having that blank piece of paper you’re starting with. It is truly wonderful. As a technologist, as an entrepreneur, I love coming to a piece of paper and starting with some ideas.

But that’s as far as it goes for me because I just don’t trust technology like everyone else does. I’m the first person to say, yeah, before I actually leverage this, I’m going to check it 25 times. Does that mean I’m going to miss an opportunity to do some of the cool new things right away? Yeah, probably. But in the end, I’ve created trust with my users. I’ve created resilience with my users that’s what we’ve gotta start thinking about . 

How do we measure software going forward? How do we really think about things like ChatGPT, putting it into our technology and really figuring out what we can trust it for. I don’t think right now I’m going to trust it for legal precedence and I don’t think any lawyer is hopefully right now.

And the question is, when will it come back? When will generative AI be able to help you  understand what’s out there and what you can do with it? When will that technology have enough maturity that it can tell you, Hey, yeah, I read. 3000 cases. In the 3000 cases, here’s what I learned, would be really useful and valuable, I think. But to read 3000 cases and then generate 3000 more is the problem that I think we’re seeing with poor Steven Schwartz is that I don’t think he anticipated that it could create fictional case law.

Mark Miller: Joel, I want to get back to Steven Schwartz now that Shannon brought his name up. I want to know from a legal perspective, what are you envisioning for him? He’s a 30 year veteran lawyer who just got his head handed to him on a platter. What do you expect the ramifications from the judge to be?

Joel MacMull: 

It’s a little difficult to say because there’s a continuum here, right? The judge could go hard or soft on him. A lot of that is discretionary. If he comes really heavy, you can guarantee there’s going to be an appeal. I would think in reading that appeal, it will say, invariably the judge, judge Castel, had the discretion to do what he did. Knowing this judge and having been before him, at least in two cases in the last two 10 years, here’s what I think. I think he’s going to sympathize with the naivete of Mr. Schwartz. But I also think that by virtue of his position, and because he recognizes this, as I said, as a seminal moment in the practice of law using this technology, he’s going to come rather heavy. So what do I mean by that? I think there will be a sanction. It may be a recommendation to the State Board of Bar Examiners that he be suspended.

I don’t exactly remember what exactly a judge can recommend in the way of suspension, because I think the suspension comes from the Bar Association, it doesn’t come from the district judge. What he can do and what other judges have done, is they can preclude him from practicing in that courthouse and issue a very terse decision that will embarrass him, until the day that he dies. I think that is a likely outcome.

I do think it’s unfortunate. I have no doubt that in retrospect, Mr. Schwartz would not engage in any of this. Of course. Why would he? as I said, this is damage to him and certainly his firm. But I just don’t see how, given the gravity of what’s at risk here, that the judge does anything less than quote unquote throwing the book at him.

Mark Miller: 

Shannon, from your perspective as a technologist, do you have sympathy for him? 

Shannon Lietz: 

I do, and I don’t have sympathy for Steven Schwartz and I think it’s this:, I still believe people are running so fast at this technology, and to your point earlier about maybe he wasn’t using it for the best case law that he could have created for his client. Maybe he was using it for productivity purposes. 

I believe, and I read a bunch of the materials that are out there, I believe that he still had this need to be able to review what was there. And that to me is the interesting piece of the puzzle is that he didn’t review it. He didn’t spend any time actually looking back over his brief and checking it as if ChatGPT were a paralegal for him, he treated ChatGPT like it was a lawyer who knew what it was doing.   

That to me is probably the more interesting part of this case where I start to go, maybe I don’t have as much sympathy for you as I once did when I first saw this, oh, this poor lawyer. Now I’m starting to say, you should have done that human review thing. You should have at least spent a little bit of your time on whether or not this was really out there.

Maybe check one of the cases at least. Didn’t he use all of them that were produced without checking a single one? That to me is where I think he falls short in terms of his intent, and I think it’s where he falls short in terms of his duty to the court, is that he didn’t even do a single review.

Joel MacMull: 

Shannon is touching on why ultimately I think Castel is going to throw the book at him because one of the questions Judge Castel asks him during the hearing is, was ChatGPT an augment to your research or was it the sole source of your research?

And the answer, of course, was the latter. And again, ChatGPT is one component here in a series of misrepresentations, some certainly more egregious than others This other guy, LoDuca, let’s step back and revisit the facts here quickly. 

LoDuca and Schwartz are partners for years. LoDuca and Schwartz trust each other implicitly. Schwartz writes, the brief relies on ChatGPT and I think LoDuca signs off on it. Something to that effect. The reply brief comes in from the other side saying, we don’t see these cases, they don’t exist. LoDuca, whose name is on the brief and who is the attorney of record never reads that document, is never alerted to the allegations that the cases that he’s signed off on are garbage.

Okay? So what he does is he then represents to the court, your Honor, I’d love to deal with this, but I’m on vacation.  At the hearing last week, it’s revealed that LoDuca wasn’t on vacation. He lied because he was trying to stall because the person who was on vacation was Schwartz and he wanted Schwartz to be able to account for what the hell had happened.

So my point is it’s just misrepresentation after misrepresentation. The idea being that the smallest of lies ends up exponentially growing into an absolute disaster. And this is that case. 

ChatGPT certainly is at the center of it, but it is not the end of the story. What the judge will say is, there was ample opportunity before my order to show cause to make this a better situation. You didn’t do that. You could have read the reply, withdrawn your brief, fallen on your sword then. But instead your lies compounded. And that’s what makes this case in its entirety, I think so egregious, 

Shannon Lietz: 

My technology curiosity’s a buzz on this one. I would love if someone could provide the ChatGPT chat log for this. Because I wonder if it actually generated this case law and then did not warn him, or if it did. That would be super curious to me because I have all my ChatGPT logs. Did it come back and say here’s, six cases that relate to what you have?

Did we get to see his prompt? Did we get to see the reply? How did ChatGPT actually tell this person whether or not they should be concerned and that they should review. 

Joel MacMull: 

That is very interesting because that might go to the willfulness argument that the defense is trying to make. Gosh, It would be a fact consistent with his innocence.

I’m not sure I’d buy it. Because I gotta tell you, fundamentally, any lawyer who’s going to submit a brief and admit that he didn’t do any research but relied on an external source for the correctness and completeness of that research, that right there’s a dead letter. Like I don’t think you have to go any further, in my estimation, to be in trouble.

But that’s a very interesting point from a defense perspective, look, your Honor, he ran it and he wasn’t prompted that, that which was going to be pumped out was necessarily going to be Fugazi. That’s interesting. 

Shannon Lietz: 

Did it actually reply. The question’s going to be in that log. it’s going to be whether or not it’s actually got it or not. 

Joel MacMull: 

It would also substantiate the representations though that he’s made in terms of, because he said, again, it’s hearsay, but as soon in the correctness of these articles that they’re saying, he apparently said, I didn’t know it could produce fake cases.

Shannon Lietz: 

I would be surprised to see this particular judge opine without those logs. I think it actually would be something that he should consider and that needs to be part of the evidence submitted to the court for this particular type of case.

 I think they should actually be subpoenaed if their information was being provided and they didn’t stop a lawyer at the very front of their gates to say, you shouldn’t be using this for a legal precedent. 

I would to understand how he is going to write his own opinion actually, to prevent him from maybe embarking on that, which he’s not understanding himself. 

Joel MacMull: 

Which is why I think in part he’s taking some time. He’s taking his time and he said to his clerks, we’ve got to get this one right. I have every expectation that this is going to be very methodical. 

Shannon Lietz: 

I do think that’s how the case should turn is based on what was provided back to this person. I understand all the other things and I believe me, like you’ve already made me believe that there’s more going on here, but, I think from a precedent perspective that what was said to a licensed professional is part of what causes them to do their job. And if we can’t figure out how to create a trust chain, this is going to end up being a place where we’re going to set the wrong precedence. 

Hey Judge Castelo, are you looking at the logs? Cause you might want to. 

Joel MacMull: 

There’s an affidavit that was filed on the seventh, the day before the hearing by Gregory Belmont in support of a motion for leave to file a proposed amicus brief. You’ve got people here that like yourself are interested in being heard on this issue. 

Shannon Lietz: 

There’s a lot here that we could help to set precedents for, honestly.

I think there’s a way to think about this, about what it means to provide this technology, what use cases, how to trust it effectively, and then how could a lawyer actually be able to leverage it. Cause I do think there’s a productivity benefit. That’s what that MIT student’s trying to make a case for, is that they’re using GPT technology for other litigation services right now.

And if this judge actually throws the book at this guy for ChatGPT usage, is that going to set a precedence against their technology? 

Joel MacMull: 

I think you’re absolutely right. I was not able to put it together as quickly as you were, but my hat is off to you, Ma’am. I think you are exactly right. If you look at the table of contents, you hit the nail right on the head. Sanctions must focus on conduct, not misuse tools. That’s right. He acutely realizes that if this goes south, lawyers are prohibited from using AI, he is S-O-L. 

Mark Miller: 

The future of this is what you said earlier, Shannon. We have to get outside of the general purpose ChatGPT, feed it 3000 case files, and see what the relationship between those files are, not to extend the files themselves in an arbitrary manner. 

Shannon Lietz: 

There’s another set of interesting things coming too, which is could it actually predict the future of case law based on what the 3000 say? And that is a fictional interpretation. 

So there are some really good use cases, but we’re just not spending enough time really understanding and bracketing and creating the box they should live in. And that’s what I think is the biggest aha moment 

But the only way that’s going to get turned around is if they actually go after ChatGPT to do a subpoena for the log, if Steven Schwartz and his partner aren’t going to be willing to give it up. If they aren’t willing to give it up, my thought process is they were warned. If they are willing to go get it, they weren’t. So if it’s defensively useful, I think they’re going to disclose. 

Mark Miller: 

I want to remind people that you’re a lawyer, but you are not their lawyer. 

Joel MacMull: 

…at least not yet.

Mark Miller:

Thanks for joining us for this week’s “You’re kidding me… that’s in my EULA??” We’d appreciate your comments on today’s show page, located at WhatsInMyEULA.com. You’ll also find information on how to get in touch with Joel. While you’re on the page, tell us what other EULAs we should investigate. If we use your suggestion, we’ll give you a shoutout in that episode.

“That’s in my EULA??” is published weekly. Special thanks today to Katy, that’s with a ‘T”, Kadi, that’s with a “D”, Edwin, and Tracy for the awesome voiceover work at the beginning of the show. Music today is provided by Hash Out from Blue Dot Sessions. 

We’ll see you next week.

This was a Sourced Network Production.

If you’re interested in talking with Joel about some of the issues in this episode, shoot him an email.

Joel G. MacMull | Partner
Chair, Intellectual Property, Brand Management and Internet Law Practice
(973) 295-3652 | JMacmull@mblawfirm.com