We Get Privacy — Episode 17: When AI Makes the Call: The CCPA’s New ADMT Regulations

Transcript

Damon Silver
Principal, New York City

Welcome to the Week at Privacy Podcast. I'm Damon Silver, and I'm joined by my co-host, Joe Lazzarotti. Joe and I co-lead the Privacy, AI and Cybersecurity Group at Jackson Lewis, and in that role, we receive a variety of questions every day from our clients, all of which boil down to the core question of how do we handle our data safely? In other words, how do we leverage all the great things that data can do for our organizations without running headfirst into a wall of legal risk? How can we manage that risk without unnecessarily hindering our business operations?

Joe Lazzarotti
Principal, Tampa

On each episode of the podcast, Damon and I talk through a common question that we're getting from clients. We talk it through in the same way we would with our clients, meaning with a focus on being practical. What are the legal risks? What options are available to manage those risks, and what should we be mindful of from an execution perspective? 

There's been a lot of talk about automated decision-making and AI in a broad range of contexts. There's also been a lot of development in privacy law, primarily with the CCPA here in the U.S. Those two things came together with some regulations that were finalized in January. We thought it made sense to start with just understanding what this rule is and when it might apply. 

Damon, just thinking about that, maybe give us like an explanation of what is ADMT? What does that mean? How should companies be thinking about that, just definitionally, in terms of what and how they may be using those technologies?

Silver

It would be helpful to start with the actual definition, Joe, from the regulations, and then we can break down some of the key components and some of the areas where most of the analysis is going to fall. 

An ADMT under the CCPA regulations is any technology that processes personal information and uses computation to replace human decision-making or substantially replace human decision-making. An example of an ADMT would be various types of AI. That said, not all AI is an ADMT, and there also are technologies that aren't AI, but that would still fall within this definition. If you’re using AI or some other technology just completely to replace human decision-making, That's going to be relatively clear-cut. 

The more interesting space and where a lot of these tools fall is whether the tool is substantially replacing human decision-making. What the regulations say is that replace means that the decision is going to be made without human involvement. The regulations then explain that human involvement means that a human, first, knows how to interpret and use the technology's output to make the decision. Second, the human is reviewing and analyzing the output of the technology along with other information that's relevant to making that or changing that decision. Then, lastly, the human has the authority to make or change the decision based on the analysis.

Joe, maybe it would be helpful to work through an example or two so that people get a sense of what this would look like in practice. One use case that comes up a lot is that there are a lot of interesting tools out there that help recruiters sift through the many applications they get for open positions. In that context, Joe, let's say we're not talking about a tool that's just making a clear-cut decision, like advancing this person or rejecting that person. Let's say it's a little more in the gray area. Maybe the tool is creating a ranking, a score, or some type of summary based on an assessment of the resume and other application materials, and that output is then being passed along to a human recruiter. If that recruiter then looks at that score, say, or that ranking and makes the decision who to advance, does that automatically mean we're outside the scope of these regulations?

Lazzarotti

It's not clear. A lot of these ranking tools present some challenges for HR folks. A lot of questions have come up around the New York City law, with ranking in terms of people feel like they're making a decision. They get a ranking, and they decide who from that ranking they're going to select. Of course, what they often do is select the ones ranked highest. Perhaps if the tool is presenting that ranking, how is it presenting that ranking? Is it knocking out any candidates even before it presents the ranking? That might be something that makes a decision, and there's no human involvement in it. But as for the ranking presented, perhaps there is human involvement if they are going through the three components you outlined. We’ve skipped a step in terms of whether we're making the types of decisions that would cause the ADMT to be considered, which is an employment decision. However, yes, you can have an application of it depending on how the person's making a decision or whether they're even presented with a decision to make to decide whether this is an ADMT or not. 

A different example is that there is an EEOC decision, and when it came out, they said it was the first EEOC AI lawsuit. It was a case where they knocked out, for lack of a better term, applicants who had been a certain age. For females it was fifty-five, and for males it was sixty. The tool basically just excluded anyone who was over those ages, respectively. What do you think about that? I am curious whether that might be an ADMT that companies or we should be concerned about as it relates to the CCPA rules.

Silver

If people are getting automatically knocked out and by the time a human is involved, certain people are just no longer under consideration. As to the people who are eliminated, the tool was making the decision. It was replacing human decision-making. As for the people who got through, then potentially a human is stepping in and making the decision. It's not necessarily going to be the case that everyone whose application is processed ends up in the same place in terms of whether these regulations apply.

Going back for a second to the ranking or scoring where no one's being eliminated, everyone appears on the list, say that is coming out of the tool for the recruiter's consideration. If as a practical matter what the recruiter then does, and this seems logical, is they start at the top with the people who get the highest ranking and start looking more closely at those applications, maybe invite them in for interviews and end up hiring from the top of that funnel the vast majority of the time, it's an interesting question as to whether in effect all the people who got lower scores are being eliminated by the AI. Because they're never actually getting the same review by a human as the people at the top. It does get complicated to determine whether the fact that a human does become involved at some point in the process is sufficient to keep you out of scope for these regulations.

As we'll talk about a little bit later, that is an important threshold determination because some of the obligations these regulations impose, in particular the risk assessment, are pretty involved. It's not the type of thing where most companies are going to say, well, why don't we just err on the side of caution and just say we're probably in scope, or we could be in scope and go ahead and do what's required. It's more a scenario where you really do want to do the front-end analysis to figure out whether your use cases or parts of your use cases are in scope so that you can have a better understanding of what obligations you really truly do have to comply with.

Lazzarotti

I don't disagree. Part of is going to depend also on how the humans are being instructed to assess that ranking and what they know about how the tool works and how they're interpreting those results. If you can put them in a position where it appears, and it's not just effectively excluding, but the person's making a decision that, hey, the people who are in the lower categories based on how the particular tool is set up really wouldn't meet the requirements of the job. It does require, to your point, really digging into the facts a little bit more and understanding how that decision's made, what level of involvement there is and feeling comfortable that there is human involvement based upon those factors. It’s a tough question for sure. 

Silver

To that point, Joe, I was looking at a tool yesterday. Again, it was in the hiring context. Clearly, the developer of the tool had given a lot of thought to how to address this issue: allowing a client, if they set up their processes correctly, to say there was sufficient human involvement and that they were outside the scope. 

What this tool does is it looks at a resume and other factors, and it matches it onto a job description. There was a scoring component, but there also was this summary where the tool highlights what specific language or facets of the resume or other materials mapped onto what specific facets of the job description. The recruiter was delivered this package not only with a score, but also with the specific inputs that the tool analyzed to make its determination that this person was a certain level of match. If the tool is delivering those types of outputs and there are proper procedures in place, you could have a scenario where the human really can check all these boxes of knowing how to interpret the output and reviewing and analyzing the output, and then obviously you have to give them the authority to overrule.

It can't be the case that the people with the highest score always automatically move on to the next round. Instead, there has to be documentation of what was actually looked at beyond just the score that was generated by the tool.

Joe, you raised a good point before. The fact that you're using an ADMT in and of itself does not necessarily mean that you're in scope for these regulations. There's another piece. You have to be using it to make a significant decision. What qualifies as a significant decision for purposes of these regs?

Lazzarotti

The regs basically lay out five areas that, if this technology is applied, would be considered a significant decision. It's financial and lending services, so if you're making credit approval or loan eligibility determinations, housing decisions around, say, rental applications or mortgage approvals, or education enrollment or opportunities, and admissions decisions. We talked a little bit about employment and independent contracting opportunities or compensation, those types of decisions. Then, healthcare services, so treatment eligibility, insurance and so on.

Whenever you're making a decision where you have to decide to provide or to deny opportunities and so on, in those areas, then you have to then say, well, is what we have an ADMT? If those two things come together, then there are some obligations that a business that's subject to the CCPA is going to have to satisfy. 

Damon, maybe you can talk a little bit about that and give us a sense of what those things are in the event that we have a business that's subject to the CCPA, has an ADMT, and is using that technology to make a significant decision.

Silver

If you are in scope for these regs, there are three main things that you need to do. The first is there is a pre-use notice requirement. You have to, prior to processing someone's personal information with an ADMT for purposes of making the significant decision, provide them a notice with certain prescribed content. You basically have to tell them why you're using the ADMT and how the ADMT is going to process that person's personal information to make the decision. What outputs is the tool going to generate? How are you going to use those outputs? You have to make the person aware that they have certain rights. 

Lastly, and this goes outside of the ADMT regs themselves, and I know we're going to tackle it more in a separate episode, but using an ADMT to make a significant decision also triggers an obligation to do a risk assessment. When I mentioned before that there are some obligations related to ADMT use that are pretty onerous and are not things that most businesses are going to want to lightly assume. That was the main thing I was referring to. Providing a notice and offering these rights is probably not going to be that heavy of a lift, but doing the risk assessment across various different uses of an ADMT to make significant decisions- that is a much heavier burden and one businesses should think carefully about whether they are comfortable assuming.

Lazzarotti

Just given the framework that we're talking about, it's obviously going to become a factor when you start to say, well, we do business not just in California. Now we have to worry about, well, do we trigger something under a different rule in California? Do we trigger the New York City rule, or the Colorado statute that was just updated? Maybe you say, well, can we design this process in a way that fits into one of the exceptions? There are a few exceptions to at least one of them, and only the exception that applies to the opt-out. 

Damon, can you address when you are exempt from the requirement to permit an opt-out from these requirements?

Damon Silver

There are a few different ways that you could potentially do that. One is building in a human appeal process. Basically, if you use an ADMT to make a significant decision, and then you obviously have notified the person about the fact that you're doing that, and the person then says, I want a human to review this decision, and that human has real authority to review and overturn the decision. That type of process would allow you to not offer the opt-out.

Another way that you could potentially not have to offer the opt-out is if you are using the ADMT to make certain types of decisions, such as decisions related to admission or acceptance to an educational institution or a hiring decision, and two conditions are met. First, you are only using the ADMT specifically to assess whether the individual is a good fit for that educational program or that job. Second, you make sure that the ADMT is working properly for that purpose and is non-discriminatory. The first of those factors is probably not that big of a deal because if you're using the ADMT for purposes of, say, figuring out whether a candidate should be admitted to your educational program, you probably are, in many cases, not going to be trying to use it for other purposes as well, so the purpose limitation is not a big deal. The second piece, making sure that it's working as intended and is non-discriminatory, that's a heavier lift because that means you're going to have to do testing to validate that the tool really works the way it is supposed to. In other words, that if you're trying to screen for a good match for a position in your educational program, the tool is actually doing that effectively. It also sounds like you need a bias audit to make sure there isn't any inherent bias built into the tool that could result in discriminatory outcomes for certain groups. This exception exists, but there are some caveats that come along with it in terms of your compliance burden.

There's a similar exception, though the last of the exceptions. If you are using the ADMT to allocate or assign work or make compensation decisions, you do not need to offer the opt-out. Again, if you are only using the ADMT for that specific purpose and you make sure that the tool actually works as intended and doesn't have discriminatory impacts.

Lazzarotti

The bias audit question is interesting because a lot of clients are asking about even the fair employment and housing regulations that came out in California. Those regulations don't require a bias audit or similar measures to avoid discrimination, although they do provide that if you are sued for a c claim that the tool is biased or discriminatory, that lack of an audit could be used as evidence. It probably would make sense, even in the absence of this exception, to think about whether you want to do that, certainly in an employment context and probably some of the other areas where the ADMT rules would apply. 

To close this one out, Damon, any other thoughts on how companies might try to balance these rules in terms of weighing the different standards for when a set of rules around a more highly risky types of AI tools might apply, this being a little unique, some of the other states being a little unique. What are some strategies to think about what we should do with, particularly, multi-state employers and how they address this across their organizations?

Silver

The key starting point is having a really good inventory of your tools and your use cases. These laws like the one in California that we've been discussing, the law in Colorado, which w was recently amended, Illinois has a couple of laws, New York City. They are not the EU AI Act, and they don't all treat this topic the same way. There are exceptions built in; there are different triggers for various obligations. To come up with a strategy that's going to put you in the best position to get benefit from these tools, but not assume obligations or risks you don't want to have, you really do need to spend some time identifying what tools you're using, how you want to use those tools, and then whether there are ways, and we talked about some of them to structure your use of those tools, so that you're basically getting most, if not all, of the benefit, but maybe you can avoid certain obligations, either entirely or at least with respect to certain of your use cases.

 Joe, I don’t know if you have any other high-level thoughts, but this is going to be the big one. It's not an area where you want to just paint with a really broad brush without having dug in first. At the end of the day, you may be able to come up with a relatively clean, comprehensive solution, but that's not really possible at the front end without understanding all the details of what the tool can do and what you are planning to do with it.

Joe Lazzarotti

Certainly looks that way at this point, and I can't imagine it getting any better as more states pass similar laws.

With that, we're going to conclude this episode. Hope it was helpful. If you have any thoughts or questions, please feel free to email us at privacy@jacksonlewis.com.